Manage Long-running Python Tasks using JupyterHub on Remote Ubuntu Server

I recently re-configured my Ubuntu server with JupyterHub so that I can manage a long-running Python task with a large dataset, which cannot be handled by my MacBook Pro. I spent quite some time figuring out how to do certain things right and decide to write it down before I forget.

The task was a sentiment analysis of ~1.4 million hotel reviews using our fine-tuned model based on Erlangshen-Roberta-330M-Sentiment, which took about 10 days to complete using my old CPU server. I developed a simplified version of the program to demo in this tutorial.

The data and code can be found in this repo.

The following are the key parts:

• install JupyterHub on the server
• install pyenv to manage different versions of Python
• setup and add virtual environments to ipykernel so that they are accessible from JupyterLab
• use a HuggingFace model to do a zero-shot sentiment analysis
• run a long-running Python task without keeping the terminal open and with logging and email notification
• check GPU usages and terminate tasks

JupyterHub Setup and SFTP

Our system admin used this tutorial to set up the JupyterHub and added my account to the sudo group.

For small files, I just use JupyterLab interface to upload them to the server. For large files, I use FileZilla to upload.

Python Versions

Different python versions might be needed for different projects. I use pyenv to manage Python versions. The following is my note for setting up pyenv on Ubuntu server, refer to my Mac tutorial if needed.

Now if I type python, I see Python 3.9.1:

Virtual Environments

Setup and activate a virtual environment (here I put all virtual environments in the folder venv and this sample one is named ml):

Next, while the virtual environment is activated, we need to add it to ipykernel so that we can access it via JupyterLab:

Then, you can use jupyter kernelspec list to check all available kernels and remove the corresponding folder to delete the kernel if needed.

Now, the new virtual environment (kernel) can be used when creating new notebooks:

Long-running Python Tasks

The program will calculate the sentiment of each review and generate a new csv file as the result. The small sample has only 500 rows. The processed result is saved every 100 rows and an email notification is sent out every 200 rows.

I use this program to show:

• how to call a HuggingFace model (NOTE: if the repo is private, you must get a token from HuggingFace and run huggingface-cli login with token first and the code also needs to be changed to use token - see my inline comments for details).
• how to log the result using Python logging
• how to send out email notifications using SendGrid (you need to get an API token)

To run the program in the background, you can ssh to the server or use the terminal from JupyterLab, activate the virtual environment, then execute:

& asks the terminal to return control to you immediately and allows the command to complete in the background ( source).

The program will keep on running even if you close the terminal or close JupyterLab.

Two files will be generated:

Three email notifications will also be sent:

The logging and email notifications are very useful for long-running tasks so that I know the program is still working fine or from where to restart if something went wrong.

Terminate Programs

Sometimes you may want to terminate a running program for whatever reasons, e.g., the program is taking too much GPU and you need to release the GPU resources.

You can check GPU usage using nvtop: install the package using sudo apt install nvtop and then run nvtop

You can list all processes by user name using ps -u [username], such as:

then you can terminate a process using its ID: kill 22112 (gracefully) or kill -9 22112 (forcefully and immediately)

ps means process status:

| pipe a command that lets you use two or more commands such that output of one command serves as input to the next. grep is a command-line utility for searching plain-text data sets for lines that match a regular expression.

ps -ef | grep python can be use to show all running processes with python.

PS. The image for this post is generated via Midjourney using the prompt “a giant cloud server in matrix calculating difficult problems”.

Originally published at https://harrywang.me on August 5, 2022.