Compute on Jupyter

Adds remote Jupyter Servers to Compute so a pipeline step can train where your notebooks run.

Name a server in .punica/compute.yaml and a Train on a Remote Machine step sends the job's files there, starts the command detached from a kernel, and brings the outputs back into your workspace. The job keeps its state in files beside it on the server, so closing the app or losing the connection loses nothing.

What you can do

• Train on a Jupyter Server you already use, such as a lab's JupyterHub server, a cloud notebook instance or a GPU box running jupyter server.
• Run in a container by naming an image, and the job runs in docker run --gpus all with its folder mounted.
• Cap a job's hours with maxHours, so a job that runs too long is stopped.
• Cancel a job, and the job and every process it started are stopped.
• Send large files: anything past 8 MB goes in chunks, and outputs stream back to disk, never through memory.

What it does not do

It does not keep your token. The target names the key the token is stored under in the secret manager, and the token travels to the server in a request header, never on a command line.

It does not leave a secret on the server. A secret the job needs reaches the kernel's shell on its input, is written under umask 077, and is read and deleted before the command starts. The kernel is asked not to keep a history of what it ran.

It does not reach outside the server's root. The job folder is a path under the server's root folder, because that is where the server's file API can read and write.

Requirements

A Jupyter Server whose default kernel is Python and whose machine has a POSIX shell, and Python 3.8 or newer on this machine. The Compute extension draws the targets and jobs, and the MLOps extension 1.29.0 or newer draws the step.

Getting started

1. Store the server's token in the secret manager under a key such as jupyter/lab.
2. Name the server in Compute (Create the file), with engine: jupyter, its url and token: jupyter/lab.
3. Press Check on its row.