Tracking Hugging Face Jobs with track.py

Oct 21, 2026 · @Clifford Anderson

What it does

track.py starts a Hugging Face Job from your computer, or attaches to one already running, and records it in Weights & Biases (W&B), MLflow, or both. It works with any job script, ours or anyone else's (Daniel van Strien's uv-scripts, for example), without changing it.

Each run records the script's URL and arguments, the job's image and hardware, and its outcome and duration. It also keeps the full job log and the CPU, GPU, memory and network use that Hugging Face reports every 15 seconds. It can't see inside a script beyond what the script prints.

Tracking runs on your computer, so your W&B and MLflow logins never enter the job. An MLflow server only needs to be reachable from your computer.

flowchart LR
    PC["<b>Your computer</b><br/>runs track.py<br/>W&B and MLflow logins never leave it"]
    HF["<b>Hugging Face Jobs</b><br/>runs the script unchanged<br/>any script, any author"]
    TR["<b>W&B and/or MLflow</b><br/>settings, full log, outcome<br/>CPU, GPU, memory, network"]
    PC -- starts the job --> HF
    HF -- log and resource use --> PC
    PC -- a run per job --> TR

Every arrow starts or ends on your computer: the job never talks to W&B or MLflow.

Set up once

Install uv, open a new terminal, and log in to Hugging Face. uvx runs the hf command without installing it.

Bash or zsh (macOS, Linux):

curl -LsSf https://astral.sh/uv/install.sh | sh
uvx hf auth login

PowerShell (Windows):

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
uvx hf auth login

Connect W&B and MLflow

Log in to W&B once with uvx wandb login, pasting your key from wandb.ai/authorize. Everything else comes from environment variables on your computer:

For MLflow on Databricks, set MLFLOW_TRACKING_URI to databricks and add DATABRICKS_HOST and DATABRICKS_TOKEN. The experiment name must then be a workspace path, such as /Users/you@yale.edu/open-theology.

Bash or zsh (add the export lines to ~/.bashrc or ~/.zshrc to keep them):

uvx wandb login
export WANDB_PROJECT=open-theology
export MLFLOW_TRACKING_URI=https://YOUR-MLFLOW-SERVER
export MLFLOW_TRACKING_USERNAME=you MLFLOW_TRACKING_PASSWORD=your-password
export MLFLOW_EXPERIMENT_NAME=open-theology

PowerShell ($env: settings last for the current window; to keep one, run [Environment]::SetEnvironmentVariable("WANDB_PROJECT", "open-theology", "User") and open a new window):

uvx wandb login
$env:WANDB_PROJECT = "open-theology"
$env:MLFLOW_TRACKING_URI = "https://YOUR-MLFLOW-SERVER"
$env:MLFLOW_TRACKING_USERNAME = "you"; $env:MLFLOW_TRACKING_PASSWORD = "your-password"
$env:MLFLOW_EXPERIMENT_NAME = "open-theology"

Run a tracked job

Put uv run $TRACK --wandb -- in front of the hf jobs command you would normally run. Use --mlflow instead for MLflow, or both flags to log to both. The job's own options (--flavor, -s HF_TOKEN, -v, --name) stay in the hf jobs part, and its log streams to your terminal as usual.

This example runs Daniel's GLM-OCR script on our page images. The script's header already picks the GPU, the Docker image and the Hugging Face token.

Bash or zsh:

TRACK=https://huggingface.co/datasets/divinity-library/scripts/resolve/main/track.py
OCR=https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py
uv run $TRACK --wandb -- hf jobs uv run --name open-theology-ocr $OCR \
  divinity-library/open-theology-pages divinity-library/open-theology-ocr

PowerShell (the backtick continues the line and must be its last character):

$TRACK = "https://huggingface.co/datasets/divinity-library/scripts/resolve/main/track.py"
$OCR = "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py"
uv run $TRACK --wandb -- hf jobs uv run --name open-theology-ocr $OCR `
  divinity-library/open-theology-pages divinity-library/open-theology-ocr

The run takes the job's name, here open-theology-ocr, and keeps the job's URL in its settings. When the job ends, the run is marked finished or failed to match it.

Resume or stop

track.py has to keep running until the job ends. Ctrl-C, closing the terminal or a sleeping laptop stops the tracking, not the job. To pick it up again, run track.py with the NAMESPACE/JOB_ID it printed at the start; it continues the same W&B or MLflow run. To stop the job itself, cancel it.

The commands are the same in Bash, zsh and PowerShell:

uv run $TRACK --wandb NAMESPACE/JOB_ID
uvx hf jobs cancel NAMESPACE/JOB_ID

The first command also follows a job started without track.py. Its log is recorded in full, but resource use only from when you attach.

Good practice

• Jobs run, and are billed, under your own Hugging Face account unless you add --namespace divinity-library to the hf jobs command. That needs write access to the organization.
• Pin other people's scripts to a commit, as in .../uv-scripts/ocr/raw/<commit>/glm-ocr.py, so the run records exactly what ran. A URL with main points at whatever the file says later.
• Keep W&B and MLflow logins out of the job: don't pass them with -s. track.py never needs them there.