Automated Testing¶

This guide is for cluster maintainers. It covers bringing a cluster into the automated smoke-test rotation.

Overview¶

rootstock smoke-test re-verifies every fetched checkpoint (one GPU forward pass each) and pushes the refreshed manifest.json to the backend. Running it on a schedule keeps each cluster's manifest honest about checkpoint health.

Each checkpoint's result is written to manifest.json the moment it is known (a short locked read-modify-write per checkpoint), so a job the scheduler kills at its wall-time limit keeps every verified/failed stamp it reached. The backend push is a separate step that happens once, after the last checkpoint. To bound how much a killed run leaves unpushed, add --push-every N: the manifest is then also pushed after every N results (each push replaces the whole manifest server-side, so an intermediate push is simply a snapshot of the stamps so far). Either way, stamps a killed run left on disk ride along with the next successful push from that install.

User-supplied weights (<family>:custom checkpoints) are never smoke-tested — there is nothing registered to verify. Every use is a fresh load through the env's setup_from_path hook, so a bad file simply fails at server start.

The mechanism is a self-scheduling batch job: a single GPU job that first re-queues its own next run, then runs the smoke-test. The only state that persists between runs is "a job is queued for next time," which the job maintains by resubmitting itself.

Two recipes with placeholders:

• scripts/nightly_smoke_test.sbatch — SLURM (e.g. Della, Delta)
• scripts/nightly_smoke_test.pbs — PBS Pro (e.g. ALCF Sophia)

The command and these recipes keep the name "nightly" for continuity, but the default cadence is weekly (CADENCE_DAYS=7).

Prerequisites¶

• A maintainer account on the cluster with a GPU allocation.
• Rootstock installed and configured on that account (~/.config/rootstock/config.toml with a valid API key and is_maintainer = true) so the manifest push succeeds.
• rootstock reachable on PATH inside a batch job (activate the venv or module load in the marked spot in the script).

Setup¶

1. Copy the recipe for your scheduler and edit the marked block at the top (account, partition/queue, walltime) and ROOTSTOCK_ROOT:

# SLURM:
$EDITOR scripts/nightly_smoke_test.sbatch   # set #SBATCH -A/-p, ROOTSTOCK_ROOT
# PBS:
$EDITOR scripts/nightly_smoke_test.pbs      # set #PBS -A/-q, filesystems, ROOTSTOCK_ROOT

1. Kick off the chain once. It self-perpetuates from there:

sbatch scripts/nightly_smoke_test.sbatch          # SLURM
qsub  scripts/nightly_smoke_test.pbs              # PBS (submit from the scripts/ dir)

1. Verify a full cycle. After the first run finishes, confirm:
2. a next run is queued — squeue --me -n rootstock-nightly (SLURM) / qstat -u $USER (PBS) shows a pending job (on PBS a job with a future start time sits in state W, waiting — not Q), and
3. the manifest landed — rootstock status shows fresh verified_at times.

If the job hit its wall-time limit before pushing, rootstock status still shows the stamps it reached (they are written per checkpoint); only the push is missing. Raise the walltime in the recipe, or pass --push-every N in the rootstock smoke-test invocation so partial runs reach the backend.

How the self-scheduling works¶

• Resubmit-first. The job queues its next run before the smoke-test, so a crash during verification can't break the chain. The only way the chain dies is a job that never starts (allocation lapsed / queue wedged).
• Drift-free cadence. The next run is a fixed wall-clock time (CADENCE_DAYS ahead at RUN_HOUR), not now+24h, so it doesn't creep later each run. SLURM uses sbatch --begin=…; PBS uses qsub -a ….
• No duplicate chains. Before resubmitting, the job checks for an already-pending instance by name and skips if found — so a stray manual kickoff won't fork the chain.

Stopping the chain¶

touch ~/.rootstock-nightly-stop          # the next run will not reschedule
# then cancel the already-queued next run:
squeue --me -n rootstock-nightly ; scancel <jobid>    # SLURM
qstat -u $USER                  ; qdel    <jobid>      # PBS

Remove the stop file to allow a fresh kickoff later.

Testing without waiting a week¶

RESCHEDULE_BEGIN=now+3minutes sbatch scripts/nightly_smoke_test.sbatch                    # SLURM
qsub -v RESCHEDULE_AT=$(date -d "+3 minutes" +%m%d%H%M) scripts/nightly_smoke_test.pbs    # PBS

Both override when generation 1 schedules generation 2, which is the handoff worth testing. (On PBS, qsub -a <time> would only delay generation 1's own start and exercises nothing about the chain.)

The override is one-shot: generation 2 recomputes its own resubmit time from CADENCE_DAYS/RUN_HOUR. On SLURM the recipe resubmits with env -u RESCHEDULE_BEGIN because sbatch defaults to --export=ALL and would otherwise carry the testing value down the chain forever; on PBS qsub -v passes only the variables it names, so RESCHEDULE_AT never propagates. CADENCE_DAYS and RUN_HOUR do persist on SLURM via the same --export=ALL.

Knobs¶

Override via env at submit time (SLURM: inline VAR=val sbatch …; PBS: qsub -v VAR=val …):

The token lookup matches the sync scripts: an HF_TOKEN already in the environment wins; otherwise the first readable of HF_TOKEN_FILE, ~/.hf_token, and ~/.cache/huggingface/token is used (whitespace stripped) and exported as both HF_TOKEN and HUGGING_FACE_HUB_TOKEN. A missing token is a warning, not an error: smoke-test never downloads, but some checkpoints re-check gating in setup() even with weights cached.

Per-cluster notes¶

• Della (SLURM, Princeton), Delta (SLURM, NCSA): use the .sbatch recipe. Delta blocks crontab/scrontab, which is exactly why the self-scheduling job (needing only normal job submission) is the right fit. Use GPU account bhhl-delta-gpu, partition gpuA100x4 on Delta. See notes/Cluster Delta.md (umbrella workspace) for filesystem/ACL gotchas — the recipe already sets umask 0002 for the shared-filesystem ACL mask.
• Sophia (PBS, ALCF): use the .pbs recipe. Keep -l filesystems=home:eagle; the recipe sets the ALCF outbound proxy defensively (smoke-test itself doesn't download).
• Other clusters: copy whichever recipe matches the scheduler. The only per-cluster edits are the directive block and ROOTSTOCK_ROOT.

date -d in the recipes is GNU date (present on Linux HPC login/compute nodes). The recipes are not meant to run on macOS.

Smoke-test passes no kwargs of its own. smoke-test calls each env's setup() with no extra kwargs; when a checkpoint needs one (e.g. UMA's required task), verification falls back to the env's VERIFY_KWARGS entry for that checkpoint (see docs/environments.md). A checkpoint whose env requires a kwarg but declares no VERIFY_KWARGS entry will appear failing even though add --kwarg ... succeeded — the remedy is to declare the entry, never to give setup() a silent default.