Skip to main content

Running on SLURM

The pipeline up to koppelen can be reproduced on an HPC cluster using two scripts:

repro.sh: pipeline reproduction

Submits the DVC pipeline as individual SLURM jobs. Each authority gets one job that runs its stages in order, and SLURM --dependency=afterok ensures correct ordering.

./repro.sh

This submits jobs in these waves:

  1. clean: removes the outputs of all stages that are run, like dvc repro does, and runs dvc pull --force for everything upstream of them; no manual pull is needed
  2. rwzi and peilbeheer (shared dependencies)
  3. 22 parallel jobs: 11 from parameterized to dynamic, 10 from feedback to forcing, and 1 with hws_demand and hws_transient
  4. samenvoegen → koppelen
  5. commit: dvc commit, pixi run check and dvc push

It does not run the input-model stages (such as lumped and boezemmodel), whose outputs clean pulls, nor the frozen bathymetry, nor the stages after koppelen. To also run those, use pixi run repro-local.

The job IDs of samenvoegen, koppelen and commit are written to repro_jobs.txt when submitting, for use with run.sh.

run.sh: isolated simulation runs

Copies a model to /p/ribasim-nl/runs/<name>/ and submits a SLURM job using the named Ribasim core configured in scripts/install_ribasim_core.py.

As an immediate preparation step, run.sh runs the archive-core Pixi task. This task depends on install-core, then archives bin/ribasim at /p/ribasim-nl/bin/<NAME>. Pixi skips the copy when the input directory and installer configuration are unchanged.

Named archives are immutable: archive-core does not replace an existing /p/ribasim-nl/bin/<NAME> directory. Change NAME when archiving a different core.

The submitted SLURM job runs this archived executable, not bin/ribasim in the repository. The repository and its installed core can therefore change while simulations are queued or running without changing the executable used by those simulations. The exact Ribasim version is written to each SLURM log before the model starts.

./run.sh <name> <model_dir> [--after=<jobid>] [key=value ...]

TOML overrides use dot notation for nested keys.

Examples

Run lhm_coupled after koppelen finishes:

./run.sh lhm_coupled data/Rijkswaterstaat/modellen/lhm_coupled \
  --after=$(grep koppelen repro_jobs.txt | cut -f2)

Run lhm_coupled with a shorter simulation period:

./run.sh lhm_coupled_1yr data/Rijkswaterstaat/modellen/lhm_coupled \
  --after=$(grep koppelen repro_jobs.txt | cut -f2) \
  endtime="2018-01-01 00:00:00"

Compare solver settings:

./run.sh lhm_coupled_tight data/Rijkswaterstaat/modellen/lhm_coupled \
  --after=$(grep koppelen repro_jobs.txt | cut -f2) \
  solver.abstol=1e-6

./run.sh lhm_coupled_qndf data/Rijkswaterstaat/modellen/lhm_coupled \
  --after=$(grep koppelen repro_jobs.txt | cut -f2) \
  solver.algorithm=QNDF solver.abstol=1e-5

All submissions are logged to /p/ribasim-nl/runs/jobs.txt. The job logs of repro.sh are gitignored.