Running locally
On a workstation with enough cores and memory, the pipeline can be reproduced in parallel with:
pixi run repro-localThis is the local counterpart of repro.sh. It runs validate_lhm_coupled, export_webmap and hws_plus and everything upstream of them, except frozen stages. So beyond the stages that repro.sh runs, it also makes the input models, and runs run_lhm_coupled, validate_lhm_coupled, delwaq, export_webmap and hws_plus. The stages and their commands are taken from dvc.yaml. delwaq needs a Delft3D installation in D3D_HOME, see other settings. It:
- Removes the outputs of all stages that are run, like
dvc reprodoes, and runsdvc pull --forcefor everything upstream of them; no manual pull is needed. - Runs each stage as soon as the stages it depends on have succeeded, so the water authorities run in parallel. If a stage fails, the stages downstream of it are skipped, and the others continue.
- Runs
dvc commitfor the stages that succeeded. If all succeeded, it runspixi run check, a second time if hooks fixed files such asdvc.lock, and with--pushalsodvc push. - If
export_webmapran, publishes the model to the model viewer as a preview of the pull request of the current branch, compared withmain. This needs a pull request for the branch, MinIO credentials with write access in.env, and a logged ingh.
Like repro.sh, it runs the stage commands directly, not with dvc repro, since parallel dvc repro processes would contend for the DVC lock. The output of each stage is written to logs/repro/<timestamp>/<stage>.log, and a summary with the status and duration of each stage is printed at the end.
Options
- Targets: stages to reproduce together with their upstream stages,
validate_lhm_coupled,export_webmapandhws_plusby default. For example,pixi run repro-local koppelenonly builds the coupled model, likerepro.sh. --jobs N: the maximum number of stages that run at once, 12 by default. Each stage is a Python process that may also run Ribasim, so watch the memory use when increasing it.--stagger SECONDS: the minimum time between stage starts, 30 by default, to not overload the GoodCloud server from whichparameterizedstages download GKW data that is not local.--incremental: only run the stages that DVC reports as changed, and everything downstream of them, rather than all stages. Data that is not in the workspace does not count as changed, so a fresh worktree does not rerun everything. A stage whose output DVC does not cache, such assamenvoegen, also runs if a stage that runs needs that output and it is not in the workspace. A stage that failed before only runs again if the outputs it left differ fromdvc.lock. Runpixi run dvc pullfirst if the workspace may be behinddvc.lock.--push: rundvc pushfor the reproduced stages if they all succeeded.--no-publish: do not publish the model to the model viewer.
To keep the run going after closing the terminal, start it in for instance tmux or with nohup.