flowchart LR
node1["bathymetry"]
node2["bergend@aa_en_maas"]
node3["bergend@brabantse_delta"]
node4["bergend@de_dommel"]
node5["bergend@drents_overijsselse_delta"]
node6["bergend@hunze_en_aas"]
node7["bergend@limburg"]
node8["bergend@noorderzijlvest"]
node9["bergend@rijn_en_ijssel"]
node10["bergend@stichtse_rijnlanden"]
node11["bergend@vallei_en_veluwe"]
node12["bergend@vechtstromen"]
node13["data/Basisgegevens/Baseline/baseline-nl_land-j23_6-v1/baseline.gdb.dvc"]
node14["data/Basisgegevens/BuitenlandseAanvoer/aangeleverd/BuitenlandseAanvoer_V5.xlsx.dvc"]
node15["data/Basisgegevens/Hydrotypen.dvc"]
node16["data/Basisgegevens/LHM/4.3/input/LHM_data.tif.dvc"]
node17["data/Basisgegevens/RWZI/aangeleverd/locaties/RWZI_coordinates.geojson.dvc"]
node18["data/Basisgegevens/Top10NL/top10nl_Compleet.gpkg.dvc"]
node19["data/Basisgegevens/VanDerGaast_QH/spafvoer1.tif.dvc"]
node20["data/Basisgegevens/profielen.dvc"]
node21["dynamic@aa_en_maas"]
node22["dynamic@brabantse_delta"]
node23["dynamic@de_dommel"]
node24["dynamic@drents_overijsselse_delta"]
node25["dynamic@hunze_en_aas"]
node26["dynamic@limburg"]
node27["dynamic@noorderzijlvest"]
node28["dynamic@rijn_en_ijssel"]
node29["dynamic@stichtse_rijnlanden"]
node30["dynamic@vallei_en_veluwe"]
node31["dynamic@vechtstromen"]
node32["feedback@amstel_gooi_en_vecht"]
node33["feedback@delfland"]
node34["feedback@hollands_noorderkwartier"]
node35["feedback@hollandse_delta"]
node36["feedback@rijnland"]
node37["feedback@rivierenland"]
node38["feedback@scheldestromen"]
node39["feedback@schieland_en_de_krimpenerwaard"]
node40["feedback@wetterskip_fryslan"]
node41["feedback@zuiderzeeland"]
node42["forcing@amstel_gooi_en_vecht"]
node43["forcing@delfland"]
node44["forcing@hollands_noorderkwartier"]
node45["forcing@hollandse_delta"]
node46["forcing@rijnland"]
node47["forcing@rivierenland"]
node48["forcing@scheldestromen"]
node49["forcing@schieland_en_de_krimpenerwaard"]
node50["forcing@wetterskip_fryslan"]
node51["forcing@zuiderzeeland"]
node52["hws_demand"]
node53["hws_transient"]
node54["koppelen"]
node55["parameterized@aa_en_maas"]
node56["parameterized@brabantse_delta"]
node57["parameterized@de_dommel"]
node58["parameterized@drents_overijsselse_delta"]
node59["parameterized@hunze_en_aas"]
node60["parameterized@limburg"]
node61["parameterized@noorderzijlvest"]
node62["parameterized@rijn_en_ijssel"]
node63["parameterized@stichtse_rijnlanden"]
node64["parameterized@vallei_en_veluwe"]
node65["parameterized@vechtstromen"]
node66["profiles@amstel_gooi_en_vecht"]
node67["profiles@delfland"]
node68["profiles@hollands_noorderkwartier"]
node69["profiles@hollandse_delta"]
node70["profiles@rijnland"]
node71["profiles@rivierenland"]
node72["profiles@scheldestromen"]
node73["profiles@schieland_en_de_krimpenerwaard"]
node74["profiles@wetterskip_fryslan"]
node75["profiles@zuiderzeeland"]
node76["rwzi"]
node77["samenvoegen"]
node1-->node52
node2-->node21
node3-->node22
node4-->node23
node5-->node24
node6-->node25
node7-->node26
node8-->node27
node9-->node28
node10-->node29
node11-->node30
node12-->node31
node13-->node52
node14-->node21
node14-->node22
node14-->node23
node14-->node24
node14-->node25
node14-->node26
node14-->node27
node14-->node28
node14-->node29
node14-->node30
node14-->node31
node15-->node66
node15-->node67
node15-->node68
node15-->node69
node15-->node70
node15-->node71
node15-->node72
node15-->node73
node15-->node74
node15-->node75
node16-->node2
node16-->node3
node16-->node4
node16-->node5
node16-->node6
node16-->node7
node16-->node8
node16-->node9
node16-->node10
node16-->node11
node16-->node12
node17-->node76
node18-->node55
node18-->node56
node18-->node57
node18-->node58
node18-->node59
node18-->node60
node18-->node61
node18-->node62
node18-->node63
node18-->node64
node18-->node65
node19-->node2
node19-->node3
node19-->node4
node19-->node5
node19-->node6
node19-->node7
node19-->node8
node19-->node9
node19-->node10
node19-->node11
node19-->node12
node20-->node66
node20-->node67
node20-->node68
node20-->node69
node20-->node70
node20-->node71
node20-->node72
node20-->node73
node20-->node74
node20-->node75
node21-->node77
node22-->node77
node23-->node77
node24-->node77
node25-->node77
node26-->node77
node27-->node77
node28-->node77
node29-->node77
node30-->node77
node31-->node77
node32-->node66
node33-->node67
node34-->node68
node35-->node69
node36-->node70
node37-->node71
node38-->node72
node39-->node73
node40-->node74
node41-->node75
node42-->node77
node43-->node77
node44-->node77
node45-->node77
node46-->node77
node47-->node77
node48-->node77
node49-->node77
node50-->node77
node51-->node77
node52-->node53
node52-->node54
node53-->node77
node55-->node2
node56-->node3
node57-->node4
node58-->node5
node59-->node6
node60-->node7
node61-->node8
node62-->node9
node63-->node10
node64-->node11
node65-->node12
node66-->node42
node67-->node43
node68-->node44
node69-->node45
node70-->node46
node71-->node47
node72-->node48
node73-->node49
node74-->node50
node75-->node51
node76-->node21
node76-->node22
node76-->node23
node76-->node24
node76-->node25
node76-->node26
node76-->node27
node76-->node28
node76-->node29
node76-->node30
node76-->node31
node76-->node42
node76-->node43
node76-->node44
node76-->node45
node76-->node46
node76-->node47
node76-->node48
node76-->node49
node76-->node50
node76-->node51
node76-->node52
node77-->node54
DVC (Data Version Control)
We use DVC to version-control large data files (models, input datasets) without storing them in Git. DVC tracks file hashes in .dvc files committed to Git, while the actual data lives on remote storage.
Authentication
Create a file .dvc/config.local (already gitignored) with credentials for the remotes:
['remote "goodcloud"']
user = nhi_api
password = <RIBASIM_NL_CLOUD_PASS>
['remote "modeldata"']
user = nhi_api
password = <RIBASIM_NL_CLOUD_PASS>
['remote "minio"']
region = eu-west-1
endpointurl = https://s3.deltares.nl
access_key_id = <MINIO_ACCESS_KEY>
secret_access_key = <MINIO_SECRET_KEY>The password is the same as RIBASIM_NL_CLOUD_PASS in your .env file. The MinIO keys correspond to MINIO_ACCESS_KEY and MINIO_SECRET_KEY in .env.
Note
Since pixi run install runs dvc pull, this configuration is needed before that step.
Pulling data
After authentication is set up, pull all tracked data:
pixi run dvc pullIf you have local changes that you want to overwrite, add --force.
Remotes
Three remotes are configured in .dvc/config:
| Remote | URL | Purpose |
|---|---|---|
minio (default) |
s3://ribasim-nl/dvc |
Primary DVC cache on MinIO |
goodcloud |
The Good Cloud /dvc |
Legacy DVC storage |
modeldata |
The Good Cloud /Ribasim modeldata |
Source data not under DVC control |
Pipeline
The DVC pipeline is defined in dvc.yaml. Stages run sequentially with dependencies:
Reproduce the full pipeline:
pixi run dvc reproReproduce a single stage:
pixi run dvc repro dynamic@aa_en_maasImporting data from remote storage
Use dvc import-url to download a file from a remote URL and take it under DVC control. The remote://modeldata alias refers to The Good Cloud storage that holds source data:
pixi run dvc import-url -f remote://modeldata/Zuiderzeeland/modellen/Zuiderzeeland_parameterized_2025_9_0 data/Zuiderzeeland/modellen/Pushing changes
After producing new outputs (running pipeline stages or adding new data), push to the remote:
pixi run dvc pushDo this before pushing the updated dvc.lock file to git, so the data hashes referenced therein are avaiable for everyone.