webmap_publish

webmap_publish

Publish the export_webmap output to MinIO for the model viewer on the documentation site.

pixi run publish-webmap            # the model of the pull request of this branch, compared with its base
pixi run publish-webmap --pr 123   # the model of the working tree as that pull request
pixi run publish-webmap --main     # only the model of origin/main
pixi run publish-webmap --commit REV [--base REV]  # the model of a commit, compared with another commit

Everything is uploaded below the anonymously readable doc-image/webmap/ prefix:

files/<hash>.<ext>: the exported files, named by their content hash, so files that did not change are uploaded once and shared by all models.
lhm_coupled/snapshots/<md5>/manifest.json: the manifest of an export, with the paths of the files in files/. The snapshot is named by the DVC hash of the export_webmap output in dvc.lock, which is how the viewer finds the model of main: it reads dvc.lock from GitHub.
lhm_coupled/main.json: the snapshot of the latest published main, for when the snapshot of the current main is not published.
lhm_coupled/diffs/<base md5>-<head md5>.json: the differences between two snapshots, see ribasim_nl.webmap_diff.
lhm_coupled/prs/<number>.json: the snapshots and differences of a pull request, which the viewer shows for ?pr=<number>. A pull request is compared with main, or in a stack of pull requests with the pull request of its base branch, so it only shows its own changes. That pull request must be published first.

The viewer finds the model of any commit on GitHub, for ?commit=<commit>, by the snapshot in its dvc.lock, and compares it with that of another commit, for ?base=<commit>&commit=<commit>, if their differences are published. It pins the link to the latest main to its commit, so a copied link keeps showing the same model.

DVC does not cache the export_webmap output, so this is the only copy that is kept. The export of another commit, to compare with, is downloaded from its published snapshot. If it is not published, the model is exported again from its run, which DVC does keep.

This needs the following environment variables to be set in .env (Deltares only): AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY They need to have write permission. A pull request is found and commented on with the GitHub CLI gh.

Classes

Name	Description
Store	The objects below `PREFIX` in the MinIO bucket.

Store

webmap_publish.Store()

The objects below PREFIX in the MinIO bucket.

Methods

Name	Description
download_snapshot	Download a published snapshot into a directory, as an export with the files named as in `files/`.
is_published	Whether snapshot `md5` is published.
metadata	The metadata of an object, or None if it does not exist.
publish_snapshot	Upload the files and manifest of an export as snapshot `md5`, unless it exists; return its name.
put_file	Upload a file named by its content hash, unless it is already there.
put_json	Upload a JSON object.

download_snapshot

webmap_publish.Store.download_snapshot(md5, target)

Download a published snapshot into a directory, as an export with the files named as in files/.

is_published

webmap_publish.Store.is_published(md5)

Whether snapshot md5 is published.

metadata

webmap_publish.Store.metadata(name)

The metadata of an object, or None if it does not exist.

publish_snapshot

webmap_publish.Store.publish_snapshot(export_dir, md5)

Upload the files and manifest of an export as snapshot md5, unless it exists; return its name.

put_file

webmap_publish.Store.put_file(name, path, content_hash)

Upload a file named by its content hash, unless it is already there.

put_json

webmap_publish.Store.put_json(name, data, cache_control)

Upload a JSON object.

Functions

Name	Description
branch_name	Assert a branch name is valid, so it is never read as an option; return it.
check_working_tree	Assert the `export_webmap` output in the working tree is up to date; return its md5 in `dvc.lock`.
checkout_export	Get the `export_webmap` output at a commit, whose hash in `dvc.lock` is `md5`; return its directory.
comment_body	The pull request comment with the viewer link and a summary of the differences with its base.
commit_hash	Assert a commit is a full commit hash, so it is never read as an option; return it.
diff_counts	The number of differing nodes and links by status, and of features with differing results.
export_hash	The DVC hash of the `export_webmap` output in a parsed `dvc.lock`, without the `.dir` suffix.
fetch_branch	Fetch a branch from GitHub and return its commit.
fetch_main	Fetch main from GitHub and return its commit.
find_pr	The number of the pull request of the current branch.
gh_json	Run a GitHub CLI command with JSON output and return the parsed result.
github_commit	The commit of a git revision, asserted to be on GitHub.
lock_at	The parsed `dvc.lock` of the working tree, or at a commit.
main	Command line entry point, see the module docstring.
manifest_entries	All file entries of a manifest, see `ribasim_nl.webmap.export_webmap`.
now	The current time in UTC, to the minute.
on_github	Assert a commit is on GitHub, where the viewer reads its `dvc.lock`; return it.
pr_number	Assert a pull request number is valid; return it.
publish_commit	Publish the model of a commit as its snapshot, unless it is published; return the snapshot md5.
publish_commits	Publish the model of a commit, and with a base commit their differences; return the viewer link.
publish_diff	Publish the differences of a model, exported to `head_dir`, from the model of a base commit.
publish_main	Publish the model of a commit of main as its snapshot and as the latest main; return the snapshot md5.
publish_pr	Publish the model of the working tree for a pull request, with its differences from its base.
snapshot_name	The name of the manifest of snapshot `md5`.
stack_base	The base branch of a pull request and the open pull request of that branch, or None if it is based on main.
store_manifest	A copy of a manifest with the paths of its files in the shared `files/` store, and the original paths.
upsert_comment	Update the viewer comment on a pull request, or add it if there is none.
viewer_url	The viewer link to the model of a commit, compared with that of a base commit if given.

branch_name

webmap_publish.branch_name(branch)

Assert a branch name is valid, so it is never read as an option; return it.

check_working_tree

webmap_publish.check_working_tree()

Assert the export_webmap output in the working tree is up to date; return its md5 in dvc.lock.

checkout_export

webmap_publish.checkout_export(store, commit, md5, cache_dir=CACHE_DIR)

Get the export_webmap output at a commit, whose hash in dvc.lock is md5; return its directory.

It is downloaded from its published snapshot. If that is not published, the model is exported again from its run in DVC, which need not be identical to the original export byte for byte.

comment_body

webmap_publish.comment_body(
    number,
    diff,
    base_commit,
    head_commit,
    base_pr=None,
)

The pull request comment with the viewer link and a summary of the differences with its base.

The base is main, or pull request base_pr in a stack.

commit_hash

webmap_publish.commit_hash(commit)

Assert a commit is a full commit hash, so it is never read as an option; return it.

diff_counts

webmap_publish.diff_counts(diff)

The number of differing nodes and links by status, and of features with differing results.

export_hash

webmap_publish.export_hash(lock)

The DVC hash of the export_webmap output in a parsed dvc.lock, without the .dir suffix.

fetch_branch

webmap_publish.fetch_branch(branch)

Fetch a branch from GitHub and return its commit.

fetch_main

webmap_publish.fetch_main()

Fetch main from GitHub and return its commit.

find_pr

webmap_publish.find_pr()

The number of the pull request of the current branch.

gh_json

webmap_publish.gh_json(*args)

Run a GitHub CLI command with JSON output and return the parsed result.

github_commit

webmap_publish.github_commit(revision)

The commit of a git revision, asserted to be on GitHub.

lock_at

webmap_publish.lock_at(commit)

The parsed dvc.lock of the working tree, or at a commit.

main

webmap_publish.main(argv=None)

Command line entry point, see the module docstring.

manifest_entries

webmap_publish.manifest_entries(manifest)

All file entries of a manifest, see ribasim_nl.webmap.export_webmap.

now

webmap_publish.now()

The current time in UTC, to the minute.

on_github

webmap_publish.on_github(commit)

Assert a commit is on GitHub, where the viewer reads its dvc.lock; return it.

pr_number

webmap_publish.pr_number(number)

Assert a pull request number is valid; return it.

publish_commit

webmap_publish.publish_commit(store, commit)

Publish the model of a commit as its snapshot, unless it is published; return the snapshot md5.

publish_commits

webmap_publish.publish_commits(store, commit, base=None)

Publish the model of a commit, and with a base commit their differences; return the viewer link.

publish_diff

webmap_publish.publish_diff(
    store,
    base_commit,
    base_md5,
    head_toml,
    head_dir,
    head_md5,
    labels,
)

Publish the differences of a model, exported to head_dir, from the model of a base commit.

Return the differences, and their name in the published model.

publish_main

webmap_publish.publish_main(store, commit)

Publish the model of a commit of main as its snapshot and as the latest main; return the snapshot md5.

publish_pr

webmap_publish.publish_pr(store, number)

Publish the model of the working tree for a pull request, with its differences from its base.

The base is main, or in a stack the pull request of the base branch.

snapshot_name

webmap_publish.snapshot_name(md5)

The name of the manifest of snapshot md5.

stack_base

webmap_publish.stack_base(number)

The base branch of a pull request and the open pull request of that branch, or None if it is based on main.

In a stack of pull requests, a pull request is based on the branch of the one below it.

store_manifest

webmap_publish.store_manifest(manifest)

A copy of a manifest with the paths of its files in the shared files/ store, and the original paths.

Files are named by their content hash, so the second result maps each new path to the exported file.

upsert_comment

webmap_publish.upsert_comment(number, body)

Update the viewer comment on a pull request, or add it if there is none.

viewer_url

webmap_publish.viewer_url(commit, base=None)

The viewer link to the model of a commit, compared with that of a base commit if given.