
Publish link's habitat flags per config as Parquet + manifest
Source:R/lnk_scenario_export.R
lnk_scenario_export.RdWrites a local tree, the layout of s3://fresh-bc/habitat/fwa/v1/, from
persisted link runs: one long-form row per stream segment, config and
species, keyed by FWA position, with the flags, the size and gradient
values the run tested, and their ratios to the run's thresholds. A
manifest.json names each part's md5 and the log rows of the builds that
produced it; lnk_catalog() reads it and lnk_db_load() loads the parts
into PostgreSQL.
Usage
lnk_scenario_export(
conn,
configs,
root,
wsgs = NULL,
species = NULL,
shared_segmentation = FALSE,
replace = FALSE
)Arguments
- conn
A DBI connection to the database holding the persist schemas.
- configs
Named character: names are link config names (as logged in
<schema>.log.config_name), values the persist schema holding each. A config may name several schemas holding different groups.- root
Directory to write.
segment/andmanifest.jsonare replaced;provenance/andconfig/accumulate.- wsgs
Watershed groups to export, or
NULLfor every group with a finished build of its config (the others are named in a message).- species
Species codes, or
NULLfor every species the config modelled in the exported groups.If
TRUE, stop unless every schema exported holds the same segmentation in the groups they share. Set it when the configs are variants meant to be compared segment for segment.- replace
An export replaces each config it names. By default it stops if that would drop a group or species the root already holds for the config, or put an older build of a group in place of a newer one;
TRUEallows it.
Details
<root>/manifest.json
<root>/segment/config=<config>/species=<sp>/part-<code>.parquet
<root>/barriers/config=<config>/part-<group>.parquet
<root>/barrier_overrides/config=<config>/part-<group>.parquet
<root>/dependencies/rule=line.parquet
<root>/inputs/<cabd_dams|pscis_assessment>/<content md5>.parquet
<root>/provenance/run_uid=<uid>/run_id=<id>/{log,log_input}.parquet
<root>/provenance/config_hash=<hex>/log_parameters_*.parquet
<root>/config/sha256-<hex>/<bundle files>Rules the export holds to:
Key.
(blue_line_key, downstream_route_measure, upstream_route_measure). Break points differ by config, so neitherlinear_feature_idnor link's per-groupid_segmentis a key; cut a segment's line from the FWA blue line at its measures.Which groups. A group is exported only when its latest log row is a finished build of the config; there is no other record of the thresholds or bundle behind its flags. Provenance is keyed per build; a build outside a dispatch (
run_uidNULL) is kept underprovenance/run_id=<run_id>/.Species. Every segment has a row for every species the config models in the exported groups; a species with no habitat in a group reads FALSE.
Discharge.
mad_m3sis the discharge the run classified on, with its logged fill, so*_ratio_madagrees with the flags.Ratios. Use the thresholds the run logged at its
config_hash. A build that logged none (before link's #282) has NA ratios for spawning and rearing.Bundle files. Published under
config/only when today's bundle hashes the same as the build; otherwise the manifest records why not.Provenance. Written once: a provenance file that already exists with different bytes stops the export.
See also
Other export:
lnk_catalog(),
lnk_db_load(),
lnk_downstream_files(),
lnk_downstream_load()
Examples
if (FALSE) { # \dontrun{
conn <- lnk_db_conn(dbname = "fwapg", host = "localhost", port = 5432L,
user = "postgres", password = "postgres")
root <- file.path(tempdir(), "habitat", "fwa", "v1")
lnk_scenario_export(conn, c(default = "fresh_default"), root,
wsgs = "DUNC", species = c("BT", "RB"))
cat <- lnk_catalog(file.path(root, "manifest.json"))
cat[, c("config", "species", "code", "rows", "md5")]
} # }