Skip to content

stacs.catalogue

The published side: which items a static catalogue links, and their bodies.

Item ids are read from the collection's item links; bodies are fetched by the published href, verbatim, never by a URL rebuilt from an id.

Read (item_id, href) for every item link in a published collection.json.

Returns hrefs verbatim — already percent-encoded, so they are usable as fetch URLs as-is. Never rebuild a fetch URL from the decoded id.

Raises rather than returning a list that silently omits items, because register and verify both read this list and would agree with each other over the omission:

  • no item links at all: a truncated file, a catalog passed for a collection, or the wrong bucket -- never a legitimately empty catalogue. Against a collection that was never registered, an empty published set compares equal to an empty registered set
  • a child link: items behind a sub-catalogue are not read here, so they would be neither registered nor reported
  • two links resolving to one id (…/2018/x.json and …/2019/x.json): an id set collapses them, and the id comparison would read in sync

Examples:

>>> import json, os, tempfile
>>> tmp = tempfile.TemporaryDirectory()
>>> path = os.path.join(tmp.name, "collection.json")
>>> links = [{"rel": "self", "href": "https://example.invalid/collection.json"},
...          {"rel": "item", "href": "items/a%20b.json"},
...          {"rel": "item", "href": "items/c.json"}]
>>> with open(path, "w") as f:
...     json.dump({"id": "my-collection", "links": links}, f)
>>> collection_item_links(path)
[('a b', 'items/a%20b.json'), ('c', 'items/c.json')]
>>> tmp.cleanup()

fetch_key(url)

The fetch file's basename for a URL -- md5 of the URL, so ids with spaces and parentheses need no quoting anywhere downstream.

Examples:

>>> fetch_key("https://example.invalid/items/a b.json")
'1af97db74cefffe0d9b97fe9da8c267f'

fetch_bodies(urls, out_dir, workers=32, retries=3, timeout=60, backoff=2.0)

Fetch each URL to out_dir/<fetch_key(url)>.json. Returns the URLs that failed after retries attempts, in input order.

In-process with a thread pool rather than one curl per item: a shell loop took ~45 min for a 102k-item catalogue; 32 threads in one process fetched 2,000 in 4.9 s (measured 2026-09-28).

A body counts only if it parses as a JSON object -- a truncated transfer is a failed attempt, retried, not a file that counts as present. Written to .part and renamed, so a failure never leaves a partial file behind.

out_dir must be EMPTY. published_digests reads "the file exists" as "this run fetched it", which holds only if nothing else wrote there: a body left by an earlier run would be digested as current, and an item the publisher rebuilt would read unchanged.

Examples:

file:// URLs read without any network:

>>> import json, os, pathlib, tempfile
>>> tmp = tempfile.TemporaryDirectory()
>>> bucket = pathlib.Path(tmp.name, "bucket")
>>> bucket.mkdir()
>>> _ = (bucket / "a.json").write_text(json.dumps({"id": "a"}))
>>> urls = [(bucket / "a.json").as_uri(), (bucket / "gone.json").as_uri()]
>>> fetch_dir = pathlib.Path(tmp.name, "fetched")
>>> fetch_dir.mkdir()
>>> failed = fetch_bodies(urls, fetch_dir, retries=1)
>>> [os.path.basename(u) for u in failed]
['gone.json']
>>> sorted(os.listdir(fetch_dir)) == [fetch_key(urls[0]) + ".json"]
True
>>> tmp.cleanup()

published_digests(links, fetch_dir)

{id: body_digest} from the fetched bodies of links [(id, href)].

fetch_dir must be one populated by a single fetch_bodies call (which refuses a non-empty directory), so a file present is a body this run read.

Every way a published digest could be absent raises -- a body not fetched, not parseable, or naming a different id than its link (which would compare the wrong pair) -- and so does an id with two links, which a dict would otherwise collapse to whichever came last.

Examples:

>>> import json, pathlib, tempfile
>>> from stacs.verify import body_digest
>>> tmp = tempfile.TemporaryDirectory()
>>> href = "https://example.invalid/items/a.json"
>>> body = {"id": "a"}
>>> fetched = pathlib.Path(tmp.name, fetch_key(href) + ".json")
>>> _ = fetched.write_text(json.dumps(body))
>>> published_digests([("a", href)], tmp.name) == {"a": body_digest(body)}
True
>>> published_digests([("b", href)], tmp.name)
Traceback (most recent call last):
...
ValueError: link for 'b' fetched a body that names id 'a' (...)
>>> tmp.cleanup()

read_hrefs(path)

[(id, href)] from a tab-separated id/href file.

Examples:

>>> import os, tempfile
>>> tmp = tempfile.TemporaryDirectory()
>>> path = os.path.join(tmp.name, "hrefs.tsv")
>>> with open(path, "w") as f:
...     _ = f.write("a b\thttps://example.invalid/items/a%20b.json\n")
>>> read_hrefs(path)
[('a b', 'https://example.invalid/items/a%20b.json')]
>>> tmp.cleanup()