# DigitalOcean Droplet Setup — STAC API Stack

STAC API for UAV imagery at [images.a11s.one](https://images.a11s.one).

> **The droplet is built from our internal infrastructure-as-code repo** (OpenTofu + cloud-init → `/opt/geoserv/docker-compose.yml`). Server build questions belong there; this directory documents the catalog-side workflow. Everything below the **Historical reference** heading describes the previous stack and is kept for context only.

## Current Stack (geoserv, 2026)

Containers on the droplet (`root@146.190.12.8`, hostname `geopro`):

| Container | Image | Port | Serves |
|---|---|---|---|
| `geoserv-stac` | `stac-fastapi-pgstac` | 8000 | `images.a11s.one` |
| `geoserv-stac-ortho` | `stac-fastapi-pgstac` | 8002 | ortho catalog (db `stac_ortho`) |
| `geoserv-db` | `pgstac:v0.9.8` | 5432 | postgres, user/db `stac` |
| `geoserv-titiler` | titiler | 8001 | `titiler.a11s.one` |
| `geoserv-caddy` | caddy | 80/443 | TLS + reverse proxy |

The public API is **read-only**: the transactions extension is off, so POST/PUT/DELETE return 405. That is deliberate — writes go through pypgstac on the droplet host (installed by the server build at `/opt/geoserv/scripts` via `uv`), driven over ssh by [`stacs`](https://github.com/NewGraphEnvironment/stacs) (#35).

## Registration and verification: stacs

`scripts/stacs.sh` runs the `stacs` CLI at the tag pinned on its one `STACS_REF` line (through `uvx`; no env to install). `stacs.toml` at the repo root declares this catalogue — API, collection id, bucket, the `image` asset every item must carry, and the transport to the droplet. No setting is a secret: the database password stays on the host and is named (`password_env`), never passed.

```bash
scripts/stacs.sh verify   --config stacs.toml --out-dir /tmp/verify   # changes nothing
scripts/stacs.sh register --config stacs.toml --mode drift             # what the API lacks or serves stale
```

Both read the **published** catalogue (`collection.json` on the bucket and the item links in it), so they run after the S3 sync, never before. `verify` compares id sets in both directions and every body by digest, and exits 1 on any drift, writing `missing.txt`, `changed.txt` and `orphaned.txt` to `--out-dir`. `register` is upsert-only, writes the collection before items, and checks that the API serves what it sent. `catalogue_release.sh` and `dataset_publish.sh` run both, so you need them by hand only for a one-off check.

The item links have only been correct since #35. Before that, every `rel: item` href was `<bucket>/<id>/<id>.json`, a key S3 never held, because `Collection.add_item()` re-homed each item under pystac's default layout. `item_create.py` now keeps the tree path and refuses to save a `collection.json` whose item links name no file in the prod tree. An additive `item_create.py` run (inside `dataset_publish.sh`) refuses until one `--rebuild` has rewritten the old links, so the first release after #35 has to come before the next publish.

The same holds for #38's field rename: an additive run refuses while any item in the prod tree still carries an `nge:` field, so the first publish after #38 waits for a full release (NEWS entry naming each old → new field, tag, `catalogue_release.sh`). Until that release the API serves the `nge:` names. A tif copied into the prod tree by a refused publish is built and published by that release, so its `sites.csv` row has to exist first.

## The Registry: data/sites.csv

`data/sites.csv` is the **source of truth** for site metadata — stream names (up to three per flight), FWA watershed group codes and names, bcfishpass `aggregated_crossings_id`, aliases, project, `published` flag, and notes. Items carry it as `title` + queryable properties under prefixes that say what each field describes (`newgraph:`, `uav:`, `fwa:`, `bcfishpass:`; the field table is in the top-level README; `nge:` names until the release that ships #38), and item geometries are true `gdal_footprint` outlines. The catalogue is a versioned build artifact of (sites.csv, COGs).

`watershed_group_name` is the atlas's name for `watershed_group_code`, not something to type: after adding or changing a row's code, run `python3 scripts/sites_fill-wsg_name.py` (reports) and then with `--write` (fills it in place from the public fwapg API). The `watershed` column is the tree's directory slug and stays as it is.

**Releasing a registry change:**

1. Edit `data/sites.csv` (and `scripts/sites_fill-wsg_name.py --write` if a watershed group code changed); add a `NEWS.md` entry; commit
2. `git tag vX.Y.Z`
3. `scripts/catalogue_release.sh` — rebuilds all items, validates + audits (`--expect` is the collection's link count, so a stray item JSON fails here), syncs, `stacs register --mode drift`, `stacs verify`, then checks the version stamp and registry coverage (~5 min, idempotent)

**Retracting a dataset** (#18 — check nothing external links the URLs first, e.g. published reports):

1. Flip its `data/sites.csv` row to `published=false` with a dated note
2. Remove the dataset dir from the prod tree and the `imagery_uav_bc` COG tree
3. `aws s3 rm s3://imagery-uav-bc/<region>/<wsg>/<year>/<item> --recursive --profile airvine`
4. `scripts/config/item_unregister.sh <item-id>...` (idempotent; API 404s confirm). Skip it and the release's `stacs verify` fails, reporting the id as orphaned: registered but no longer published
5. Release as usual (NEWS + tag + `catalogue_release.sh`) — the rebuild drops the collection links

## Adding New Imagery — the Recipe

Two commands with a human QC gate between them:

1. Stitch — `caffeinate -s scripts/odm_process-batch.sh <project-dir>...` (skips already-processed dirs; resume-safe after interruption)
2. QC the ortho — check `odm_report/stats.json` (all images reconstructed? reprojection error ~1-2 px?) and eyeball a preview
3. Publish — `scripts/dataset_publish.sh <project-dir>...` (COG convert + validate → prod tree → items with flight datetimes → whole-tree `stacs validate` + `audit --expect` → durable S3 upload → collection extent → `stacs register --mode drift` → `stacs verify`; idempotent, safe to re-run)

The underlying single-purpose tools these orchestrate, for one-off use:

- `scripts/item_create.py <tifs relative to prod tree>` — items + collection.json (conda env `titiler`)
- `scripts/stacs.sh register --config stacs.toml --mode drift` — upsert the collection, then every item the API lacks or serves stale (reads the published catalogue: sync first)
- `scripts/stacs.sh verify --config stacs.toml --out-dir <dir>` — what the API serves against what is published
- `find <prod tree> -name '*.json' ! -name collection.json | scripts/stacs.sh validate` (or `audit --config stacs.toml`) — pystac validation / collection + asset audit; `--dir` does not recurse, so pass paths on stdin
- `conda run -n dff rio cogeo create <in> <out>` — COG conversion (see `scripts/cog_convert.R` for the batch-log history)
- Dev tree/bucket retired 2026-07 — prod only

### Repeat flights of one site (before/after a replacement)

A site reflown after a culvert replacement produces two datasets that share a stream name **and** a
year, so their titles collide — `item_create.py` builds `"<stream_name> — <year> <product>"`. The
`alias` column is what separates them: set it on **both** rows (e.g. `moose pre-replacement` /
`moose post-replacement`) and the title becomes `"<stream_name> (<alias>) — <year> <product>"`.
Aliases are optional; a row without one titles exactly as before.

Give the second flight its own `item` (a `_post_replacement` suffix on the first one's reads well),
and cross-reference the pair in both rows' `notes` — nothing in the schema links them otherwise.

### Renaming a published dataset

A rename is a retraction + republish, since the item id is built from the directory path. Order
matters, and two steps are easy to miss:

1. Rename the dataset dir in **all three** trees — raw `uav_imagery/`, COG `imagery_uav_bc/`, prod
   `stac/prod/imagery_uav_bc/`. `dataset_publish.sh` derives the prod path (and so the item id) from
   the raw path, so a partial rename silently republishes under the old id.
2. **Delete the old-id JSONs from the renamed prod dir.** `item_create.py --rebuild` writes
   `<new-id>.json` beside each tif but never removes JSONs whose id no longer matches, so the stale
   ones survive the rebuild and get pushed to S3 alongside the new ones.
3. Update the row's `item` in `data/sites.csv` — the registry key is `(region, watershed, year, item)`.
4. `item_unregister.sh` the old ids **before** the publish or release, whose closing `stacs verify`
   otherwise fails on them. The final `aws s3 sync --delete` clears the old S3 prefix on its
   own (prod is authoritative for the bucket), but pgstac is not driven by the sync — and stacs is
   upsert-only, so `stacs verify` reports the old ids as orphaned until they are unregistered.
   Skipping step 2 is caught too: the release audit refuses an item JSON the collection does not link.

Check for external dependents first, exactly as in the retraction recipe above.

### Large datasets (>~300 images)

Single-pass ODM memory scales with the whole flight and the meshing stage can run 10+ hours or OOM (observed: 817 images → 20+ h, >70 GB RAM). Use split-merge instead:

```bash
caffeinate -s scripts/odm_process-batch.sh --split 250 <project-dir>
```

ODM carves the flight into ~250-image submodels (100 m overlap), processes each like a normal dataset, and merges orthos/DEMs at the end — memory bounded per submodel, progress checkpointed between them. Other escape hatches: `--fast-orthophoto` (skips dense reconstruction; ortho only, no usable DEMs) and, when a meshing-stage OOM needs rescue, rerun with a lower `--mesh-octree-depth` from `--rerun-from odm_meshing`. For routinely-large campaigns consider a NodeODM ephemeral droplet (tracked in the infrastructure repo).

### Gotchas

- **s3://imagery-uav-bc has ACLs disabled** — public read comes from the bucket policy. Never pass `--acl`; `put-object-acl` fails with `AccessControlListNotSupported`.
- **TiTiler caches failed lookups**: if a COG is probed before its upload finishes, GDAL caches the 403 and the URL stays broken. Fix: `ssh root@146.190.12.8 docker restart geoserv-titiler`.
- Full collection rebuild and collection deletion are **server-side operations** — that tooling lives with the server build in our internal infrastructure repo, not here.

---

## Historical reference (previous stack)

> **Note:** The numbered setup scripts (`01_server.sh`, `02_server.sh`, etc.) are `.gitignore`d — they contain credentials and server config.

## Infrastructure

- **Provider:** DigitalOcean (single droplet)
- **OS:** Ubuntu (18.04+ scripts, currently running 22.04)
- **Domain:** `a11s.one` with subdomains via Nginx reverse proxy (current), Caddy recommended for OpenTofu rebuild
- **SSL:** Let's Encrypt via Certbot (current), automatic via Caddy (recommended)

## Services

| Service | Runtime | Port | Subdomain | Image / Package |
|---|---|---|---|---|
| STAC FastAPI (pgstac) | Python venv | 8000 | `images.a11s.one` | `stac-fastapi.pgstac` (pip) |
| PostgreSQL + PostGIS + pgstac | Docker | 5432 | — | `postgis/postgis` |
| TiTiler | Docker | 8001 | `titiler.a11s.one` | `developmentseed/titiler:latest` |
| RStudio Server | System | 8787 | `rstudio.a11s.one` | (installed separately) |
| Static viewer | Nginx | 443 | `viewer.a11s.one` | Files at `/var/www/html/viewer` |

## Networking

- **Nginx** reverse-proxies all subdomains to localhost ports
- **UFW** firewall — allows OpenSSH, port 8001 (TiTiler)
- All HTTP traffic redirects to HTTPS
- CORS headers on `titiler.a11s.one` and `viewer.a11s.one` (GET, OPTIONS)
- ACME challenge passthrough on all port-80 server blocks

## Database

- **Container name:** `postgis`
- **Credentials:** user=`stac`, password=`stacpw`, db=`stac`
- **Extensions:** PostGIS
- **Schema:** `pgstac` (applied via `pypgstac migrate`)
- **Roles:** `pgstac_admin`, `pgstac_ingest`, `pgstac_read`
- Search path set to `pgstac, public` for the `stac` user

## Python Environment

- Virtual env at `/home/airvine/stac-env/`
- Key packages: `stac-fastapi.pgstac`, `pypgstac`, `psycopg`, `psycopg_pool`, `stac-validator`
- STAC API started via: `stac-fastapi-pgstac --host 0.0.0.0 --port 8000`
- Optional rate-limit middleware in `main.py` (caps `/search` POST to 1000 items)

## System Tuning

- 2GB swap file at `/swapfile` (persisted in `/etc/fstab`)
- `vm.swappiness=10`
- `vm.vfs_cache_pressure=50`

## Setup Order

1. `01_server.sh` — User creation, SSH hardening, UFW
2. `02_server.sh` — Docker, Python venv, PostgreSQL/pgstac, STAC FastAPI, swap
3. `02b_stac_fast_api.sh` — Alternative STAC API startup (from venv, no Docker)
4. `03_certbot.sh` — SSL certificates for all subdomains
5. `04_titiler.sh` — TiTiler Docker container on port 8001

Post-setup:
- Collection registration/unregistration scripts (since moved to the internal infrastructure repo)

## Data Flow

```
S3 buckets (COGs)                    Clients (rstac, QGIS)
  imagery-uav-bc.s3.amazonaws.com           │
        │                                   ▼
        │                          images.a11s.one (Nginx)
        │                                   │
        │                                   ▼
        │                          STAC FastAPI :8000
        │                                   │
        │                                   ▼
        │                          PostgreSQL/pgstac :5432
        │
        └──────────────────────► TiTiler :8001 (tile serving)
                                   titiler.a11s.one (Nginx)
```

## Migration Strategy

Use a **DigitalOcean reserved IP** for zero-downtime cutover between the current droplet and the new OpenTofu-managed one.

1. Assign a reserved IP to the **current** droplet, update DNS A records for `a11s.one` and all subdomains to point to it
2. Build new droplet with OpenTofu, test via its direct IP
3. Reassign the reserved IP to the new droplet (instant cutover)
4. Caddy auto-provisions SSL certs within seconds of receiving traffic
5. Keep old droplet around briefly as fallback, then destroy

DNS A records point to the reserved IP permanently — future rebuilds just reassign it.

```hcl
resource "digitalocean_reserved_ip" "main" {
  region = "sfo3"
}

resource "digitalocean_reserved_ip_assignment" "main" {
  ip_address = digitalocean_reserved_ip.main.ip_address
  droplet_id = digitalocean_droplet.stac.id
}
```

**Cert chicken-and-egg:** Caddy needs the domain pointing at it to get certs (HTTP-01 challenge). On the new server, test via IP first (Caddy serves self-signed), then reassign the reserved IP and real certs provision automatically. Alternatively, use DNS-01 challenges with Caddy's DigitalOcean plugin to provision certs before cutover.

## Notes for OpenTofu Translation

- **Replace Nginx + Certbot with Caddy** — eliminates `03_certbot.sh`, the 160-line nginx config, renewal timers, and ACME challenge blocks. Caddy handles HTTPS automatically. Target Caddyfile:

```caddyfile
images.a11s.one {
	reverse_proxy localhost:8000
}

rstudio.a11s.one {
	reverse_proxy localhost:8787
}

titiler.a11s.one {
	reverse_proxy localhost:8001
	header Access-Control-Allow-Methods "GET, OPTIONS"
	header Access-Control-Allow-Headers "Content-Type"
}

viewer.a11s.one {
	root * /var/www/html/viewer
	file_server
	header Access-Control-Allow-Methods "GET, OPTIONS"
	header Access-Control-Allow-Headers "Content-Type"
}
```

- PostgreSQL and TiTiler are Docker containers — could become managed services or remain containerized
- STAC FastAPI runs from a Python venv, not Docker — consider containerizing for consistency
- The `02_server.sh` script was assembled incrementally and may not run cleanly end-to-end
- Database credentials are hardcoded — use secrets management in IaC
- `.msmtprc` is a Gmail SMTP template (not currently active in the stack)