What crate knows about
Every file or table crate knows about is a row in its registry, keyed
by a source family and a file name. A file entry is one
crate reads and normalizes with crt_ingest(). A
schema_only entry declares a canonical shape for data crate
does not read, which you hold to that shape yourself with
crt_schema_conform().
crt_files()
#> # A tibble: 16 × 6
#> source file_name kind handler_fn schema_yaml canonical_cols
#> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 bcfp user_habitat_classificati… file crt_handl… schemas/bc… blue_line_key…
#> 2 nge track_sessions sche… NA schemas/ng… project,sessi…
#> 3 nge track_vertices sche… NA schemas/ng… project,sessi…
#> 4 nge track_annotations sche… NA schemas/ng… project,sessi…
#> 5 nge form_capture sche… NA schemas/ng… project,form,…
#> 6 nge discharge/fwa/v1/segment_… sche… NA schemas/ng… linear_featur…
#> 7 nge discharge/fwa/v1/segment_… sche… NA schemas/ng… linear_featur…
#> 8 nge discharge/fwa/v1/watershe… sche… NA schemas/ng… watershed_fea…
#> 9 nge form_pscis sche… NA schemas/ng… project,form,…
#> 10 nge form_fiss_site sche… NA schemas/ng… project,form,…
#> 11 nge form_monitoring_fish_pass… sche… NA schemas/ng… project,form,…
#> 12 nge form_monitoring_general sche… NA schemas/ng… project,form,…
#> 13 nge form_monitoring sche… NA schemas/ng… project,form,…
#> 14 nge fish_data file crt_handl… schemas/ng… local_name,sp…
#> 15 nge photo_files sche… NA schemas/ng… path,checksum…
#> 16 nge photo_links sche… NA schemas/ng… checksum,proj…One call, either shape
bcfishpass
changed user_habitat_classification.csv on 2026-04-26 from
a long format, one row per segment, species and habitat type with a text
habitat_ind of "t" or "f", to a
wide format with integer spawning and rearing
columns. crate ships a small fixture of each:
long_path <- system.file(
"extdata/examples/bcfp/user_habitat_classification_long.csv",
package = "crate"
)
wide_path <- system.file(
"extdata/examples/bcfp/user_habitat_classification_wide.csv",
package = "crate"
)
names(readr::read_csv(long_path, show_col_types = FALSE))
#> [1] "blue_line_key" "downstream_route_measure"
#> [3] "upstream_route_measure" "watershed_group_code"
#> [5] "species_code" "habitat_type"
#> [7] "habitat_ind" "reviewer_name"
#> [9] "review_date" "source"
#> [11] "notes"
names(readr::read_csv(wide_path, show_col_types = FALSE))
#> [1] "blue_line_key" "downstream_route_measure"
#> [3] "upstream_route_measure" "watershed_group_code"
#> [5] "species_code" "spawning"
#> [7] "rearing" "reviewer_name"
#> [9] "review_date" "source"
#> [11] "notes"The same call reads both to the canonical wide shape:
long <- crt_ingest("bcfp", "user_habitat_classification", long_path)
wide <- crt_ingest("bcfp", "user_habitat_classification", wide_path)
identical(names(long), names(wide))
#> [1] TRUE
str(long[c("spawning", "rearing")])
#> tibble [5 × 2] (S3: tbl_df/tbl/data.frame)
#> $ spawning: int [1:5] 1 NA 1 NA 1
#> $ rearing : int [1:5] 1 1 NA 1 1Five pieces at runtime
When you call
crt_ingest("bcfp", "user_habitat_classification", path),
five pieces wire together:
-
Registry (
inst/extdata/crate_registry.csv) maps each(source, file_name)pair to its kind, its schema YAML and, for afileentry, its handler function. It is whatcrt_files()prints above. -
Schema YAML (
inst/extdata/schemas/bcfp/user_habitat_classification.yaml) declares the canonical columns (names, types, required flags) and each known upstream variant: a set of column names and an id.crt_schema_read()loads it. -
Handler (
R/crt_handler_bcfp_user_habitat_classification.R), one function perfileentry. crate matches the file’s column names against each variant’s (the first set-equal match wins) and passes the matched variant’s id to the handler. Here2026-04-26-widepasses the data through, andpre-2026-04-26-longpivots the long rows to wide, turning thehabitat_indtext into integer indicators in the same step. -
Validation (
crt_schema_validate()) checks that every column the schema declaresrequired: trueis present, and fails listing all that are missing. -
Type enforcement (
crt_schema_apply()) coerces each declared column the data has to its schema type (integer,double,string,logicalordatetime). Whatever types a handler returns, the schema’s are the ones that come out, and readr’s guesses (a double for an integer column, aDatefor a declared string) do not leak through.
Steps 4 and 5 are crt_schema_conform(), which
crt_ingest() calls rather than repeats, so a file crate
reads and a data frame you bring get the same treatment. The handler,
crt_schema_read(), crt_schema_validate() and
crt_schema_apply() are internal: you call
crt_ingest() or crt_schema_conform(), and they
call the rest.
A datetime is an instant
A datetime column is an instant, and crate never moves
it. A POSIXct has its timezone attribute stamped
to UTC rather than converted. Several readers return correct UTC
instants with no timezone label, which R then renders in the session’s
zone, so they read as a clock in another timezone. Converting them to
“fix” that rendering would put a whole-hour error into every join
downstream. Stamping fixes the label, which was the only thing
wrong.
Variant matching uses column names only
crt_ingest() picks the upstream variant by the set of
column names alone. It does not compare column types at that step, so an
upstream file that kept its column names and changed a type would reach
the handler misshapen. crt_schema_apply() then coerces the
output, and some coercions turn a value into NA without an
error (as.integer("yes")). Checking types at dispatch is crate#55.
Required columns and output types are still enforced, by steps 4 and 5.
Registering a file or table
How to add a (source, file_name) pair, a schema, a
registry row, a handler for a file entry, a decision entry
and tests with fixtures, is in inst/extdata/schemas/README.md.
