Geospatial Compliance Metadata Automation Hub

Writing a Declarative Crosswalk Table in YAML

Declare each mapping as a YAML record carrying a source path, a target path, a named transform, a cardinality, an absence policy and a provenance flag — six fields, no conditionals — and the crosswalk becomes something a domain expert can review in a pull request rather than something only its author can change.

The pull towards putting crosswalk logic in Python is strong and worth resisting for a specific reason: the people who know whether a mapping is correct are usually not the people who can read the code. A metadata specialist can check that idinfo/citation/citeinfo/title should become identificationInfo/citation/title and that a missing title is a failure rather than a default. They cannot check that assertion inside a function with three branches. This guide is part of metadata crosswalks and field mapping tables, under Automated Metadata Generation & Schema Mapping.

The six fields of a crosswalk ruleEach rule carries locating fields, a transform, a multiplicity declaration, an absence policy and a provenance flag.Provenancefor the auditcarried or suppliedrule idnoteBehaviourfor the awkward recordscardinalityon_absentdefaultValuetransformvocabularyLocationthe obvious partsourcetarget
Six fields, and the bottom three are the ones hand-written crosswalks omit.

Automated Python Implementation

The loader below validates the rule file against a schema before anything is applied. A crosswalk with a typo in a transform name should fail on load, not on the four-hundredth record.

#!/usr/bin/env python3
"""Load and validate a declarative crosswalk table."""
import yaml
from jsonschema import Draft202012Validator

RULE_SCHEMA = {
    "type": "object",
    "required": ["id", "source", "target", "transform", "cardinality", "on_absent"],
    "additionalProperties": False,
    "properties": {
        "id": {"type": "string", "pattern": "^[A-Z]{2,4}[0-9]{3}$"},
        "source": {"type": "string", "minLength": 1},
        "target": {"type": "string", "minLength": 1},
        "transform": {"type": "string"},
        "vocabulary": {"type": "string"},
        "cardinality": {"enum": ["one", "many"]},
        "on_absent": {"enum": ["fail", "omit", "default"]},
        "default": {},
        "note": {"type": "string"},
    },
}

CROSSWALK_SCHEMA = {
    "type": "object",
    "required": ["version", "source_standard", "target_standard", "rules"],
    "properties": {
        "version": {"type": "string"},
        "source_standard": {"type": "string"},
        "target_standard": {"type": "string"},
        "rules": {"type": "array", "minItems": 1, "items": RULE_SCHEMA},
    },
}


def load_crosswalk(path, transforms, vocabularies):
    with open(path, encoding="utf-8") as fh:
        doc = yaml.safe_load(fh)

    errors = sorted(
        Draft202012Validator(CROSSWALK_SCHEMA).iter_errors(doc),
        key=lambda e: list(e.path),
    )
    problems = [f"{'/'.join(str(p) for p in e.path)}: {e.message}" for e in errors]

    seen = set()
    for rule in doc.get("rules", []):
        if rule.get("id") in seen:
            problems.append(f"duplicate rule id: {rule['id']}")
        seen.add(rule.get("id"))
        if rule.get("transform") not in transforms:
            problems.append(f"{rule.get('id')}: unknown transform {rule.get('transform')!r}")
        if "vocabulary" in rule and rule["vocabulary"] not in vocabularies:
            problems.append(f"{rule['id']}: unknown vocabulary {rule['vocabulary']!r}")
        if rule.get("on_absent") == "default" and "default" not in rule:
            problems.append(f"{rule['id']}: on_absent is 'default' but no default given")

    if problems:
        raise SystemExit("crosswalk invalid:\n  " + "\n  ".join(problems))
    return doc

A rule file that passes that loader looks like this. The id is not decoration: it appears in every error message, in the provenance of every value the rule produced, and in any suppression a team records — which is why it must never be reused for a different mapping.

EXAMPLE = """
version: "2026.1"
source_standard: "FGDC CSDGM"
target_standard: "ISO 19115-3"
rules:
  - id: IDN001
    source: idinfo/citation/citeinfo/title
    target: identificationInfo/citation/title
    transform: text
    cardinality: one
    on_absent: fail

  - id: IDN014
    source: idinfo/keywords/theme/themekey
    target: identificationInfo/descriptiveKeywords/keyword
    transform: text
    cardinality: many
    on_absent: omit

  - id: IDN021
    source: idinfo/status/progress
    target: identificationInfo/status
    transform: vocabulary
    vocabulary: progress_codes
    cardinality: one
    on_absent: default
    default: unknown
    note: "CSDGM 'Suspended' has no ISO counterpart; see the residue policy"

  - id: MDM003
    source: metainfo/metstdn
    target: metadataStandardName
    transform: constant
    cardinality: one
    on_absent: default
    default: "ISO 19115-3"
    note: "supplied by the pipeline; never carried from the source"
"""
What each absence policy means in practiceThe three on_absent values compared by what the run does, what the output contains and how the record is reported.Run doesOutput containsReported asfailstops the recordnothinga failure with the rule idomitcontinuesno target elementan entry in the residue listdefaultcontinuesthe declared defaultan entry in the supplied list
The default policy is the only one that puts a value in the output that no source supplied — which is why it forces a provenance flag.

Validation and Pipeline Integration

Three checks belong in CI, and they run in well under a second because they read the rule file rather than any data.

def test_crosswalk_loads():
    load_crosswalk("crosswalk.yaml", TRANSFORMS, VOCABULARIES)


def test_every_mandatory_target_has_a_rule():
    doc = load_crosswalk("crosswalk.yaml", TRANSFORMS, VOCABULARIES)
    targets = {rule["target"] for rule in doc["rules"]}
    missing = MANDATORY_TARGETS - targets
    assert not missing, f"no rule writes: {sorted(missing)}"


def test_rule_ids_are_stable():
    """Rule ids appear in provenance records; renaming one rewrites history."""
    doc = load_crosswalk("crosswalk.yaml", TRANSFORMS, VOCABULARIES)
    current = {rule["id"]: rule["target"] for rule in doc["rules"]}
    for rule_id, target in RECORDED_RULE_TARGETS.items():
        assert current.get(rule_id, target) == target, \
            f"rule {rule_id} changed target; mint a new id instead"

The second test is the one that finds real gaps. A crosswalk is usually written by working through the source standard, which guarantees that every source field is considered and says nothing about whether every mandatory target is reached. Asserting from the target side finds the elements that no rule writes, which is where records fail validation for reasons the crosswalk author never saw.

Choosing an absence policy for a ruleThree questions decide whether an absent source should fail the record, be defaulted, or simply omit the target.Is the target mandatory in theschema?fail, or defaultthe record is invalid without ityesnoIs a defensible default available?a property of the record, not of the datadefaultand flag it as suppliedyesnoomitrecorded in the residue list
Defaulting is the option that requires a justification, because it puts data in the output that nobody supplied.

Keeping the File and the Standard in Step

A crosswalk file is a claim about two external standards, and both move. Two habits keep the claim honest without turning maintenance into a project.

Pin both standard versions in the header and treat a version bump as a code change. source_standard: "FGDC CSDGM" is a family; "FGDC CSDGM 1998" is a fact. When a target standard revises — adding a mandatory element, deprecating a code list — the pinned version is what tells a reviewer whether the file has been reconciled with it.

Generate the mandatory-target list from the schema rather than maintaining it. The test above needs a set of mandatory targets, and hand-maintaining that set reintroduces exactly the drift the declarative file was meant to remove. Extracting it from the target XSD or SHACL shapes at test time costs a few lines and means a new mandatory element in the standard fails the build the day the vendored schema is updated.

Together these produce a useful property: updating the vendored target schema is the event that surfaces every consequence of a standard revision, in one build, with the missing rules named. That is the opposite of the usual experience, in which a standard revision is discovered record by record over the following months.

Long-Term Compliance Best Practices

  • Never reuse a rule id. Ids end up in provenance records and suppression lists; retiring one and minting another costs nothing and keeps history interpretable.
  • Keep vocabularies in separate files, referenced by name. They change on a different schedule from the mappings and are frequently shared between crosswalks.
  • Comment the decisions, not the mechanics. A note explaining why a source has no target earns its line; a note restating the rule does not.
  • Order rules by target section in schema order. It makes missing mandatory targets visible to a reviewer scanning the file, which no amount of tooling replaces.
  • Version the file and record the version on every output. A corrected rule then identifies exactly the records that need regenerating.