# Metadata Crosswalks & Field Mapping Tables

A crosswalk is the table that says which field in one metadata standard becomes which field in another, and everything hard about metadata interoperability is a consequence of that table being treated as code rather than as configuration. This topic sits under Automated Metadata Generation & Schema Mapping and covers how to declare a crosswalk, apply it deterministically, and handle the fields on both sides that have no counterpart.

# Prerequisites

1. Python 3.11+ with lxml==5.2.2 for XML sources and PyYAML==6.0.1 for the crosswalk declaration.
2. A source record you can read and a target schema you can validate against. A crosswalk with no validation on the far side is a transformation nobody can check.
3. Vocabulary files vendored locally for both standards, since code list mapping is the second-largest source of crosswalk failures.
4. A decision, in advance, about what happens to unmappable fields. This is a policy question, not an implementation detail, and it is addressed in handling unmappable fields in a metadata crosswalk.

# Concept & Spec Reference

A crosswalk rule is not simply a pair of field names. Four properties are needed before a rule can be applied without a human present, and omitting any one of them is what turns a declared crosswalk back into code.

The transform property deserves particular attention because it covers three distinct kinds of change that are often conflated.

Type transforms convert a value’s representation: a date string in an unspecified format into an ISO 8601 date, a decimal degree string into a number. These are the easy case and they fail loudly.

Vocabulary transforms map a value from one controlled list to another. Progress: Complete in one standard becomes a codeListValue in the other, drawn from a list that may not contain an exact counterpart. These fail quietly, because an unmapped value that is written through unchanged produces a document that validates structurally and means nothing.

Structural transforms change the shape rather than the value: one source element becoming three target elements, or four source fields collapsing into one composite. These cannot be expressed as a path pair at all, and a crosswalk format that does not accommodate them forces the hard cases into code where they become invisible.

# Implementation Walkthrough

# Step 1 — Flatten the source

Working against a nested tree makes every rule a navigation problem. Flattening to path-keyed values makes every rule a lookup, which is what allows the rules to live in a data file.

"""Flatten an XML document to a dictionary of path -> list of values."""
from collections import defaultdict

from lxml import etree


def flatten(xml_path, namespaces=None):
    tree = etree.parse(xml_path)
    root = tree.getroot()
    values = defaultdict(list)
    for element in root.iter():
        if element.text and element.text.strip():
            path = tree.getelementpath(element)
            # Strip positional predicates so repeated elements share a path.
            key = "/".join(part.split("[")[0] for part in path.split("/"))
            values[key].append(element.text.strip())
    return dict(values)

Stripping the positional predicate is the decision that makes multiplicity explicit: every path maps to a list, and a rule that expects one value has to say so.

# Step 2 — Declare the rules

The rules are data. Anything that cannot be expressed as data belongs in a named transform function, referenced by name from the data.

import yaml

CROSSWALK = yaml.safe_load("""
- source: idinfo/citation/citeinfo/title
  target: identificationInfo/citation/title
  transform: first
  on_absent: fail
- source: idinfo/timeperd/timeinfo/sngdate/caldate
  target: identificationInfo/citation/date
  transform: iso_date
  on_absent: omit
- source: idinfo/status/progress
  target: identificationInfo/status
  transform: progress_code
  on_absent: default
  default: unknown
- source: idinfo/keywords/theme/themekey
  target: identificationInfo/descriptiveKeywords/keyword
  transform: all
  on_absent: omit
""")

# Step 3 — Apply them

The applier is small on purpose. A crosswalk engine that grows conditionals is one whose rule format is not expressive enough.

import datetime

PROGRESS = {"complete": "completed", "in work": "onGoing", "planned": "planned"}


def t_first(values):
    return values[0] if values else None


def t_all(values):
    return list(values)


def t_iso_date(values):
    raw = t_first(values)
    if raw is None:
        return None
    for fmt in ("%Y%m%d", "%Y-%m-%d", "%Y"):
        try:
            return datetime.datetime.strptime(raw, fmt).date().isoformat()
        except ValueError:
            continue
    raise ValueError(f"unparseable date: {raw!r}")


def t_progress_code(values):
    raw = (t_first(values) or "").strip().lower()
    if raw and raw not in PROGRESS:
        raise KeyError(f"unmapped progress value: {raw!r}")
    return PROGRESS.get(raw)


TRANSFORMS = {"first": t_first, "all": t_all,
              "iso_date": t_iso_date, "progress_code": t_progress_code}


def apply_crosswalk(flat, rules):
    out, residue, supplied = {}, [], []
    for rule in rules:
        values = flat.get(rule["source"], [])
        result = TRANSFORMS[rule["transform"]](values) if values else None
        if result in (None, []):
            policy = rule.get("on_absent", "omit")
            if policy == "fail":
                raise ValueError(f"required source absent: {rule['source']}")
            if policy == "default":
                result = rule["default"]
                supplied.append(rule["target"])
            else:
                residue.append(rule["source"])
                continue
        out[rule["target"]] = result
    return out, {"omitted": residue, "supplied": supplied}

The second return value is what makes the run auditable. A crosswalk that reports only its output cannot answer the two questions anyone asks afterwards: what did we lose, and what did the pipeline make up.

# Step 4 — Assemble in schema order

Target standards constrain element sequence as well as presence, so the assembler walks the schema’s order rather than the crosswalk’s.

TARGET_ORDER = [
    "identificationInfo/citation/title",
    "identificationInfo/citation/date",
    "identificationInfo/abstract",
    "identificationInfo/status",
    "identificationInfo/descriptiveKeywords/keyword",
]


def assemble(mapped, order=TARGET_ORDER):
    """Emit path/value pairs in the target schema's declared sequence."""
    return [(path, mapped[path]) for path in order if path in mapped]

# Validation & CI Integration

A crosswalk needs three assertions, and the third is the one that catches silent damage.

def test_every_rule_names_a_known_transform():
    for rule in CROSSWALK:
        assert rule["transform"] in TRANSFORMS


def test_unmapped_vocabulary_value_raises():
    try:
        t_progress_code(["Suspended"])
    except KeyError:
        return
    raise AssertionError("an unmapped code list value must not pass through")


def test_multi_valued_source_is_not_truncated():
    flat = {"idinfo/keywords/theme/themekey": ["flood", "risk", "zone"]}
    rules = [{"source": "idinfo/keywords/theme/themekey",
              "target": "identificationInfo/descriptiveKeywords/keyword",
              "transform": "all"}]
    mapped, _ = apply_crosswalk(flat, rules)
    assert len(mapped["identificationInfo/descriptiveKeywords/keyword"]) == 3

Run the crosswalk over a fixture corpus in CI and fail on any change to the residue list that was not intended. The residue is the crosswalk’s real output surface: a rule that stops matching because a source schema changed will produce a smaller output and a larger residue, and only the residue makes the loss visible.

# Derivative & Lineage Management

A crosswalked record is a derivative of its source, and the same obligations apply as to derived data: the output should say what produced it.

Three facts are enough. The source document identity — a content hash rather than a path, since paths move. The crosswalk version that was applied, so that a corrected rule identifies the set of records to regenerate. And the residue and supplied lists from the run, which together state exactly where the output differs from a faithful representation of the source.

Recording these turns a regeneration into a targeted operation. When a vocabulary mapping is corrected, the records to reprocess are those whose run recorded that transform, which is a query rather than a full re-run of the archive. This is the same discipline described in spatial data lineage and provenance tracking, applied to metadata rather than to geometry.

The one rule worth stating absolutely: never treat the crosswalked output as the new source of truth. Keeping the source records and regenerating on demand means a crosswalk bug is a re-run; adopting the output as canonical means it is a data-entry project. The output is a build artifact, and build artifacts are disposable.

# Directionality, and Why Round-Tripping Fails

Crosswalks are frequently described as though they were bidirectional — a mapping between two standards — and almost none of them are. Being explicit about direction avoids a class of expectation failure that is expensive to discover late.

A crosswalk is written from a specific source to a specific target, and reversing it is not a matter of swapping the two path columns. Three asymmetries make the reverse direction a separate piece of work.

Enrichment cannot be reversed. A forward crosswalk supplies target-mandatory elements the source never carried — a character set declaration, a metadata standard name, a default contact. Running the reverse crosswalk over that output produces a source-shaped record containing values that never existed in any source. They are indistinguishable, in the reversed record, from real data, and the only defence is the supplied list from the forward run, which the reverse crosswalk has no access to.

Structural collapse cannot be reversed. Where four source fields were combined into one composite target element, the reverse mapping has to split a string. It will do so by a rule that is a guess, and the guess will be right for the records the rule was tested on. This is the mechanism by which a round trip produces a record that is structurally identical to the original and differs in a field nobody checks.

Vocabulary mappings are rarely bijective. Two source progress values may both map to one target code, at which point the reverse mapping has to pick one. Whichever it picks is wrong half the time, and the loss is silent because both answers are valid values.

The practical consequence is that a round trip is not a test of crosswalk correctness, though it is frequently used as one. A crosswalk that survives a round trip may simply have avoided every field where the asymmetries bite. A crosswalk that fails one may be entirely correct and merely honest about enrichment. Neither result tells you what you wanted to know.

What does test a crosswalk is a corpus with expected outputs: a few dozen source records, hand-checked target records, and an assertion that the run reproduces them. Building that corpus is a day of tedious work and it is the only thing that catches a mapping which is plausible and wrong. Take the records from the awkward end of the archive — the ones with missing dates, repeated contacts, unmapped vocabulary values and empty mandatory elements — because the well-formed records are the ones any crosswalk handles.

Where both directions genuinely are needed, write them as two crosswalks with two rule files and two test corpora, and accept that they will not compose to the identity. Naming that expectation up front is considerably cheaper than discovering it when somebody reports that a record survived a migration with its update frequency subtly changed.

# Keeping the Rule File Reviewable

A crosswalk file for a real standard pair runs to two or three hundred rules, which is past the point where a reviewer can hold it in their head. Two conventions keep it reviewable at that size.

Group rules by target section and keep the groups in schema order. A reviewer checking the identification block should find every rule that writes into it adjacent, and should be able to see at a glance which mandatory targets have no rule at all. Ordering by source path instead scatters them, and the missing-target gaps become invisible.

Comment the decisions, not the mechanics. # source has no equivalent; defaulting per the 2026 archive policy is worth a line. # maps title to title is noise that makes the file longer without making it clearer. The rules that need comments are exactly the ones a reviewer would otherwise query, which is a useful test of whether the comment earns its place.

# Pitfalls & Resolution Table

# Related

• Writing a Declarative Crosswalk Table in YAML — the rule format in full, with cardinality and absence policy
• Crosswalking ISO 19115 to STAC Item Properties — a worked crosswalk between two very differently shaped standards
• Handling Unmappable Fields in a Metadata Crosswalk — the policy question behind the residue list
• FGDC-to-ISO 19115 Conversion Pipelines — the largest crosswalk most spatial teams will run
• Automated Metadata Generation & Schema Mapping — the parent guide to standards, generation sources and validation