# Handling Unmappable Fields in a Metadata Crosswalk

Route every field that has no counterpart into one of four declared outcomes — carried into a free-text element, defaulted with a provenance flag, deferred to a human queue, or explicitly dropped and logged — and never allow a fifth outcome in which a field simply disappears because no rule mentioned it.

Unmappable fields are not an edge case in metadata conversion; they are most of the interesting work. Every pair of standards was designed by different people for different purposes, and the overlap is smaller than the union in both directions. What separates a defensible conversion from a lossy one is not how many fields mapped, but whether the ones that did not are accounted for. This guide is part of metadata crosswalks and field mapping tables, under Automated Metadata Generation & Schema Mapping.

# Automated Python Implementation

The residue tracker below wraps a crosswalk run and guarantees that every source path is accounted for by one of the four outcomes. Its central assertion is coverage: the union of mapped, carried, deferred and dropped paths must equal the set of paths present in the source.

#!/usr/bin/env python3
"""Account for every source field in a crosswalk run."""
import json
from dataclasses import dataclass, field


@dataclass
class Residue:
    """What happened to the parts of the source that did not map cleanly."""
    carried: dict = field(default_factory=dict)     # path -> value, into free text
    supplied: dict = field(default_factory=dict)    # target -> default written
    deferred: dict = field(default_factory=dict)    # path -> reason a human is needed
    dropped: dict = field(default_factory=dict)     # path -> reason for the drop

    def accounted(self):
        return set(self.carried) | set(self.deferred) | set(self.dropped)


# Declared policy for the source paths that no mapping rule covers.
UNMAPPED_POLICY = {
    "idinfo/native": {"action": "carry", "into": "supplementalInformation"},
    "idinfo/crossref": {"action": "carry", "into": "supplementalInformation"},
    "dataqual/attracc": {"action": "defer", "reason": "no structured ISO equivalent"},
    "distinfo/custom": {"action": "drop", "reason": "site-specific, superseded"},
}


def account_for_source(flat, mapped_sources, policy=UNMAPPED_POLICY):
    """Classify every source path that no rule consumed."""
    residue = Residue()
    for path, values in sorted(flat.items()):
        if path in mapped_sources:
            continue
        rule = policy.get(path)
        if rule is None:
            # The important case: an unknown field is deferred, never dropped.
            residue.deferred[path] = "no policy declared for this source path"
        elif rule["action"] == "carry":
            residue.carried[path] = {"into": rule["into"], "values": values}
        elif rule["action"] == "defer":
            residue.deferred[path] = rule["reason"]
        else:
            residue.dropped[path] = rule["reason"]
    return residue


def assert_full_coverage(flat, mapped_sources, residue):
    """Every source path must be mapped or accounted for. No exceptions."""
    unexplained = set(flat) - set(mapped_sources) - residue.accounted()
    if unexplained:
        raise AssertionError(
            "source paths neither mapped nor accounted for: "
            + ", ".join(sorted(unexplained))
        )


def render_carried(residue, separator="\n\n"):
    """Compose the free-text block from the carried fields, deterministically."""
    blocks = []
    for path in sorted(residue.carried):
        entry = residue.carried[path]
        label = path.rsplit("/", 1)[-1]
        blocks.append(f"{label}: " + "; ".join(entry["values"]))
    return separator.join(blocks)

The default for an unknown path is defer, not drop, and that choice is the whole design. A crosswalk encountering a source field nobody has classified has found something its authors did not know about, which is precisely the case where silent behaviour is most damaging. Deferring makes it visible; dropping makes it disappear with the appearance of success.

# Validation and Pipeline Integration

Coverage is the assertion; everything else is reporting.

def test_every_source_path_is_accounted_for():
    flat = {"idinfo/citation/citeinfo/title": ["A"], "idinfo/native": ["ArcInfo 7.2"]}
    mapped = {"idinfo/citation/citeinfo/title"}
    residue = account_for_source(flat, mapped)
    assert_full_coverage(flat, mapped, residue)


def test_unknown_path_defers_rather_than_drops():
    flat = {"idinfo/somethingnew": ["x"]}
    residue = account_for_source(flat, set())
    assert "idinfo/somethingnew" in residue.deferred
    assert not residue.dropped


def test_carried_block_is_deterministic():
    flat = {"idinfo/crossref": ["b"], "idinfo/native": ["a"]}
    residue = account_for_source(flat, set())
    assert render_carried(residue) == render_carried(residue)

Emit the residue as a machine-readable artifact from every run and diff it between runs in CI. A residue that grows without an accompanying policy change means an upstream source has started emitting fields nobody has classified — which is the earliest possible warning that a source schema has changed, and considerably cheaper to act on than the validation failures that follow.

# Writing the Drop Policy Down

Dropping a field is a decision with consequences that outlive everyone involved, and it should read like one. A policy entry with a reason of "not needed" is not a decision; it is the absence of one, recorded.

A usable entry answers three questions. What was in the field — a short description, because the source standard’s documentation will not always be at hand in ten years. Why it has no target — whether the target standard genuinely lacks the concept, or has it somewhere the crosswalk has chosen not to populate, which are different situations. And what would need to change for the decision to be revisited: a new target element, a profile extension, or simply somebody deciding the information matters.

The category that most often deserves a second look is the one dismissed as site-specific. Native environment strings, custom distribution notes and local processing remarks look like clutter and are frequently the only surviving record of how a dataset was produced. Carrying them into a free-text element costs a few hundred bytes per record and preserves something that cannot be reconstructed. Dropping them is defensible; dropping them without recording that they existed is not.

One further habit is worth adopting: review the drop list when the target standard revises. Elements get added, and a field dropped in 2019 for want of a target may have one now. Nothing will surface that automatically, because the crosswalk works and produces no complaint. A ten-minute review at each standard update is what turns a permanent loss into a temporary one.

# Long-Term Compliance Best Practices

• Default unknown paths to defer. The one behaviour that must never be silent is the appearance of a field nobody has classified.
• Keep the residue artifact with the converted record. It is the answer to “what did we lose”, and it is only credible if it was produced by the run rather than reconstructed afterwards.
• Count deferrals as a work queue, not a failure rate. A conversion with two hundred deferrals is not going badly; it has a list.
• Never let a default silently satisfy a mandatory target twice. Two records defaulting the same contact are fine; two hundred means the field should be sourced properly or the default should be reviewed.
• Diff the residue between runs. A growing residue with an unchanged policy is a source schema change, detected before it becomes a validation failure.

# Related

• Metadata Crosswalks & Field Mapping Tables — the applier that produces the residue this guide classifies
• Writing a Declarative Crosswalk Table in YAML — the rule format whose absence policy this extends
• FGDC-to-ISO 19115 Conversion Pipelines — the conversion where these decisions are made in bulk
• Batch Converting an FGDC Archive with lxml — running the conversion at archive scale