# Detecting Orphaned Metadata Records in a Catalog

Reconcile the catalogue against the data store in both directions on a schedule: records whose dataset no longer exists are orphans, datasets with no record are unlisted, and the two failures need different owners and different responses.

Catalogues drift from the things they describe because the two are updated by different processes. A dataset is deleted by a retention job, moved by a migration, or renamed during a reorganisation, and nothing tells the catalogue. The record continues to be returned by searches, harvested by partners and counted in coverage figures, describing something that is not there. This guide sits under automated broken link and reference detection, within CI/CD Validation & Policy Enforcement for Spatial Data.

# Automated Python Implementation

The reconciler compares two identifier sets and classifies the difference. Its only real subtlety is deciding what counts as the identity of a dataset, which is where naive implementations go wrong.

#!/usr/bin/env python3
"""Reconcile a metadata catalogue against the datasets it describes."""
import json
import pathlib
from dataclasses import dataclass, field


@dataclass
class Reconciliation:
    healthy: set = field(default_factory=set)
    orphaned: dict = field(default_factory=dict)   # record id -> last known location
    unlisted: dict = field(default_factory=dict)   # dataset id -> where it was found
    ambiguous: dict = field(default_factory=dict)  # id -> why it could not be resolved


def catalogue_identities(records):
    """Identity comes from a recorded identifier, never from a file path."""
    identities, ambiguous = {}, {}
    for record in records:
        identifier = record.get("dataset_identifier")
        if not identifier:
            ambiguous[record["id"]] = "record carries no dataset identifier"
            continue
        if identifier in identities:
            ambiguous[identifier] = "two records claim the same dataset identifier"
            continue
        identities[identifier] = record
    return identities, ambiguous


def store_identities(root, read_identifier):
    """Read the identifier the dataset carries, not the name it happens to have."""
    identities, ambiguous = {}, {}
    for path in sorted(pathlib.Path(root).rglob("*")):
        if path.suffix.lower() not in {".gpkg", ".tif", ".geojson"}:
            continue
        try:
            identifier = read_identifier(path)
        except Exception as exc:
            ambiguous[str(path)] = f"unreadable: {exc}"
            continue
        if not identifier:
            ambiguous[str(path)] = "dataset carries no identifier"
            continue
        identities[identifier] = str(path)
    return identities, ambiguous


def reconcile(records, root, read_identifier):
    catalogue, cat_bad = catalogue_identities(records)
    store, store_bad = store_identities(root, read_identifier)

    result = Reconciliation()
    result.ambiguous = {**cat_bad, **store_bad}
    for identifier, record in catalogue.items():
        if identifier in store:
            result.healthy.add(identifier)
        else:
            result.orphaned[identifier] = record.get("last_known_location")
    for identifier, path in store.items():
        if identifier not in catalogue:
            result.unlisted[identifier] = path
    return result


def report(result):
    return {
        "healthy": len(result.healthy),
        "orphaned": result.orphaned,
        "unlisted": result.unlisted,
        "ambiguous": result.ambiguous,
        "coverage": len(result.healthy) / max(
            1, len(result.healthy) + len(result.unlisted)),
    }

The design decision that matters is that identity comes from an identifier the dataset itself carries, not from its path or filename. Reconciling on paths produces a flood of false orphans on the day a directory is reorganised, and the response to a flood of false orphans is to stop running the reconciler. An identifier written into the data — the GeoPackage metadata table, a raster tag, a sidecar — survives moves and renames, which is the property the whole exercise depends on.

# Validation and Pipeline Integration

def test_moved_dataset_is_not_reported_as_orphaned():
    records = [{"id": "r1", "dataset_identifier": "urn:uuid:abc"}]
    store = {"urn:uuid:abc": "/new/location/roads.gpkg"}
    result = reconcile(records, "/new/location", lambda _p: "urn:uuid:abc")
    assert not result.orphaned


def test_duplicate_identifier_is_ambiguous_not_healthy():
    records = [{"id": "r1", "dataset_identifier": "urn:uuid:abc"},
               {"id": "r2", "dataset_identifier": "urn:uuid:abc"}]
    _, ambiguous = catalogue_identities(records)
    assert "urn:uuid:abc" in ambiguous


def test_unlisted_dataset_is_reported_separately():
    result = reconcile([], "/data", lambda _p: "urn:uuid:def")
    assert result.unlisted and not result.orphaned

Run the reconciliation on a schedule and route the two findings differently. An orphaned record is a catalogue problem, owned by whoever maintains the catalogue, and the resolution is usually a tombstone rather than a deletion. An unlisted dataset is a publishing problem, owned by the team that produced it, and the resolution is a record. Sending both to the same queue means one of them is always somebody else’s and neither gets done.

# Tombstones, and Why Deletion Is Not Enough

Removing an orphaned record from the catalogue feels like the obvious fix and leaves the problem in place for everyone downstream. Harvesters synchronise by fetching what exists; absence is not a signal they receive. A record deleted from your catalogue remains in every partner’s copy indefinitely, and those copies are what users search.

A tombstone is a record that asserts the deletion: the same identifier, a deleted flag, the date, and ideally a reason and a successor identifier where one exists. It is a few hundred bytes, it propagates through the same harvest mechanism as any other record, and it is the only way a downstream copy can learn that a dataset is gone.

Two properties make tombstones work in practice. They must keep the original identifier — a tombstone with a new identifier is a new record describing nothing, and the original remains live downstream. And they must persist: a tombstone deleted after six months because the catalogue looks untidy re-creates the problem for any harvester that had not synchronised in that window. Tombstones are cheap enough to keep indefinitely and the alternative is unbounded downstream inaccuracy.

Where a dataset has been replaced rather than withdrawn, naming the successor turns the tombstone from a dead end into a redirect. A user who finds the retired record learns both that it is gone and what to use instead, which is more than most catalogues manage for datasets that are still live.

# When the Store Cannot Be Enumerated

The reconciliation above assumes both sides can be listed. Sometimes the data store cannot be — it is a partner’s service, an object store the catalogue team has no credentials for, or a database whose enumeration would be prohibitively expensive.

One-directional reconciliation still works and is worth running. Checking that each catalogued identifier resolves to something is a per-record probe rather than a full listing, and it finds orphans. What it cannot find is unlisted datasets, so the coverage figure becomes unavailable and should be reported as unknown rather than as 100 percent — the failure mode here is a dashboard that shows perfect coverage because it only ever looked at what it already knew about.

Where the store is a partner’s, the reconciliation becomes a request rather than a scan: an identifier list exchanged periodically, compared locally. That is slower and less frequent, and it is the only mechanism available. Recording when the last exchange happened, and treating a stale exchange as reduced confidence rather than as a pass, keeps the reported state honest.

# Long-Term Compliance Best Practices

• Identify datasets by an identifier they carry, not by path. Reorganisations are routine; false orphans destroy trust in the reconciler.
• Report unlisted datasets as loudly as orphans. An unlisted dataset is undiscoverable, which is the failure the catalogue exists to prevent.
• Track coverage over time. The absolute number matters less than the direction.
• Treat a duplicate identifier as ambiguous, not healthy. Two records claiming one dataset is a defect that reconciliation is well placed to catch.
• Never delete without a tombstone. Absence does not propagate; assertions do.

# Related

• Automated Broken Link & Reference Detection — the parent workflow for reference integrity
• Checking Catalog Distribution URLs in CI — the same discipline applied to external links
• Aggregating Compliance Metrics Across a Dataset Inventory — reporting coverage alongside the other catalogue metrics
• Automated Metadata Generation & Schema Mapping — identifier stability, which this reconciliation depends on