Geospatial Compliance Metadata Automation Hub

Writing Custom Metadata Lint Rules in Python

Write each rule as a small function that takes a parsed record and returns findings — never a boolean — with a stable rule id, a severity and a message naming the element and what to do about it, and register the functions rather than chaining them, so adding a rule never means editing the runner.

Schema validation answers whether a record is legal. Almost every question an organisation actually has about its metadata is a different one: whether the abstract says anything, whether the contact resolves, whether the extent is plausible for the stated coverage. Those are house rules, they change more often than any standard, and they belong in a linter that is easy to extend. This guide sits under metadata schema validation and linting, within Automated Metadata Generation & Schema Mapping.

What a finding must carry to be actionableA lint finding carries an identity, a location, a severity and a remedy, each serving a different consumer.Identityfor trackingrule idrule versionrecord idLocationfor fixingelement pathline, where knownvalue seenSeverityerrorwarninginfoRemedythe part usually missingwhat to changeexpected form
Drop the bottom band and the finding is a complaint. Drop the top one and it cannot be suppressed or tracked.

Automated Python Implementation

#!/usr/bin/env python3
"""A registry-based metadata linter."""
import re
from dataclasses import dataclass

RULES = []


@dataclass(frozen=True)
class Finding:
    rule_id: str
    severity: str          # "error" | "warning" | "info"
    path: str
    message: str
    value: str | None = None


def rule(rule_id, severity="warning"):
    """Register a rule. The function yields Findings; it never returns a bool."""
    def decorate(func):
        func.rule_id = rule_id
        func.severity = severity
        RULES.append(func)
        return func
    return decorate


@rule("MD001", severity="error")
def abstract_is_substantive(record):
    """An abstract that restates the title describes nothing."""
    abstract = (record.get("abstract") or "").strip()
    title = (record.get("title") or "").strip()
    if not abstract:
        yield Finding("MD001", "error", "identificationInfo/abstract",
                      "abstract is empty; describe what the dataset contains")
    elif abstract.lower() == title.lower():
        yield Finding("MD001", "error", "identificationInfo/abstract",
                      "abstract repeats the title; describe content, not the name",
                      value=abstract[:60])
    elif len(abstract.split()) < 12:
        yield Finding("MD001", "warning", "identificationInfo/abstract",
                      "abstract is under twelve words; expand it", value=abstract)


@rule("MD002", severity="error")
def contact_is_functional(record):
    """Personal addresses stop working; team addresses are redirected."""
    email = (record.get("contact_email") or "").strip().lower()
    if not email:
        yield Finding("MD002", "error", "contact/electronicMailAddress",
                      "no contact address recorded")
        return
    local = email.split("@")[0]
    if re.fullmatch(r"[a-z]+\.[a-z]+", local) or re.fullmatch(r"[a-z]\.[a-z]+", local):
        yield Finding("MD002", "warning", "contact/electronicMailAddress",
                      "looks like a personal address; use a team mailbox", value=email)


@rule("MD003", severity="error")
def extent_is_plausible(record):
    """A bounding box outside its CRS bounds usually means swapped axes."""
    bbox = record.get("bbox")
    if not bbox:
        yield Finding("MD003", "error", "extent/EX_GeographicBoundingBox",
                      "no geographic extent recorded")
        return
    west, south, east, north = bbox
    if not (-180 <= west <= 180 and -180 <= east <= 180):
        yield Finding("MD003", "error", "extent/westBoundLongitude",
                      "longitude outside -180..180; check axis order", value=str(bbox))
    if not (-90 <= south <= 90 and -90 <= north <= 90):
        yield Finding("MD003", "error", "extent/southBoundLatitude",
                      "latitude outside -90..90; check axis order", value=str(bbox))
    if west > east or south > north:
        yield Finding("MD003", "error", "extent/EX_GeographicBoundingBox",
                      "bounds are inverted; min must be less than max", value=str(bbox))


def lint(record, suppressed=frozenset()):
    findings = []
    for func in RULES:
        for finding in func(record) or []:
            if finding.rule_id in suppressed:
                continue
            findings.append(finding)
    return findings

Two structural decisions are what make this maintainable at fifty rules rather than five.

Rules yield findings and never return booleans. A boolean rule can say only that something is wrong; a finding says which element, what value, and what the expected form is. The difference determines whether the linter is fixed or disabled.

Registration is a decorator, not a list in the runner. Adding a rule means adding a function, so a contributor never has to understand the runner to extend it. That is also what makes rules easy to move into a shared package later.

What a linter should check that a schema cannotFour classes of house rule with what the schema says about each and what the linter adds.Schema saysLinter addsAbstract qualitypresent, a stringnot a repeat of the titleContact forma valid addressa team mailbox, not a personExtent plausibilityfour decimalsinside CRS bounds, not invertedKeyword vocabularyany stringsdrawn from the agreed scheme
In every row the schema is satisfied. That is precisely the gap the linter fills.

Validation and Pipeline Integration

Every rule needs a test that it fires, and one that it does not fire on a clean record. A rule that has never rejected anything is indistinguishable from a rule that does not work.

def test_every_rule_has_a_stable_id():
    ids = [func.rule_id for func in RULES]
    assert len(ids) == len(set(ids)), "duplicate rule ids"
    assert all(re.fullmatch(r"MD[0-9]{3}", i) for i in ids)


def test_abstract_rule_fires_on_a_repeat():
    findings = lint({"title": "Roads", "abstract": "Roads", "contact_email": "gis@x.gov",
                     "bbox": [-1, 50, 1, 52]})
    assert any(f.rule_id == "MD001" for f in findings)


def test_clean_record_produces_no_findings():
    findings = lint({
        "title": "Road centrelines",
        "abstract": "Centrelines for adopted highways, surveyed and maintained quarterly.",
        "contact_email": "gis@example.gov",
        "bbox": [-1.2, 50.4, 0.9, 52.1],
    })
    assert findings == []


def test_suppression_is_by_rule_id():
    record = {"title": "Roads", "abstract": "Roads", "contact_email": "gis@x.gov",
              "bbox": [-1, 50, 1, 52]}
    assert not lint(record, suppressed={"MD001"})

Report findings as annotations on the changed records in a pull request rather than as a log, using the same routing described in metadata schema validation and linting. Fail the build on errors and count warnings, so that a new rule can be introduced at warning severity and promoted once the backlog is cleared.

Choosing a severity for a new ruleWhether a new lint rule should be an error, a warning, or informational.Does it make the record wrong?inverted extent, absent contacterrorafter a warning periodyesnoDoes it make the record lessuseful?thin abstract, no keywordswarningcounted, not blockingyesnoinforeported, never gated
New rules start as warnings regardless of eventual severity, or the first run blocks everyone.

Keeping Rules from Becoming Folklore

A linter accumulates rules, and rules accumulate reasons that are known only to whoever added them. Two years on, a rule fires, nobody can say why it exists, and the response is to suppress it — which is how a rule that was catching something real stops.

The docstring is the cheapest defence and the one most often skipped. A rule’s docstring should state the failure it prevents rather than what it checks: """Personal addresses stop working; team addresses are redirected.""" says why, whereas """Check the contact email.""" says nothing the code does not. Where a rule exists because of a specific incident, name it — a sentence referring to the publication that had to be withdrawn is worth more than any amount of abstract justification.

Suppressions deserve the same treatment and rarely get it. A suppression recorded as a bare rule id is an assertion with no author and no expiry. Recording the record, the rule, who suppressed it and why, in a file that a reviewer sees, turns an accumulating pile of exceptions into something that can be audited — and the count of suppressions per rule is a useful signal in itself: a rule suppressed on forty records is a rule that is wrong, not forty records that are.

Retire rules deliberately. When a rule has fired on nothing for a year, it is either preventing a problem invisibly or checking something that stopped mattering, and the two are distinguishable only by asking. Deleting it and seeing whether anything breaks is not a safe test, because the failure it prevented may take another year to recur.

Long-Term Compliance Best Practices

  • Never reuse a rule id. Ids appear in suppressions and dashboards; retiring one and minting another costs nothing.
  • Keep rules pure. A rule that reads a database is untestable and slow; resolve external facts before linting and pass them in.
  • State the remedy in the message. The difference between a fixed record and a suppressed rule is usually whether the author knew what to change.
  • Introduce every rule at warning severity. Promotion after the backlog clears is a non-event; immediate enforcement is a revolt.
  • Track suppressions per rule. A rule suppressed everywhere is a defect in the rule.