# Batch Converting an FGDC Archive with lxml

Convert an archive as a resumable job over a work list rather than as a single pass over a directory: record one result row per source document, retry only the failures, and keep the source documents as the source of truth so that a crosswalk fix means a re-run rather than a data-entry project.

The scripts that convert one CSDGM document to ISO 19115-3 are straightforward. What breaks on an archive of several thousand is everything around them — a document that hangs the parser, a run that dies at record 2,800 with no record of what succeeded, a memory footprint that grows until the process is killed. This guide sits under FGDC-to-ISO 19115 conversion pipelines, within Automated Metadata Generation & Schema Mapping.

# Automated Python Implementation

#!/usr/bin/env python3
"""Convert an FGDC CSDGM archive to ISO 19115-3, resumably."""
import hashlib
import pathlib
import sqlite3
import traceback

from lxml import etree

SCHEMA = """
CREATE TABLE IF NOT EXISTS conversion (
    source_path TEXT PRIMARY KEY,
    source_sha256 TEXT NOT NULL,
    status TEXT NOT NULL,
    crosswalk_version TEXT,
    output_path TEXT,
    error TEXT,
    converted_at TEXT DEFAULT (datetime('now'))
)
"""

def sha256(path):
    digest = hashlib.sha256()
    with open(path, "rb") as fh:
        for chunk in iter(lambda: fh.read(1 << 20), b""):
            digest.update(chunk)
    return digest.hexdigest()

def build_work_list(con, root):
    """One row per source document. Re-running is idempotent."""
    for path in sorted(pathlib.Path(root).rglob("*.xml")):
        con.execute(
            "INSERT OR IGNORE INTO conversion (source_path, source_sha256, status)"
            " VALUES (?, ?, 'pending')",
            (str(path), sha256(path)),
        )
    con.commit()

def parse_defensively(path):
    """A recovering parser gets a tree out of documents strict parsing refuses."""
    parser = etree.XMLParser(recover=True, huge_tree=False, resolve_entities=False)
    tree = etree.parse(str(path), parser)
    if tree.getroot() is None:
        raise ValueError("no root element after recovery")
    return tree

def convert_one(path, crosswalk, version, out_dir):
    tree = parse_defensively(path)
    mapped, residue = crosswalk(tree)
    out_path = pathlib.Path(out_dir) / (pathlib.Path(path).stem + ".iso.xml")
    out_path.write_bytes(etree.tostring(mapped, pretty_print=True, encoding="UTF-8",
                                        xml_declaration=True))
    return out_path, residue

def run(db_path, root, out_dir, crosswalk, version, retry_failed=False):
    con = sqlite3.connect(db_path)
    con.execute(SCHEMA)
    build_work_list(con, root)

    states = ("pending", "failed") if retry_failed else ("pending",)
    placeholders = ",".join("?" * len(states))
    rows = con.execute(
        f"SELECT source_path FROM conversion WHERE status IN ({placeholders})",
        states,
    ).fetchall()

    ok = failed = 0
    for (source_path,) in rows:
        try:
            out_path, _ = convert_one(source_path, crosswalk, version, out_dir)
            con.execute(
                "UPDATE conversion SET status='ok', output_path=?, error=NULL,"
                " crosswalk_version=? WHERE source_path=?",
                (str(out_path), version, source_path),
            )
            ok += 1
        except Exception:
            con.execute(
                "UPDATE conversion SET status='failed', error=?, crosswalk_version=?"
                " WHERE source_path=?",
                (traceback.format_exc(limit=3), version, source_path),
            )
            failed += 1
        con.commit()
    con.close()
    return {"ok": ok, "failed": failed}

Three properties of that loop are what make it survive an archive.

Each document is converted independently and its result committed immediately. A crash loses at most one record’s work, and the next run picks up exactly where the last stopped without any bookkeeping by the operator.

Failures are recorded with their traceback rather than raised. The run completes and produces a classified failure list, which is a work queue. A run that stops at the first bad document produces one error and no information about the other four thousand.

The parser recovers rather than refusing. Legacy archives contain documents with unescaped ampersands, mismatched tags and truncated tails, and a recovering parser extracts what is there. Where recovery yields nothing, the failure is explicit — which is the correct outcome, arrived at after trying.

# Validation and Pipeline Integration

The checks that matter are about the job rather than about any document.

def test_work_list_is_idempotent():
    con = sqlite3.connect(":memory:")
    con.execute(SCHEMA)
    build_work_list(con, "fixtures/archive")
    first = con.execute("SELECT count(*) FROM conversion").fetchone()[0]
    build_work_list(con, "fixtures/archive")
    assert con.execute("SELECT count(*) FROM conversion").fetchone()[0] == first

def test_failure_does_not_stop_the_run():
    result = run(":memory:", "fixtures/archive_with_bad_doc", "out", CROSSWALK, "1.0")
    assert result["ok"] > 0 and result["failed"] > 0

def test_every_source_is_accounted_for():
    con = sqlite3.connect("conversion.db")
    pending = con.execute(
        "SELECT count(*) FROM conversion WHERE status='pending'").fetchone()[0]
    assert pending == 0, "the run finished with documents never attempted"

Report three numbers after every run — converted, failed, and pending — and treat a non-zero pending count as a defect in the job rather than in the data. It means documents were enumerated and never attempted, which is the failure mode that produces an archive believed to be converted and quietly is not.

# Memory and the Documents That Do Not Fit

Most CSDGM documents are a few kilobytes. Archives reliably contain a handful that are not: a record with ten thousand keyword elements, or one whose entity and attribute section describes four hundred fields. etree.parse holds the whole tree, and a few of those in flight is enough to end a run on a constrained runner.

Two adjustments handle it without complicating the common case.

Free each tree explicitly. lxml releases memory when the tree goes out of scope, but a loop holding a reference to the last tree while parsing the next keeps two alive. Assigning None after use, or scoping the parse inside a function as above, keeps the footprint to one document.

Set a size threshold and route large documents through iterative parsing. Above a few megabytes, etree.iterparse with element clearing extracts the same values at constant memory. The crosswalk applier does not need to change if it consumes the flattened path dictionary rather than the tree, which is another argument for the flattening step described in metadata crosswalks and field mapping tables.

One further safeguard is worth the line it costs: disable entity resolution. Legacy XML occasionally carries external entity declarations, and a parser that resolves them will read files off the filesystem or attempt network access while converting what looks like an ordinary metadata record.

# Long-Term Compliance Best Practices

• Keep the source archive immutable and the output disposable. Every crosswalk correction is then a re-run rather than a migration.
• Stamp the crosswalk version on every result row. After a fix, the documents to reprocess are those converted under the old version.
• Retry failures separately from pending work. Mixing them means a fix that resolves one failure class re-runs the whole archive.
• Keep the failure traceback, not just a message. The class of failure is what groups the queue into fixable batches.
• Re-hash sources on each run. A source that changed under a completed conversion should return to pending, and only a hash comparison notices.

# Related

• FGDC-to-ISO 19115 Conversion Pipelines — the crosswalk this job runs at scale
• FGDC Field Mapping Reference Table for ISO Conversion — the mapping table itself
• Validating FGDC Metadata Against XML Schemas — checking the sources before converting them
• Handling Unmappable Fields in a Metadata Crosswalk — accounting for what the conversion cannot carry