# Spatial Data Lineage & Provenance Tracking

Every automated geoprocessing pipeline silently destroys the story of how a dataset came to be: a reproject overwrites the CRS, a clip discards the source extent, a join fuses two licensing regimes into one file, and a rasterize collapses vector attributes into pixels — yet the output carries none of that history. Lineage and provenance tracking rebuilds that story as a machine-readable graph so an auditor can answer “which sources produced this layer, under what license, through which transformations” without opening a single script. This guide sits within Spatial Data Audit Reporting & Compliance Governance and shows how to capture geoprocessing steps as W3C PROV-O triples with rdflib, serialize them for archival, and project them into the ISO 19115 LI_Lineage structure that catalogs expect.

# Prerequisites

1. Python 3.11+ in an isolated virtual environment (python -m venv .venv)
2. rdflib>=7.0 — RDF graph construction and Turtle/JSON-LD serialization
3. geopandas>=0.14 — vector processing steps (reproject, clip, join) that generate lineage
4. rasterio>=1.3 — raster steps (rasterize, warp) that generate lineage
5. pyproj>=3.6 — CRS resolution used when recording reprojection parameters
6. lxml>=5.1 — emitting the ISO 19115 LI_Lineage XML fragment
7. pyshacl>=0.25 — validating the provenance graph against PROV constraints
8. Environment variable PROV_BASE_URI pointing at a stable namespace you control, e.g. https://data.example.gov/prov/
9. Write access to a durable artifact store for the serialized .ttl provenance files

Install the stack:

pip install "rdflib>=7.0" "geopandas>=0.14" "rasterio>=1.3" \
    "pyproj>=3.6" "lxml>=5.1" "pyshacl>=0.25"

# Concept & Spec Reference

Provenance is the record of the entities, activities, and agents involved in producing a dataset. The W3C PROV Ontology (PROV-O) gives it three core classes — prov:Entity (a thing, such as a dataset version), prov:Activity (something that happened over a period, such as a clip), and prov:Agent (something responsible, such as a pipeline or an analyst) — plus a small set of relating properties. Because it is plain RDF, a provenance graph is queryable with SPARQL, mergeable across datasets, and serializable to the same JSON-LD your catalog already emits. The identical PROV vocabulary already appears in the lineage triples produced during DCAT-AP spatial profile mapping, so a single provenance model can feed both your audit trail and your open-data catalog.

# PROV-O terms used for geoprocessing lineage

# ISO 19115 lineage projection

# Namespace registry

Bind the vocabulary once at import time so serialization and SPARQL stay collision-free:

from rdflib import Namespace

PROV = Namespace("http://www.w3.org/ns/prov#")
DCTERMS = Namespace("http://purl.org/dc/terms/")
GEO = Namespace("http://www.opengis.net/ont/geosparql#")
RDFS = Namespace("http://www.w3.org/2000/01/rdf-schema#")
XSD = Namespace("http://www.w3.org/2001/XMLSchema#")

# Implementation Walkthrough

# Step 1 — Give every dataset a stable, content-addressed identity

An entity URI must identify a specific version of a dataset, not a moving file path. Deriving the URI from a SHA-256 of the file contents makes provenance reproducible: rerun the pipeline on the same input and you get the same entity node, so graphs merge cleanly instead of accumulating duplicates.

import hashlib
import os
from rdflib import URIRef

PROV_BASE = os.environ.get("PROV_BASE_URI", "https://data.example.gov/prov/")

def entity_uri(path: str) -> URIRef:
    """Content-addressed URI for a dataset version."""
    digest = hashlib.sha256()
    with open(path, "rb") as fh:
        for chunk in iter(lambda: fh.read(1 << 20), b""):
            digest.update(chunk)
    return URIRef(f"{PROV_BASE}entity/{digest.hexdigest()[:32]}")

# Step 2 — Record each processing step as an Activity

Wrap the geoprocessing call so the lineage is captured as a side effect of the work itself. The rationale: provenance recorded after the fact drifts from reality, whereas provenance recorded at the call site cannot lie about parameters or timing.

import datetime as dt
import uuid
from contextlib import contextmanager
from rdflib import Graph, URIRef, Literal

def new_graph() -> Graph:
    g = Graph()
    g.bind("prov", PROV)
    g.bind("dcterms", DCTERMS)
    g.bind("rdfs", RDFS)
    return g

@contextmanager
def record_activity(g: Graph, label: str, agent: URIRef, params: dict):
    """Emit a prov:Activity with timing, agent, and parameters."""
    activity = URIRef(f"{PROV_BASE}activity/{uuid.uuid4().hex}")
    started = dt.datetime.now(dt.timezone.utc)
    g.add((activity, RDFS.label, Literal(label)))
    g.add((activity, PROV.wasAssociatedWith, agent))
    for key, value in params.items():
        g.add((activity, DCTERMS.description, Literal(f"{key}={value}")))
    try:
        yield activity
    finally:
        ended = dt.datetime.now(dt.timezone.utc)
        g.add((activity, PROV.startedAtTime, Literal(started, datatype=XSD.dateTime)))
        g.add((activity, PROV.endedAtTime, Literal(ended, datatype=XSD.dateTime)))

# Step 3 — Link inputs, outputs, and derivation

With the activity captured, connect the entities. A derived layer is prov:wasGeneratedBy the activity, the activity prov:used each source, and the derived entity prov:wasDerivedFrom the source. This triangle is the minimal, queryable core of any lineage claim.

def link_derivation(
    g: Graph,
    activity: URIRef,
    source_uri: URIRef,
    derived_uri: URIRef,
) -> None:
    """Wire the used / generated / derived triangle."""
    g.add((source_uri, RDFS.label, Literal("source")))
    g.add((activity, PROV.used, source_uri))
    g.add((derived_uri, PROV.wasGeneratedBy, activity))
    g.add((derived_uri, PROV.wasDerivedFrom, source_uri))

Putting it together for a real reproject step with geopandas:

import geopandas as gpd

def reproject_with_lineage(
    g: Graph,
    agent: URIRef,
    in_path: str,
    out_path: str,
    target_crs: str = "EPSG:4326",
) -> URIRef:
    """Reproject a layer and record the operation as PROV-O."""
    source = entity_uri(in_path)
    gdf = gpd.read_file(in_path)
    with record_activity(
        g, "reproject", agent,
        {"from_crs": str(gdf.crs), "to_crs": target_crs},
    ) as activity:
        gdf = gdf.to_crs(target_crs)
        gdf.to_file(out_path)
    derived = entity_uri(out_path)
    link_derivation(g, activity, source, derived)
    return derived

# Step 4 — Serialize the provenance graph

Emit both Turtle (compact, human-auditable, ideal for archival) and JSON-LD (consumable by the same harvesters that ingest your catalog records). Write them next to the dataset so the provenance travels with the data.

def serialize_provenance(g: Graph, base_path: str) -> None:
    """Write Turtle and JSON-LD provenance artifacts."""
    g.serialize(destination=f"{base_path}.prov.ttl", format="turtle")
    g.serialize(destination=f"{base_path}.prov.jsonld", format="json-ld", indent=2)

# Step 5 — Project PROV-O into ISO 19115 LI_Lineage

Catalogs speak ISO 19115, not PROV-O, so project the activities into LI_ProcessStep elements. Build the fragment with lxml and splice it into the metadata record’s dataQualityInfo block. A structured ISO 19115 metadata template gives you the surrounding record to insert this lineage into.

from lxml import etree

GMD = "http://www.isotc211.org/2005/gmd"
GCO = "http://www.isotc211.org/2005/gco"
NSMAP = {"gmd": GMD, "gco": GCO}

def build_li_lineage(steps: list[dict]) -> etree._Element:
    """Render a gmd:LI_Lineage element from process-step dicts."""
    lineage = etree.Element(f"{{{GMD}}}LI_Lineage", nsmap=NSMAP)
    for step in steps:
        proc = etree.SubElement(lineage, f"{{{GMD}}}processStep")
        li = etree.SubElement(proc, f"{{{GMD}}}LI_ProcessStep")
        desc = etree.SubElement(li, f"{{{GMD}}}description")
        cs = etree.SubElement(desc, f"{{{GCO}}}CharacterString")
        cs.text = step["description"]
        when = etree.SubElement(li, f"{{{GMD}}}dateTime")
        dtv = etree.SubElement(when, f"{{{GCO}}}DateTime")
        dtv.text = step["dateTime"]
    return lineage

# Validation & CI Integration

# Validate the provenance graph

A provenance graph is only trustworthy if it is complete: every derived entity must have exactly one generating activity, and every activity must have timing and an agent. Encode these as SHACL constraints, or assert them directly in Python for a fast pre-commit check.

from rdflib import Graph
from rdflib.namespace import RDFS

def check_provenance(g: Graph) -> list[str]:
    """Return a list of provenance completeness violations."""
    problems: list[str] = []
    derived = set(g.subjects(PROV.wasGeneratedBy, None))
    for entity in derived:
        activities = list(g.objects(entity, PROV.wasGeneratedBy))
        if len(activities) != 1:
            problems.append(f"{entity} has {len(activities)} generating activities")
    for activity in set(g.objects(None, PROV.wasGeneratedBy)):
        if (activity, PROV.endedAtTime, None) not in g:
            problems.append(f"{activity} is missing prov:endedAtTime")
        if (activity, PROV.wasAssociatedWith, None) not in g:
            problems.append(f"{activity} has no associated agent")
    return problems

# Pre-commit and GitHub Actions

Gate provenance completeness the same way you gate schema validity. Wire it into the broader CI/CD validation and policy enforcement framework so a pipeline change that drops lineage fails the build.

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: provenance-completeness
        name: PROV-O completeness check
        language: python
        entry: python ci/check_provenance.py
        files: "\\.prov\\.ttl$"
        additional_dependencies:
          - "rdflib>=7.0"

# .github/workflows/provenance.yml
name: Provenance validation
on:
  pull_request:
    paths:
      - '**/*.prov.ttl'
      - 'pipelines/**/*.py'
jobs:
  check-provenance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install "rdflib>=7.0" "pyshacl>=0.25"
      - run: python ci/check_provenance.py provenance/

# Pytest regression coverage

import pytest
from rdflib import Graph, URIRef
from pipelines.lineage import new_graph, link_derivation, check_provenance, PROV_BASE

def test_complete_graph_has_no_violations():
    g = new_graph()
    activity = URIRef(f"{PROV_BASE}activity/a1")
    g.add((activity, PROV.endedAtTime, __import__("rdflib").Literal("2025-04-08T00:00:00Z")))
    g.add((activity, PROV.wasAssociatedWith, URIRef(f"{PROV_BASE}agent/pipeline")))
    source = URIRef(f"{PROV_BASE}entity/src")
    derived = URIRef(f"{PROV_BASE}entity/out")
    link_derivation(g, activity, source, derived)
    assert check_provenance(g) == []

def test_missing_agent_is_flagged():
    g = new_graph()
    activity = URIRef(f"{PROV_BASE}activity/a2")
    derived = URIRef(f"{PROV_BASE}entity/out2")
    g.add((derived, PROV.wasGeneratedBy, activity))
    assert any("agent" in msg for msg in check_provenance(g))

# Derivative & Lineage Management

Provenance is most valuable exactly where compliance risk concentrates: at transformation boundaries where geometry, CRS, schema, or license change. Manage those boundaries deliberately.

• Reproject: record from_crs and to_crs as activity parameters. A derived entity in EPSG:4326 must not silently inherit the source’s projected-CRS metadata.
• Clip: record the clipping boundary’s own entity URI as a second prov:used input. The clip extent is itself a dataset with its own license, and auditors will ask where it came from.
• Join / enrich: the derived entity prov:wasDerivedFrom both the geometry source and the attribute source. When a join merges two licenses, propagate the most restrictive term — the same obligation covered in the automated attribution mapping workflows guidance.
• Rasterize: the vector-to-raster step loses attributes, so record which attribute drove the burn value as an activity parameter, and note the output resolution. Downstream users cannot reconstruct this from the raster alone.

Because the same PROV graph underlies your audit trail, feed it into compliance dashboards for spatial catalogs to report the fraction of the catalog that carries complete lineage, and hash the serialized graph into your audit trail and evidence retention store so the provenance itself is tamper-evident.

# Pitfalls & Resolution

# Related

• Spatial Data Audit Reporting & Compliance Governance — the parent guide framing where lineage fits in the audit program
• Capturing Processing Lineage with PROV-O in Python — a self-contained rdflib script for a single geoprocessing step
• Compliance Dashboards for Spatial Catalogs — aggregate lineage completeness into a catalog-wide metric
• Audit Trail & Evidence Retention — make the serialized provenance itself tamper-evident
• DCAT-AP Spatial Profile Mapping — reuses the same PROV vocabulary in catalog records
• ISO 19115 Metadata Template Generation — supplies the record that hosts the LI_Lineage fragment