Geospatial Compliance Metadata Automation Hub

Automating License Checks with Python and OGR

Automating license checks with Python and OGR eliminates manual EULA verification bottlenecks by programmatically reading embedded geospatial metadata, normalizing license strings, and validating them against a compliance allowlist before ingestion.

This technique sits at the operational heart of commercial EULA compliance tracking — the broader discipline of translating proprietary license obligations into machine-readable compliance states. Manually reviewing license metadata across hundreds of vendor drops, open-data portals, and batch deliveries is error-prone and leaves no auditable trail. Embedding validation directly into ingestion scripts — using the osgeo GDAL/OGR Python bindings — converts a legal review bottleneck into a repeatable pipeline gate. This approach is foundational to the wider obligations covered in Geospatial Data Licensing & Compliance Fundamentals.

License validation pipeline flow Datasets enter the pipeline through OGR/GDAL open, then metadata is extracted and normalized to SPDX, then checked against an allowlist. Compliant files proceed; non-compliant files are rejected to a manual queue. Dataset Input OGR/GDAL Open Extract & Normalize (SPDX) Allowlist Check Compliant → Ingest Rejected → Review queue

Automated Python Implementation

The script below uses GDAL’s unified architecture to handle both raster and vector formats. It opens the dataset, scans common metadata keys for license information across all known domains, normalizes the string to SPDX conventions, and validates against an organizational allowlist.

import os
import logging
from pathlib import Path
from typing import Dict, List, Optional, Tuple
from osgeo import gdal, ogr

# Enable exception-based error handling (required for clean try/except logic)
gdal.UseExceptions()
ogr.UseExceptions()

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

# SPDX-compatible allowlist — version-control this in YAML/JSON for quarterly review
ALLOWED_LICENSES = {
    "CC-BY-4.0", "CC-BY-SA-4.0", "ODBL-1.0", "MIT", "APACHE-2.0",
    "US-GOV-PD", "EUPL-1.2"
}

# Keys checked in the default metadata domain; driver-specific domains handled below
LICENSE_KEYS = [
    "LICENSE", "LICENCE", "COPYRIGHT", "DATA_LICENSE",
    "GDAL_LICENSE", "OGC_LICENSE", "AUTHOR"
]


def open_dataset(filepath: str):
    """Attempt raster open first, then vector, returning (dataset, type) or (None, None)."""
    try:
        ds = gdal.Open(filepath)
        if ds:
            return ds, "raster"
    except RuntimeError:
        pass
    try:
        ds = ogr.Open(filepath)
        if ds:
            return ds, "vector"
    except RuntimeError:
        pass
    return None, None


def collect_metadata(ds) -> Dict[str, str]:
    """Merge metadata from the default domain and all driver-specific domains."""
    merged: Dict[str, str] = {}
    # Default domain first
    default = ds.GetMetadata() or {}
    merged.update(default)
    # Driver-specific domains (e.g. IMAGE_STRUCTURE, SUBDATASETS, xml:ESRI)
    try:
        domains = ds.GetMetadataDomainList() or []
        for domain in domains:
            if domain:
                merged.update(ds.GetMetadata(domain) or {})
    except Exception:
        pass
    return merged


def normalize_license_string(raw: str) -> Optional[str]:
    """Strip boilerplate prefixes and align to SPDX hyphen-separated uppercase format."""
    if not raw:
        return None
    cleaned = raw.strip().upper()
    for prefix in ("LICENSE:", "COPYRIGHT:", "LICENCE:", "DATA LICENSE:"):
        if cleaned.startswith(prefix):
            cleaned = cleaned[len(prefix):].strip()
    # Normalize whitespace and underscores to SPDX hyphens
    cleaned = cleaned.replace(" ", "-").replace("_", "-")
    return cleaned if cleaned else None


def extract_and_validate(filepath: str) -> Tuple[str, str, Optional[str], bool]:
    """
    Open dataset, extract license metadata from all domains, normalize and validate.

    Returns: (filepath, ds_type, normalized_license, is_compliant)
    """
    ds, ds_type = open_dataset(filepath)
    if not ds:
        logging.warning("Could not open: %s", filepath)
        return filepath, "unknown", None, False

    metadata = collect_metadata(ds)
    raw_license = None
    for key in LICENSE_KEYS:
        if key in metadata:
            raw_license = metadata[key]
            logging.debug("Found key %s in %s", key, filepath)
            break

    normalized = normalize_license_string(raw_license)
    is_compliant = normalized in ALLOWED_LICENSES if normalized else False

    # Explicit cleanup releases file locks — critical for Shapefiles and Windows paths
    if ds_type == "raster":
        ds = None  # GDAL raster datasets are garbage-collected via reference drop
    else:
        ds.Destroy()

    if not normalized:
        logging.warning("No license metadata found in: %s — route to manual review", filepath)
    elif not is_compliant:
        logging.warning("Non-compliant license '%s' in: %s", normalized, filepath)

    return filepath, ds_type, normalized, is_compliant

Key design decisions:

  • collect_metadata() iterates GetMetadataDomainList() so that licenses buried in driver-specific domains (e.g., xml:ESRI in File Geodatabases, SUBDATASETS in NetCDF) are not silently missed.
  • gdal.UseExceptions() is mandatory. Without it, GDAL returns None on failure instead of raising, making format detection logic ambiguous.
  • Destroying the OGR dataset via ds.Destroy() rather than relying on garbage collection is critical for Shapefile .lck file release and for PostGIS connection hygiene in long-running processes.

Validation and Pipeline Integration

After writing the core function, verify it behaves correctly before wiring it into a production pipeline.

Step 1 — Smoke-test with ogrinfo:

ogrinfo -al -so my_dataset.gpkg | grep -i licen

If ogrinfo surfaces a license key, your Python extraction will too. If it does not, the dataset lacks embedded metadata and will return None — confirm your manual review routing handles this case.

Step 2 — Unit tests covering the normalization edge cases:

import pytest
from your_module import normalize_license_string

@pytest.mark.parametrize("raw,expected", [
    ("LICENSE: CC-BY-4.0",        "CC-BY-4.0"),
    ("copyright: cc by 4.0",      "CC-BY-4.0"),
    ("  ODbL-1.0 ",               "ODBL-1.0"),
    ("DATA LICENSE: Apache 2.0",  "APACHE-2.0"),
    ("",                          None),
    (None,                        None),
])
def test_normalize(raw, expected):
    assert normalize_license_string(raw) == expected

Run with python -m pytest -v.

Step 3 — Batch validation with parallel execution and CSV reporting:

import concurrent.futures
import csv
from typing import Iterable

def process_batch(filepaths: Iterable[str], output_csv: str) -> None:
    """Run license checks in parallel and write a structured compliance report."""
    results = []
    # ThreadPoolExecutor — GDAL ops are I/O-bound; ProcessPoolExecutor adds
    # serialization overhead and risks forking shared GDAL state
    with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
        futures = {executor.submit(extract_and_validate, fp): fp for fp in filepaths}
        for future in concurrent.futures.as_completed(futures):
            results.append(future.result())

    # Sort non-compliant and unknown results to the top for triage
    results.sort(key=lambda x: (x[3], x[0]))  # compliant last, then alpha by path

    with open(output_csv, "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["Filepath", "Type", "Normalized_License", "Compliant"])
        writer.writerows(results)

    compliant = sum(1 for r in results if r[3])
    logging.info(
        "Processed %d files — %d compliant, %d flagged. Report: %s",
        len(results), compliant, len(results) - compliant, output_csv,
    )

Step 4 — CI gate integration:

Add a pre-merge check that fails the pipeline if any newly added dataset is non-compliant. In a GitHub Actions workflow, call process_batch() and exit non-zero when the CSV contains any False in the Compliant column. This pattern aligns with policy enforcement gates for data PRs and ensures license violations never reach the main branch.

import sys, csv

def assert_all_compliant(report_csv: str) -> None:
    with open(report_csv) as f:
        rows = list(csv.DictReader(f))
    failures = [r for r in rows if r["Compliant"] == "False"]
    if failures:
        for r in failures:
            print(f"FAIL: {r['Filepath']} — license: {r['Normalized_License'] or 'MISSING'}")
        sys.exit(1)
    print(f"All {len(rows)} datasets passed license check.")

Long-term Compliance Best Practices

  • Version-control the allowlist. Store ALLOWED_LICENSES in a YAML or JSON config file committed to your repository. Tag each change with the date and the policy decision that drove it. When a vendor’s dataset arrives with CC-BY-NC-4.0 and your organization decides to accept it, the allowlist edit is the auditable record.
  • Capture raw metadata alongside normalized results. Your compliance CSV should include a Raw_License column in addition to Normalized_License. When a vendor disputes a flag, you can show the exact string the dataset reported rather than defending a transformation.
  • Route None licenses to a secondary review queue, not to rejection. Many legitimate public-domain datasets (e.g., US federal agency deliveries tagged US-GOV-PD in a sidecar rather than embedded metadata) will return None on GDAL extraction. Automatic rejection discards valid data; a review queue preserves it while maintaining traceability.
  • Re-validate on format conversion. Reprojecting, clipping, or converting a dataset to a new format (e.g., GeoPackage → GeoTIFF) can strip embedded metadata entirely. Run extract_and_validate() on the output file as well as the input, and log a warning when metadata disappears. This connects directly to mapping commercial GIS data usage rights obligations that survive format transformation.
  • Pin GDAL versions in CI. GDAL driver behavior — especially metadata key names — changes between minor releases. Pin gdal>=3.4,<4 in your requirements.txt and test on each GDAL minor version bump before upgrading.
  • Escalate unsupported formats to procurement, not to the pipeline. Closed-source LiDAR formats, certain CAD deliveries, and encrypted commercial rasters may expose zero metadata via OGR. Log these as unsupported_format with the file’s SHA-256 fingerprint and escalate to the team responsible for commercial EULA compliance tracking procurement workflows.