# Spatial Data Schema Linting in CI

Broken geometries, undeclared coordinate reference systems, and missing mandatory attributes rarely surface during data authoring — they surface when a downstream map service crashes or an audit flags a non-compliant dataset. Embedding automated schema linting directly into pull-request pipelines converts these silent failures into explicit, pre-merge gate failures. This page is part of the broader CI/CD Validation & Policy Enforcement for Spatial Data framework; it focuses on the structural integrity gate that runs first in that pipeline, before metadata or licensing checks proceed.

Prerequisites

1. Python 3.10+ with a venv or conda environment
2. GDAL/OGR 3.6+ compiled with PROJ 9+ (gdal-bin, libgdal-dev on Ubuntu)
3. Python packages: geopandas>=0.14, shapely>=2.0, fiona>=1.9, pyproj>=3.6, jsonschema>=4.21, pydantic>=2.5, lxml>=5.1, ruff>=0.3
4. A machine-readable schema file (JSON Schema draft 2020-12) defining your organisation’s mandatory fields, geometry type whitelist, CRS constraint, and SPDX license field
5. CI runner with ≥4 GB RAM and ≥2 vCPUs for large GeoPackage files
6. Branch protection rules enabled on the target repository so gate failures actually block merges

# Concept & Spec Reference

Schema linting for spatial data combines two distinct validation concerns: structural schema validation (are the declared fields present and typed correctly?) and spatial integrity validation (do the geometries and coordinate systems make physical sense?). Neither check subsumes the other.

# JSON Schema constraints for spatial datasets

JSON Schema draft 2020-12 is the standard vehicle for structural rules. For a GeoJSON FeatureCollection the minimum compliant schema enforces:

For GeoPackage and Shapefile workflows the same rules are checked programmatically rather than via JSON Schema, because those binary formats must be read by GDAL/OGR before attribute constraints can be evaluated.

# CRS authority and bounding-box constraints

Every spatial file must declare a CRS whose EPSG authority entry defines a valid coordinate domain. For WGS 84 (EPSG:4326), longitudes must fall within −180 to 180° and latitudes within −90 to 90°. A common error is exporting a dataset in EPSG:3857 (Web Mercator) but leaving the declared CRS as EPSG:4326: coordinates of (−20,037,508, 20,037,508) are numerically valid metres but are wildly out of range for degrees, and a linter that checks bounding-box plausibility will catch this in milliseconds.

# Implementation Walkthrough

# Step 1 — Author the schema file

Store the schema alongside the data under schemas/vector_feature.schema.json. This file is the single source of truth for your attribute dictionary.

# schemas/build_schema.py
# Generates schemas/vector_feature.schema.json programmatically
import json, pathlib

ALLOWED_GEOMETRY_TYPES = [
    "Point", "MultiPoint",
    "LineString", "MultiLineString",
    "Polygon", "MultiPolygon",
]

schema = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "OrgVectorFeature",
    "type": "object",
    "required": ["type", "features"],
    "properties": {
        "type": {"enum": ["FeatureCollection"]},
        "features": {
            "type": "array",
            "minItems": 1,
            "items": {
                "type": "object",
                "required": ["type", "geometry", "properties"],
                "properties": {
                    "type": {"enum": ["Feature"]},
                    "geometry": {
                        "type": "object",
                        "required": ["type", "coordinates"],
                        "properties": {
                            "type": {"enum": ALLOWED_GEOMETRY_TYPES},
                            "coordinates": {"type": "array", "minItems": 1},
                        },
                    },
                    "properties": {
                        "type": "object",
                        "required": ["dataset_id", "crs", "license", "source_date"],
                        "properties": {
                            "dataset_id": {"type": "string", "minLength": 1},
                            "crs": {
                                "type": "string",
                                "pattern": "^(EPSG:[0-9]+|urn:ogc:def:crs:EPSG::[0-9]+)$",
                            },
                            "license": {
                                "type": "string",
                                "enum": ["ODbL-1.0", "CC-BY-4.0", "CC0-1.0", "proprietary"],
                            },
                            "source_date": {"type": "string", "format": "date"},
                        },
                    },
                },
            },
        },
    },
}

out = pathlib.Path("schemas/vector_feature.schema.json")
out.parent.mkdir(exist_ok=True)
out.write_text(json.dumps(schema, indent=2))
print(f"Schema written to {out}")

# Step 2 — Write the spatial linting script

This script is the core validation module invoked both by the pre-commit hook and by the CI job.

#!/usr/bin/env python3
"""scripts/validate_spatial.py — spatial schema linter for CI gates.

Usage:
    python scripts/validate_spatial.py --files path/to/file.geojson [more.gpkg ...]
    python scripts/validate_spatial.py --changed  # auto-detect changed files via git diff
"""
from __future__ import annotations

import argparse
import json
import subprocess
import sys
from pathlib import Path

import fiona
import geopandas as gpd
import jsonschema
from pyproj import CRS, Transformer
from shapely.validation import make_valid

SCHEMA_PATH = Path("schemas/vector_feature.schema.json")
CRS_BOUNDS: dict[int, tuple[float, float, float, float]] = {
    4326: (-180.0, -90.0, 180.0, 90.0),
    3857: (-20_037_508.34, -20_048_966.1, 20_037_508.34, 20_048_966.1),
    27700: (-103_976.3, -16_703.87, 652_897.98, 1_199_851.44),  # British National Grid
}


def load_schema() -> dict:
    if not SCHEMA_PATH.exists():
        raise FileNotFoundError(f"Schema not found: {SCHEMA_PATH}")
    return json.loads(SCHEMA_PATH.read_text())


def changed_spatial_files() -> list[Path]:
    """Return spatial files changed relative to origin/main."""
    result = subprocess.run(
        ["git", "diff", "--name-only", "origin/main...HEAD"],
        capture_output=True, text=True, check=True,
    )
    suffixes = {".geojson", ".gpkg", ".shp"}
    return [Path(p) for p in result.stdout.splitlines() if Path(p).suffix in suffixes and Path(p).exists()]


def validate_geojson(path: Path, schema: dict) -> list[str]:
    errors: list[str] = []
    try:
        data = json.loads(path.read_text())
    except json.JSONDecodeError as exc:
        return [f"{path}: JSON parse error — {exc}"]

    validator = jsonschema.Draft202012Validator(schema)
    for err in sorted(validator.iter_errors(data), key=lambda e: list(e.path)):
        errors.append(f"{path}: schema — {err.message} at {list(err.path)}")

    # Spatial integrity
    try:
        gdf = gpd.read_file(path)
    except Exception as exc:  # noqa: BLE001
        errors.append(f"{path}: GDAL read error — {exc}")
        return errors

    errors.extend(_spatial_checks(path, gdf))
    return errors


def validate_vector(path: Path) -> list[str]:
    """Validate GeoPackage or Shapefile (non-JSON formats)."""
    errors: list[str] = []
    try:
        with fiona.open(path) as src:
            declared_crs_str = src.crs_wkt or ""
            if not declared_crs_str:
                errors.append(f"{path}: missing CRS declaration")
    except Exception as exc:  # noqa: BLE001
        errors.append(f"{path}: fiona open error — {exc}")
        return errors

    try:
        gdf = gpd.read_file(path)
    except Exception as exc:  # noqa: BLE001
        errors.append(f"{path}: geopandas read error — {exc}")
        return errors

    errors.extend(_spatial_checks(path, gdf))
    return errors


def _spatial_checks(path: Path, gdf: gpd.GeoDataFrame) -> list[str]:
    errors: list[str] = []
    if gdf.empty:
        errors.append(f"{path}: file contains zero features")
        return errors

    # Geometry validity
    invalid_mask = ~gdf.geometry.is_valid
    if invalid_mask.any():
        count = int(invalid_mask.sum())
        errors.append(f"{path}: {count} invalid geometry/geometries — run shapely.make_valid() before committing")

    # CRS bounds check
    if gdf.crs is None:
        errors.append(f"{path}: CRS is None after read")
    else:
        epsg = gdf.crs.to_epsg()
        if epsg in CRS_BOUNDS:
            minx, miny, maxx, maxy = CRS_BOUNDS[epsg]
            bounds = gdf.total_bounds  # (minx, miny, maxx, maxy)
            if bounds[0] < minx or bounds[1] < miny or bounds[2] > maxx or bounds[3] > maxy:
                errors.append(
                    f"{path}: coordinates {bounds.tolist()} exceed declared CRS {epsg} bounds — "
                    "possible CRS mismatch or unit error"
                )

    # Mandatory attribute check for non-GeoJSON (GeoJSON checked via JSON Schema above)
    if path.suffix != ".geojson":
        for col in ("dataset_id", "license", "source_date"):
            if col not in gdf.columns:
                errors.append(f"{path}: missing mandatory column '{col}'")

    return errors


def main() -> None:
    parser = argparse.ArgumentParser(description="Spatial schema linter")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--files", nargs="+", type=Path, help="Explicit file list")
    group.add_argument("--changed", action="store_true", help="Lint only git-changed files")
    parser.add_argument("--report-format", choices=["text", "json"], default="text")
    args = parser.parse_args()

    schema = load_schema()
    files: list[Path] = args.files if args.files else changed_spatial_files()

    all_errors: list[str] = []
    for f in files:
        if f.suffix == ".geojson":
            all_errors.extend(validate_geojson(f, schema))
        else:
            all_errors.extend(validate_vector(f))

    if args.report_format == "json":
        import pathlib
        report_dir = pathlib.Path("reports")
        report_dir.mkdir(exist_ok=True)
        (report_dir / "validation_errors.json").write_text(
            json.dumps({"errors": all_errors, "file_count": len(files)}, indent=2)
        )

    if all_errors:
        for msg in all_errors:
            print(f"ERROR: {msg}", file=sys.stderr)
        sys.exit(1)

    print(f"OK: {len(files)} file(s) passed all spatial schema checks.")


if __name__ == "__main__":
    main()

# Step 3 — Configure the pre-commit hook

Run the linter locally on every staged spatial file before the push reaches CI, keeping the feedback loop tight.

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: spatial-schema-lint
        name: Validate geospatial assets
        entry: python scripts/validate_spatial.py --files
        language: python
        additional_dependencies:
          - "geopandas>=0.14"
          - "shapely>=2.0"
          - "fiona>=1.9"
          - "pyproj>=3.6"
          - "jsonschema>=4.21"
        types_or: [file]
        files: \.(geojson|gpkg|shp)$
        pass_filenames: true

# Step 4 — GitHub Actions workflow

Path-scoped triggers ensure that documentation-only commits never queue unnecessary spatial validation. For integration with setting up GitHub Actions for ISO 19115 validation, run this structural gate first and the metadata check as a dependent job.

# .github/workflows/spatial-schema-lint.yml
name: Spatial Schema Lint

on:
  pull_request:
    paths:
      - 'data/**/*.geojson'
      - 'data/**/*.gpkg'
      - 'data/**/*.shp'
      - 'schemas/**'

jobs:
  validate:
    runs-on: ubuntu-22.04
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # needed for git diff --name-only origin/main...HEAD

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install GDAL system libraries
        run: |
          sudo apt-get update -qq
          sudo apt-get install -y --no-install-recommends gdal-bin libgdal-dev

      - name: Install Python dependencies
        run: |
          pip install --upgrade pip
          pip install \
            "geopandas>=0.14" \
            "shapely>=2.0" \
            "fiona>=1.9" \
            "pyproj>=3.6" \
            "jsonschema>=4.21" \
            "pydantic>=2.5" \
            "lxml>=5.1"

      - name: Run schema validation (changed files only)
        run: python scripts/validate_spatial.py --changed --report-format json

      - name: Upload validation report on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: spatial-lint-report
          path: reports/validation_errors.json
          retention-days: 14

# Validation & CI Integration

# Verifying locally before push

# Check a single file
python scripts/validate_spatial.py --files data/boundaries/admin_level2.geojson

# Check all spatial files changed relative to main
python scripts/validate_spatial.py --changed

# Quick GDAL sanity check (no Python required)
ogrinfo -al -so data/boundaries/admin_level2.gpkg

ogrinfo -al -so prints layer name, feature count, geometry type, and CRS — a rapid sanity check that confirms GDAL can open the file and that the declared CRS is parseable before the Python linter runs.

# JSON Schema validation as a standalone step

# scripts/check_schema_only.py — useful for validating schema file itself
import json, jsonschema, pathlib, sys

draft = json.loads(pathlib.Path("schemas/vector_feature.schema.json").read_text())
meta_schema_url = "https://json-schema.org/draft/2020-12/schema"

# Validate schema file structure (requires jsonschema[format-nspkg])
try:
    jsonschema.Draft202012Validator.check_schema(draft)
    print("Schema file is valid JSON Schema draft 2020-12")
except jsonschema.SchemaError as exc:
    print(f"Schema file invalid: {exc.message}", file=sys.stderr)
    sys.exit(1)

# Integrating with policy enforcement gates

Configure the GitHub Actions job as a required status check in the branch protection settings. This wires schema linting into the policy enforcement gates for data PRs workflow: a non-zero exit from the linter automatically blocks the merge button and prevents silent data quality regressions from reaching the main branch.

For repositories that mix code and spatial data, run the linter as a separate job from your Python unit tests. This lets reviewers see spatial failures and code failures independently, which shortens triage time.

# Derivative & Lineage Management

Schema linting obligations do not end at the source dataset boundary. Any transformation that produces a new geospatial file — reprojection, clip, spatial join, rasterization — creates a derivative that must independently satisfy the schema before it is committed.

# Reprojection

When reprojecting from EPSG:27700 to EPSG:4326 the crs attribute field must be updated in the output file’s attribute table, not just in the file’s projection header. A common lineage error is re-running a pipeline that writes EPSG:4326 coordinates while leaving a stale "crs": "EPSG:27700" attribute value, which then fails the schema linter’s pattern constraint on the attribute field.

import geopandas as gpd

src = gpd.read_file("data/boundaries/admin_level2_osgb.gpkg")
dst = src.to_crs(epsg=4326)
# Keep attribute table in sync with actual projection
dst["crs"] = "EPSG:4326"
dst.to_file("data/boundaries/admin_level2_wgs84.gpkg", driver="GPKG")

# Clip and spatial join

Clipping a dataset to a bounding box may produce slivers or degenerate geometries at the clip boundary. Run shapely.make_valid over the output before committing, and re-assert the geometry type constraint — a Polygon clip can produce GeometryCollections if the clip mask touches a vertex exactly, which then fails the enum whitelist in the schema.

from shapely.validation import make_valid
import geopandas as gpd

clipped = gpd.read_file("data/clipped_output.gpkg")
clipped["geometry"] = clipped["geometry"].apply(make_valid)
# Explode geometry collections down to single type
clipped = clipped.explode(index_parts=False).reset_index(drop=True)
# Re-filter to polygon-only if schema requires it
clipped = clipped[clipped.geometry.geom_type == "Polygon"].copy()
clipped.to_file("data/clipped_output_clean.gpkg", driver="GPKG")

# Rasterization lineage

Rasterized outputs (GeoTIFF, COG) require a parallel linting track covering TIFF header validation, CRS authority checks, and nodata value consistency. For workflows that produce both vector and raster derivatives, add a separate CI job rather than extending the vector linter — mixing GDAL vector and raster validation paths in one script introduces complex dependency chains that are difficult to cache efficiently.

# Pitfalls & Resolution Table

# Related

• CI/CD Validation & Policy Enforcement for Spatial Data — parent section covering the full pipeline architecture, from versioned checkout through to immutable artefact generation
• Policy Enforcement Gates for Data PRs — how to wire schema lint results into branch protection rules and reviewer approval workflows
• Automated Broken Link and Reference Detection — companion gate that verifies external href, url, and path fields inside spatial attributes resolve correctly
• Integrating PyLint with Spatial Metadata Validators — extend linting to cover the Python scripts that process spatial data, not just the data files themselves
• Setting up GitHub Actions for ISO 19115 Validation — run ISO 19115-1:2014 metadata compliance as a dependent CI job after this structural gate passes