# Running Spatial Validation on a Matrix of Formats

Use a matrix when the same check must run against genuinely different drivers — GeoPackage, GeoJSON, shapefile, GeoTIFF — set fail-fast: false so one driver’s failure does not hide the others, and do not use a matrix to parallelise a handful of files, which costs more in provisioning than it saves.

Matrices are the most over-applied feature in CI configuration. Their real value in spatial work is not speed but coverage: a validator that behaves correctly on a GeoPackage may quietly do nothing on a shapefile, because the driver reports different capabilities and the code path silently short-circuits. Running the same check across drivers is how that gets discovered. This guide sits under GitHub Actions workflows for spatial data, within CI/CD Validation & Policy Enforcement for Spatial Data.

# Automated Python Implementation

The runner below takes a format specification and executes the shared check set against fixtures for that format. The point of the design is that the check code is identical across legs; only the reader differs.

#!/usr/bin/env python3
"""Run one shared check set against a single spatial format."""
import json
import sys

from osgeo import ogr, gdal

gdal.UseExceptions()

FORMATS = {
    "gpkg": {"driver": "GPKG", "glob": "*.gpkg", "supports_metadata_table": True},
    "geojson": {"driver": "GeoJSON", "glob": "*.geojson", "supports_metadata_table": False},
    "shapefile": {"driver": "ESRI Shapefile", "glob": "*.shp",
                  "supports_metadata_table": False},
    "gtiff": {"driver": "GTiff", "glob": "*.tif", "supports_metadata_table": False},
}


def open_dataset(path, driver_name):
    """Open with an explicit driver so a format mismatch fails loudly."""
    driver = ogr.GetDriverByName(driver_name) or gdal.GetDriverByName(driver_name)
    if driver is None:
        raise RuntimeError(f"driver {driver_name} is not available in this build")
    dataset = gdal.OpenEx(path)
    actual = dataset.GetDriver().ShortName
    if actual != driver_name:
        raise RuntimeError(f"{path}: opened as {actual}, expected {driver_name}")
    return dataset


def check_crs_declared(dataset):
    """Every layer must declare a spatial reference."""
    findings = []
    for i in range(dataset.GetLayerCount() or 0):
        layer = dataset.GetLayer(i)
        if layer.GetSpatialRef() is None:
            findings.append(f"layer {layer.GetName()!r} has no spatial reference")
    if dataset.GetLayerCount() in (0, None) and not dataset.GetProjection():
        findings.append("raster has no projection")
    return findings


def check_licence_recorded(dataset, spec):
    """Where the format cannot carry it, say so rather than passing silently."""
    meta = dataset.GetMetadata() or {}
    if any(k.lower() == "license" for k in meta):
        return []
    if spec["supports_metadata_table"]:
        return ["no licence recorded in the metadata table"]
    return ["no licence in dataset metadata (format has no metadata table)"]


CHECKS = [check_crs_declared]


def run_for_format(format_key, paths):
    spec = FORMATS[format_key]
    results = []
    for path in paths:
        dataset = open_dataset(path, spec["driver"])
        findings = []
        for check in CHECKS:
            findings.extend(check(dataset))
        findings.extend(check_licence_recorded(dataset, spec))
        results.append({"format": format_key, "path": path, "findings": findings})
    return results


if __name__ == "__main__":
    key = sys.argv[1]
    print(json.dumps(run_for_format(key, sys.argv[2:]), indent=2))

The open_dataset check — comparing the driver that actually opened the file against the one expected — is the assertion that makes the matrix worth running. Without it, a shapefile leg handed a GeoPackage by a glob mistake will open it happily and report a pass, and the leg proves nothing about the format it was named for.

WORKFLOW = """
jobs:
  validate:
    strategy:
      fail-fast: false
      matrix:
        format: [gpkg, geojson, shapefile, gtiff]
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/restore-spatial-env
      - name: Validate ${{ matrix.format }}
        run: .venv/bin/python validate.py ${{ matrix.format }} $(cat changed-${{ matrix.format }}.txt)
      - uses: actions/upload-artifact@v4
        with:
          name: report-${{ matrix.format }}
          path: report.json
"""

fail-fast: false is not a stylistic preference here. With the default, the first failing driver cancels the rest, so a run reports one problem and hides three — and the next run, after that one is fixed, reports the second. A four-driver matrix with fail-fast enabled can take four rounds to surface four problems.

# Validation and Pipeline Integration

The matrix needs its own tests, and they are about the legs rather than the checks.

def test_every_format_key_has_a_driver():
    for key, spec in FORMATS.items():
        assert spec["driver"], f"{key} has no driver"


def test_wrong_driver_is_refused(tmp_path):
    path = str(tmp_path / "not_a_shapefile.gpkg")
    make_empty_gpkg(path)
    try:
        open_dataset(path, "ESRI Shapefile")
    except RuntimeError:
        return
    raise AssertionError("a format mismatch must not pass silently")


def test_formats_without_a_metadata_table_report_the_limitation():
    findings = check_licence_recorded(EMPTY_GEOJSON, FORMATS["geojson"])
    assert findings and "no metadata table" in findings[0]

Aggregate the legs into a single status rather than letting each post its own. Four separate checks on a pull request produce four rows in the review interface, and a reviewer reads the first one. One status with a per-format breakdown produces one row and the same information.

# Not Every Axis Deserves a Leg

Matrix dimensions multiply, and a matrix with three axes of four values is forty-eight runs of an environment that takes a minute to construct. Two questions keep it proportionate.

Does this axis change behaviour, or only configuration? Running the same check against four drivers changes which code path executes and is worth a leg each. Running it against four CRS values does not: the code path is identical and the difference belongs in a parameterised test inside one job.

Would a failure in this leg be diagnosed differently? The purpose of separating legs is that a failure names its context. If a failure in the gpkg leg and the geojson leg would send the same person to the same file with the same fix, the separation is decorative.

The axis most often added without justification is the Python version. It earns a leg when the repository is a library that others install, and it does not when the workflow validates data with a pinned environment — in that case the pinned version is the only one that will ever run, and testing three proves nothing about the one in use.

Where a matrix genuinely is needed but the minutes are uncomfortable, the lever is the fixture set rather than the axes. Each leg needs enough data to exercise the code path, not a copy of the catalogue, and a leg running against three small fixtures completes in the time it takes to provision.

# Long-Term Compliance Best Practices

• Set fail-fast: false on any matrix intended for coverage. A cancelled leg is an unknown result, not a passing one.
• Assert the driver that actually opened the file. A leg fed the wrong format proves nothing and reports a pass.
• Aggregate into one status. Reviewers read the first row; make it the informative one.
• Report per-leg results even when all pass. A leg that silently checked nothing is invisible in a green tick.
• Keep the check set identical across legs. The moment a leg has its own checks, the matrix has become four workflows sharing a file.

# Related

• GitHub Actions Workflows for Spatial Data — the surrounding workflow structure
• Caching GDAL and Python Dependencies in GitHub Actions — what makes a multi-leg matrix affordable
• Posting Validation Results as Pull Request Comments — aggregating the legs into something readable
• Spatial Data Schema Linting in CI — the check set the matrix runs