# Caching GDAL and Python Dependencies in GitHub Actions

Key the cache on a hash of the lock file and the runner image, restore into a virtual environment rather than a package directory, and install the binary wheels that bundle GDAL instead of building it — the difference between a warm cache and a cold build on a spatial workflow is typically three minutes per run, every run.

Dependency installation is the dominant cost in almost every spatial CI workflow, and it is dominant by a wide margin. A validation step that takes four seconds sits behind an environment that takes four minutes to construct, and no amount of optimising the check changes the number anybody experiences. This guide sits under GitHub Actions workflows for spatial data, within CI/CD Validation & Policy Enforcement for Spatial Data.

# Automated Python Implementation

The cache key is the whole design, and it has to answer one question: would a cache built under these conditions still be correct now? Three inputs determine that.

#!/usr/bin/env python3
"""Compute a dependency cache key for a spatial CI environment."""
import hashlib
import pathlib
import platform
import sys

def file_digest(path):
    return hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()[:16]

def cache_key(lock_files=("requirements.lock", "constraints.txt"),
              runner_image=None, prefix="spatialenv"):
    """A key that changes exactly when the resolved environment would."""
    parts = [
        prefix,
        runner_image or platform.platform(terse=True),
        f"py{sys.version_info.major}.{sys.version_info.minor}",
    ]
    for lock in lock_files:
        path = pathlib.Path(lock)
        parts.append(file_digest(path) if path.exists() else "absent")
    return "-".join(parts)

def restore_keys(key):
    """Progressively less specific prefixes, so a near-miss still helps."""
    segments = key.split("-")
    return ["-".join(segments[:n]) + "-" for n in range(len(segments) - 1, 1, -1)]

The restore keys matter as much as the exact key. A lock file change invalidates the exact match, and without prefix fallbacks the run installs everything from scratch. With them, the run restores the previous environment for the same Python and runner, and pip installs only what actually changed — which for a one-package bump is seconds rather than minutes.

WORKFLOW_STEPS = """
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Restore the environment
        id: venv-cache
        uses: actions/cache@v4
        with:
          path: .venv
          key: spatialenv-${{ runner.os }}-py3.11-${{ hashFiles('requirements.lock') }}
          restore-keys: |
            spatialenv-${{ runner.os }}-py3.11-
            spatialenv-${{ runner.os }}-

      - name: Install if the cache missed
        if: steps.venv-cache.outputs.cache-hit != 'true'
        run: |
          python -m venv .venv
          .venv/bin/pip install --require-hashes -r requirements.lock

      - name: Verify the environment is what we think
        run: |
          .venv/bin/python -c "from osgeo import gdal; print(gdal.__version__)"
          .venv/bin/python -c "import rasterio, geopandas; print(rasterio.__gdal_version__)"
"""

Two details in those steps are worth stating explicitly.

Cache the virtual environment, not the pip cache. Caching ~/.cache/pip saves the download and still pays the install, which for compiled packages is most of the cost. Caching .venv restores an environment ready to use. The trade is that the cache is larger and tied to the exact Python version, which the key already encodes.

Verify after restoring. A restored environment can be subtly wrong — a wheel built against a different libgdal, a partially written cache from a cancelled run. Printing the GDAL version that rasterio was linked against costs a second and turns a bizarre downstream failure into an obvious one.

# Validation and Pipeline Integration

The cache is a correctness risk as well as a speed feature, so the assertions are about correctness.

def test_key_changes_with_the_lock_file(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    pathlib.Path("requirements.lock").write_text("gdal==3.8.4\n")
    first = cache_key(("requirements.lock",), runner_image="ubuntu-24.04")
    pathlib.Path("requirements.lock").write_text("gdal==3.9.0\n")
    assert cache_key(("requirements.lock",), runner_image="ubuntu-24.04") != first

def test_key_changes_with_the_runner_image():
    a = cache_key((), runner_image="ubuntu-22.04")
    b = cache_key((), runner_image="ubuntu-24.04")
    assert a != b

def test_restore_keys_are_progressively_shorter():
    keys = restore_keys("spatialenv-ubuntu-24.04-py3.11-abc123")
    assert keys == sorted(keys, key=len, reverse=True)

The second test guards a failure that is confusing when it happens. A cache built on one runner image and restored on another can contain wheels linked against a system library the new image does not have, producing an import error in a step that has worked for months. Including the image in the key makes the upgrade a cache miss rather than a mystery.

# When the Cache Is the Problem

Caching introduces a class of failure that is worth recognising quickly, because the instinct — to rerun the job — makes it worse by an hour.

A poisoned cache repeats. If a cache entry was written from a partially installed environment, every subsequent run restores the same broken state, and reruns fail identically. The tell is a failure that appeared without any change to the lock file or the code. Changing one character of the key prefix invalidates every entry and is the fastest resolution; deleting caches through the interface works and is slower.

A cache hit can hide a broken lock file. A run that restores a warm environment never executes the install step, so a lock file that no longer resolves will pass for as long as the cache survives — and fail on the day it expires, in an unrelated pull request. A weekly scheduled job that installs from scratch with the cache disabled catches this while it is still cheap.

Caches expire on their own schedule. Entries are evicted after a period of disuse and when a repository exceeds its cache allowance. A workflow that is fast on active branches and slow on a rarely touched one is not misconfigured; it is simply cold, and the fix is either accepting it or keeping a scheduled run warm.

The general principle is that a cache should be an optimisation you can remove without changing correctness. If disabling the cache makes the workflow fail rather than merely slow, the environment is not actually described by the lock file, and that is worth fixing before it is worth speeding up.

# Long-Term Compliance Best Practices

• Pin with hashes, not just versions. --require-hashes turns a compromised or republished package into a build failure rather than a silent substitution.
• Include the runner image in the key. Image upgrades change system libraries that wheels are linked against.
• Verify the restored environment. One line printing the GDAL version separates an environment problem from a data problem.
• Run one uncached build a week. It is the only thing that proves the lock file still resolves.
• Record the resolved versions in the run report. A verdict is only reproducible if the tools that produced it are known.

# Related

• GitHub Actions Workflows for Spatial Data — the workflow structure this optimises
• Running Spatial Validation on a Matrix of Formats — where cache reuse across legs pays for itself
• Setting Up GitHub Actions for ISO 19115 Validation — vendoring the schema set, the other large fixed cost
• Pre-commit Hooks for Spatial Metadata — the same dependency problem at commit time