Geospatial Compliance Metadata Automation Hub

Checking Catalog Distribution URLs in CI

Check distribution URLs on a schedule rather than on every pull request, compare the final host against the recorded one so a redirect to a parking page is not counted as reachable, and cache negative results for hours rather than days so a ten-minute outage does not mark a live dataset dead for a week.

A distribution URL is the one part of a catalogue record that depends on somebody else continuing to do something. Everything else — the title, the extent, the licence — stays true once written. The URL decays silently, and the catalogue keeps asserting it works. This guide sits under automated broken link and reference detection, within CI/CD Validation & Policy Enforcement for Spatial Data.

Whether a distribution URL check belongs in the pull requestThree tests decide whether a URL check runs per change, on a schedule, or not at all.Was the URL added or changed inthis diff?Check it nowthe author can fix a typoyesnoIs the host inside your control?Scheduled, dailyyour outage, your alertyesnoScheduled, weeklynever blocks a merge
Only a newly added URL belongs in a pull request; the rest is scheduled work.

Automated Python Implementation

#!/usr/bin/env python3
"""Check catalogue distribution URLs, politely and with useful verdicts."""
import collections
import time
import urllib.parse
import urllib.request

USER_AGENT = "example-catalogue-linkcheck/1.0 (+https://example.org/about/linkcheck)"
TIMEOUT = 20


def normalise(url):
    """Canonical form for deduplication: lowercase host, no fragment."""
    parts = urllib.parse.urlsplit(url)
    return urllib.parse.urlunsplit(
        (parts.scheme.lower(), parts.netloc.lower(), parts.path, parts.query, ""))


def group_by_host(urls):
    grouped = collections.defaultdict(list)
    for url in {normalise(u) for u in urls}:
        grouped[urllib.parse.urlsplit(url).netloc].append(url)
    return grouped


def probe(url, method="HEAD"):
    """One request. Returns (status, final_url) or raises."""
    request = urllib.request.Request(url, method=method)
    request.add_header("User-Agent", USER_AGENT)
    with urllib.request.urlopen(request, timeout=TIMEOUT) as response:
        return response.status, response.geturl()


def classify(url):
    """Verdict for one URL, with the host comparison that catches parking pages."""
    try:
        status, final = probe(url, "HEAD")
    except urllib.error.HTTPError as exc:
        if exc.code in (403, 405):                 # HEAD refused, not a dead link
            try:
                status, final = probe(url, "GET")
            except Exception as inner:
                return {"url": url, "verdict": "unreachable", "detail": str(inner)}
        elif exc.code in (404, 410):
            return {"url": url, "verdict": "gone", "detail": f"HTTP {exc.code}"}
        elif exc.code == 429:
            return {"url": url, "verdict": "throttled",
                    "detail": exc.headers.get("Retry-After", "no Retry-After")}
        else:
            return {"url": url, "verdict": "error", "detail": f"HTTP {exc.code}"}
    except Exception as exc:
        return {"url": url, "verdict": "unreachable", "detail": str(exc)}

    original_host = urllib.parse.urlsplit(url).netloc
    final_host = urllib.parse.urlsplit(final).netloc
    if final_host != original_host:
        return {"url": url, "verdict": "redirected_offsite", "detail": final,
                "final_host": final_host}
    return {"url": url, "verdict": "live", "detail": f"HTTP {status}"}


def check_all(urls, per_host_delay=0.5):
    """Deduplicate, group by host, and space requests within each host."""
    results = []
    for host, host_urls in group_by_host(urls).items():
        for url in sorted(host_urls):
            results.append(classify(url))
            time.sleep(per_host_delay)
    return results

The redirected_offsite verdict is the one that repays the effort. A retired dataset frequently redirects to a portal home page or, worse, to a domain that has been re-registered; both return 200 and both are indistinguishable from success without comparing hosts. Reporting the final host lets a reviewer see immediately that a link to data.example.gov now lands on something else.

The verdicts and what the catalogue should do with eachSix link-check verdicts mapped to the action a catalogue maintainer should take.MeaningCatalogue actionlivereachable at the recorded hostrecord the check dateredirected_offsite200, but somewhere elsereview — often a retirementgone404 or 410fail the recordthrottled429, we are asking too fastback off, re-checkunreachabletimeout or DNS failureretry before believing iterror5xx at the far endtheir problem, note it
Two of the six are failures. Treating the others as failures is how link checking gets disabled.

Validation and Pipeline Integration

def test_head_refusal_falls_back_to_get():
    result = classify("https://example.org/refuses-head.zip")
    assert result["verdict"] in {"live", "unreachable"}
    assert result["verdict"] != "gone"


def test_offsite_redirect_is_not_live():
    result = classify("https://old.example.gov/dataset.zip")
    if result["verdict"] == "redirected_offsite":
        assert result["final_host"] != "old.example.gov"


def test_urls_are_deduplicated_before_probing():
    urls = ["https://Example.org/a.zip", "https://example.org/a.zip#frag"]
    assert len(group_by_host(urls)["example.org"]) == 1

Run the checker from a stable address on a schedule and publish the results as a report rather than a gate. Where a URL was added in the diff, check that one URL in the pull request — a typo in a newly added link is the author’s to fix and costs one request to catch.

How a distribution URL decays, and when a check would noticeA URL is published, the site is reorganised behind a redirect, the redirect is retired, and the record continues to assert the original link.publicationRecorded livechecked once, believed forevermonths laterReorganisedredirect in place, still 200a year onRedirect retirednow a 404next scheduled runDetectedthe only mechanism that will
None of these transitions emits an event. Only a scheduled check turns them into information.

Recording the Check, Not Just the Failure

A link checker that reports only failures throws away most of what it learned. The positive results are the more useful half, for two reasons.

A last-checked date is what makes a record’s claim honest. A distribution that says “available” with no date is an assertion about an unknown moment. One that says “available, checked three days ago” is a fact with an expiry, and a consumer can weigh it. Writing the check date and verdict back into the catalogue record costs one field and turns the checker’s output into part of the record rather than a report nobody reads.

Coverage is otherwise invisible. A run that checked forty of five hundred URLs and found no failures reports a clean result identical to one that checked all five hundred. Recording every verdict, including the successes, makes the coverage countable — and coverage is the number that goes wrong first when a checker is silently failing to enumerate part of the catalogue.

The same records make the flapping problem tractable. A URL that alternates between live and unreachable across successive runs is usually a rate limit or an intermittent host rather than a broken link, and only a history distinguishes it from a genuine failure. A simple rule — fail a URL only after it has been unreachable on two consecutive scheduled runs at least a day apart — removes almost all of the noise without slowing the detection of a real removal by more than a day.

Checking Cloud Object URLs

A growing share of distribution URLs point at object storage rather than at a web server, and object stores behave differently in ways that produce misleading verdicts.

A signed URL expires. A distribution recorded with a pre-signed link will return a 403 once the signature lapses, which the checker reports as unreachable. The URL was never durable and the record should not have contained it; the check has found a cataloguing error rather than an outage. Recording signed URLs in a published catalogue is worth flagging as its own rule.

Requester-pays buckets refuse anonymous requests. A 403 from a requester-pays bucket is the correct response to an unauthenticated probe and says nothing about whether the object exists. Without a per-host exception the checker will report the entire bucket as gone.

A missing object and a missing bucket look different. Most object stores distinguish them by error code, and the distinction matters: one dataset removed is a record to fix, an entire bucket gone is an incident affecting every record that references it. Grouping failures by host before reporting turns four hundred individual failures into one finding.

Long-Term Compliance Best Practices

  • Identify the checker in the user agent. A crawler an administrator can identify is allowed; one they cannot is blocked.
  • Bound concurrency per host, not overall. Forty requests across forty domains is polite; forty at one is an incident.
  • Cache negatives briefly and positives longer. A ten-minute outage should not mark a dataset dead for a week.
  • Compare the final host. A redirect to a parking page returns 200 and is the failure most worth catching.
  • Run from a stable address. Ephemeral runner addresses hit per-address rate limits that a fixed scheduled job never sees.