# Publishing Validation Artifacts to Object Storage

Write validation artifacts to a content-addressed key derived from the commit and run, set the storage class and lifecycle rule at write time rather than retrofitting them, and store the small verdict document separately from any large payload so the two can be retained on different schedules.

CI providers keep artifacts for weeks; compliance evidence is needed for years. That gap is why validation output ends up in object storage, and the move is usually done badly: everything in one bucket under a key derived from a build number, with no lifecycle policy and no separation between the verdict and the data that produced it. This guide sits under metadata artifact retention strategies, within CI/CD Validation & Policy Enforcement for Spatial Data.

# Automated Python Implementation

#!/usr/bin/env python3
"""Publish validation artifacts with keys and storage classes set at write time."""
import gzip
import hashlib
import io
import json
import os

import boto3

BUCKET = os.environ["ARTIFACT_BUCKET"]
PREFIXES = {"verdict": "verdicts", "report": "reports", "payload": "payloads"}
STORAGE_CLASS = {"verdict": "STANDARD_IA", "report": "STANDARD", "payload": "STANDARD"}


def key_for(kind, repo, commit_sha, run_id, name):
    """Sortable, traceable, collision-free: date is not in the key on purpose."""
    return f"{PREFIXES[kind]}/{repo}/{commit_sha[:12]}/{run_id}/{name}"


def put(client, kind, repo, commit_sha, run_id, name, body, content_type):
    key = key_for(kind, repo, commit_sha, run_id, name)
    buffer = io.BytesIO()
    with gzip.GzipFile(fileobj=buffer, mode="wb", mtime=0) as fh:
        fh.write(body)
    payload = buffer.getvalue()

    client.put_object(
        Bucket=BUCKET,
        Key=key,
        Body=payload,
        ContentType=content_type,
        ContentEncoding="gzip",
        StorageClass=STORAGE_CLASS[kind],
        ChecksumAlgorithm="SHA256",
        Metadata={
            "commit": commit_sha,
            "run-id": str(run_id),
            "artifact-kind": kind,
            "content-sha256": hashlib.sha256(body).hexdigest(),
        },
    )
    return key


def publish_run(client, repo, commit_sha, run_id, verdicts, report_html, payloads):
    written = {}
    written["verdict"] = put(
        client, "verdict", repo, commit_sha, run_id, "verdicts.json",
        json.dumps(verdicts, sort_keys=True, separators=(",", ":")).encode(),
        "application/json")
    written["report"] = put(
        client, "report", repo, commit_sha, run_id, "report.html",
        report_html.encode(), "text/html")
    for name, data in payloads.items():
        written.setdefault("payloads", []).append(
            put(client, "payload", repo, commit_sha, run_id, name,
                data, "application/octet-stream"))
    return written

Three choices in that writer are worth defending, because each addresses a failure that shows up months later.

The key contains the commit and run, not a date. Dates in keys are tempting for lifecycle rules and wrong for retrieval: the question anyone asks is “what did the checks say about this commit”, and a key derived from the commit answers it directly. Lifecycle rules operate on object age, which the store already knows.

mtime=0 in the gzip header. Without it, compressing identical content twice produces different bytes, so a content hash over the stored object changes on every run. That defeats deduplication and makes the stored checksum useless as an identity.

The content hash is stored as metadata alongside the object’s own checksum. The store’s checksum covers the compressed bytes; the metadata hash covers the content. Verifying evidence years later means comparing content, and the compressed representation may not survive a storage migration.

# Validation and Pipeline Integration

def test_key_is_derived_from_the_commit():
    key = key_for("verdict", "org/repo", "abcdef1234567890", 42, "verdicts.json")
    assert "abcdef123456" in key and key.startswith("verdicts/")


def test_identical_content_compresses_identically():
    body = b'{"result":"pass"}'
    first = compress(body)
    second = compress(body)
    assert first == second, "gzip mtime must be pinned for reproducible bytes"


def test_payloads_and_verdicts_use_different_prefixes():
    verdict = key_for("verdict", "org/repo", "a" * 40, 1, "v.json")
    payload = key_for("payload", "org/repo", "a" * 40, 1, "bad.gpkg")
    assert verdict.split("/")[0] != payload.split("/")[0]

Grant the workflow write access to the artifact prefixes and nothing else. A CI role that can delete objects is a role that can delete evidence, and the deletion should be performed by the lifecycle policy rather than by anything holding credentials.

Where the evidence needs to be tamper-evident, enable object lock in governance mode on the verdict prefix for the retention period. That makes an accidental cleanup script impossible rather than merely unlikely, which is a materially different claim to make to an auditor. The hashing discipline described in hashing datasets for tamper-evident audit logs then covers the content.

# Costs That Surprise People

Object storage is cheap enough that teams stop thinking about it, and then encounter one of three bills.

Request charges on many small objects. A run writing one object per checked file produces thousands of PUTs, and at catalogue scale the request charges exceed the storage charges by an order of magnitude. Writing one verdict document per run rather than one per file removes the problem entirely and makes the evidence easier to query.

Retrieval charges on cold storage. Infrequent-access and archive tiers are cheap to store and expensive to read, and a compliance report that scans a year of verdicts pays that cost every time it runs. Keeping verdicts in a standard tier for the period they are actively reported on, and transitioning afterwards, costs slightly more in storage and considerably less in total.

Early-deletion charges. Cold tiers bill a minimum storage duration, so an object transitioned to archive and deleted a week later is charged for the full minimum. A lifecycle rule that transitions at thirty days and expires at sixty is paying for ninety, and the fix is to choose one or the other.

The general shape is that the retention policy and the storage tiers have to be designed together. A policy written by compliance and a lifecycle rule written by engineering will produce a configuration that satisfies neither, and the discrepancy will surface as a bill rather than as an incident.

# Long-Term Compliance Best Practices

• Write the storage class at PUT time. Retrofitting a class means rewriting every object, which is a bill and a new set of modification dates.
• Keep verdicts small and separate. They are the part that must survive; anything large is almost always regenerable.
• Pin gzip metadata for reproducible bytes. Otherwise identical content stores differently on every run.
• Let the lifecycle policy do all deleting. Credentials that can delete evidence undermine the evidence.
• Record the artifact keys in the run report. An artifact nobody can find is not retained in any useful sense.

# Related

• Metadata Artifact Retention Strategies — the policy this implements
• Pruning CI Metadata Artifacts on a Retention Schedule — the deletion side of the same policy
• Audit Trail & Evidence Retention — what makes stored artifacts admissible as evidence
• GitHub Actions Workflows for Spatial Data — the run that produces these artifacts