# Pruning CI Metadata Artifacts on a Retention Schedule

Run the prune as a dry run that reports what it would delete, require an explicit flag to act, and reconcile monthly against what the policy says should exist — a deletion job that has never been reviewed is indistinguishable from a deletion job that is quietly removing the wrong prefix.

Retention has two halves and the second is where the risk sits. Writing artifacts is additive and forgiving; deleting them is neither, and a pruning job encountering an artifact class it was never taught about will either skip it forever or match it too broadly. Both failures are silent. This guide sits under metadata artifact retention strategies, within CI/CD Validation & Policy Enforcement for Spatial Data.

# Automated Python Implementation

#!/usr/bin/env python3
"""Prune CI artifacts against a declared retention policy. Dry run by default."""
import argparse
import datetime
import fnmatch

import boto3

# Retention in days per artifact class, keyed by the prefix it lives under.
POLICY = {
    "payloads/*": {"days": 7, "reason": "regenerable from the commit"},
    "reports/*": {"days": 90, "reason": "one release cycle"},
    "verdicts/*": {"days": 2555, "reason": "evidence, seven years"},
    "logs/*": {"days": 30, "reason": "debugging window"},
}


def classify(key, age_days, policy=POLICY):
    """keep, delete, or unclassified. Unknown prefixes are never deleted."""
    for pattern, rule in policy.items():
        if fnmatch.fnmatch(key, pattern):
            return ("delete" if age_days > rule["days"] else "keep"), pattern
    return "unclassified", None


def plan(client, bucket, now=None, policy=POLICY):
    now = now or datetime.datetime.now(datetime.timezone.utc)
    buckets = {"keep": [], "delete": [], "unclassified": []}
    paginator = client.get_paginator("list_objects_v2")
    for page in paginator.paginate(Bucket=bucket):
        for obj in page.get("Contents", []):
            age = (now - obj["LastModified"]).days
            verdict, pattern = classify(obj["Key"], age, policy)
            buckets[verdict].append({
                "key": obj["Key"], "age_days": age,
                "size": obj["Size"], "rule": pattern,
            })
    return buckets


def summarise(buckets):
    return {
        state: {
            "count": len(items),
            "bytes": sum(i["size"] for i in items),
            "sample": [i["key"] for i in items[:5]],
        }
        for state, items in buckets.items()
    }


def execute(client, bucket, to_delete, batch=1000):
    deleted = 0
    for start in range(0, len(to_delete), batch):
        chunk = [{"Key": item["key"]} for item in to_delete[start:start + batch]]
        client.delete_objects(Bucket=bucket, Delete={"Objects": chunk, "Quiet": True})
        deleted += len(chunk)
    return deleted


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("bucket")
    parser.add_argument("--apply", action="store_true",
                        help="actually delete; omit for a dry run")
    parser.add_argument("--max-delete", type=int, default=5000,
                        help="refuse runs larger than this without review")
    args = parser.parse_args()

    client = boto3.client("s3")
    buckets = plan(client, args.bucket)
    report = summarise(buckets)
    print(report)

    if buckets["unclassified"]:
        raise SystemExit(
            f"{len(buckets['unclassified'])} object(s) match no policy rule; "
            "extend the policy before pruning")

    if not args.apply:
        return
    if len(buckets["delete"]) > args.max_delete:
        raise SystemExit(
            f"{len(buckets['delete'])} deletions exceeds --max-delete; review first")
    print("deleted", execute(client, args.bucket, buckets["delete"]))


if __name__ == "__main__":
    main()

Three safeguards in that script are the difference between a prune and an incident.

Unclassified objects abort the run. An object matching no rule is evidence that the policy is incomplete, and continuing means either leaving it forever or, after somebody adds a catch-all rule in frustration, deleting things nobody classified. Stopping forces the policy to be extended deliberately.

A dry run is the default. The flag is --apply, not --dry-run, so the mistake of forgetting a flag results in a report rather than a deletion.

A deletion cap requires review. A run proposing to delete fifty thousand objects when it usually deletes two hundred has almost certainly matched something it should not. The cap turns that into a stop rather than a discovery.

# Validation and Pipeline Integration

def test_unknown_prefix_is_unclassified_not_deleted():
    verdict, _ = classify("newthing/2026/x.json", age_days=9999)
    assert verdict == "unclassified"


def test_evidence_survives_a_long_age():
    verdict, _ = classify("verdicts/org/abc/1/verdicts.json", age_days=400)
    assert verdict == "keep"


def test_every_rule_matches_something(client_stub):
    buckets = plan(client_stub, "bucket")
    used = {item["rule"] for item in buckets["keep"] + buckets["delete"]}
    assert set(POLICY) - used == set(), "a rule that matches nothing is a dead rule"

The third test is the one that catches the quiet failure. A rule whose prefix was renamed matches nothing, so the job reports zero deletions for that class and looks like it is working. Asserting that every rule matched at least one object turns a dead rule into a test failure.

Run the prune on a schedule with --apply, and run the same script in report mode as part of the monthly reconciliation described in metadata artifact retention strategies. The two runs use identical code, which is what keeps the report honest about what the job will actually do.

# Deleting Is Not the Only Option

Framing retention as delete-or-keep discards the middle ground where most artifacts belong, and the middle options are cheaper than either extreme.

Transition rather than delete. An artifact moved to a colder storage class costs a fraction of standard storage and remains available. For anything that might be needed once a year, this is strictly better than deletion, and it is a lifecycle rule rather than a job.

Compact rather than delete. A hundred per-file verdict documents from one run compress into a single document, which reduces both storage and request costs while losing nothing. Compaction is a good answer for older evidence where the per-file granularity has stopped being useful.

Keep the hash and drop the content. Where an artifact’s value is that it can be proved to have said something, retaining a hash record and deleting the payload preserves the proof at negligible cost. This is the pattern described in audit trail and evidence retention, and it converts an unbounded storage obligation into a bounded one.

The one option that should never be chosen implicitly is deletion by omission — an artifact class nobody wrote a rule for, sitting in a bucket, eventually removed by whoever is cleaning up next quarter. That is what the unclassified abort exists to prevent.

# Long-Term Compliance Best Practices

• Make the dry run the default. A forgotten flag should produce a report, never a deletion.
• Abort on unclassified objects. They are the signal that the policy has fallen behind the pipeline.
• Assert that every rule matches something. A dead rule silently retains a class forever.
• Cap the deletion size. A run proposing an unusual volume has matched something unexpected.
• Log what was deleted, and keep that log longer than the artifacts. A deletion nobody recorded is indistinguishable from a loss.

# Related

• Metadata Artifact Retention Strategies — the policy this job enforces
• Publishing Validation Artifacts to Object Storage — the write side, and the prefixes this prunes
• Audit Trail & Evidence Retention — keeping the hash when the content goes
• Hashing Datasets for Tamper-Evident Audit Logs — what a retained hash can and cannot prove