Reduce release latency without weakening release gates

[jeremi]

Context

Beta-14 exposed avoidable runner contention and a missing executable release check. PR #428 delivered the useful foundation: PR-only concurrency cancellation, conservative path routing, read-only release planning, immutable-tag checks, OCI-label smoke coverage, and release timing telemetry.

Beta-16/v0.12.2 then provided the measurement baseline and completed successfully from exact target e25f081ce800ade13e892503cc19b96588e081ef. Its standard workflow evidence, published checksums, cosign signatures, SLSA provenance, and post-publish reconciliation established that the core release controls work.

Accepted closeout scope

Completed in PR #431:

• patch fixable docs dependency advisories
• pin the remaining GitHub Pages actions to reviewed SHAs
• stagger routine Dependabot updates and limit each ecosystem to one open version-update PR
• refuse an existing or ambiguously queried GitHub Release before builds and image publication
• remove the asset replacement path and require a new patch version after publication failure
• align the internal release manual and Registry Engineering Skills with the lean operating rule

Cost decisions

The following designs are not planned:

• a dedicated finalization classifier, because the additional trusted policy surface is disproportionate and current path routing already avoids unnecessary product matrices
• a content-addressed Docs archive cache, because the implementation and state are not justified for an occasional full sweep
• a bespoke published-release evidence bundle, because Signed Releases already passes and the workflow already creates and reconciles checksums, SBOMs, signatures, provenance, and release assets
• duplicate advisory suppressions or other changes made only to improve a Scorecard number

The stale stacked PRs #432 and #433 were closed. PR #431 was rebuilt directly on current main, passed all required checks, and merged as 9e960d091df162709daf39e8bf606cae0c04ff45.

Operating rule

Keep source and tag binding, protected-branch checks, vulnerability policy, checksums, SBOMs, signatures, provenance, image labels, non-root execution, and publish-once behavior on every release.

Perform extra human evidence work only after a release-system change, a failed automated control, or a stable milestone. Do not repeat independent local rebuilds, exhaustive manual asset inventories, or release rehearsals for an unchanged beta pipeline. Run the full historical Docs sweep when dependencies, archive tooling, or docset pins change; ordinary content work should use current-site and affected-page checks.

Completion

PR #431 is merged, required checks passed, and the release manual and skills record the same boundary. No additional release rehearsal or beta publication is required.

[jeremi]

Independent workflow audit identified these concrete implementation targets:

• .github/workflows/ci.yml currently calls its broad mark_all() path for any workflow-file change. Classify .github/workflows/release.yml separately so it runs release tooling, source proof, workflow lint/contract tests, secret scanning, and other required checks without automatically scheduling every product/fuzz/tutorial job. Changes to ci.yml itself should continue to trigger the full matrix.
• Add first-class registry-release prepare and registry-release finalize commands. They should derive the next release/version from the previous manifest, validate all version/changelog/Crosswalk surfaces, repoint only the selected release refs after promotion, regenerate committed JSON mirrors, and emit a machine-readable plan.
• Add duration and cache telemetry before changing the tag build. Recent successful release workflows have grown from roughly 15 minutes to more than 30 minutes, with release binary compilation dominating. Evaluate reproducible cache warming or pinned sccache only after measuring it, and never reuse unsigned PR binaries as release evidence.
• Machine-check release-manual command examples against the public CLI help so procedural drift fails before an operator reaches a release window.

A sensible first implementation slice is: exact-PR concurrency cancellation, conservative release.yml path classification, executable OCI smoke, prepare/finalize planning commands, and duration telemetry.

[jeremi]

Clarification for the proposed preparation command: it may calculate and display the expected next beta/version, but it must require the operator to pass an explicit --version and --release-id. Automation should propagate and validate a release decision, not silently make release policy or permit version reuse.

[jeremi]

The first implementation slice has landed in PR #428 at protected-main commit d6d2d16.

Delivered in that slice:

• exact-PR CI cancellation and conservative release-path routing
• deterministic read-only prepare/finalize plans with explicit release identity
• fail-closed local and remote immutable-tag checks
• executable two-image OCI label smoke with clear stub-binary boundaries
• release cache and duration telemetry
• final review fixes for exact artifact inventory, registryctl lock coverage, candidate commit resolution, and safe evidence-URL rewriting

Beta-15 preparation PR #436 is now exercising the landed workflow. The remaining recommendations stay open in #431, #432, #433, #434, and #435 and are not being added to this release unless they become confirmed release blockers.

[jeremi]

Open-PR review against the beta-16 main line:

• fix(release): make release builds reproducible #440 is the confirmed release-reproducibility blocker. It has been rebased onto current main, preserves BuildKit provenance on registry-pushed images, and is running full CI on its final head.
• security: retire stale dependency exceptions #434 is now treated as a beta-16 security blocker, but must merge only after its rebase, minimal lockfile refresh, retired advisory-exception proof, and green Rust/Notary checks.
• maintenance: reduce release overhead and close Scorecard gaps #431 should wait for a rebase after the release. It currently conflicts with the canonical wrappers and can regress exact artifact-inventory validation.
• docs: cache immutable archive builds #432 should wait for the release and fix its production Pages policy. Main pushes currently use incremental archives where the intended contract says full rebuild.
• release: verify and attest published release evidence #433 should wait for the release and close the stale-signed-evidence path before merge, then rebase onto the final release workflow.
• validation: run CI and CodeQL for issue 427 stack #435 is obsolete validation evidence and must not be merged. A fresh main-targeted validation PR is needed after the remaining stack is rebuilt.

Concrete fix requests are recorded on each PR. The beta-16 scope remains limited to #440, #434, and the required refreshed preparation/finalization metadata.

[jeremi]

Beta-16/v0.12.2 completed the #440 reproducibility fix-forward and is the first measured release after #428.

Release workflow 29774136957 completed successfully from exact T e25f081ce800ade13e892503cc19b96588e081ef in 2,728 seconds to the telemetry collector. It used 4,482 completed runner-seconds (74.7 minutes), excluding the collector itself. The canonical Linux binary build dominated at 2,307 seconds and reported exact_key_hit: false for the new release-script cache key. Image build/push/scan took 123 seconds, release assembly/signing took 163 seconds, and provenance generation plus verification completed successfully.

Independent closeout verified the exact 106-asset inventory, all 35 cosign signatures, all 35 tag-bound SLSA subjects, byte-identical published Linux binaries and image inputs against both retained T builds, and published OCI application manifests/layers against retained T.

This does not close #427. Draft PRs #431-#433 still require their post-release rebases and integrity fixes, followed by the planned before/after rehearsal.

[jeremi]

Post-beta-16 implementation has started under a new cleanup goal.

Current plan:

• Make the Relay, Notary, and programme decision boundary normative #418 is closed as superseded with exact patch-equivalence evidence.
• Rebuild maintenance: reduce release overhead and close Scorecard gaps #431 from current main, carrying only the unique finalization and Dependabot controls while preserving beta-16 source-ref, artifact-inventory, registryctl lockfile, OCI, and reproducible-build invariants.
• Rebuild docs: cache immutable archive builds #432 from current main with production pushes forced to a full immutable archive rebuild; incremental caching will be manual-only.
• Rebuild release: verify and attest published release evidence #433 from current main with fail-closed evidence invalidation before payload clobber and final live-byte/hash comparison against the signed evidence bundle.
• Run the complete release-tool, source-proof, docs, dependency, security, and protected-branch gates, then record the before/after rehearsal required by this issue.

This issue remains open until those changes are merged and the acceptance evidence is recorded.

[jeremi]

Post-beta review complete. The implementation has been reduced to the lean replacement in #431, and the larger classifier, archive-cache, and evidence-bundle designs are explicitly not planned. This preserves the core release and Scorecard controls while making duplicate evidence work exception-based.