diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 7a2ed4e..8ca9d81 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1325,6 +1325,25 @@ **AND `dc-mirror.sh check dc0` PASSES while this is true** -- it verifies sync STATUS, not that the mirror carries every suite the deployed image asks for. That is the checker-that-cannot-see-the-real-failure class again, and the gap is SUITE COVERAGE. + **^ RULED 2026-07-31 (GA-R5). Question as presented: add `jammy-backports` to the mirror and + re-sync, or remove backports from the nodes' sources? Operator answer, exact utterance: + "Sync the backports into the mirror".** Recorded as a **D-135 AMENDMENT** dated 2026-07-31; + OPS under GA-R3, no new D-number, and D-135's per-DC strategy split (dc0 full mirror / dc1 + caching proxy) is UNCHANGED. CONSEQUENCE: the mirror's declared scope becomes the jammy + triple PLUS `jammy-backports`. **Fixed at SOURCE (`scripts/dc-mirror.sh`), not only on the + live rack**, so a reinstall or a future DC standup cannot silently regenerate the old scope; + `tests/dc-mirror` T10 asserted the old dist string verbatim and is RE-POINTED to the new + invariant rather than deleted. **COST MEASURED BEFORE THE RULING WAS PUT: 1,070,687,402 + bytes (~1.00 GiB) across 461 packages** -- main 859 MB / 338 pkgs, universe 212 MB / 123 + pkgs, restricted and multiverse both EMPTY in backports -- against a **951 GB** existing + mirror with **1.8 T free**, i.e. ~0.1%. Sanity-checked rather than trusted: the five largest + entries were listed to confirm the `Size:` field was read correctly, and a separate + `binary-all` index probes **404**, so arch:all is already inside `binary-amd64` (301 of + main's 338) and is neither double-counted nor missed. **~46% of it is LibreOffice**, which + no control-plane node installs -- recorded because it matters if backports is ever mirrored + for another release or across many DCs. **A SUITE-COVERAGE ASSERTION IS OWED** on + `dc-mirror.sh check`: nothing compares the suites the mirror serves against the suites the + deployed image's `sources.list` requests, which is why it read PASS throughout. **>>> QUEUED BY OPERATOR DIRECTION 2026-07-31, NOT BUILT: a per-DC Tailscale subnet router is now a STANDING DC-STANDUP REQUIREMENT. <<<** Operator, verbatim: **"A closer real work analog would be to install tailscale in DC0, DC1, and any future DCs as they would be running diff --git a/docs/design-decisions.md b/docs/design-decisions.md index f70e144..850872e 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -5880,6 +5880,51 @@ convention, the staged class order, the measure-then-rule posture on charmhub/snaps, and the per-DC-strategy rule that a utility service owns its own net prerequisites. +### D-135 -- AMENDMENT (2026-07-31): the apt mirror carries `jammy-backports` + +**Status:** RULED 2026-07-31 (operator, GA-R5). **OPS under GA-R3** -- a scope correction to +an already-adopted mirror, doubt resolves DOWN, no new D-number. D-135's per-DC artifact +strategy (dc0 = full mirror, dc1 = caching proxy) is UNCHANGED. + +Question as presented: bundle deploy attempt 3 succeeded, then 22 of 33 units failed their +`install` hook on a single cause, MEASURED end to end -- +`E: The repository 'http://10.12.8.4/ubuntu jammy-backports Release' does not have a Release +file.` The mirror serves `jammy` / `jammy-security` / `jammy-updates` **200** and +`jammy-backports` **404**, which is exactly the "jammy triple" scope +`scripts/dc-mirror.sh:179` was built with, while every deployed node's +`/etc/apt/sources.list` carries `deb http://10.12.8.4/ubuntu jammy-backports main restricted +universe multiverse` -- because the Step-3.5 `apt-mirror` model-config rewrites the stock +Ubuntu sources to point EVERY suite at the DC mirror, including one the mirror was never +built to hold. Options put: (a) add `jammy-backports` to the mirror and re-sync; +(b) remove backports from the nodes' sources. + +**Operator answer, exact utterance: "Sync the backports into the mirror".** + +**Effect.** `--dist=` in the generated `-mirror-sync` gains `jammy-backports`, and the +mirror's declared scope becomes the jammy triple PLUS backports. **Fixed at SOURCE** +(`scripts/dc-mirror.sh`), not only on the live rack, so a reinstall or a future DC standup +cannot silently regenerate the old scope. `tests/dc-mirror` T10 asserted the old dist string +verbatim and is RE-POINTED to the new invariant rather than deleted -- the standing rule that +a test asserting a literal finding gets re-pointed, never removed to go green. + +**Cost, measured before the ruling was put:** 1,070,687,402 bytes (~1.00 GiB) across 461 +packages -- main 859 MB / 338 pkgs, universe 212 MB / 123 pkgs, restricted and multiverse +both EMPTY in backports. Against a 951 GB existing mirror with 1.8 T free that is ~0.1%. +Roughly 46% of it is LibreOffice, which no OpenStack control-plane node installs; recorded +because it matters if backports is ever mirrored for another release or across many DCs. + +**The generalisable defect, and it outlives this fix: `dc-mirror.sh check` verifies sync +STATUS, not SUITE COVERAGE.** It read PASS throughout, because nothing compares the suites +the mirror serves against the suites the deployed image's `sources.list` actually requests. +That is the same checker-cannot-see-the-real-failure class the mirror gate was already fixed +for once (2026-07-27, the `last-sync.status` existence check). A coverage assertion is OWED. + +**Roosevelt analog:** an artifact mirror's scope is a CONTRACT with the node image. Whenever +the image or its `sources.list` changes, the mirror's suite list is part of that change -- +and the gate should compare the two rather than trusting either. + +--- + ## D-136: NetBox-coupled per-DC render pipeline for bundle overlays and tfvars [ARCH] **Status:** **ADOPTED 2026-07-27 -- option (D)** (GA-R5). Question as presented: options (A)