===================================================================== Stage-5 grounding audit -- LIVE MEASUREMENT SWEEP Date: 2026-07-27 Branch: dc-dc-stage5-grounding-audit Run by: the audit session itself (not a subagent), read-only throughout. Charter: docs/audit/stage5-grounding-audit-scope-20260727.md ===================================================================== This file records MEASUREMENTS with their commands. Under GA-R1 amendment C2, a fresh measurement WINS over the document and corrects it in the capture's own commit. --------------------------------------------------------------------- 1. CONVERGENCE -- all three OpenTofu roots, ZERO DIFF --------------------------------------------------------------------- The standing expectation (CURRENT-STATE section 5) is ZERO DIFF, and ANY diff is a STOP. All three roots were planned this session. OUTER (on vcloud): $ tofu -chdir=opentofu plan -no-color -> "No changes. Your infrastructure matches the configuration." capture: docs/audit/outer-plan-20260727-stage5-audit.txt exit 0 INNER dc0 (ON voffice1 -- D-128 Plane 2, its tfstate lives there): $ ssh voffice1 'cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate \ && tofu plan -no-color' -> "No changes. Your infrastructure matches the configuration." exit 0 (tfvars present: d124-inner.auto.tfvars; 10 domains refreshed -- 9 nodes + vr1-dc0-opnsense) INNER dc1 (ON voffice1): $ ssh voffice1 'cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc1-substrate \ && tofu plan -no-color' -> "No changes. Your infrastructure matches the configuration." exit 0 VERDICT: the substrate is genuinely converged across all three roots. This is the strongest single as-built test available and it passes cleanly. VALIDITY NOTE (important): the voffice1 clone is 105 commits behind main (finding 5 below), so an inner plan run there uses that clone's config. It was verified FIRST that the inner roots did not change: $ git diff --stat 61c416e..main -- opentofu/vr1-dc0-substrate/ \ opentofu/vr1-dc1-substrate/ opentofu/modules/ -> EMPTY (no differences) So the inner plans are valid convergence evidence. Had that diff been non-empty, they would have been UNMEASURED, not green. --------------------------------------------------------------------- 2. GAUNTLET AND LINT --------------------------------------------------------------------- $ bash scripts/run-tests-all.sh -> "GAUNTLET: ALL GREEN (81 harnesses)" exit 0 $ bash scripts/repo-lint.sh -> 0 fail, 1 warn (L1 legacy non-ASCII carve-out in design-decisions.md) This matches the expected result. NOTE: during the sweep repo-lint reported "1 fail" -- L10 firing on this session's own UNCOMMITTED capture files under docs/audit/. That is the rule WORKING (GA-R1 C1: a commit touching a state-bearing surface must touch CURRENT-STATE.md in the same commit), not a defect. It is recorded here because a future session will see the same thing and should not treat it as a regression. --------------------------------------------------------------------- 3. PREFLIGHT -- red for EXACTLY the known reasons, nothing new --------------------------------------------------------------------- $ bash scripts/preflight.sh -> exit 1 capture: docs/audit/preflight-20260727-stage5-audit.txt P1 repo lint ............ FAIL (this session's uncommitted captures; see 2) P2 bundle invariants .... PASS (Pattern A / D-052-D-053) P3 channel assert ....... WARN (33 pins, 0 fail, 33 warn) ** AMENDED after lens 4 -- this line was WRONG to wave through. The COUNT was in the expected set; the CAUSE was not, and it is consequential. All 33 warns read "juju info failed ([Errno 2] No such file or directory: 'juju') -- verify manually". `channel_assert.py:41` folds a MISSING BINARY into the same soft verdict as an unreachable charmhub, and `:53` sets WARN whenever `fails` is empty. So ZERO of the 33 committed Caracal channel pins are currently verified against charmhub by any executable check -- on the eve of the deploy stage. The harness cannot see it: tests/preflight T7 simulates only charmhub-UNREACHABLE and assumes the binary exists. Ties directly to S-1 below: neither `juju` nor `openstack` is installed on the host preflight runs on. ** P4 live pre-flight CHECK 0 octavia-pki .. FAIL overlays/octavia-pki.yaml MISSING (gitignored secret; known, expected) CHECK 1 bundle VIPs .. PASS vip: line count = 11; triples OK=11 bad=0 MAAS reachability .... FAIL 'maas admin subnets read' from the jumphost (expected: the region is ON voffice1) P5 credential matrix .... FAIL 82 rows, 15 groups clean, 7 findings (matches the recorded residual 7 exactly) VERDICT: preflight was ALREADY red before this session and remains red for the same reasons. NOTHING NEW has joined the set. P5's 7 findings match the recorded residual exactly (dc0-edge-api x2, S5 x3, S6 conflation x1, E4 uncheckable x2). CONSEQUENTIAL SIDE-FINDING -- see finding 6 below: CHECK 1 PASSING is not the good news it looks like. --------------------------------------------------------------------- 4. PER-DC ARTIFACT PATHS -- both genuinely healthy --------------------------------------------------------------------- OWNED MEASUREMENT ERROR, recorded rather than quietly corrected: this session first ran $ bash scripts/dc-mirror.sh check dc0 $ bash scripts/dc-cache-proxy.sh check dc1 LOCALLY on vcloud. Both printed a comprehensive FAIL with every probe MISS ("apt-cacher-ng not installed", "nginx not active", utility address absent). That output is INDISTINGUISHABLE from "the service was never installed". It was wrong. Both scripts are designed to be PIPED TO THE RACK: ssh -i -J voffice1 @ 'sudo bash -s' -- check \ < scripts/dc-.sh (header, scripts/dc-cache-proxy.sh:13-15). Run locally they check vcloud, where of course nothing is installed. The scripts are NOT at fault. True exit codes were then measured (the first reading had captured `tail`'s status through a pipe, not the script's): $ bash scripts/dc-mirror.sh check dc0 >/dev/null 2>&1 ; echo $? -> 1 $ bash scripts/dc-cache-proxy.sh check dc1 >/dev/null 2>&1 ; echo $? -> 1 So they DO fail closed. The canonical piped invocation needs remote sudo, which is an `ask` rule and the operator is away -- so it is UNMEASURED (see section 6). The FUNCTIONAL question was measured instead, without sudo, from each rack host: dc0 rack (172.31.0.2, transit IP taken from scripts/lib-hosts.sh:127): hostname vvr1-dc0; up 1 week 6 h; nginx ACTIVE virbr2 10.12.8.2/22 10.12.8.3/22 10.12.8.4/22 ; virbr5 UP 10.12.4.2/22 $ curl http://10.12.8.4/ -> 200 $ curl http://10.12.8.4/ubuntu/dists/jammy/Release -> 200 $ cat /var/lib/dc-mirror/last-sync.status -> OK 2026-07-27T08:43:46Z ubuntu=0 uca=0 [attested current] dc1 rack (172.31.0.6, from scripts/lib-hosts.sh:150): hostname vvr1-dc1; up 4 days 14 h; apt-cacher-ng ACTIVE virbr4 UP 10.12.64.2/22 ; virbr6 10.12.68.2/22 10.12.68.3/22 10.12.68.4/22 $ curl --proxy http://10.12.68.4:3142 http://archive.ubuntu.com/...jammy/Release -> 200 $ curl --proxy http://10.12.68.4:3142 http://ubuntu-cloud.archive.canonical.com/ ubuntu/dists/jammy-updates/caracal/Release -> 200 VERDICT: dc0's full mirror and dc1's caching proxy are BOTH genuinely serving, and dc0's sync is attested current. CURRENT-STATE's claim holds. The D-135-amended per-DC split is intact (dc0 mirror / dc1 proxy). OBSERVATION, not a defect: on both racks the metal-admin bridge (virbr2 / virbr6) reads DOWN while carrying its addresses. That is expected -- all 18 node VMs are powered off in `Ready` under the READY-handoff ruling, so the bridge has no attached running domain. It is noted because it is exactly the kind of state that looks alarming to a fresh reader. --------------------------------------------------------------------- 5. OFFICE1 BASE LEG --------------------------------------------------------------------- $ bash scripts/site-baseleg.sh check office1 ok leg 10.10.0.10/24 on virbr2 ok route 10.10.1.0/24 via 10.10.0.20 exit 0 --------------------------------------------------------------------- 6. THE TWO CONSEQUENTIAL FINDINGS FROM THIS SWEEP --------------------------------------------------------------------- FINDING A -- THE OFFICE1 HEADEND CLONE IS 105 COMMITS STALE, AND STAGE 5 EXECUTES FROM IT. $ ssh voffice1 'cd ~/openstack-caracal-dc-dc && git rev-parse --abbrev-ref HEAD' -> dc-dc-g12-dc1-substrate $ ... git log -1 --format="%H %ci %s" -> 61c416e... 2026-07-22 21:38:06 +0000 "G12 dc1 step B COMPLETE: transit live, rack enrolled, SEC-010 both ends" $ ... git rev-list --left-right --count origin/main...HEAD -> 105 0 [105 BEHIND origin/main, 0 ahead] $ git diff --stat 61c416e..main | tail -1 -> 133 files changed, 19381 insertions(+), 432 deletions(-) D-128 fixes Plane 2 (MAAS / NetBox / inner tofu / tenant tests) to EXECUTE on voffice1, and Stage 5 is a Plane-2 activity: `juju bootstrap` and `juju deploy` run from the Office1 headend. The deploy artifacts there are therefore the PRE-RENDER ones. Measured: $ ssh voffice1 '... for f in ...; do [ -e $f ] && echo PRESENT || echo ABSENT' PRESENT bundle.yaml ABSENT overlays/vr1-dc1-vips.yaml ABSENT overlays/vr1-dc1-machines.yaml PRESENT overlays/dc-ha-scaleup.yaml PRESENT scripts/provider-bundle-check.py ABSENT scripts/creds-matrix.py ABSENT creds-matrix.tsv bundle.yaml sha256 (first 16): vcloud/main 7d6261c55d50f362 voffice1 40dc685eaccfa1d2 voffice1's machines block: machines: "8": constraints: arch=amd64 tags=openstack "9": constraints: arch=amd64 tags=openstack "10": constraints: arch=amd64 tags=openstack "11": constraints: arch=amd64 tags=openstack That is the VR0 4-node HYPERCONVERGED layout -- precisely the pre-render state the 2026-07-24 committee identified as the deploy blocker. The 9-node role-separated render (machines "0"-"8", role+DC tags, nova-compute 3->2, ceph-osd on 4 storage) exists ONLY on vcloud/main. Consequence if unaddressed: an operator following the phase-4 runbook on voffice1 today would either fail on the two absent overlay files or deploy the WRONG TOPOLOGY. Note also that both retired branches still exist as stale remote-tracking refs in that clone (it has never been pruned), while ground truth from vcloud is: $ git ls-remote --heads origin -> only refs/heads/main and refs/heads/dc-dc-stage5-grounding-audit so a naive `git pull` there operates on a branch that no longer exists upstream. FINDING B -- A GA-R5 RULING FROM 2026-07-25 WAS NEVER EXECUTED, AND THE CHECKER THAT WOULD HAVE CAUGHT IT PASSES ONLY BECAUSE IT WAS NOT. Ruling 3 of 2026-07-25 states that bundle.yaml becomes VIP-FREE (topology only), every per-DC value arrives via overlay including dc0's, and dc0's VIPs extract into `overlays/vr1-dc0-vips.yaml` in two commits. $ ls overlays/ dc-dc-ipv6-family-matrix.yaml dc-ha-scaleup.yaml vr1-dc1-machines.yaml vr1-dc1-vips.yaml -> overlays/vr1-dc0-vips.yaml is ABSENT $ grep -c 'vip:' bundle.yaml -> 11 So bundle.yaml still carries its 11 inline VIPs and the ruled extraction has not happened. This reframes BLOCKER-1: preflight P4 CHECK 1 reports "vip: line count = 11 ... OK=11 bad=0" and PASSES today ONLY BECAUSE the ruled change was never made. BLOCKER-1 is LATENT, not live -- it becomes live the moment ruling 3 is executed. A green CHECK 1 is therefore evidence that a ruling is outstanding, not evidence of health. This is the "a checker that cannot fail is not a gate" family seen from a new angle: here the checker CAN fail, but its current PASS encodes an un-executed ruling rather than a good state. --------------------------------------------------------------------- 7. UNMEASURED (could not look -- NOT "nothing there") --------------------------------------------------------------------- - The canonical piped `dc-mirror.sh check dc0` / `dc-cache-proxy.sh check dc1` invocation, which needs `ssh ... 'sudo bash -s'` (an `ask` rule; the operator is away). The functional substitute in section 4 covers the gate question -- the artifact source answering on its own address with an attested-current sync -- but does NOT cover the units/timers/persistence half that the real checker asserts. Re-run both on the operator's return. - Everything gated behind jumphost sudo, tofu apply, any maas write verb, juju bootstrap/deploy, and opnsense POST. None were attempted.