=====================================================================
Stage-5 grounding audit -- LIVE MEASUREMENT SWEEP
Date: 2026-07-27   Branch: dc-dc-stage5-grounding-audit
Run by: the audit session itself (not a subagent), read-only throughout.
Charter: docs/audit/stage5-grounding-audit-scope-20260727.md
=====================================================================

This file records MEASUREMENTS with their commands. Under GA-R1 amendment
C2, a fresh measurement WINS over the document and corrects it in the
capture's own commit.

---------------------------------------------------------------------
1. CONVERGENCE -- all three OpenTofu roots, ZERO DIFF
---------------------------------------------------------------------
The standing expectation (CURRENT-STATE section 5) is ZERO DIFF, and ANY
diff is a STOP. All three roots were planned this session.

OUTER (on vcloud):
  $ tofu -chdir=opentofu plan -no-color
  -> "No changes. Your infrastructure matches the configuration."
  capture: docs/audit/outer-plan-20260727-stage5-audit.txt   exit 0

INNER dc0 (ON voffice1 -- D-128 Plane 2, its tfstate lives there):
  $ ssh voffice1 'cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate \
      && tofu plan -no-color'
  -> "No changes. Your infrastructure matches the configuration."   exit 0
  (tfvars present: d124-inner.auto.tfvars; 10 domains refreshed --
   9 nodes + vr1-dc0-opnsense)

INNER dc1 (ON voffice1):
  $ ssh voffice1 'cd ~/openstack-caracal-dc-dc/opentofu/vr1-dc1-substrate \
      && tofu plan -no-color'
  -> "No changes. Your infrastructure matches the configuration."   exit 0

VERDICT: the substrate is genuinely converged across all three roots. This
is the strongest single as-built test available and it passes cleanly.

VALIDITY NOTE (important): the voffice1 clone is 105 commits behind main
(finding 5 below), so an inner plan run there uses that clone's config. It
was verified FIRST that the inner roots did not change:
  $ git diff --stat 61c416e..main -- opentofu/vr1-dc0-substrate/ \
      opentofu/vr1-dc1-substrate/ opentofu/modules/
  -> EMPTY (no differences)
So the inner plans are valid convergence evidence. Had that diff been
non-empty, they would have been UNMEASURED, not green.

---------------------------------------------------------------------
2. GAUNTLET AND LINT
---------------------------------------------------------------------
  $ bash scripts/run-tests-all.sh
  -> "GAUNTLET: ALL GREEN (81 harnesses)"   exit 0

  $ bash scripts/repo-lint.sh
  -> 0 fail, 1 warn  (L1 legacy non-ASCII carve-out in design-decisions.md)
  This matches the expected result. NOTE: during the sweep repo-lint
  reported "1 fail" -- L10 firing on this session's own UNCOMMITTED capture
  files under docs/audit/. That is the rule WORKING (GA-R1 C1: a commit
  touching a state-bearing surface must touch CURRENT-STATE.md in the same
  commit), not a defect. It is recorded here because a future session will
  see the same thing and should not treat it as a regression.

---------------------------------------------------------------------
3. PREFLIGHT -- red for EXACTLY the known reasons, nothing new
---------------------------------------------------------------------
  $ bash scripts/preflight.sh          -> exit 1
  capture: docs/audit/preflight-20260727-stage5-audit.txt

  P1 repo lint ............ FAIL (this session's uncommitted captures; see 2)
  P2 bundle invariants .... PASS  (Pattern A / D-052-D-053)
  P3 channel assert ....... WARN  (33 pins, 0 fail, 33 warn)
      ** AMENDED after lens 4 -- this line was WRONG to wave through. The COUNT
      was in the expected set; the CAUSE was not, and it is consequential.
      All 33 warns read "juju info failed ([Errno 2] No such file or directory:
      'juju') -- verify <channel> manually". `channel_assert.py:41` folds a
      MISSING BINARY into the same soft verdict as an unreachable charmhub, and
      `:53` sets WARN whenever `fails` is empty. So ZERO of the 33 committed
      Caracal channel pins are currently verified against charmhub by any
      executable check -- on the eve of the deploy stage. The harness cannot see
      it: tests/preflight T7 simulates only charmhub-UNREACHABLE and assumes the
      binary exists. Ties directly to S-1 below: neither `juju` nor `openstack`
      is installed on the host preflight runs on. **
  P4 live pre-flight
     CHECK 0 octavia-pki .. FAIL  overlays/octavia-pki.yaml MISSING
                                  (gitignored secret; known, expected)
     CHECK 1 bundle VIPs .. PASS  vip: line count = 11; triples OK=11 bad=0
     MAAS reachability .... FAIL  'maas admin subnets read' from the jumphost
                                  (expected: the region is ON voffice1)
  P5 credential matrix .... FAIL  82 rows, 15 groups clean, 7 findings
                                  (matches the recorded residual 7 exactly)

VERDICT: preflight was ALREADY red before this session and remains red for
the same reasons. NOTHING NEW has joined the set. P5's 7 findings match the
recorded residual exactly (dc0-edge-api x2, S5 x3, S6 conflation x1,
E4 uncheckable x2).

CONSEQUENTIAL SIDE-FINDING -- see finding 6 below: CHECK 1 PASSING is not
the good news it looks like.

---------------------------------------------------------------------
4. PER-DC ARTIFACT PATHS -- both genuinely healthy
---------------------------------------------------------------------
OWNED MEASUREMENT ERROR, recorded rather than quietly corrected: this
session first ran
  $ bash scripts/dc-mirror.sh check dc0
  $ bash scripts/dc-cache-proxy.sh check dc1
LOCALLY on vcloud. Both printed a comprehensive FAIL with every probe MISS
("apt-cacher-ng not installed", "nginx not active", utility address absent).
That output is INDISTINGUISHABLE from "the service was never installed".
It was wrong. Both scripts are designed to be PIPED TO THE RACK:
  ssh -i <dc-key> -J voffice1 <user>@<rack-ip> 'sudo bash -s' -- check <site> \
      < scripts/dc-<name>.sh
(header, scripts/dc-cache-proxy.sh:13-15). Run locally they check vcloud,
where of course nothing is installed. The scripts are NOT at fault.

True exit codes were then measured (the first reading had captured `tail`'s
status through a pipe, not the script's):
  $ bash scripts/dc-mirror.sh check dc0 >/dev/null 2>&1 ; echo $?      -> 1
  $ bash scripts/dc-cache-proxy.sh check dc1 >/dev/null 2>&1 ; echo $? -> 1
So they DO fail closed.

The canonical piped invocation needs remote sudo, which is an `ask` rule and
the operator is away -- so it is UNMEASURED (see section 6). The FUNCTIONAL
question was measured instead, without sudo, from each rack host:

dc0 rack (172.31.0.2, transit IP taken from scripts/lib-hosts.sh:127):
  hostname vvr1-dc0; up 1 week 6 h; nginx ACTIVE
  virbr2 10.12.8.2/22 10.12.8.3/22 10.12.8.4/22 ; virbr5 UP 10.12.4.2/22
  $ curl http://10.12.8.4/                        -> 200
  $ curl http://10.12.8.4/ubuntu/dists/jammy/Release -> 200
  $ cat /var/lib/dc-mirror/last-sync.status
    -> OK 2026-07-27T08:43:46Z ubuntu=0 uca=0     [attested current]

dc1 rack (172.31.0.6, from scripts/lib-hosts.sh:150):
  hostname vvr1-dc1; up 4 days 14 h; apt-cacher-ng ACTIVE
  virbr4 UP 10.12.64.2/22 ; virbr6 10.12.68.2/22 10.12.68.3/22 10.12.68.4/22
  $ curl --proxy http://10.12.68.4:3142 http://archive.ubuntu.com/...jammy/Release
    -> 200
  $ curl --proxy http://10.12.68.4:3142 http://ubuntu-cloud.archive.canonical.com/
         ubuntu/dists/jammy-updates/caracal/Release -> 200

VERDICT: dc0's full mirror and dc1's caching proxy are BOTH genuinely
serving, and dc0's sync is attested current. CURRENT-STATE's claim holds.
The D-135-amended per-DC split is intact (dc0 mirror / dc1 proxy).

OBSERVATION, not a defect: on both racks the metal-admin bridge (virbr2 /
virbr6) reads DOWN while carrying its addresses. That is expected -- all 18
node VMs are powered off in `Ready` under the READY-handoff ruling, so the
bridge has no attached running domain. It is noted because it is exactly the
kind of state that looks alarming to a fresh reader.

---------------------------------------------------------------------
5. OFFICE1 BASE LEG
---------------------------------------------------------------------
  $ bash scripts/site-baseleg.sh check office1
  ok   leg   10.10.0.10/24 on virbr2
  ok   route 10.10.1.0/24 via 10.10.0.20            exit 0

---------------------------------------------------------------------
6. THE TWO CONSEQUENTIAL FINDINGS FROM THIS SWEEP
---------------------------------------------------------------------
FINDING A -- THE OFFICE1 HEADEND CLONE IS 105 COMMITS STALE, AND STAGE 5
EXECUTES FROM IT.

  $ ssh voffice1 'cd ~/openstack-caracal-dc-dc && git rev-parse --abbrev-ref HEAD'
  -> dc-dc-g12-dc1-substrate
  $ ... git log -1 --format="%H %ci %s"
  -> 61c416e... 2026-07-22 21:38:06 +0000
     "G12 dc1 step B COMPLETE: transit live, rack enrolled, SEC-010 both ends"
  $ ... git rev-list --left-right --count origin/main...HEAD
  -> 105   0            [105 BEHIND origin/main, 0 ahead]
  $ git diff --stat 61c416e..main | tail -1
  -> 133 files changed, 19381 insertions(+), 432 deletions(-)

D-128 fixes Plane 2 (MAAS / NetBox / inner tofu / tenant tests) to EXECUTE
on voffice1, and Stage 5 is a Plane-2 activity: `juju bootstrap` and
`juju deploy` run from the Office1 headend. The deploy artifacts there are
therefore the PRE-RENDER ones. Measured:

  $ ssh voffice1 '... for f in ...; do [ -e $f ] && echo PRESENT || echo ABSENT'
    PRESENT bundle.yaml
    ABSENT  overlays/vr1-dc1-vips.yaml
    ABSENT  overlays/vr1-dc1-machines.yaml
    PRESENT overlays/dc-ha-scaleup.yaml
    PRESENT scripts/provider-bundle-check.py
    ABSENT  scripts/creds-matrix.py
    ABSENT  creds-matrix.tsv

  bundle.yaml sha256 (first 16): vcloud/main 7d6261c55d50f362
                                 voffice1     40dc685eaccfa1d2
  voffice1's machines block:
    machines:
      "8":  constraints: arch=amd64 tags=openstack
      "9":  constraints: arch=amd64 tags=openstack
      "10": constraints: arch=amd64 tags=openstack
      "11": constraints: arch=amd64 tags=openstack

That is the VR0 4-node HYPERCONVERGED layout -- precisely the pre-render
state the 2026-07-24 committee identified as the deploy blocker. The
9-node role-separated render (machines "0"-"8", role+DC tags, nova-compute
3->2, ceph-osd on 4 storage) exists ONLY on vcloud/main.

Consequence if unaddressed: an operator following the phase-4 runbook on
voffice1 today would either fail on the two absent overlay files or deploy
the WRONG TOPOLOGY. Note also that both retired branches still exist as
stale remote-tracking refs in that clone (it has never been pruned), while
ground truth from vcloud is:
  $ git ls-remote --heads origin
  -> only refs/heads/main and refs/heads/dc-dc-stage5-grounding-audit
so a naive `git pull` there operates on a branch that no longer exists
upstream.

FINDING B -- A GA-R5 RULING FROM 2026-07-25 WAS NEVER EXECUTED, AND THE
CHECKER THAT WOULD HAVE CAUGHT IT PASSES ONLY BECAUSE IT WAS NOT.

Ruling 3 of 2026-07-25 states that bundle.yaml becomes VIP-FREE (topology
only), every per-DC value arrives via overlay including dc0's, and dc0's
VIPs extract into `overlays/vr1-dc0-vips.yaml` in two commits.

  $ ls overlays/
    dc-dc-ipv6-family-matrix.yaml
    dc-ha-scaleup.yaml
    vr1-dc1-machines.yaml
    vr1-dc1-vips.yaml
  -> overlays/vr1-dc0-vips.yaml is ABSENT
  $ grep -c 'vip:' bundle.yaml    -> 11

So bundle.yaml still carries its 11 inline VIPs and the ruled extraction has
not happened. This reframes BLOCKER-1: preflight P4 CHECK 1 reports
"vip: line count = 11 ... OK=11 bad=0" and PASSES today ONLY BECAUSE the
ruled change was never made. BLOCKER-1 is LATENT, not live -- it becomes
live the moment ruling 3 is executed. A green CHECK 1 is therefore evidence
that a ruling is outstanding, not evidence of health.

This is the "a checker that cannot fail is not a gate" family seen from a
new angle: here the checker CAN fail, but its current PASS encodes an
un-executed ruling rather than a good state.

---------------------------------------------------------------------
7. UNMEASURED (could not look -- NOT "nothing there")
---------------------------------------------------------------------
- The canonical piped `dc-mirror.sh check dc0` / `dc-cache-proxy.sh check
  dc1` invocation, which needs `ssh ... 'sudo bash -s'` (an `ask` rule; the
  operator is away). The functional substitute in section 4 covers the gate
  question -- the artifact source answering on its own address with an
  attested-current sync -- but does NOT cover the units/timers/persistence
  half that the real checker asserts. Re-run both on the operator's return.
- Everything gated behind jumphost sudo, tofu apply, any maas write verb,
  juju bootstrap/deploy, and opnsense POST. None were attempted.
