STAGE-5 PHASE 0 -- execution-environment preconditions: EXECUTED 2026-07-27
==========================================================================
Operator direction (GA-R5 not required -- OPS execution of the already-ruled
R10, plus readiness-doc Phase 0): "Merge to main, then start Phase 0", then
"Approve A, B, and C, and retire the audit branch".
Scope executed: readiness-doc preconditions 0.1, 0.2, 0.3
(docs/audit/stage5-readiness-20260727.md), plus retirement of the merged
audit branch. Host: voffice1 (the D-128 Plane-2 execution host).
Status authority for anything below is docs/CURRENT-STATE.md (GA-R1). This
file is a capture, not a status surface.
0. MERGE (prerequisite, done first)
-----------------------------------
merge commit 607813b 2 parents (NOT squashed), 33 commits from
dc-dc-stage5-grounding-audit
recorded by 6495cfb (a --no-ff merge cannot carry the doc edit;
same shape as Stage 4's 6f5701d / 1023596)
containment git branch --merged main -> listed, before deleting
ON main gauntlet ALL GREEN (81 harnesses); repo-lint 0 fail / 1 warn
branch RETIRED local + remote; git ls-remote --heads origin -> main only
NO STAGE OPENED OR CLOSED.
A. 0.1 -- ADVANCE THE voffice1 CLONE
------------------------------------
PRE (measured read-only, before touching anything):
HEAD 61c416e on dc-dc-g12-dc1-substrate (deleted upstream)
distance 105 0 vs its CACHED origin/main (itself pre-merge 71519a2)
dirty 1 untracked: opentofu/vr1-dc1-substrate/.terraform.lock.hcl
SAFETY CHECK RUN FIRST -- the inner tfstate for BOTH DCs lives INSIDE this
working tree (it is the state-of-record; CURRENT-STATE section 1 records the
location). A branch switch here is therefore not a routine checkout.
git check-ignore -> IGNORED, all of:
opentofu/vr1-dc0-substrate/terraform.tfstate
opentofu/vr1-dc0-substrate/terraform.tfstate.backup
opentofu/vr1-dc1-substrate/terraform.tfstate
opentofu/vr1-dc1-substrate/terraform.tfstate.backup
opentofu/vr1-dc{0,1}-substrate/d124-inner.auto.tfvars
git ls-tree origin/main -> tracks the SAME file set at those paths as the
outgoing branch, and does NOT track vr1-dc1-substrate/.terraform.lock.hcl,
so the one untracked-not-ignored file cannot collide.
=> the switch cannot touch the state-of-record. PROVEN, not assumed.
EXECUTED:
git fetch origin --prune
- [deleted] origin/dc-dc-g12-dc1-substrate
- [deleted] origin/dc-dc-stage3-phase2-dc-substrate
- [deleted] origin/dc-dc-stage5-grounding-audit
71519a2..6495cfb main -> origin/main
git switch main
(a plain `git pull` would NOT have worked -- the tracked branch is gone
upstream; recovery shape per finding L5-4)
POST -- ASSERTIONS:
HEAD 6495cfb
origin/main 6495cfb ASSERT PASS
tfstate sha256, dc0 2e140b74...894608c3 IDENTICAL to PRE
tfstate sha256, dc1 aa91f103...dc671645 IDENTICAL to PRE
overlays/vr1-dc1-vips.yaml NOW PRESENT (was ABSENT)
overlays/vr1-dc1-machines.yaml NOW PRESENT (was ABSENT)
bundle.yaml machines block NOW the VR1 9-node role-separated layout
(was VR0 4-node hyperconverged "8"-"11")
dirty 1 untracked (unchanged, as predicted)
=> readiness blocker 1 ("you would deploy the wrong topology") CLEARED.
B. 0.3 -- PRUNE STALE REFS
--------------------------
Remote-tracking refs: pruned by the fetch above (3 deleted).
Local branches: voffice1 carried TWO stale locals, not the one the record
named. Containment proven before deletion (`git rev-list --count
origin/main..<branch>` -> 0 for both; `-d` used, which refuses if unmerged).
Deleted dc-dc-g12-dc1-substrate (was 61c416e)
Deleted dc-dc-stage3-phase2-dc-substrate (was 57836b1)
FINAL: local `main` only; remotes/origin/main only.
Ordering note: the audit branch was retired on origin BEFORE this fetch, so
advancing the clone could not hand it a FRESH stale ref -- the exact condition
that produced the 105-commit problem.
C. 0.2 -- THE openstack CLIENT
------------------------------
MEASURED BEFORE PRESENTING (the options were not assumed):
snap `openstackclients` NO Caracal channel. Newest stable = zed (2023-03).
latest/stable = xena (2021). REFUTED for this
deployment. Corroborating in-repo prior art:
docs/design-decisions.md:638 records the snap's
home-only confinement (paths under /tmp unreadable).
apt noble/main python3-openstackclient 6.6.0-0ubuntu2 <-- MATCH
6.6.0 IS the Caracal 2024.1 client (verified
upstream, not from memory: Launchpad
python-openstackclient 6.6.0-0ubuntu1, packaged
for the 2024.1 coordinated release).
archive reachability http://archive.ubuntu.com/ubuntu/dists/noble/Release -> 200
simulated install 0 upgraded, 68 newly installed, 0 to remove
UCA not configured on this host (not needed -- noble's
native OpenStack release IS Caracal 2024.1)
EXECUTED: sudo apt-get install -y python3-openstackclient
POST -- VERIFIED BEHAVIOURALLY, not just by presence:
command -v openstack /usr/bin/openstack
openstack --version openstack 6.6.0
openstack --help exit 0
openstack server list "Missing value auth-url required for auth plugin
password" <- fails CLEANLY on absent config,
which is correct (no cloud configured yet); it
does not crash.
dpkg versions (the pin):
python3-openstackclient 6.6.0-0ubuntu2
python3-openstacksdk 3.0.0-0ubuntu2
python3-novaclient 2:18.5.0-0ubuntu1
=> readiness blocker 2 CLEARED. Pin recorded in CURRENT-STATE section 7.
D. THE PAYOFF RUNS (deliberately deferred until the clone was current)
----------------------------------------------------------------------
CURRENT-STATE recorded these as OWED; running them against a 105-commit-stale
tree would have measured nothing. All three run here for the first time.
D.1 repo-lint ON voffice1 0 fail / 1 warn (the standing legacy
D-001..018 non-ASCII carve-out). Matches
vcloud exactly.
D.2 gauntlet ON voffice1 *** 2/81 FAILED *** opentofu-validate,
site-headend-install
-- on the IDENTICAL commit that reports
ALL GREEN (81) on vcloud.
FINDING P0-1: "GAUNTLET ALL GREEN" IS A HOST-DEPENDENT CLAIM.
Both failures are host-portability defects, NOT tree defects. Every
GA-R6 stage close in this project's history has cited a gauntlet figure
measured on vcloud only.
(a) opentofu-validate -- every sub-check PASSES (root, 12 modules, both
extra roots) and the aggregate still reports FAIL. Cause:
`tofu fmt -check -recursive` walks the FILESYSTEM, not the git tree,
and trips on alignment drift in
opentofu/vr1-dc0-substrate/d124-inner.auto.tfvars
which is GITIGNORED and exists ONLY on voffice1 (it carries that
host's real deploy inputs). vcloud cannot see this file, so vcloud
can never fail on it -- and equally, vcloud's green can never
attest the fmt state of the inputs the deploy actually uses.
Verbatim drift: `opnsense_base_path = ` wants two more spaces to
align with `vvr1_dc0_ssh_keyfile`.
(b) site-headend-install -- tests/site-headend-install/run-tests.sh:36
if snap list lxd >/dev/null 2>&1 || snap list maas >/dev/null 2>&1; then
bad "--dry-run appears to have INSTALLED a snap on the test host"
The assertion takes an UNCONDITIONAL snapshot of the host and never
compares it to a pre-state. On voffice1 -- which IS the headend, where
lxd and maas are installed BY DESIGN -- it fires as a false positive.
It works only on a host that is not a headend. Same defect class the
R15 work names (an assertion that does not measure what it claims),
in its false-RED direction.
NEITHER is fixed here (hard rule 1 -- Phase 0's ruled scope is 0.1/0.2/0.3).
Both are logged for a ruling/OPS decision.
D.3 preflight ON voffice1 TRUE EXIT 1 ("PREFLIGHT: FAIL -- do NOT deploy")
(exit code read from the script itself, NOT through a pipe -- the 07-27
audit owned exactly that error by reading `tail`'s status instead.)
R10's RULED CONSEQUENCE IS DELIVERED, both halves:
P3 ALL 33 charm-channel pins now VERIFY [ok] -- 2024.1/stable,
squid/stable, 2.4/stable resolved against charmhub. Previously
ZERO of 33 were verified by any executable check, because `juju`
is absent on vcloud. This is the first executable confirmation of
the committed Caracal pins.
P4 "MAAS reachable (profile=admin)" PASS. The former "MAAS
unreachable" was the missing `maas` binary on vcloud, as ruled.
P4 residual FAILs -- and ONE OF THESE WAS INITIALLY MIS-FILED BY THIS
CAPTURE AS "already known". Corrected here by measurement:
- MISSING overlays/octavia-pki.yaml (deliberately absent; host-independent).
Genuinely known.
- metal-internal VID=0 (want 103) -> readiness item 3.7: pre-flight-checks.sh
is VR0-FROZEN and its VID-103 / br-internal assertions CONTRADICT D-133,
so they can never pass on either VR1 DC. Genuinely known.
- metal-admin gateway=none (want 10.12.8.1) -> **NOT item 3.7, and NOT
previously recorded anywhere.** Item 3.7 is specifically the VID-103 /
br-internal assertions; this is a separate expectation. See P0-5.
FINDING P0-5: lib-net DECLARES A metal-admin GATEWAY THAT HAS NEVER EXISTED.
scripts/lib-net.sh:37 (the DEFAULT / VR0-dc0 arm):
declare -A PLANE_GW=( ["10.12.4.0/22"]="10.12.4.1"
["10.12.8.0/22"]="10.12.8.1" )
so pre-flight-checks.sh CHECK 3 hard-FAILs metal-admin for having no
gateway. MEASURED from the dc0 rack (ssh -J voffice1, dc0 svc key):
ping 10.12.8.1 -> 2 transmitted, 0 received, 100% loss
ip neigh -> 10.12.8.1 dev virbr2 INCOMPLETE (ARP never
resolved -- nothing HOLDS the address; this is
not a device that is merely down)
rack legs on the plane: 10.12.8.2 (rack), .3 (D-131 DNS forwarder),
.4 (D-134 utility / mirror)
CONTROL: ping 10.12.4.1 (the edge, provider gw) -> 0% loss
The control proves the instrument. So MAAS is RIGHT to carry no gateway
on metal-admin and the CHECKER's expectation is the defect -- a VR0
inheritance. Consistent with the record of the D-134 carve, which ruled
`.1` gateways for the two PROVIDER subnets only; the 8 plane subnets were
created without them. Nothing routes off metal-admin (nodes reach the
mirror .4 and DNS .3 on-link), so no gateway is wanted.
Adjacent to item 3.7 -- same script, same VR0-frozen class -- but a
DISTINCT assertion that 3.7's remediation would not touch. Logged, not
fixed (hard rule 1). NOTE the dc1 arm (lib-net.sh:149) carries the same
shape at 10.12.68.1 and will fail identically.
FINDING P0-2: P5 IS HOST-BLIND, AND PREFLIGHT HAS NO CORRECT HOST.
P5 on voffice1 -> 34 findings
P5 on vcloud -> 7 findings (the known residual set)
SAME matrix (82 rows), SAME commit.
Cause, measured:
vcloud has ~/vr1-dc0-creds, ~/vr1-dc1-creds, ~/vr1-office1-creds
voffice1 has ~/vr1-dc0-creds, ~/vr1-dc1-creds and NO vr1-office1-creds
(its two ARE the SEC-022 shadow stores, not the jumphost set)
The matrix's `jumphost` role carries no assertion about WHICH host is the
jumphost, so every `jumphost/*` location silently re-points to whatever
host runs the checker. On voffice1 that manufactures ~27 spurious
EXPECTED-BUT-ABSENT findings against the headend's shadow stores.
This is the committee's "tier 2 is SITE-BLIND" defect one level up:
HOST-blind. Note the direction -- here it produces false RED, but the
same hole would produce false GREEN for any row whose credential happens
to exist on the wrong host under a matching name.
One finding in the voffice1 run is REAL and worth keeping:
E3 UNDECLARED: 'maas-virsh_ed25519' present at jumphost/vr1-dc0 but in
NO matrix row for that site
-- i.e. an undeclared file in the headend shadow store, the SEC-022 class.
CONSEQUENCE FOR R10: the ruling ("fix the stale commit issue and make sure
all working directories are current") is EXECUTED and its predicted
P3/P4 clearance is CONFIRMED. But the measurement also shows the reds are
not merely an artifact of the wrong host -- vcloud has the credentials and
not the clients; voffice1 has the clients and not the credentials. There
is NO single host on which preflight is currently correct. Not ruled here.
E. FINDINGS LOGGED, NOT ACTIONED (hard rule 1)
----------------------------------------------
P0-1 gauntlet is host-dependent; two harnesses encode "I am not the
headend" assumptions (detail in D.2). The stage-close citation
"gauntlet ALL GREEN (N)" should state the HOST it was measured on.
P0-2 creds-matrix `jumphost` role is host-blind; preflight has no
correct host (detail in D.3).
P0-3 opentofu/vr1-dc1-substrate/.terraform.lock.hcl is untracked on main
while its dc0 twin IS tracked -- an asymmetry, harmless today, but it
is why the dc1 root shows a permanent untracked file on every host.
P0-4 No logged window (scripts/run-logged.sh) was opened for this session;
it was offered and the operator approved the steps directly. Recorded
for the as-executed trail rather than left silent.
P0-5 lib-net declares a metal-admin gateway (.8.1 / dc1 .68.1) that no
device holds -- measured 100% loss + INCOMPLETE ARP against a working
control (detail in D.3). Both DC arms affected.
QUESTION QUEUED FOR THE OPERATOR (not a finding -- an interlock the R15
rulings do not currently reach; one GA-R5 exchange, no D-number self-assigned,
GA-R3 doubt resolves DOWN to OPS unless the A1 Roosevelt test is met):
WHICH HOST IS AUTHORITATIVE FOR THE GATES?
R10 ruled preflight belongs on voffice1; R15(3) rules preflight must stop
failing open. Executed together as ruled, and given P0-2, preflight on
voffice1 becomes PERMANENTLY UNPASSABLE -- its 34 findings are host
artifacts, not defects. Likewise R15(2) pins a harness MANIFEST, which
catches renames but does nothing about P0-1: `tofu fmt -check -recursive`
walks the filesystem, so implementing R15(2) exactly as ruled still leaves
voffice1 red. So the R15 execution scope needs to answer, first, which host
each gate is authoritative on -- and for the credential matrix, whether
`jumphost` is a ROLE that must be pinned to a named host.
STANDING CONSEQUENCE either way, worth stating plainly: STAGE 5 EXECUTES
FROM voffice1, and the gauntlet cannot be green there until P0-1 is
resolved. Every future GA-R6 close citing a gauntlet figure measured there
inherits a known red, so the citation must name its host.
Captured 2026-07-27 on branch dc-dc-stage5-preconditions.
