diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 83d1204..4192f6d 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -898,6 +898,55 @@ 81 on record (R15(2) pins a manifest, not built), and repo-lint emits NO files-scanned figure at all -- R15(1)'s finding, observed again here rather than assumed. Execution of every ruling remains LOGGED-NOT-EXECUTED; what changed is only where the record lives. +- **STAGE-5 PHASE 0 (execution environment) EXECUTED 2026-07-27**, operator-approved step by + step ("Approve A, B, and C, and retire the audit branch"). Readiness-doc preconditions + 0.1/0.2/0.3. Capture `docs/audit/stage5-phase0-20260727.txt`. **NO STAGE OPENED** -- "start + Phase 0" authorises precondition work, not a stage transition. + **0.1 the voffice1 clone is CURRENT**: `HEAD == origin/main == 6495cfb`, asserted. A plain + `pull` could not have worked (tracked branch deleted upstream); `fetch --prune` + `switch main` + per L5-4. **The safety question was the real one and was PROVEN before the switch, not + assumed:** both DCs' inner tfstate -- the substrate's state-of-record -- lives INSIDE that + working tree, and every state artifact was confirmed `git check-ignore`-IGNORED with + `origin/main` tracking an identical file set at those paths. sha256 of both tfstates is + BYTE-IDENTICAL before and after. Consequence: the two dc1 overlays are now present and + `bundle.yaml` is the 9-node role-separated layout, so the "you would deploy the wrong + topology" blocker is CLEARED. **0.3** three stale remote-tracking refs pruned and TWO stale + local branches deleted (the record named one), containment proven first; the audit branch was + retired on origin BEFORE the fetch so the clone could not be handed a fresh stale ref. + **0.2** the `openstack` client is installed on voffice1 -- see the section 7 pin row. + **THE DEFERRED PAYOFF RUNS ARE NOW DONE, and two produced NEW findings:** + **(i) `repo-lint` on voffice1 matches vcloud** (0 fail / 1 standing warn). + **(ii) THE GAUNTLET IS HOST-DEPENDENT -- 2/81 FAILED on voffice1, on the IDENTICAL commit + that reports ALL GREEN (81) on vcloud.** Neither failure is a tree defect. `opentofu-validate` + passes every sub-check (root, 12 modules, both extra roots) and STILL reports FAIL, because + `tofu fmt -check -recursive` walks the FILESYSTEM not the git tree and trips on alignment + drift in `opentofu/vr1-dc0-substrate/d124-inner.auto.tfvars` -- a GITIGNORED file that exists + only on voffice1 and carries that host's real deploy inputs, so vcloud can neither fail on it + nor attest it. `site-headend-install` fails at `tests/site-headend-install/run-tests.sh:36`, + which asserts a `--dry-run` installed no snap by taking an UNCONDITIONAL snapshot of the host + and never comparing it to a pre-state -- on the actual headend, where lxd and maas are + installed by design, it is a false positive. **Every GA-R6 stage close in this project has + cited a gauntlet figure measured on vcloud only; that citation should name its host.** Both + logged, NOT fixed (hard rule 1 -- outside Phase 0's ruled scope). + **(iii) preflight: R10's ruled consequence is CONFIRMED, and a new gap is measured.** True + exit 1, read from the script rather than through a pipe. **P3 now verifies ALL 33 + charm-channel pins** (`2024.1/stable`, `squid/stable`, `2.4/stable`) -- the first executable + confirmation of the committed Caracal pins, where previously ZERO of 33 were checked; and + **P4 reports "MAAS reachable"**. Both were missing-binary artifacts on vcloud exactly as + ruled. P4's residual FAILs are the known set (the deliberately-absent octavia-pki overlay; + the VID-103/gateway assertions that readiness item 3.7 records as VR0-frozen and + D-133-contradicting, so they can never pass on a VR1 DC). **BUT P5 INVERTS: 34 findings on + voffice1 against 7 on vcloud, same 82-row matrix, same commit.** Measured cause: the matrix's + `jumphost` role carries NO assertion about which host is the jumphost, so every `jumphost/*` + location silently re-points to whatever host runs the checker -- and voffice1 has + `vr1-dc0-creds`/`vr1-dc1-creds` (which are the SEC-022 SHADOW stores) but no + `vr1-office1-creds`. This is the committee's "tier 2 is SITE-BLIND" defect one level up: + HOST-blind. It surfaces here as false RED, but the same hole yields false GREEN for any row + whose credential exists on the wrong host under a matching name. One finding in that run is + REAL: `E3 UNDECLARED 'maas-virsh_ed25519'` in the headend shadow store (SEC-022 class). + **So R10 is EXECUTED and its prediction held, but the fuller answer is that there is NO single + host on which preflight is currently correct** -- vcloud has the credentials and not the + clients, voffice1 the reverse. Logged, not ruled. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B @@ -1381,7 +1430,8 @@ | OPNsense edge | 26.7.1 (FreeBSD base 15.1) | MEASURED 2026-07-23 via the gated API (`GET core/firmware/status` -> product_version 26.7.1, capture `docs/audit/g13-close-20260723.txt`); updated 26.7 -> 26.7.1 in the G13 bundle. DC edges (vr1-dc0/dc1) remain 26.7 | office1-opnsense | | NetBox (Office1 apex) | 4.6.4 per as-built `docs/vr1-office1-as-built.md:44`; service UP verified (HTTP 302) this session | `ssh office1-netbox 'curl ... localhost:8000'` | office1-netbox | | Juju | **3.6.27 (rev 35621, `3/stable`) MEASURED 2026-07-27 on voffice1** -- the headend is the D-128 Plane-2 execution host and this is the client that will bootstrap the controller. Supersedes the 3.6.25 figure recorded 2026-07-24 (also at line 162, kept there as history): an in-channel patch refresh, which D-071 ADOPTED 2026-07-21 explicitly permits (patch-only jumps, in-channel-only refreshes), so this is policy-compliant drift and NOT an incident. The jumphost has NO juju client (measured ABSENT). | `ssh voffice1 'snap list'` (capture `docs/audit/stage5-live-measurement-20260727.txt`) | voffice1 | -| OpenStack client | **ABSENT ON BOTH HOSTS, MEASURED 2026-07-27** -- `command -v openstack` returns nothing on vcloud AND on voffice1, and voffice1's snap list carries only core24/juju/lxd/maas/postgresql/snapd. Ten Stage-5/6/7 scripts invoke it (`phase-03-admin-openrc.sh`, `phase-04-network-{create,verify}.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-{amphora-pipeline,octavia-verify}.sh`, `phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh`), so every post-deploy step from Stage 5 Step 7 onward has no client to run. Recorded here because it is a measured Stage-5 precondition that no prior surface carried. | `command -v openstack`; `ssh voffice1 'command -v openstack; snap list'` | vcloud + voffice1 | +| OpenStack client | **6.6.0 (`python3-openstackclient 6.6.0-0ubuntu2`, noble/main) INSTALLED ON voffice1 2026-07-27** -- Stage-5 Phase 0 precondition 0.2, operator-approved. voffice1 is the D-128 Plane-2 host every Stage-5+ script runs from. Verified behaviourally, not by presence: `openstack --version` -> `openstack 6.6.0`, `--help` exit 0, and `server list` fails CLEANLY on absent auth config rather than crashing. Companion pins: `python3-openstacksdk 3.0.0-0ubuntu2`, `python3-novaclient 2:18.5.0-0ubuntu1`. **The snap was REFUTED by measurement, not preference:** `openstackclients` has NO Caracal channel (newest stable `zed`, 2023-03; `latest/stable` is `xena`, 2021), and `docs/design-decisions.md:638` already records its home-only confinement trap. 6.6.0 is the Caracal 2024.1 client, verified upstream rather than from memory. noble's native OpenStack release IS Caracal 2024.1, so no UCA is needed on this host. **STILL ABSENT ON vcloud** -- deliberately: D-128 puts this work on the headend. Capture `docs/audit/stage5-phase0-20260727.txt`. | `ssh voffice1 'openstack --version; dpkg-query -W ...'` | voffice1 | +| OpenStack client (prior state, history) | **ABSENT ON BOTH HOSTS, MEASURED 2026-07-27** -- `command -v openstack` returns nothing on vcloud AND on voffice1, and voffice1's snap list carries only core24/juju/lxd/maas/postgresql/snapd. Ten Stage-5/6/7 scripts invoke it (`phase-03-admin-openrc.sh`, `phase-04-network-{create,verify}.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-{amphora-pipeline,octavia-verify}.sh`, `phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh`), so every post-deploy step from Stage 5 Step 7 onward has no client to run. Recorded here because it is a measured Stage-5 precondition that no prior surface carried. | `command -v openstack`; `ssh voffice1 'command -v openstack; snap list'` | vcloud + voffice1 | The known-stale pin sites this table used to enumerate (the GA-F03/F04/ F05 tofu, OPNsense, and jumphost-name values -- stated token-free here so diff --git a/docs/audit/stage5-phase0-20260727.txt b/docs/audit/stage5-phase0-20260727.txt new file mode 100644 index 0000000..8ab9ac2 --- /dev/null +++ b/docs/audit/stage5-phase0-20260727.txt @@ -0,0 +1,231 @@ +STAGE-5 PHASE 0 -- execution-environment preconditions: EXECUTED 2026-07-27 +========================================================================== +Operator direction (GA-R5 not required -- OPS execution of the already-ruled +R10, plus readiness-doc Phase 0): "Merge to main, then start Phase 0", then +"Approve A, B, and C, and retire the audit branch". + +Scope executed: readiness-doc preconditions 0.1, 0.2, 0.3 +(docs/audit/stage5-readiness-20260727.md), plus retirement of the merged +audit branch. Host: voffice1 (the D-128 Plane-2 execution host). + +Status authority for anything below is docs/CURRENT-STATE.md (GA-R1). This +file is a capture, not a status surface. + + +0. MERGE (prerequisite, done first) +----------------------------------- + merge commit 607813b 2 parents (NOT squashed), 33 commits from + dc-dc-stage5-grounding-audit + recorded by 6495cfb (a --no-ff merge cannot carry the doc edit; + same shape as Stage 4's 6f5701d / 1023596) + containment git branch --merged main -> listed, before deleting + ON main gauntlet ALL GREEN (81 harnesses); repo-lint 0 fail / 1 warn + branch RETIRED local + remote; git ls-remote --heads origin -> main only + + NO STAGE OPENED OR CLOSED. + + +A. 0.1 -- ADVANCE THE voffice1 CLONE +------------------------------------ +PRE (measured read-only, before touching anything): + HEAD 61c416e on dc-dc-g12-dc1-substrate (deleted upstream) + distance 105 0 vs its CACHED origin/main (itself pre-merge 71519a2) + dirty 1 untracked: opentofu/vr1-dc1-substrate/.terraform.lock.hcl + +SAFETY CHECK RUN FIRST -- the inner tfstate for BOTH DCs lives INSIDE this +working tree (it is the state-of-record; CURRENT-STATE section 1 records the +location). A branch switch here is therefore not a routine checkout. + git check-ignore -> IGNORED, all of: + opentofu/vr1-dc0-substrate/terraform.tfstate + opentofu/vr1-dc0-substrate/terraform.tfstate.backup + opentofu/vr1-dc1-substrate/terraform.tfstate + opentofu/vr1-dc1-substrate/terraform.tfstate.backup + opentofu/vr1-dc{0,1}-substrate/d124-inner.auto.tfvars + git ls-tree origin/main -> tracks the SAME file set at those paths as the + outgoing branch, and does NOT track vr1-dc1-substrate/.terraform.lock.hcl, + so the one untracked-not-ignored file cannot collide. + => the switch cannot touch the state-of-record. PROVEN, not assumed. + +EXECUTED: + git fetch origin --prune + - [deleted] origin/dc-dc-g12-dc1-substrate + - [deleted] origin/dc-dc-stage3-phase2-dc-substrate + - [deleted] origin/dc-dc-stage5-grounding-audit + 71519a2..6495cfb main -> origin/main + git switch main + (a plain `git pull` would NOT have worked -- the tracked branch is gone + upstream; recovery shape per finding L5-4) + +POST -- ASSERTIONS: + HEAD 6495cfb + origin/main 6495cfb ASSERT PASS + tfstate sha256, dc0 2e140b74...894608c3 IDENTICAL to PRE + tfstate sha256, dc1 aa91f103...dc671645 IDENTICAL to PRE + overlays/vr1-dc1-vips.yaml NOW PRESENT (was ABSENT) + overlays/vr1-dc1-machines.yaml NOW PRESENT (was ABSENT) + bundle.yaml machines block NOW the VR1 9-node role-separated layout + (was VR0 4-node hyperconverged "8"-"11") + dirty 1 untracked (unchanged, as predicted) + + => readiness blocker 1 ("you would deploy the wrong topology") CLEARED. + + +B. 0.3 -- PRUNE STALE REFS +-------------------------- +Remote-tracking refs: pruned by the fetch above (3 deleted). +Local branches: voffice1 carried TWO stale locals, not the one the record +named. Containment proven before deletion (`git rev-list --count +origin/main..` -> 0 for both; `-d` used, which refuses if unmerged). + Deleted dc-dc-g12-dc1-substrate (was 61c416e) + Deleted dc-dc-stage3-phase2-dc-substrate (was 57836b1) +FINAL: local `main` only; remotes/origin/main only. + +Ordering note: the audit branch was retired on origin BEFORE this fetch, so +advancing the clone could not hand it a FRESH stale ref -- the exact condition +that produced the 105-commit problem. + + +C. 0.2 -- THE openstack CLIENT +------------------------------ +MEASURED BEFORE PRESENTING (the options were not assumed): + snap `openstackclients` NO Caracal channel. Newest stable = zed (2023-03). + latest/stable = xena (2021). REFUTED for this + deployment. Corroborating in-repo prior art: + docs/design-decisions.md:638 records the snap's + home-only confinement (paths under /tmp unreadable). + apt noble/main python3-openstackclient 6.6.0-0ubuntu2 <-- MATCH + 6.6.0 IS the Caracal 2024.1 client (verified + upstream, not from memory: Launchpad + python-openstackclient 6.6.0-0ubuntu1, packaged + for the 2024.1 coordinated release). + archive reachability http://archive.ubuntu.com/ubuntu/dists/noble/Release -> 200 + simulated install 0 upgraded, 68 newly installed, 0 to remove + UCA not configured on this host (not needed -- noble's + native OpenStack release IS Caracal 2024.1) + +EXECUTED: sudo apt-get install -y python3-openstackclient + +POST -- VERIFIED BEHAVIOURALLY, not just by presence: + command -v openstack /usr/bin/openstack + openstack --version openstack 6.6.0 + openstack --help exit 0 + openstack server list "Missing value auth-url required for auth plugin + password" <- fails CLEANLY on absent config, + which is correct (no cloud configured yet); it + does not crash. + dpkg versions (the pin): + python3-openstackclient 6.6.0-0ubuntu2 + python3-openstacksdk 3.0.0-0ubuntu2 + python3-novaclient 2:18.5.0-0ubuntu1 + + => readiness blocker 2 CLEARED. Pin recorded in CURRENT-STATE section 7. + + +D. THE PAYOFF RUNS (deliberately deferred until the clone was current) +---------------------------------------------------------------------- +CURRENT-STATE recorded these as OWED; running them against a 105-commit-stale +tree would have measured nothing. All three run here for the first time. + +D.1 repo-lint ON voffice1 0 fail / 1 warn (the standing legacy + D-001..018 non-ASCII carve-out). Matches + vcloud exactly. + +D.2 gauntlet ON voffice1 *** 2/81 FAILED *** opentofu-validate, + site-headend-install + -- on the IDENTICAL commit that reports + ALL GREEN (81) on vcloud. + + FINDING P0-1: "GAUNTLET ALL GREEN" IS A HOST-DEPENDENT CLAIM. + Both failures are host-portability defects, NOT tree defects. Every + GA-R6 stage close in this project's history has cited a gauntlet figure + measured on vcloud only. + + (a) opentofu-validate -- every sub-check PASSES (root, 12 modules, both + extra roots) and the aggregate still reports FAIL. Cause: + `tofu fmt -check -recursive` walks the FILESYSTEM, not the git tree, + and trips on alignment drift in + opentofu/vr1-dc0-substrate/d124-inner.auto.tfvars + which is GITIGNORED and exists ONLY on voffice1 (it carries that + host's real deploy inputs). vcloud cannot see this file, so vcloud + can never fail on it -- and equally, vcloud's green can never + attest the fmt state of the inputs the deploy actually uses. + Verbatim drift: `opnsense_base_path = ` wants two more spaces to + align with `vvr1_dc0_ssh_keyfile`. + + (b) site-headend-install -- tests/site-headend-install/run-tests.sh:36 + if snap list lxd >/dev/null 2>&1 || snap list maas >/dev/null 2>&1; then + bad "--dry-run appears to have INSTALLED a snap on the test host" + The assertion takes an UNCONDITIONAL snapshot of the host and never + compares it to a pre-state. On voffice1 -- which IS the headend, where + lxd and maas are installed BY DESIGN -- it fires as a false positive. + It works only on a host that is not a headend. Same defect class the + R15 work names (an assertion that does not measure what it claims), + in its false-RED direction. + + NEITHER is fixed here (hard rule 1 -- Phase 0's ruled scope is 0.1/0.2/0.3). + Both are logged for a ruling/OPS decision. + +D.3 preflight ON voffice1 TRUE EXIT 1 ("PREFLIGHT: FAIL -- do NOT deploy") + (exit code read from the script itself, NOT through a pipe -- the 07-27 + audit owned exactly that error by reading `tail`'s status instead.) + + R10's RULED CONSEQUENCE IS DELIVERED, both halves: + P3 ALL 33 charm-channel pins now VERIFY [ok] -- 2024.1/stable, + squid/stable, 2.4/stable resolved against charmhub. Previously + ZERO of 33 were verified by any executable check, because `juju` + is absent on vcloud. This is the first executable confirmation of + the committed Caracal pins. + P4 "MAAS reachable (profile=admin)" PASS. The former "MAAS + unreachable" was the missing `maas` binary on vcloud, as ruled. + + P4 residual FAILs are the already-known, already-queued set: + - MISSING overlays/octavia-pki.yaml (deliberately absent; host-independent) + - metal-internal VID=0 (want 103) and metal-admin gateway=none + -> readiness item 3.7: pre-flight-checks.sh is VR0-FROZEN and its + VID-103 / br-internal assertions CONTRADICT D-133, so they can + never pass on either VR1 DC. Expected, not new. + + FINDING P0-2: P5 IS HOST-BLIND, AND PREFLIGHT HAS NO CORRECT HOST. + P5 on voffice1 -> 34 findings + P5 on vcloud -> 7 findings (the known residual set) + SAME matrix (82 rows), SAME commit. + Cause, measured: + vcloud has ~/vr1-dc0-creds, ~/vr1-dc1-creds, ~/vr1-office1-creds + voffice1 has ~/vr1-dc0-creds, ~/vr1-dc1-creds and NO vr1-office1-creds + (its two ARE the SEC-022 shadow stores, not the jumphost set) + The matrix's `jumphost` role carries no assertion about WHICH host is the + jumphost, so every `jumphost/*` location silently re-points to whatever + host runs the checker. On voffice1 that manufactures ~27 spurious + EXPECTED-BUT-ABSENT findings against the headend's shadow stores. + This is the committee's "tier 2 is SITE-BLIND" defect one level up: + HOST-blind. Note the direction -- here it produces false RED, but the + same hole would produce false GREEN for any row whose credential happens + to exist on the wrong host under a matching name. + One finding in the voffice1 run is REAL and worth keeping: + E3 UNDECLARED: 'maas-virsh_ed25519' present at jumphost/vr1-dc0 but in + NO matrix row for that site + -- i.e. an undeclared file in the headend shadow store, the SEC-022 class. + + CONSEQUENCE FOR R10: the ruling ("fix the stale commit issue and make sure + all working directories are current") is EXECUTED and its predicted + P3/P4 clearance is CONFIRMED. But the measurement also shows the reds are + not merely an artifact of the wrong host -- vcloud has the credentials and + not the clients; voffice1 has the clients and not the credentials. There + is NO single host on which preflight is currently correct. Not ruled here. + + +E. FINDINGS LOGGED, NOT ACTIONED (hard rule 1) +---------------------------------------------- + P0-1 gauntlet is host-dependent; two harnesses encode "I am not the + headend" assumptions (detail in D.2). The stage-close citation + "gauntlet ALL GREEN (N)" should state the HOST it was measured on. + P0-2 creds-matrix `jumphost` role is host-blind; preflight has no + correct host (detail in D.3). + P0-3 opentofu/vr1-dc1-substrate/.terraform.lock.hcl is untracked on main + while its dc0 twin IS tracked -- an asymmetry, harmless today, but it + is why the dc1 root shows a permanent untracked file on every host. + P0-4 No logged window (scripts/run-logged.sh) was opened for this session; + it was offered and the operator approved the steps directly. Recorded + for the as-executed trail rather than left silent. + +Captured 2026-07-27 on branch dc-dc-stage5-preconditions.