diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 4192f6d..7b21b18 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -933,9 +933,19 @@ charm-channel pins** (`2024.1/stable`, `squid/stable`, `2.4/stable`) -- the first executable confirmation of the committed Caracal pins, where previously ZERO of 33 were checked; and **P4 reports "MAAS reachable"**. Both were missing-binary artifacts on vcloud exactly as - ruled. P4's residual FAILs are the known set (the deliberately-absent octavia-pki overlay; - the VID-103/gateway assertions that readiness item 3.7 records as VR0-frozen and - D-133-contradicting, so they can never pass on a VR1 DC). **BUT P5 INVERTS: 34 findings on + ruled. Of P4's residual FAILs, two are the known set (the deliberately-absent octavia-pki + overlay; the VID-103 assertion that readiness item 3.7 records as VR0-frozen and + D-133-contradicting, so it can never pass on a VR1 DC). **The third was MIS-FILED as known by + this entry's first draft and is corrected here by measurement: `metal-admin gateway=none + (want 10.12.8.1)` is NOT item 3.7** (that item is specifically the VID-103/`br-internal` + assertions) **and was recorded nowhere.** `scripts/lib-net.sh:37` declares + `PLANE_GW["10.12.8.0/22"]="10.12.8.1"`, and measured from the dc0 rack that address is held + by NOTHING -- 100% loss with an `INCOMPLETE` ARP entry, i.e. never resolved, against a + control ping to the edge at `10.12.4.1` returning 0% loss. MAAS is therefore RIGHT to carry + no gateway there and the CHECKER is the defect: a VR0 inheritance, consistent with the D-134 + carve having ruled `.1` gateways for the two PROVIDER subnets only. The dc1 arm + (`lib-net.sh:149`) carries the same shape at `10.12.68.1` and will fail identically. Logged, + not fixed. **BUT P5 INVERTS: 34 findings on voffice1 against 7 on vcloud, same 82-row matrix, same commit.** Measured cause: the matrix's `jumphost` role carries NO assertion about which host is the jumphost, so every `jumphost/*` location silently re-points to whatever host runs the checker -- and voffice1 has @@ -1430,8 +1440,7 @@ | OPNsense edge | 26.7.1 (FreeBSD base 15.1) | MEASURED 2026-07-23 via the gated API (`GET core/firmware/status` -> product_version 26.7.1, capture `docs/audit/g13-close-20260723.txt`); updated 26.7 -> 26.7.1 in the G13 bundle. DC edges (vr1-dc0/dc1) remain 26.7 | office1-opnsense | | NetBox (Office1 apex) | 4.6.4 per as-built `docs/vr1-office1-as-built.md:44`; service UP verified (HTTP 302) this session | `ssh office1-netbox 'curl ... localhost:8000'` | office1-netbox | | Juju | **3.6.27 (rev 35621, `3/stable`) MEASURED 2026-07-27 on voffice1** -- the headend is the D-128 Plane-2 execution host and this is the client that will bootstrap the controller. Supersedes the 3.6.25 figure recorded 2026-07-24 (also at line 162, kept there as history): an in-channel patch refresh, which D-071 ADOPTED 2026-07-21 explicitly permits (patch-only jumps, in-channel-only refreshes), so this is policy-compliant drift and NOT an incident. The jumphost has NO juju client (measured ABSENT). | `ssh voffice1 'snap list'` (capture `docs/audit/stage5-live-measurement-20260727.txt`) | voffice1 | -| OpenStack client | **6.6.0 (`python3-openstackclient 6.6.0-0ubuntu2`, noble/main) INSTALLED ON voffice1 2026-07-27** -- Stage-5 Phase 0 precondition 0.2, operator-approved. voffice1 is the D-128 Plane-2 host every Stage-5+ script runs from. Verified behaviourally, not by presence: `openstack --version` -> `openstack 6.6.0`, `--help` exit 0, and `server list` fails CLEANLY on absent auth config rather than crashing. Companion pins: `python3-openstacksdk 3.0.0-0ubuntu2`, `python3-novaclient 2:18.5.0-0ubuntu1`. **The snap was REFUTED by measurement, not preference:** `openstackclients` has NO Caracal channel (newest stable `zed`, 2023-03; `latest/stable` is `xena`, 2021), and `docs/design-decisions.md:638` already records its home-only confinement trap. 6.6.0 is the Caracal 2024.1 client, verified upstream rather than from memory. noble's native OpenStack release IS Caracal 2024.1, so no UCA is needed on this host. **STILL ABSENT ON vcloud** -- deliberately: D-128 puts this work on the headend. Capture `docs/audit/stage5-phase0-20260727.txt`. | `ssh voffice1 'openstack --version; dpkg-query -W ...'` | voffice1 | -| OpenStack client (prior state, history) | **ABSENT ON BOTH HOSTS, MEASURED 2026-07-27** -- `command -v openstack` returns nothing on vcloud AND on voffice1, and voffice1's snap list carries only core24/juju/lxd/maas/postgresql/snapd. Ten Stage-5/6/7 scripts invoke it (`phase-03-admin-openrc.sh`, `phase-04-network-{create,verify}.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-{amphora-pipeline,octavia-verify}.sh`, `phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh`), so every post-deploy step from Stage 5 Step 7 onward has no client to run. Recorded here because it is a measured Stage-5 precondition that no prior surface carried. | `command -v openstack`; `ssh voffice1 'command -v openstack; snap list'` | vcloud + voffice1 | +| OpenStack client | **6.6.0 (`python3-openstackclient 6.6.0-0ubuntu2`, noble/main) INSTALLED ON voffice1 2026-07-27** -- Stage-5 Phase 0 precondition 0.2, operator-approved. voffice1 is the D-128 Plane-2 host every Stage-5+ script runs from. Verified behaviourally, not by presence: `openstack --version` -> `openstack 6.6.0`, `--help` exit 0, and `server list` fails CLEANLY on absent auth config rather than crashing. Companion pins: `python3-openstacksdk 3.0.0-0ubuntu2`, `python3-novaclient 2:18.5.0-0ubuntu1`. **The snap was REFUTED by measurement, not preference:** `openstackclients` has NO Caracal channel (newest stable `zed`, 2023-03; `latest/stable` is `xena`, 2021), and `docs/design-decisions.md:638` already records its home-only confinement trap. 6.6.0 is the Caracal 2024.1 client, verified upstream rather than from memory. noble's native OpenStack release IS Caracal 2024.1, so no UCA is needed on this host. **STILL ABSENT ON vcloud** -- deliberately: D-128 puts this work on the headend. Capture `docs/audit/stage5-phase0-20260727.txt`. **Supersedes the "ABSENT ON BOTH HOSTS" figure measured earlier the same day** (kept here as history, per the Juju row's precedent for in-row supersession): the ten Stage-5/6/7 scripts that invoke it (`phase-03-admin-openrc.sh`, `phase-04-network-{create,verify}.sh`, `phase-04-internal-cert-san-verify.sh`, `phase-05-{amphora-pipeline,octavia-verify}.sh`, `phase-06-{bootstrap,capi-stack,mgmt-vm,net-setup}.sh`) now have a client on the host D-128 runs them from. | `ssh voffice1 'openstack --version; dpkg-query -W ...'` | voffice1 | The known-stale pin sites this table used to enumerate (the GA-F03/F04/ F05 tofu, OPNsense, and jumphost-name values -- stated token-free here so diff --git a/docs/audit/stage5-phase0-20260727.txt b/docs/audit/stage5-phase0-20260727.txt index 8ab9ac2..e6bffc1 100644 --- a/docs/audit/stage5-phase0-20260727.txt +++ b/docs/audit/stage5-phase0-20260727.txt @@ -178,12 +178,40 @@ P4 "MAAS reachable (profile=admin)" PASS. The former "MAAS unreachable" was the missing `maas` binary on vcloud, as ruled. - P4 residual FAILs are the already-known, already-queued set: - - MISSING overlays/octavia-pki.yaml (deliberately absent; host-independent) - - metal-internal VID=0 (want 103) and metal-admin gateway=none - -> readiness item 3.7: pre-flight-checks.sh is VR0-FROZEN and its - VID-103 / br-internal assertions CONTRADICT D-133, so they can - never pass on either VR1 DC. Expected, not new. + P4 residual FAILs -- and ONE OF THESE WAS INITIALLY MIS-FILED BY THIS + CAPTURE AS "already known". Corrected here by measurement: + - MISSING overlays/octavia-pki.yaml (deliberately absent; host-independent). + Genuinely known. + - metal-internal VID=0 (want 103) -> readiness item 3.7: pre-flight-checks.sh + is VR0-FROZEN and its VID-103 / br-internal assertions CONTRADICT D-133, + so they can never pass on either VR1 DC. Genuinely known. + - metal-admin gateway=none (want 10.12.8.1) -> **NOT item 3.7, and NOT + previously recorded anywhere.** Item 3.7 is specifically the VID-103 / + br-internal assertions; this is a separate expectation. See P0-5. + + FINDING P0-5: lib-net DECLARES A metal-admin GATEWAY THAT HAS NEVER EXISTED. + scripts/lib-net.sh:37 (the DEFAULT / VR0-dc0 arm): + declare -A PLANE_GW=( ["10.12.4.0/22"]="10.12.4.1" + ["10.12.8.0/22"]="10.12.8.1" ) + so pre-flight-checks.sh CHECK 3 hard-FAILs metal-admin for having no + gateway. MEASURED from the dc0 rack (ssh -J voffice1, dc0 svc key): + ping 10.12.8.1 -> 2 transmitted, 0 received, 100% loss + ip neigh -> 10.12.8.1 dev virbr2 INCOMPLETE (ARP never + resolved -- nothing HOLDS the address; this is + not a device that is merely down) + rack legs on the plane: 10.12.8.2 (rack), .3 (D-131 DNS forwarder), + .4 (D-134 utility / mirror) + CONTROL: ping 10.12.4.1 (the edge, provider gw) -> 0% loss + The control proves the instrument. So MAAS is RIGHT to carry no gateway + on metal-admin and the CHECKER's expectation is the defect -- a VR0 + inheritance. Consistent with the record of the D-134 carve, which ruled + `.1` gateways for the two PROVIDER subnets only; the 8 plane subnets were + created without them. Nothing routes off metal-admin (nodes reach the + mirror .4 and DNS .3 on-link), so no gateway is wanted. + Adjacent to item 3.7 -- same script, same VR0-frozen class -- but a + DISTINCT assertion that 3.7's remediation would not touch. Logged, not + fixed (hard rule 1). NOTE the dc1 arm (lib-net.sh:149) carries the same + shape at 10.12.68.1 and will fail identically. FINDING P0-2: P5 IS HOST-BLIND, AND PREFLIGHT HAS NO CORRECT HOST. P5 on voffice1 -> 34 findings @@ -227,5 +255,26 @@ P0-4 No logged window (scripts/run-logged.sh) was opened for this session; it was offered and the operator approved the steps directly. Recorded for the as-executed trail rather than left silent. + P0-5 lib-net declares a metal-admin gateway (.8.1 / dc1 .68.1) that no + device holds -- measured 100% loss + INCOMPLETE ARP against a working + control (detail in D.3). Both DC arms affected. + + QUESTION QUEUED FOR THE OPERATOR (not a finding -- an interlock the R15 + rulings do not currently reach; one GA-R5 exchange, no D-number self-assigned, + GA-R3 doubt resolves DOWN to OPS unless the A1 Roosevelt test is met): + WHICH HOST IS AUTHORITATIVE FOR THE GATES? + R10 ruled preflight belongs on voffice1; R15(3) rules preflight must stop + failing open. Executed together as ruled, and given P0-2, preflight on + voffice1 becomes PERMANENTLY UNPASSABLE -- its 34 findings are host + artifacts, not defects. Likewise R15(2) pins a harness MANIFEST, which + catches renames but does nothing about P0-1: `tofu fmt -check -recursive` + walks the filesystem, so implementing R15(2) exactly as ruled still leaves + voffice1 red. So the R15 execution scope needs to answer, first, which host + each gate is authoritative on -- and for the credential matrix, whether + `jumphost` is a ROLE that must be pinned to a named host. + STANDING CONSEQUENCE either way, worth stating plainly: STAGE 5 EXECUTES + FROM voffice1, and the gauntlet cannot be green there until P0-1 is + resolved. Every future GA-R6 close citing a gauntlet figure measured there + inherits a known red, so the citation must name its host. Captured 2026-07-27 on branch dc-dc-stage5-preconditions. diff --git a/docs/changelog-20260727-stage5-phase0.md b/docs/changelog-20260727-stage5-phase0.md new file mode 100644 index 0000000..f93e998 --- /dev/null +++ b/docs/changelog-20260727-stage5-phase0.md @@ -0,0 +1,164 @@ +# Session changelog 2026-07-27 -- grounding-audit merge + Stage-5 Phase 0 + +ONE changelog per session (GA-R2/D1). Under blanket approval this is the review +surface: every item states WHAT, WHY (evidence), and HOW TO REVERT. + +**No logged window was opened for this session** (`scripts/run-logged.sh` was +offered; the operator approved the steps directly). This file plus +`docs/audit/stage5-phase0-20260727.txt` are therefore the ENTIRE as-executed +record of five live mutations. Reverts below are stated with that in mind. + +Status authority for everything here is `docs/CURRENT-STATE.md` (GA-R1). This +file is session-scoped scratch and is NOT citable as status or decision +authority. + +--- + +## 1. Merged `dc-dc-stage5-grounding-audit` to `main` + +**What.** Merge commit `607813b`, 2 parents (NOT squashed), 33 commits. +Recorded on `main` by follow-up commit `6495cfb` -- a `--no-ff` merge cannot +carry the doc edit, so this is the Stage-4 shape (`6f5701d` recorded by +`1023596`). + +**Why.** Operator direction, exact utterance: "Merge to main, then start Phase +0". The branch carried 14 GA-R5 rulings and the audit record; precondition work +must branch off a `main` that contains them. **No stage opened or closed.** + +**Evidence.** `git log -1 --pretty=%P` -> two hashes. `git branch --merged main` +listed the branch before deletion. Post-merge ON `main`: gauntlet ALL GREEN +(81 harnesses), repo-lint 0 fail / 1 standing warn. + +**Revert.** `git revert -m 1 607813b` on `main` (then revert `6495cfb`). +Prefer this over a reset -- `main` is pushed and shared. + +## 2. Retired the audit branch (local + origin) + +**What.** Deleted `dc-dc-stage5-grounding-audit` from origin and locally on +vcloud. Last commit was **`39e8988`**. + +**Why.** Contained in `main` after item 1, and leaving it alive would hand the +voffice1 clone a FRESH stale remote-tracking ref during item 3 -- the exact +condition that produced the 105-commit problem being fixed. + +**Evidence.** Containment re-checked immediately before deletion; `git branch -d` +(not `-D`) used, which refuses an unmerged branch. `git ls-remote --heads origin` +afterwards -> `main` only. + +**Revert.** `git push origin 39e8988:refs/heads/dc-dc-stage5-grounding-audit`. +Nothing is lost meanwhile -- the commits are in `main`. + +## 3. Advanced the voffice1 clone to `main` (precondition 0.1) + +**What.** On `voffice1:~/openstack-caracal-dc-dc`: `git fetch origin --prune` +then `git switch main`. Was `61c416e` on `dc-dc-g12-dc1-substrate`, a branch +deleted upstream; now `6495cfb`. + +**Why.** voffice1 is the D-128 Plane-2 host Stage 5 EXECUTES from, and it was +105 commits behind with both dc1 overlays ABSENT and `bundle.yaml` still the VR0 +4-node hyperconverged layout -- readiness blocker 1, "you would deploy the wrong +topology". R10 ruled this fix. A plain `git pull` could not work (tracked branch +gone upstream); the shape is `fetch` + `switch` per finding L5-4. + +**Evidence / the safety argument.** Both DCs' inner tfstate -- the substrate's +state-of-record -- lives INSIDE that working tree, so this was not a routine +checkout. PROVEN before the switch: every state artifact is +`git check-ignore`-IGNORED, and `origin/main` tracks an identical file set at +those paths (so no collision, and the one untracked-not-ignored file, +`vr1-dc1-substrate/.terraform.lock.hcl`, is untracked on `main` too). AFTER: +both tfstate sha256s BYTE-IDENTICAL to before (`2e140b74...`, `aa91f103...`); +`HEAD == origin/main` asserted; both dc1 overlays now present; `bundle.yaml` now +the 9-node role-separated layout. + +**Revert.** `ssh voffice1 'cd ~/openstack-caracal-dc-dc && git switch -c +dc-dc-g12-dc1-substrate 61c416e'`. The ignored state artifacts are untouched by +either direction, which is the point of the check above. + +## 4. Deleted two stale local branches on voffice1 (precondition 0.3) + +**What.** `dc-dc-g12-dc1-substrate` (was **`61c416e`**) and +`dc-dc-stage3-phase2-dc-substrate` (was **`57836b1`**). Three stale +remote-tracking refs were pruned by item 3's `--prune`. + +**Why.** Both tracked branches deleted upstream. The record named ONE stale +local branch; there were two. + +**Evidence.** `git rev-list --count origin/main..` -> **0** for both +(fully contained, nothing unique lost). `git branch -d` used, which refuses if +unmerged. + +**Revert.** `ssh voffice1 'cd ~/openstack-caracal-dc-dc && git branch +dc-dc-g12-dc1-substrate 61c416e && git branch dc-dc-stage3-phase2-dc-substrate +57836b1'`. + +## 5. Installed the `openstack` client on voffice1 (precondition 0.2) + +**What.** `sudo apt-get install -y python3-openstackclient` on voffice1. +Landed `python3-openstackclient 6.6.0-0ubuntu2`, `python3-openstacksdk +3.0.0-0ubuntu2`, `python3-novaclient 2:18.5.0-0ubuntu1`. 68 packages newly +installed, 0 upgraded, 0 removed. + +**Why.** Readiness blocker 2: the client was absent on BOTH hosts while ten +Stage-5/6/7 scripts invoke it. voffice1 is where D-128 runs them. + +**Evidence -- the method was measured, not assumed.** The snap was REFUTED: +`openstackclients` has NO Caracal channel (newest stable `zed`, 2023-03; +`latest/stable` is `xena`, 2021), and `docs/design-decisions.md:638` already +records its home-only confinement trap. 6.6.0 IS the Caracal 2024.1 client, +verified upstream rather than from memory. Verified behaviourally after install: +`openstack --version` -> `openstack 6.6.0`; `--help` exit 0; `server list` fails +CLEANLY on absent auth config rather than crashing. + +**Revert.** `ssh voffice1 'sudo apt-get purge -y python3-openstackclient +python3-openstacksdk python3-osc-placement && sudo apt-get autoremove -y'`. +Scope the autoremove by reviewing `/var/log/apt/history.log` for this +transaction first -- 68 packages were pulled and some (e.g. `python3-pil`, +`libopenjp2-7`) may be shared with other consumers on that host. + +**Note this is now a PIN.** Recorded in `docs/CURRENT-STATE.md` section 7, +which carried no row for this component before. + +--- + +## Findings LOGGED, NOT ACTIONED (hard rule 1 -- outside Phase 0's ruled scope) + +Full detail in `docs/audit/stage5-phase0-20260727.txt`. + +- **P0-1 the gauntlet is HOST-DEPENDENT.** 2/81 FAILED on voffice1 on the + IDENTICAL commit that is ALL GREEN (81) on vcloud. Neither is a tree defect. + `opentofu-validate` passes every sub-check and still reports FAIL because + `tofu fmt -check -recursive` walks the FILESYSTEM and trips on a GITIGNORED + file that exists only on voffice1. `site-headend-install:36` asserts a + `--dry-run` installed no snap by snapshotting the host without comparing to a + pre-state -- on the real headend, where lxd/maas are installed by design, a + false positive. **Every GA-R6 stage close in this project cites a gauntlet + figure measured on vcloud only.** +- **P0-2 preflight P5 is HOST-BLIND.** 34 findings on voffice1 vs 7 on vcloud, + same 82-row matrix and commit. The matrix's `jumphost` role has no assertion + about WHICH host is the jumphost, so those locations re-point to whatever host + runs the checker. Surfaces here as false RED; the same hole yields false GREEN + for a credential present on the wrong host under a matching name. +- **P0-3** `opentofu/vr1-dc1-substrate/.terraform.lock.hcl` is untracked on + `main` while its dc0 twin IS tracked. +- **P0-4** no logged window opened for this session (see the header). +- **P0-5** `lib-net.sh:37` declares a metal-admin gateway `10.12.8.1` that NO + device holds -- measured 100% loss + `INCOMPLETE` ARP against a control ping + to `10.12.4.1` at 0% loss. MAAS is right; the checker is wrong. **This capture + initially MIS-FILED it as covered by readiness item 3.7; it is not** (3.7 is + the VID-103/`br-internal` assertions). The dc1 arm carries the same shape at + `10.12.68.1`. + +## Question QUEUED for the operator (one GA-R5 exchange; no D-number assigned) + +**Which host is authoritative for the gates?** R10 ruled preflight belongs on +voffice1; R15(3) rules preflight must stop failing open. Executed together as +ruled, and given P0-2, preflight on voffice1 becomes PERMANENTLY UNPASSABLE -- +its 34 findings are host artifacts. Similarly R15(2) pins a harness MANIFEST, +which catches renames but does nothing about P0-1, so R15(2) as ruled still +leaves voffice1 red. GA-R3 resolves doubt DOWN to OPS; whether "which host is +authoritative for a gate" clears the A1 Roosevelt test is the operator's call. + +**Standing consequence either way:** Stage 5 executes from voffice1 and the +gauntlet cannot be green there until P0-1 is resolved, so any GA-R6 close citing +a gauntlet figure measured there inherits a known red -- the citation must name +its host.