diff --git a/docs/changelog-20260721-netem-install-verify.md b/docs/changelog-20260721-netem-install-verify.md index c718814..897f828 100644 --- a/docs/changelog-20260721-netem-install-verify.md +++ b/docs/changelog-20260721-netem-install-verify.md @@ -122,7 +122,61 @@ ## 7. Queued (logged, not executed) - DOCFIX: phase2 runbook Step 11 leg-selection text (item 5's flagged - divergence). + divergence). -> DELIVERED as item 9 below. - G16 mechanism ruling: state-only surgery (G6 precedent) vs ride the edge's next scheduled restart (with the D-129 office1 qga retrofit). -- ledger-scan D-131 status-phrasing blind spot (item 1). +- ledger-scan D-131 status-phrasing blind spot (item 1). -> DELIVERED + as item 8 below. + +## 8. ledger-scan: PARTIALLY-RULED decisions now surface (stage-close batch) + +- The last-Status-line match gained PARTIAL alongside PROPOSED|OPEN -- + D-131 ("PARTIALLY RULED", the word OPEN on a continuation line) was + invisible to the scan while its sub-2..4 are open. Blast radius + checked before widening: exactly two PARTIAL status lines exist + (D-129 already caught via OPEN; D-131 now caught). Harness gained a + regression fixture (D-052, numbered below the fixture high-water so + the next-free check stays true) -- 46 checks ALL PASS. Live scan now + lists D-068/D-071/D-129/D-131. +- **Revert:** git revert this hunk; D-131 disappears from the scan + again (the ledger's hand-seeded block still carries it). + +## 9. DOCFIX-197: phase2 runbook Step 11 leg selection corrected to executed reality + +- Two spots (the step-E summary line + the Step 11 body): target is the + dc0<->dc1 leg (re-measure bridge at apply time), NOT dc0<->office1, + which now carries live rack<->region transit; example block updated + to the as-executed module shape (local mode, drifting-ID caveat, + capture cites). The netem-parameters PLACEHOLDER note was already + accurate and is untouched. +- **Revert:** git revert (docs only). + +## 10. D-103 + D-123 amendments and maas-vm-host header: pod-mechanism refutation RECORDED + +- Records the 2026-07-20 MEASURED refutation (snap confinement kills + the local pod; qemu+ssh pod dies on volume-ref domblkinfo; pod + unnecessary -- PXE delivered discovery) and the already-made + 2026-07-20 ruling (per-machine power_type=virsh). This is recording, + not a new ruling: both facts and the ruling predate this session and + are cited to their captures/changelogs. modules/maas-vm-host header + gains a DO-NOT-WIRE-FOR-DC note; retire-or-keep stays an OPEN + stage-close ruling (SEC-013 tied). +- **Revert:** git revert (docs + one module comment; no resource HCL + touched). + +## 11. GA-R7 memory review (stage-close item) DONE + +- Index (MEMORY.md) matches the two entries; both are compliant + pointer/convention records (re-vetted in the 07-19 sweep). The one + flagged possibly-memory-only fact (sandbox NetBox token location) + verified REPO-CARRIED (vr1-office1-as-built rows 130-131) and + confirmed live by name-only directory listing; memory updated to + drop its caveat. No memory-only facts remain; nothing new needs + memorizing (all stage lessons are repo-carried). + +## 12. Stage-close read-only fact gathering (for the pending rulings) + +- Stale MAAS pod object CONFIRMED on the region: `vm-hosts read` -> + id=4 `vr1-dc0-inner` type=virsh (beside the legitimate office1-lxd + id=1). Its deletion is a gated mutation awaiting the retire-or-keep + ruling cluster (with SEC-013's on-disk key + tfstate disposition). diff --git a/docs/design-decisions.md b/docs/design-decisions.md index e62a651..5e50671 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2028,6 +2028,21 @@ lifecycle is otherwise unchanged; only the libvirt HOST the nodes land on moves from vcloud to `vvr1-dc0`. Office1's placement is unaffected. +**AMENDED 2026-07-20 (VM-host registration MECHANISM refuted by measurement; INTENT stands):** the +"registers ... as a virsh VM-host" bullet below is DEAD for the DCs, on two measured grounds from the +step-D deploy: (1) snap-confined MAAS gets `Permission denied` on `/var/run/libvirt/libvirt-sock` and +`snap connections maas` lists NO libvirt interface, so a LOCAL `qemu:///system` pod is impossible with +snap MAAS; (2) the `qemu+ssh` replacement pod then fails at refresh on `domblkinfo ... missing storage +backend for 'volume'` (reproduced locally on the rack with an active pool) -- MAAS virsh pods are +incompatible with `modules/node-vm`'s pool+volume disk refs. The pod was also UNNECESSARY: its job in +this seam was DISCOVERY, and PXE enlistment across the transit already delivers it (all 9 nodes +enlisted 2026-07-20, `docs/audit/stepD-enlistment-20260720.txt`). **RULED 2026-07-20: per-machine +`power_type=virsh`** (`scripts/maas-node-power.sh`, MAC-matched, write-verified by +`query-power-state`) -- which is ALSO the Roosevelt shape (per-node IPMI/BMC, no pod). Everything +else in this seam (OpenTofu creates, MAAS owns commission/deploy/power/release, never composes) +SURVIVES and is proven end to end. `modules/maas-vm-host` and `opentofu/vr1-dc0-maas/` are refuted +for DC use; their retire-or-keep is a stage-close question (SEC-013 tied). + **Decision:** - OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt. - OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs. @@ -3651,6 +3666,22 @@ retarget `modules/maas-vm-host`. **What still stands:** the MAAS model (ONE region on Office1 + a rack controller per DC -- `vvr1-dc0` is that rack, now also the node host) is operator-confirmed and unaffected. +### D-123 -- AMENDMENT (2026-07-20): the Model-B virsh-POD mechanism is REFUTED by measurement; per-machine virsh power replaces it (RULED) + +Model B itself (nodes inside `vvr1-dc0`; site-down = one `virsh destroy`; depth-4) is BUILT and PROVEN +-- this amendment corrects only the "vm-host to register is vvr1-dc0's OWN (inner, local +`qemu:///system`) virsh" mechanism this entry and `modules/maas-vm-host`'s header stated. Measured +during step D (2026-07-20): (1) snap MAAS cannot open `/var/run/libvirt/libvirt-sock` (confinement; +no libvirt snap interface exists), so the LOCAL pod is impossible; (2) a `qemu+ssh` pod (dedicated +operator-ruled key, reachable from both snaps) still fails -- `domblkinfo ... missing storage backend +for 'volume'` -- because MAAS virsh pods cannot read `modules/node-vm`'s pool+volume disk refs. The +pod's only job here was DISCOVERY, which PXE enlistment already achieved (9/9 enlisted, shapes exact +to D-121 Option C). **RULED 2026-07-20: per-machine `power_type=virsh`** via +`scripts/maas-node-power.sh` (MAC-matched, dry-by-default, each write verified by a live +`query-power-state`) -- the same per-node shape Roosevelt uses (IPMI/BMC per machine, no pod). +D-103 carries the matching amendment; `opentofu/vr1-dc0-maas/` (the pod root) is retained UNUSED +pending the stage-close retire-or-keep ruling (SEC-013 tied). + ## D-124: the Office1-region <-> DC-rack management overlay -- addressing for the MAAS control path [ARCH] **Status:** ADOPTED (2026-07-16, operator ruling). **Scheme A** (transit-numbered mesh) + rack sizing diff --git a/opentofu/modules/maas-vm-host/main.tf b/opentofu/modules/maas-vm-host/main.tf index 7d1931d..d1f9a39 100644 --- a/opentofu/modules/maas-vm-host/main.tf +++ b/opentofu/modules/maas-vm-host/main.tf @@ -10,6 +10,14 @@ # provider block forces every plan to demand creds); wire in its own gated step after the region # + inner nodes exist, registering to the Office1 REGION. # +# REFUTED FOR DC USE BY MEASUREMENT (2026-07-20 step D; D-103/D-123 amendments of that date are +# the authority): snap MAAS cannot open the local libvirt socket (confinement, no libvirt snap +# interface), and a qemu+ssh pod fails on `domblkinfo ... missing storage backend for 'volume'` +# (virsh pods cannot read node-vm's pool+volume disk refs). Discovery -- this module's whole D-103 +# job -- is delivered by PXE enlistment instead, and power is RULED per-machine +# (`power_type=virsh` via scripts/maas-node-power.sh; the Roosevelt per-node-BMC shape). Do NOT +# wire this module for a DC. Retained pending the stage-close retire-or-keep ruling (SEC-013). +# # DELIBERATELY uses `maas_vm_host`, NOT `maas_vm_host_machine` -- confirmed # by reading both resources' real schemas (canonical/terraform-provider-maas, # fetched 2026-07-09): `maas_vm_host_machine` takes `cores`/`memory`/ diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md index cce4ca6..ae87d26 100644 --- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md +++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md @@ -161,7 +161,11 @@ vvr1-dc0's libvirt, NOT vcloud. `[MUTATION: gated]` - **D. maas-vm-host** (Step 9) -- register vvr1-dc0's OWN (inner, local `qemu:///system`) virsh to the Office1 REGION; MAAS discovers the inner node VMs (Step 10). Still DEFERRED (DOCFIX-179). -- **E. netem** (Step 11) -- UNCHANGED: runs on the vcloud-level mesh bridges (virbr7), outer. +- **E. netem** (Step 11) -- runs on the vcloud-level mesh bridges, outer. TARGET + CORRECTED (DOCFIX-197, 2026-07-21; executed reality): the dc0<->dc1 leg + (measured `virbr5` at apply time), NOT the dc0<->office1 leg this line used to + name -- that leg carries the LIVE rack<->region transit (MAAS, node DNS, the + inner root's qemu+ssh) and shaping it would perturb operations. See Step 11. - **Edge plugin profile (D-129, pinned 2026-07-18).** After the DC edge boots and its REST API is up (inner apply C), apply the VR1 edge base profile **{`os-qemu-guest-agent`, `os-iperf`}** via the @@ -623,24 +627,32 @@ profile, use that instead and cite the ruling; do not use this placeholder language once a real ruling exists. -**MUTATION -- edit `opentofu/main.tf`**, one block per relevant mesh leg. The -only leg with substrate on both ends this pass is `vr1-dc0`<->`office1` -(`vr1-dc0`<->`vr1-dc1` and `vr1-dc1`<->`office1` have no `vr1-dc1`-side -substrate while `vr1-dc1` remains gated): +**LEG SELECTION CORRECTED (DOCFIX-197, 2026-07-21 -- this is how step E was +actually executed).** This step's original text targeted `vr1-dc0`<->`office1` +as "the only leg with substrate on both ends". Superseded by operations: that +leg now CARRIES the live rack<->region transit (MAAS enlistment/commissioning, +the D-131 node-DNS forwarder path, the inner root's `qemu+ssh`), so shaping it +perturbs a production path. The step-E target is the `vr1-dc0`<->`vr1-dc1` +leg -- zero traffic while `vr1-dc1` is gated, which exercises the netem +MECHANISM (this step's actual goal, per the 2026-07-16 placeholder ruling) +with no operational impact. The netem-tc sudoers fragment +(`scripts/sudoers.d/netem-tc`) grants all three mesh bridges, so extending to +other legs later is config-only -- but shaping a leg that carries live +transit gets its own gated decision first. + +**MUTATION -- edit `opentofu/main.tf`** (as executed 2026-07-21; captures +`docs/audit/outer-{plan,apply}-20260721-netem*.txt` + `stepE-netem-20260721.txt`): ```hcl -module "netem_vr1_dc0_office1" { - source = "./modules/netem-link" - link_name = "vr1-dc0-office1" - vcloud_host_ssh_target = "" - bridge_name = "" - netem_args = "" +module "netem_vr1_dc0_vr1_dc1" { + source = "./modules/netem-link" + link_name = "vr1-dc0-vr1-dc1" + bridge_name = "" + netem_args = "" + # vcloud_host_ssh_target: leave at its empty default = LOCAL mode. The outer + # root executes ON vcloud (D-128 Plane 1), where the bridges live; the SSH + # hop exists only for a root that runs remotely (module amendment 2026-07-21). } ``` -(A `vr1-dc0`<->`vr1-dc1` block -- `module "netem_vr1_dc0_vr1_dc1"`, -`link_name "vr1-dc0-vr1-dc1"` -- follows the same shape once `vr1-dc1` itself -is ungated; until then, applying netem on that leg has no `vr1-dc1`-side -substrate to matter for, so it is naturally deferred alongside the rest of -`vr1-dc1`'s work.) ```bash cd opentofu diff --git a/scripts/ledger-scan.sh b/scripts/ledger-scan.sh index 53024b2..ad6249f 100644 --- a/scripts/ledger-scan.sh +++ b/scripts/ledger-scan.sh @@ -76,10 +76,14 @@ # (A later RESOLVED/CLOSED/SUPERSEDED amendment closes it -- so we track the last # status per block, not the first; this is the D-063-amendment drift the first # real run surfaced.) + # PARTIAL is included because a "PARTIALLY RULED" decision still has open + # sub-decisions (the D-131 blind spot, found 2026-07-21: its Status line + # begins "PARTIALLY RULED" and the word OPEN falls on a continuation line + # this per-line match never sees). OPEN_DEC="$(awk ' - /^## D-[0-9]+:/ { if (hdr!="" && laststat ~ /PROPOSED|OPEN/) print " " hdr; sub(/^## /,"",$0); hdr=$0; laststat="" } + /^## D-[0-9]+:/ { if (hdr!="" && laststat ~ /PROPOSED|OPEN|PARTIAL/) print " " hdr; sub(/^## /,"",$0); hdr=$0; laststat="" } /\*\*Status:\*\*/ { laststat=$0 } - END { if (hdr!="" && laststat ~ /PROPOSED|OPEN/) print " " hdr } + END { if (hdr!="" && laststat ~ /PROPOSED|OPEN|PARTIAL/) print " " hdr } ' "$DD")" if [ -n "$OPEN_DEC" ]; then printf '%s\n' "$OPEN_DEC"; else echo " (none)"; fi else echo " (design-decisions.md not found)"; fi diff --git a/tests/ledger-scan/run-tests.sh b/tests/ledger-scan/run-tests.sh index 3567194..c7c9b75 100644 --- a/tests/ledger-scan/run-tests.sh +++ b/tests/ledger-scan/run-tests.sh @@ -22,6 +22,9 @@ **Status:** PROPOSED / OPEN (recorded). Some text. **Status:** RESOLVED / CLOSED (amendment 2026-07-03). +## D-052: partially ruled -- open sub-decisions, OPEN off the status line +**Status:** PARTIALLY RULED. Sub-decision 1 RULED 2026-07-21; the word for the rest +falls on this continuation line: sub-2..4 OPEN (the D-131 blind-spot shape, fixture-numbered below high-water). DD cat > "$W/docs/security-ledger.md" <<'SL' | id | date | item | src | owner | status | @@ -51,6 +54,7 @@ nochk 'D-063' "omits D-063 CLOSED" nochk 'D-070' "omits D-070 ADOPTED (from decisions section)" nochk 'D-063b' "omits amendment-CLOSED decision (last-status wins)" +chk 'D-052' "surfaces D-052 PARTIALLY RULED (open sub-decisions; D-131 blind-spot regression)" chk 'SEC-001' "surfaces SEC-001 OPEN" chk 'SEC-003' "surfaces SEC-003 OPEN" nochk 'SEC-004' "omits SEC-004 CLOSED"