diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index 133cb13..7b4422a 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -132,7 +132,17 @@ service net -- if it is an ISOLATED host-local net, add a MEASURED leg row + `install` the unit as part of standup definition-of-done (NEVER an inferred DC IP -- the harness fails on an un-commented DC supernet); a routed/NAT net or an already-reachable -`qemu+ssh` path needs none. Register item 20 tracks this. +`qemu+ssh` path needs none. Register item 20 tracks this. **Rack-side twin invariant +(D-131 sub-1, RULED 2026-07-21): every internet-isolated DC rack gets the repo-carried +`scripts/dc-rack-net.sh install ` at standup (reboot-persistent rack bridge legs ++ the node-DNS dnsmasq forwarder aliased on metal-admin, forwarding to region BIND over +the rack's own transit) -- MAAS 3.7's rack-only agent resolver SERVFAILs on isolated +racks, so WITHOUT this, commissioning silently times out (appendix-A carries the +symptom pair).** Two measured MAAS traps with standing consequences: MAAS virsh PODS +are refuted for this repo's DCs (per-machine `power_type=virsh` via +`scripts/maas-node-power.sh` is the ruled shape -- D-103/D-123 amendments 2026-07-20), +and a pod delete CASCADES to its linked machine records -- read the pod's machine list +BEFORE any `vm-host delete`; non-empty = STOP (appendix-A, 2026-07-21 incident). **Change-delivery loop:** grep for prior art (zeroth decision) -> grep design-decisions for the governing D-NNN -> edit -> `bash scripts/repo-lint.sh` @@ -168,8 +178,9 @@ (local + remote). **ALSO sweep THIS skill in the same close:** fold in any new INVARIANT the stage established and reconcile any divergence from repo HEAD (facts delegate to the repo and go stale; invariants live here and must be kept current). The next stage branches off post-merge `main`. `main` is -trunk-of-record only for stages that have merged (precedent: Stage 2 -> PR #1 `e468d2e`, -2026-07-15). See `docs/dc-dc-deployment-workflow.md` "Cross-cutting discipline". +trunk-of-record only for stages that have merged (precedents: Stage 2 -> PR #1 `e468d2e` +2026-07-15; Stage 3 -> merge commit 2026-07-21, vr1-dc0 scope, dc1 HELD under G12). See +`docs/dc-dc-deployment-workflow.md` "Cross-cutting discipline". **Incident loop:** capture the verbatim error -> `bash scripts/cloud-assert.sh` (the service-own-verdict sweep localizes the layer) -> appendix-A by exact diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index b29ebfa..34fc4db 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -24,10 +24,16 @@ virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal Roosevelt deployment (D-100, `docs/design-decisions.md:1946`). - Stage: Stage 3 / Phase 2 -- "OpenTofu builds each DC substrate" - (`docs/dc-dc-deployment-workflow.md:148`; branch name - `dc-dc-stage3-phase2-dc-substrate`; runbook - `runbooks/dc-dc-phase2-tofu-dc-substrate.md`). Stages 0-2 precede it; - stages 4-7 are authored, not executed (workflow doc:873). + (`docs/dc-dc-deployment-workflow.md:148`; runbook + `runbooks/dc-dc-phase2-tofu-dc-substrate.md`) -- **CLOSED 2026-07-21 + for its vr1-dc0 scope** (operator-ruled "Close and merge"; dc1 is + the stage's designed HELD remainder, gate G12). Close-out set: + gauntlet ALL GREEN + repo-lint 0-fail + this consolidation commit + + GA-R7 memory review + merge of `dc-dc-stage3-phase2-dc-substrate` + to `main` (merge commit) + branch retirement; stage record + `docs/archive/stage-records/vr1-stage3-record.md`. Stages 0-2 + precede it; stages 4-7 are authored, not executed (workflow + doc:873). The NEXT stage branches off post-merge `main`. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/archive/changelogs/changelog-20260719-dc0-deploy-stepB.md b/docs/archive/changelogs/changelog-20260719-dc0-deploy-stepB.md new file mode 100644 index 0000000..c039ff1 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260719-dc0-deploy-stepB.md @@ -0,0 +1,838 @@ +# Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap) + +Session changelog (GA-R2/D1: ONE per session). Logged window: `dc0-deploy` +(same-day append, per the A/B-boundary handoff). Status lives ONLY in +`docs/CURRENT-STATE.md`; this file is narrative + reverts. + +## 1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124) + +- **Question as presented (2026-07-19):** "Who should hold 172.31.0.1 (the + Office1 end of the dc0 transit /30), and by what mechanism? This is the + prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS + rack-to-region enrollment path." +- **Operator answer, exact utterance (option selected):** "voffice1 via IaC + + restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to + `module "voffice1"` in the outer root; apply in a planned voffice1 restart + window (graceful shutdown first; region/NetBox/Tailscale return via proven + autostart). +- Classification: OPS mechanism decision implementing D-124's already-ruled + addressing ("region/peer .1" -- `docs/design-decisions.md:3727`); no new + D-number (GA-R3, doubt resolves DOWN). +- **Measured basis (this session, logged):** mesh-vr1-dc0-office1 (virbr7) is + ISOLATED (no ``/``) with exactly ONE attached interface + (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing + holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only + route `10.10.0.0/22 via 172.31.0.1` (d124-rack.auto.tfvars, consumed by + step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached + by `-J voffice1` (D-126 shape) -- NO vcloud host leg / site-baseleg DC row + needed for this net. + +## 2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated) + +- **What:** `opentofu/main.tf` `module "voffice1"`: `network_names` gains + `module.mesh_vr1_dc0_office1.network_name` SECOND (office1-local stays + first -- PCI order preserves enp1s0); `network_config` rewritten from the + single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4; + transit=enp2s0 static `${vr1_dc0_rack_transit_peer_ip}/${prefix}` -- same + D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init- + re-run caveats documented in-block: the seed netplan is REBUILD correctness; + the live transit config is an in-guest netplan drop-in at the gated attach + step. +- **Why:** section 1's measured gap -- step B (ssh reach + rack enrollment) + has no L3 path until the region end of the transit exists. +- **Revert:** restore `network_names = [module.office1_network.network_name]` + and the previous "en*" glob `network_config` (git revert of this commit); + if already applied, a follow-up plan shows the interface removal and the + in-guest drop-in `/etc/netplan/60-transit.yaml` is deleted. + +## 3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation + +- Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply + (1/1/1 exact, `docs/audit/outer-apply-20260719-voffice1-transit.txt`) -> + domain up with both NICs, office1-local index 0. +- **INCIDENT (found+fixed in-window):** the provider REGENERATED both NIC + MACs on the interface-list rebuild (old lan MAC `52:54:00:6a:87:e5` -> + new `52:54:00:89:e1:19`). The Kea reservation (MAC-keyed) missed and + voffice1 leased pool address `.108` (measured via edge ARP). FIX: re-keyed + reservation uuid `b12621e6-63e9-4714-9726-2e130c84e069` to the new MAC via + the D-113 API (`kea/dhcpv4/set_reservation` -> saved; + `kea/service/reconfigure` -> ok), then a detached + `networkctl reconfigure enp1s0` -> voffice1 back on `10.10.0.20`. +- **LOGGED FINDING (not executed):** `modules/cloudinit-vm` does not pin + MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed + DHCP reservations. Roosevelt-relevant; candidate module `mac` var or a + documented ops step. Also: live transit config landed as in-guest netplan + drop-in `/etc/netplan/60-transit.yaml` (narrows the lan glob to enp1s0 + + transit enp2s0 static) -- the drop-in must ALSO narrow `lan`, else the + 50-cloud-init `en*` glob claims enp2s0 (measured: first drop-in attempt + without it left enp2s0 addressless). +- **Revert:** `kea/dhcpv4/set_reservation` back to the old MAC (senseless + unless the NIC change is also reverted); delete the drop-in. + +## 4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix + +- Symptom: `172.31.0.2` unreachable; MEASURED: both rack NICs emitted ZERO + frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive + (ACPI-responsive; cloud-init completed per offline logs). +- Guest is a SEALED BOX: `modules/cloudinit-vm` domains have NO serial + console, NO qga channel, NO graphics; qga is NOT in the noble base image + and its cloud-init install failed (no egress) -- so no console, no agent, + no network. **LOGGED FINDING: add opt-in serial console (+ the D-129-style + qga channel var) to `modules/cloudinit-vm` -- observability gap bites every + future DC VM. NOT executed mid-step.** +- Offline inspection (libguestfs, operator-run, captures + `~/vvr1-dc0-offline-inspect{,2}.txt`): NIC names enp1s0/enp2s0 CORRECT + (naming trap closed); root cause = netplan `match.name` + `set-name`: + boot 1 renames don't apply (.link written after udev add) so the rendered + config matched nothing; after removing set-name, boot 2 STILL failed + because the early-boot netplan generator ran against the OLD config and + renamed the devices to mgmt/uplink before cloud-init wrote the new file + (one-boot lag, measured in syslog). Boot 3 (current config, no renames) + is the verification boot. +- FIX (this commit + applied): `network_config` drops set-name, keeps + MEASURED kernel names; `instance-id` bumped `vvr1-dc0-d123 -> -d123-r2` + (side effects noted in-block); seed replaced via explicit + `-replace=module.vvr1_dc0.libvirt_volume.seed` (the sanctioned override + of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2 + exact (`docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt`). +- CONSEQUENCE for step B: bootstrap runs with + `--transit-if enp1s0 --uplink-if enp2s0` (script defaults mgmt/uplink no + longer match reality); runbook prose naming "mgmt" needs a DOCFIX at + stage close. +- **Revert:** git revert of the netplan/instance-id edit + one more + seed `-replace` apply + reboot. + +## 4b. VERIFICATION (2026-07-20, logged window) + +- Boot 3 with the fixed config: `enp1s0 UP 172.31.0.2/30`, region route + present, `br-vr1-dc0-wan` up. +- voffice1 -> 172.31.0.2 ping 0% loss; ssh + `-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2` + works (D-126 -J shape, per-env key); rack -> region `ping 10.10.0.20` + 0% loss (the MAAS-enrollment path for the G10 bootstrap). +- Outer plan CONVERGED back to ZERO DIFF + (`docs/audit/outer-plan-20260720-postfix-converged.txt`); CURRENT-STATE + section 5 updated in the same commit. +- Follow-items (logged, not executed): D-126 DC ssh-config alias row for + vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key); + cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC + pinning; DOCFIX for runbook "mgmt/uplink" naming. + +## 4c. Step B bootstrap EXECUTED on vvr1-dc0 (2026-07-20, logged window) + +- Snap proxy set to the region maas-proxy (measured serving: HTTP 200 to + archive.ubuntu.com, CONNECT to api.snapcraft.io) -- the D-107-shaped + answer to the rack's no-egress bootstrap; apt used the same proxy via + per-invocation env. Enrollment secret piped region->rack + (`install -m600 /dev/stdin`), verified by byte-count only (32), never + in context. +- Run 1 (`docs/audit/stepB-bootstrap-20260720.txt`): maas snap 3.7/stable + installed, **rack ENROLLED to http://10.10.0.20:5240/MAAS**; FAILED at + qemu apt install on stale image package lists (404s via proxy). + Fix: `apt-get update` (proxied), re-run. +- Run 2 (`docs/audit/stepB-bootstrap-20260720-run2.txt`): EXIT 0 -- + libvirt/qemu + nested KVM on, inner pool + AppArmor, SEC-010 + FORWARD-drop loaded on enp1s0 + `sec010-fw.service` enabled + (boot-persistent), `br-vr1-dc0-wan` verified with enp2s0 enslaved. +- `--check` (`docs/audit/stepB-check-20260720.txt`): all items ok EXCEPT + opnsense base (the remaining step-B item); rackd running, enrolled. +- Region-side residue for the MAAS phase (per script exit text): enable + DHCP on metal-admin at the Office1 region naming this rack primary_rack + (step D territory, NOT done). +- **Revert:** rack enrollment: region-side `maas ... rack-controllers + delete`; SEC-010: disable sec010-fw.service + rm /etc/nftables-sec010.nft; + packages: apt/snap remove; snap proxy: `snap unset system proxy.http + proxy.https`. + +## 4d. RULING RECORD (GA-R5) -- DC edge base version (2026-07-20) + +- **Question as presented:** reuse verified 26.1 artifact vs fresh download + (first exchange); then, sharpened after the operator noted "Office1 is + running OPNsense 26.7": 26.1-base-then-upgrade vs fresh 26.7 nano base. +- **Operator answer, exact utterance (option selected):** "Fresh 26.7 nano + base". +- Consequences implemented: prep runs 26.7 -> `opnsense-26.7-nano.qcow2`; + `vr1-dc0-substrate` `opnsense_base_path` default updated (no inner state + exists; pre-apply edit); step-B `--check` passes `--opnsense-base` for + the 26.7 path. LOGGED follow-items: site-headend-install.sh's 26.1 + default + hint strings (script change + harness, stage close); the + D-112/D-113 serial/API boot path is REVALIDATED on 26.7 at the step-C + boot gate (office1 precedent: 26.1-proven, upgraded in place to 26.7). +- Mirror: `https://mirror.ams1.nl.leaseweb.net/opnsense` -- MEASURED from + the vendor's live download page this session (not memory), HEAD-verified + serving the 26.7 nano (~468 MiB). LOGGED FINDING: opnsense-prep-image.sh + performs no checksum verification of the downloaded image (same property + as the original 26.1 prep; candidate hardening). + +## 4e. Step B CLOSED: 26.7 base staged; final check EXIT 0; SEC-010 CLOSED both ends + +- OPNsense 26.7 nano prepped ON the rack (leaseweb mirror via region proxy, + 468M; python3 bz2 fallback -- bunzip2 never needed, DOCFIX-184; + `docs/audit/stepB-opnsense-prep-20260720.txt`). Inner-root + `opnsense_base_path` default -> 26.7 (validate PASS). +- Final `--check` EXIT 0, ALL items ok + (`docs/audit/stepB-check-20260720-final.txt`). +- SEC-010 voffice1-side pin (the row's "same pin follows" clause, due since + this session wired voffice1's transit leg; voffice1 runs LXD so + ip_forward=1): identical scoped nft artifact + sec010-fw.service on + enp2s0, verified loaded + persistent; rack proxy/region paths re-verified + working post-pin. **SEC-010 row CLOSED (operator ruling, utterance in the + row).** Open SEC count 8 -> 7. +- LOGGED follow-item: site-headend-install.sh region-role support for the + transit-if pin (voffice1's was hand-applied to the script's exact + artifact shape; fold into the script + harness at stage close). +- **Revert:** rm the two files + `systemctl disable sec010-fw` on voffice1; + `nft delete table inet sec010`; ledger row reopens citing this item. + +## 7. Step C: voffice1 becomes the Plane-2 executor; INNER APPLY COMPLETE (28/28) + +- voffice1 setup (gated, logged): tofu 1.12.4 via vcloud's exact apt + source (keyrings byte-copied; version == section-7 pin); repo cloned at + `29cf7bf` (branch); dc0 private key staged to `~/vr1-dc0-creds/` + (0700/0600, piped -- SEC-009 convention now spans hosts); ssh config + entry for 172.31.0.2; known_hosts seeded via ssh-keyscan with the + ED25519 fingerprint cross-checked against the accept-new-trusted entry + (MATCH). +- Inner-root fixes measured at the FIRST REAL dial/apply (each committed): + (i) provider needs `keyfile`+`sshauth=privkey` URI params -- its Go ssh + ignores ~/.ssh/config AND default identities; single-type known_hosts + fails as "key mismatch" (multi-type scan required); + (ii) `modules/wan-bridge`: `` is ILLEGAL in bridge-mode networks -- + removed (MTU belongs to the host bridge; DOCFIX-194 class: parsed, never + applied); + (iii) base-image location premise WRONG under remote provider -- content + UPLOADS from the executing host; 26.7 qcow2 copied rack->voffice1 + (`~/vr1-dc0-images/`), tfvar override + variables.tf description + corrected; + (iv) edge serial-log dir `/var/lib/libvirt/vr1/staging/` is a HARDCODED + vcloud literal in modules/opnsense-edge (:223) -- dir created on the rack + (755); LOGGED findings: parameterize the path; fold staging-dir creation + into site-headend-install --host-nodes; opentofu-validate.sh does not + cover the inner root. +- Applies (captures `docs/audit/inner-{plan,apply}-20260720-stepC*.txt`): + run 1 = 25/28 (nodes+planes+pool; wan-bridge mtu abort); run 2 = wan + bridge (edge disk stat abort); run 3 = edge disk (staging-dir abort; + provider self-cleaned the failed domain -- operator undefine found + nothing); run 4 = edge domain, EXIT 0. CONVERGED: inner plan zero diff + (`docs/audit/inner-converge-20260720-stepC.txt`). +- VERIFIED: 10/10 domains RUNNING inside vvr1-dc0 (9 nodes depth-4 + edge); + edge serial log = FreeBSD login prompt on ttyu0 -- 26.7 boots the D-112 + serial path (first revalidation datapoint; full D-112(c) bootstrap is the + next window). +- NEW STATE OF RECORD: the inner tfstate lives on voffice1 + (`~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/terraform.tfstate`) + -- back it up with the site; follow-item for the backup set. +- **Revert:** `tofu destroy` in the inner root from voffice1 (site-down + alternative: `virsh destroy vvr1-dc0`, D-122); module fixes revert by + commit. + +## 9. DC edge: D-112(c) console bootstrap + API key minted (26.7) -- REPLACEMENT CHAIN PROVEN + +- Per-env edge key minted on vcloud (`~/vr1-dc0-creds/vr1-dc0-edge_ed25519`, + 0600/0644, SEC-009 convention), staged to the rack the same way. +- Rack legs added (operator-ruled "10.12.4.2, manual now + script later"): + `10.12.4.2/22` on the provider-public inner bridge (virbr5, MEASURED) + + the D-124-ruled `10.12.8.2/22` on metal-admin (virbr2). INTERIM: `ip addr + add`, not persistent. **LOGGED FINDING (queued):** site-headend-install.sh + never implemented the rack legs despite main.tf's comment claiming it + consumes `vr1_dc0_rack_metal_admin_ip` -- add rack-leg support + harness + before step D closes. Also interim: `192.168.1.2/24` on virbr5 to reach + the edge's FACTORY LAN; REMOVE once the edge is re-addressed. +- **D-112(c) console bootstrap: DONE** (`~/d112c-console.log` on the rack). + Driver evolved v1->v6 against MEASURED failures, each real and worth + keeping for the runbook: + (i) serial console wraps long lines -> ship payload as <=160-char base64 + chunks (`openssl base64 -d`), never one long line; + (ii) `set-name` in the edge netplan is irrelevant here but the same class + bit the rack (item 4); + (iii) tcsh on the edge (documented trap) breaks nested-quoted `sed` -- + "Unmatched '"'"'" -- so NEVER patch files via quoted one-liners; re-ship + the whole payload instead; + (iv) `write_config()` needs an include beyond config.inc -- MEASURED + (`grep -l shell_safe /usr/local/etc/inc/*.inc` -> auth/console/filter/ + interfaces/rrd/system/util.inc), not guessed; + (v) writing config.xml does NOT materialize `/root/.ssh/authorized_keys` + -- must call the vendor's `local_user_set()` (auth.inc). This was the + actual reason key auth failed after a "successful" config write. +- VERIFIED: `ssh -i vr1-dc0-edge_ed25519 root@192.168.1.1` -> key-only auth + OK, `uname -r` = 15.1-RELEASE-p1, `ifconfig -l` = vtnet0/vtnet1. + INTERFACE MAPPING MEASURED: vtnet0 = LAN = provider-public plane (it + answered on virbr5), vtnet1 = WAN -- same order as the office1 edge. +- **API key minted via the vendor model** (`opnsense-bootstrap-apikey.sh` + + `opnsense-mint-apikey.php`, run ON the rack): key/secret 80 chars -> + `~/vr1-dc0-creds/opnsense-api.txt` (0600, secret never printed). + SMOKE TEST PASS: `GET core/firmware/status` -> HTTP 200, `CORE_ABI 26.7`. + **This is the first proof the D-113(a2) API path works on 26.7** (the + 26.7 revalidation the operator's base-version ruling called for). + Script note: the bootstrap script requires the minter BESIDE it (its + own dir), not in `scripts/` -- worth a usage-line DOCFIX. +- **Revert:** delete the API key (`POST auth/user/del_api_key/`, the + proven negative-tested path) + creds file; `ip addr del` the three + interim addresses; the console bootstrap reverts by re-imaging the edge + (inner `tofu destroy`/re-apply of the edge module). + +## 10. D-125 egress isolation gate: EXECUTED as written -- PARTIAL (bridge-in PROVEN, +## upstream NAT egress FAILS). Gate NOT closed; double-NAT fallback NOT invoked. + +- Method: operator-ruled "throwaway VM, as written" -- noble cloud image + fetched ON the rack via the region proxy, one cloud-init guest, SINGLE + NIC bridged to `br-vr1-dc0-wan`, serial-log probe, then destroyed + (`leftover-domains=0` asserted). Script: scratchpad + `d125-egress-test.sh`; capture `docs/audit/d125-egress-gate-20260720.txt`. +- Run 1 was a TEST ARTIFACT, not a gate failure: the probe used DHCP and + came up address-less. MEASURED cause: `vr1-dc0-uplink` carries an `` + but NO `` block -- there is no DHCP server on that /24 by design + (the edge WAN takes a STATIC .2, D-113/D-125). **DOCFIX QUEUED:** the + gate's wording "must GET a vcloud-ISP address" implies DHCP; it must say + STATIC, or the /24 needs a range. Re-run used static 172.30.2.50/24. +- RESULT (run 2): guest configured 172.30.2.50/24 + default via 172.30.2.1; + **gateway ping 3/3, 0% loss** -- a FOREIGN MAC inside vvr1-dc0 reaches + the vcloud ISP NAT across the nested bridge. That is the D-125 bridge-in + property, and it PASSES. But `ping 1.1.1.1` FAILED (NET-PING-RC=1). +- **CORRECTION (2026-07-20, same-day): the rack-bridge follow-up is + CONFOUNDED and is NOT evidence.** It sourced from 172.30.2.51 on the + rack, but the rack has NO DEFAULT ROUTE (netplan gives it only + `10.10.0.0/22 via 172.31.0.1`; the WAN bridge is IP-less by design), so + traffic to 1.1.1.1 fails in the rack's own routing table before the NAT + is ever consulted. Its ICMP+TCP failures say nothing about virbr4. The + ONLY clean signal is the GUEST's (172.30.2.50) failed egress, and that + guest tested ICMP only -- if the rule dump comes back identical for both + bridges, re-run the guest probe with `curl` before concluding anything. + (Recorded per GA-R1/C2: a wrong measurement is corrected in the record, + not quietly dropped.) +- Controls: vcloud egress 0% loss; the office1 edge + egresses through the SAME-SHAPE `office1-wan` NAT 0% loss. The two + network definitions are byte-equivalent in substance (`net-dumpxml`: + both `forward mode=nat` + ``; only netmask-vs- + prefix notation differs) -- so the delta is in vcloud's LIVE firewall + rules for virbr4, NOT in the design or the config. +- **GATE STATUS: PASS / CLOSED 2026-07-20** (runs 3+4, capture + `docs/audit/d125-egress-gate-20260720-matrix.txt`). The "missing rule" + hypothesis was DISPROVED by measurement first: the operator-run dumps + show virbr4's rule set is structurally IDENTICAL to the WORKING virbr11 + (only counters differ; virbr11's one extra rule is a DHCP checksum + fixup it has because it carries a DHCP range). All three masquerade + rules exist for 172.30.2.0/24 INCLUDING the generic (non-tcp/udp) one, + and it had FIRED (1 packet/84 bytes = one ICMP conntrack entry), with + FORWARD accept counting 136 packets. So `net-destroy/net-start` would + have fixed nothing -- the pre-staged fix was correctly NOT run. +- Decisive re-probe used a protocol/destination MATRIX instead of counter + archaeology. Result, IDENTICAL on two consecutive runs: gateway ping 0, + **internet ping 0**, `curl http://1.1.1.1` -> 301, `curl + archive.ubuntu.com` -> 200. Egress works for ICMP AND TCP to real + hosts, so the D-125 bridge-in chain is PROVEN end to end and the + double-NAT fallback is NOT needed. +- **Run-2 anomaly, recorded UNEXPLAINED (not swept):** the same path + failed ICMP-to-internet once while its gateway ping passed 3/3. Runs 3+4 + passed identically with no intervening change to the uplink or its + rules. Most likely a first-use conntrack/ARP transient on vcloud. It is + not reproducible now; if a DC edge ever shows first-boot egress failure, + START HERE rather than re-deriving. +- **Security-positive side observation** (extra probe, outside the gate): + the guest could NOT reach the Office1 management LAN through the ISP NAT + (`10.10.0.20:8000` -> 000) on all runs, while internet egress worked. + That is the D-122/SEC-010 intent holding: the DC's WAN segment is an ISP + uplink, not a path into Office1. Worth an explicit assertion in the + hardening tests rather than leaving it an incidental observation. +- Prior status line (superseded, kept for the audit trail): gate was held + OPEN pending measurement rather than FAILED. The runbook's FAIL remedy + (revert to + double-NAT) is for a DESIGN failure; invoking it against what is very + likely a host firewall-rule gap would be the wrong remedy and would add + Roosevelt delta. Next step is a read-only measurement needing operator + sudo: FULL dumps of BOTH `nft list ruleset` and `iptables-save` (which + backend is authoritative here is itself unmeasured -- do not assume), + diffing virbr4 against the WORKING virbr11. Two rule families must both + be checked, and a subnet-only grep would MISS the second: POSTROUTING + masquerade (keyed on `172.30.2.0/24`) and FORWARD accept (keyed on the + INTERFACE `virbr4`). Pre-staged fix-forward IF the rules are absent or + malformed vs virbr11: `virsh net-destroy vr1-dc0-uplink && virsh + net-start vr1-dc0-uplink` reinstalls libvirt's rules from scratch -- + cheap and safe (nothing rides that uplink yet but the probe). If instead + the dump shows libvirt never installed them correctly, the restart will + not help and the dig moves to the firewall backend / conntrack. +- **Revert:** nothing to revert -- the probe guest and both temporary + addresses were removed; the only residue is `/tmp/noble.img` on the rack + (kept deliberately: it is the base for any future throwaway probe). + +## 11. NEW TOOLING + edge re-addressed: opnsense-set-interface-v4 pair (53/53) + +- **RULING RECORD (GA-R5), 2026-07-20.** Question as presented: base-iface + IPv4 is not REST-covered on 26.7 (measured); how should LAN 10.12.4.1/22 + and WAN 172.30.2.2/24 + gw 172.30.2.1 be set -- build the v4 setter pair + / console menu now / console now + build at stage close? Operator answer, + exact utterance (option selected): "Build the v4 setter pair + (Recommended)". Classification: OPS tooling under the existing D-113 + amendment; no new D-number (GA-R3). +- **D-113 AMENDMENT RE-MEASURED ON 26.7** (feeds the item-8 review): the + amendment was measured on 26.1 only. On this 26.7 edge: + `interfaces/settings/get` returns ONLY global offload flags, while + `interfaces/overview/list` and `interfaces/lan/get` both return 404 + "Endpoint not found". Base-interface addressing is still legacy + `/interfaces.php`. The amendment now holds on TWO majors, measured. +- SHIPPED: `scripts/opnsense-set-iface-v4.php` + + `scripts/opnsense-set-interface-v4.sh` + `tests/opnsense-set-interface-v4/` + (53/53 PASS first run; gauntlet 70 -> **71 harnesses ALL GREEN**; + repo-lint 0 fail). Mirrors the proven v6 sibling exactly -- vendor Config + singleton (never a config.xml push), dry-by-default, interface-count + guard, fresh read-back verify, `sh -s` for the tcsh trap, KERNEL as + ground truth. Two deliberate improvements over the v6 sibling: + (i) the GATEWAY half (creates/updates at most ONE `gateway_item`, + count-guarded exactly like the interface count, opt-in via a 5th arg); + (ii) the interface's DEVICE is MEASURED from the edge's own config + instead of the v6 sibling's hardcoded `lan->vtnet0 / wan->vtnet1` sed -- + that mapping is per-edge and hardcoding it is a hard-rule-2 violation + waiting to happen. Harness pins BOTH (test 12 fails if the hardcode + returns). **LOGGED FINDING:** backport the measured-device fix to the v6 + sibling at stage close. +- APPLIED to vr1-dc0-opnsense, WAN first per the script's own ordering + warning (re-addressing the LAN drops the session you arrived on): + - WAN: dhcp -> `172.30.2.2/24` + `WAN_GW` = 172.30.2.1 default. Note the + edge had shipped with WAN on **dhcp**, which could NEVER have worked -- + the uplink /24 has no DHCP server (same fact that broke probe run 1). + Verified on the kernel: `inet 172.30.2.2`, `default 172.30.2.1 UGS + vtnet1`, and **the EDGE ITSELF pings 1.1.1.1 at 0% loss** -- the D-125 + chain now proven through the real OPNsense edge, not just a probe. + - LAN: `192.168.1.1/24` -> `10.12.4.1/22` (the D-115/lib-net ruled + provider-public gateway). Session dropped as designed; edge verified at + the new address from the rack: `inet 10.12.4.1 netmask 0xfffffc00`. + - Post-move reconciliation at the NEW address: API `GET + core/firmware/status` -> 200 `CORE_ABI 26.7`; edge egress still 0% loss. +- CLEANUP: the interim `192.168.1.2/24` on the rack's provider-public + bridge is REMOVED (virbr5 now carries only the ruled `10.12.4.2/22`). +- **Revert:** re-run the setter with the previous values + (`--commit lan 192.168.1.1 24`, `--commit wan ...`); the + gateway item is named `WAN_GW` and is the only one this tool created. + +## 12. D-129 edge profile APPLIED on 26.7 + qga channel shipped + a FALSE-SUCCESS bug fixed + +- **RULING RECORD (GA-R5), 2026-07-20.** Question: the qga virtio channel is + absent from the edge domain and unsupported by `modules/opnsense-edge` + (both MEASURED), so installing os-qemu-guest-agent alone yields a package + that can never answer. Options presented: add channel var now / iperf only + + defer / install both and leave qga inert. Operator answer, exact + utterance (option selected): "Add channel var now, then install both + (Recommended)". [OPS under D-129; no new D-number, GA-R3.] +- **SHIPPED `expose_qga_channel`** (opt-in, DEFAULT OFF) in + `modules/opnsense-edge`; `vr1-dc0-substrate` sets it true so NEW DC edges + are BORN with the channel (the runbook's no-retrofit intent). Default OFF + is load-bearing: a device change alters the domain, so defaulting ON would + bounce every existing edge on the next plan. Shape taken from + `tofu providers schema -json` (dmacvicar 0.9.8: + `devices.channels[].target.virt_io{name}` + `source.unix{mode}`), NOT from + a blog; `source.unix` carries no path, letting libvirt allocate the socket + (same reasoning the serial block documents for pty). +- APPLY was SAFE: plan showed **0 add / 1 change / 0 destroy -- an IN-PLACE + domain update**, so `libvirt_volume.disk` (holding the console + API + bootstrap) was never at risk. Verified after: channel present (grep count + 2), domain running, edge still reachable at 10.12.4.1. +- **BUG FOUND ON THE FIRST-EVER LIVE RUN -- `opnsense-plugins.sh apply` had + NEVER installed anything.** `api()` used `bash "$API" ${DRY_RUN:+--dry-run}`: + `${VAR:+word}` expands when VAR is set and NON-EMPTY, and `DRY_RUN=0` is + non-empty -- so every apply silently dry-ran while printing + "OK: profile 'vr1-edge' applied". A textbook false success, and the reason + it survived review is that the harness drove ONLY `--dry-run`, leaving the + live path with zero coverage. FIX tests the VALUE; harness gains two + live-path tests against a stub api client, **verified to FAIL 2/2 on the + buggy version before restoring** (a regression test that cannot fail is + worthless). 20/20. This is exactly the class D-129's own "package present + is not success" warning is about -- here the SCRIPT was the false reporter. +- REAL install then ran: `{"status":"ok","msg_uuid":...}` per plugin (not + dry-run echoes), capture `docs/audit/d129-plugins-20260720.txt` (the + earlier capture of the same name recorded the false dry-run and is + REPLACED). Measured on the edge: `os-qemu-guest-agent-1.3`, + `qemu-guest-agent-11.0.1`, `os-iperf-1.0_2`, `iperf3-3.21`. +- Service was NOT auto-started (the script's own NOTE); started via the + plugin's OWN API (`POST qemuguestagent/service/start` -> `{"response":"OK"}`, + status -> running) rather than by hand-editing rc.conf. +- **GATE MET, to the runbook's exact bar (not package-presence):** + `virsh qemu-agent-command vr1-dc0-opnsense '{"execute":"guest-ping"}'` -> + `{"return":{}}`, and `virsh domifaddr --source agent` reports vtnet0 + 10.12.4.1/22 + vtnet1 172.30.2.2/24. The agent ANSWERS over the channel. +- **SECURITY FINDING (queued for the item-8 hardening review):** the plugin + model exposes `DisabledRPCs`, including **`guest-exec`/`guest-exec-status` + -- arbitrary command execution on the edge from the libvirt host**. On a + commercial multi-tenant cloud the DC edge should almost certainly disable + the exec RPCs (we need guest-ping/fsfreeze/shutdown coordination, not + remote exec). Not changed mid-step; propose with options at the review. +- **Revert:** `POST qemuguestagent/service/stop`; uninstall via + `core/firmware/remove/`; set `expose_qga_channel = false` and re-apply + (in-place domain update again). + +## 13. Step D part 1 DONE (DHCP configured) -- then INCIDENT: MAAS DHCP is DOWN +## REGION-WIDE (Temporal wedged). Not caused by this deploy. + +Step-D work completed before the incident surfaced: +- Both controllers confirmed registered: `voffice1` (region, mtstwf) and + **`vvr1-dc0` (rack, 7chphy, rackd running)**. +- MAAS auto-discovered the inner planes from the rack's interfaces: + metal-admin `10.12.8.0/22` = VLAN 5005/fabric-4, provider-public + `10.12.4.0/22` = VLAN 5006/fabric-5. (Stale `192.168.1.0/24` also sits on + fabric-5 -- the edge's old factory LAN, now vacated; cleanup queued.) +- **RULING RECORD (GA-R5), 2026-07-20.** Question: metal-admin is a /22 but + D-120's band layout is written for a compose /24; what dynamic range? + Operator answer, exact utterance (option selected): "Inherit D-120 band: + 10.12.8.100-10.12.8.200 (Recommended)". Range CREATED and verified on the + correct subnet; VLAN 5005 updated `dhcp_on=true primary_rack=7chphy`, + verified by read-back (`10.12.8.0/22 dhcp=True primary=7chphy`). + +**INCIDENT (found by verification, not by a self-report):** +- Canary `virsh reset vr1-dc0-control-01` did NOT enlist. MEASURED cause + chain: rack service_set shows `dhcpd: off`; no dhcpd process on the rack; + no `/var/snap/maas/common/maas/dhcpd.conf` generated -- while MAAS's own + API reports the VLAN as `dhcp_on=True` with the correct primary rack and + the interface correctly linked (VLAN 5005, static 10.12.8.2). Config + state and running state DISAGREE. +- Root cause, from the region's journal: MAAS 3.7 drives DHCP through + **Temporal workflows**, and Temporal is wedged -- + `"Not enough hosts to serve the request"`, `poll_workflow_task_queue + retried 2807 times`, task queue `mtstwf@region:...`. No worker is polling, + so the workflow that pushes DHCP config to a rack can never run. +- **SCOPE IS WIDER THAN THIS DEPLOY:** `voffice1` itself has **NO dhcpd + process** (`pgrep -c dhcpd` -> 0) and reports no dhcpd service, while + Office1's compose net `10.10.1.0/24` reads `dhcp=True`. So MAAS DHCP has + been DOWN REGION-WIDE -- almost certainly since the 2026-07-17 host reboot + -- and went unnoticed because office1-netbox/office1-tailscale were + already Deployed and kept their addresses. **This is NOT caused by the DC0 + deploy; the deploy is what exposed it.** +- Consequence for the as-built record: any claim that Office1 MAAS DHCP is + "working" is currently FALSE (last positive evidence predates the reboot). +- **Detection gap (queued):** `cloud-assert.sh` and the Office1 health + checks did not catch a dead DHCP server, because MAAS's own service_set + reported the region healthy. A behavioral DHCP assertion belongs in the + health sweep -- this is precisely the "service's own verdict" principle + the repo already applies elsewhere. +- No prior art: appendix-A has no entry for this symptom. NEW incident + material once the fix is known. +**RESOLVED 2026-07-20 (operator-approved: "Go ahead with the restart").** +- `sudo snap restart maas` on voffice1 -> region back (HTTP 301 on + localhost:5240). Recovery verified BEHAVIORALLY, not by service status: + `pgrep -c dhcpd` on voffice1 0 -> **1**, and on the DC rack 0 -> **1**; + MAAS then reported `dhcpd: running` on BOTH controllers (was `off` on the + rack, absent entirely on the region). One restart fixed both sites, which + confirms the single wedged-Temporal root cause rather than two faults. +- **PROOF IT WAS THE REAL BLOCKER:** the identical canary that had failed + before (`virsh reset vr1-dc0-control-01`) enlisted immediately after. +- APPENDIX-A MATERIAL (queued, new incident class): symptom "MAAS says + dhcp_on=True but no dhcpd runs and nothing PXEs"; check `pgrep -c dhcpd` + on each controller and the region journal for Temporal + `"Not enough hosts to serve the request"`; remedy `snap restart maas`; + verify behaviorally on every controller, never by service_set. + +## 14. G10 DEPTH-4 NESTED BOOT GATE: **PASS** -- all 9 DC0 nodes enlisted in MAAS + +- After the DHCP fix, all nine node VMs were reset and **all nine enlisted** + (`docs/audit/stepD-enlistment-20260720.txt`): count 9, status 8 + Commissioning + 1 New, shapes `{16cpu/64GiB: 3, 12cpu/48GiB: 2, + 8cpu/24GiB: 4}` -- an EXACT match to the D-121 Option C / R-3 layout + (3 control + 2 compute + 4 storage). +- **This closes the D-114/D-123 depth-4 boot gate, the flagged-risk item of + Model B.** A node VM inside `vvr1-dc0` (itself a VM on vcloud) PXE-booted + from the Office1 region across the D-124 transit and is running MAAS's + ephemeral commissioning kernel -- i.e. real nested KVM at depth 4, proven + by behavior rather than by inference from `svm` being present. +- Still open in step D: commissioning completes with a power-off, and MAAS + has no power driver for these VMs yet -- the `maas-vm-host` registration + (module BUILT 2026-07-09, still UNVALIDATED) is the next piece, and it + needs `TF_VAR_maas_api_key`. +- **Revert:** `maas admin machines delete` per machine (or delete the pod + once registered); the VMs themselves are inner-root state. + +- PROPOSED REMEDY (superseded by the RESOLVED block above; kept for the + audit trail): restart MAAS on the region + (`sudo snap restart maas` on voffice1) to re-establish Temporal workers, + then re-verify dhcpd on BOTH controllers behaviorally. Blast radius is + low right now (no OpenStack workloads; both Office1 VMs already deployed + and unaffected by a brief DHCP/API outage), which makes this the cheapest + moment to fix it. + +## 15. Step D part 2: NEW root for MAAS registration; and a MEASURED REFUTATION +## of D-123's "power_address = qemu:///system LOCAL to the rack" + +- SHIPPED `opentofu/vr1-dc0-maas/` -- a THIRD root holding the MAAS + registration alone. Rationale is DOCFIX-179's own measured lesson applied + rather than repeated: a `provider "maas"` block forces EVERY plan in its + root to demand `maas_api_url` + the SENSITIVE key, even for plans creating + zero MAAS resources (the DOCFIX-175 plaintext-secret surface). Putting it + in the inner substrate root would make routine substrate work require MAAS + creds. Substrate roots stay credential-free. Validated explicitly -- + **note `opentofu-validate.sh` covers only the outer root + modules, so + neither extra root is gauntlet-covered; extending it stays QUEUED and is + now more load-bearing (two uncovered roots).** +- Secret handling: key supplied ONLY as `TF_VAR_maas_api_key` from the + operator-placed 0600 file; verified by FORMAT alone (71 bytes, 3 + colon-separated parts), never printed. Caveat recorded in the root's + header: the key DOES land in that root's tfstate (unavoidable with this + provider), so that state inherits credential handling. +- Plan was clean (1 to add). **APPLY FAILED:** + `ServerError: 503 ... Failed talking to pod: Failed to login to virsh + console.` +- **ROOT CAUSE, MEASURED (not inferred):** the MAAS snap is strictly + confined. From inside it, `virsh -c qemu:///system` -> + `Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission + denied`. And `snap connections maas` lists **NO libvirt interface at all** + -- there is nothing to `snap connect`. So a LOCAL `qemu:///system` VM host + is ARCHITECTURALLY IMPOSSIBLE with snap-based MAAS. +- **THIS REFUTES A DOCUMENTED DESIGN LINE.** D-123 Model B and + `modules/maas-vm-host`'s header both state the DC pod is registered with + `power_address = qemu:///system` LOCAL to the rack ("vvr1-dc0 is BOTH the + MAAS rack AND the libvirt host, so no cross-fiber virsh dial"). The + no-cross-fiber INTENT survives; the stated MECHANISM cannot work. Both + surfaces need an amendment once the replacement is ruled. Roosevelt- + relevant: the same constraint hits every DC pod and any snap-MAAS metal + build, so this is not a VR1 quirk. +- Feasibility of the replacement, MEASURED: the snap DOES ship an ssh client + (`/snap/maas/41649/usr/bin/ssh`), and `jessea123` on the rack is in the + `libvirt` group (113) -- so `qemu+ssh://jessea123@172.31.0.2/system` + reaches libvirt through sshd (unconfined) instead of the blocked socket. + It needs a key the snap's root can read (`/var/snap/maas/current/root/.ssh/` + does not exist yet) plus that pubkey in the rack user's authorized_keys. +- **Revert:** the failed apply created nothing (state empty); `git rm -r + opentofu/vr1-dc0-maas` removes the root. + +## 16. The pod path, chased to its end -- and a SECOND refutation: MAAS virsh +## pods are incompatible with this repo's node-vm disks + +Sequence, each step measured (operator ruled the DEDICATED key: "Dedicated +MAAS->libvirt key (Recommended)"): +1. Minted `maas-virsh_ed25519` on the rack (0600, SEC-009 folder), pubkey into + the rack user's authorized_keys, private half into the RACK's MAAS snap. + Rack snap could then `virsh -c qemu+ssh://...` fine -- **but the apply still + failed the same way.** +2. MEASURED why: MAAS attempts the pod connection from the **REGION**, not the + rack. voffice1's snap had no key and no known_hosts -> `Host key + verification failed`. Same key + keyscan installed there; region snap then + listed the inner domains. (Worth keeping: for a virsh pod, the credential + belongs wherever MAAS dials FROM, which is not necessarily the rack that + owns the hardware.) +3. Next apply hit `Pod vr1-dc0-inner ... already exist` -- the two earlier 503s + had EACH created a pod server-side before erroring, so MAAS held two broken + `cores=0` pods outside tofu state. Deleted both (own residue), re-applied. +4. Next apply reached the storage inventory and failed: + `Virsh command ['domblkinfo','vr1-dc0-compute-01','vda'] failed: internal + error: missing storage backend for 'volume' storage`. +- **ROOT CAUSE, reproduced LOCALLY on the rack** (so not a snap/SSH artifact): + `virsh domblkinfo vr1-dc0-compute-01 vda` fails identically at the console + with an ACTIVE, autostart pool. libvirtd cannot run domblkinfo against disks + declared as pool+volume references -- which is exactly how `modules/node-vm` + builds them. **MAAS virsh pods are therefore incompatible with this repo's + node-vm disk shape**; making the pod work would mean converting node-vm to + file-path disks and re-applying all nine domains. +- **BUT THE POD IS UNNECESSARY.** Its designed job (D-103) was DISCOVERY, and + discovery already happened via PXE enlistment. All MAAS still needs is + POWER. MEASURED, working: per-machine `power_type=virsh` + + `power_parameters_power_{address,id}` on the canary (`ace-oyster` = + `vr1-dc0-control-01`, matched by MAC) -> `maas admin machine + query-power-state` returns `{"state": "off"}`. No pod, no domblkinfo, no + module change, no touching nine live domains. +- Roosevelt note that makes this more than expedience: on bare metal each + machine carries its OWN power parameters (IPMI/BMC). Per-machine power is + the SHAPE Roosevelt uses; the virsh pod is a VR1-only convenience. So the + per-machine route is the smaller Roosevelt delta as well as the smaller + change here. +- Both routes are recorded for the operator's ruling; nothing further applied. + +## 17. STEP D COMPLETE: per-machine power shipped as tooling; nodes COMMISSIONING to Ready + +- **RULING RECORD (GA-R5), 2026-07-20.** Question: pods are incompatible with + node-vm's pool+volume disks (measured) and the pod's discovery role is + already satisfied by PXE -- per-machine virsh power, or convert node-vm to + file-path disks and keep the pod? Operator answer, exact utterance (option + selected): "Per-machine virsh power (Recommended)". [Amends D-103/D-123's + stated vm-host mechanism; amendment text queued for stage close.] +- SHIPPED `scripts/maas-node-power.sh` + `tests/maas-node-power/` (24/24 PASS + first run; **gauntlet 71 -> 72 harnesses ALL GREEN**; repo-lint 0 fail) -- + delivered as TOOLING rather than nine hand-typed CLI calls so dc1 and any + re-run are repeatable. Design points the harness pins: + (i) matching is **BY MAC, never by name** -- MAAS assigns random hostnames + at enlistment (`ace-oyster`, not `vr1-dc0-control-01`), so a name match + would map nothing or, worse, the wrong node; + (ii) dry by default; + (iii) every write is verified by an actual `query-power-state` -- a stored + power parameter proves nothing about whether MAAS can REACH the hypervisor; + (iv) the measured pod-refutation reasoning travels in the script header. +- Prereqs measured and fixed on the region: it had **no `virsh`** (it is the + MAAS/LXD host, not a libvirt host) -> installed `libvirt-clients`; and the + `maas` CLI profile was root-only -> created a user profile with `maas login + ... -` reading the key from STDIN (never printed, never in argv). +- LIVE RESULT (`docs/audit/stepD-power-20260720.txt`): 10 domains found, **9 + mapped and set, each confirmed `power state: off` by query**. The 10th + (the edge) is correctly unmapped -- it is not a MAAS machine. +- **COMMISSIONING NOW WORKS END TO END:** all 9 commissioned; at the time of + writing **3 Ready, 6 Commissioning**, shapes still exactly D-121 Option C. + `Ready` means MAAS commissioned the node and it is deployable -- the whole + point of step D. The remaining six finish unattended. +- **Revert:** `maas admin machine update power_type=` (clears power) per + machine; the script itself reverts by commit. + +## 18. OPEN: 6 of 9 nodes fail commissioning ("timed out"); NOT contention; blocked +## on node observability. Step E not started. + +- After power was wired, all 9 were commissioned at once: **3 reached Ready, + 6 hit MAAS's 30-minute timeout** (`Marking node failed - Node operation + 'Commissioning' timed out after 30 minutes`). +- FIRST HYPOTHESIS (resource contention -- 9 nodes x depth-4 nested, 384 GiB + of 416 GiB, all pulling ephemeral images through one /30 transit) was + TESTED AND REFUTED: a single failed node re-commissioned ALONE sat in + `Loading ephemeral` for 20+ minutes with the rack at **385 GiB free and + load 0.04**. It is not contention and not memory. +- Also RULED OUT by measurement: rack boot-image sync (`list-boot-images` + -> `status: synced`, 10 images), and DHCP itself (these same nodes + enlisted fine over PXE earlier, which is what proved the depth-4 gate). +- **BLOCKER IS OBSERVABILITY, and it is the same gap twice:** + `modules/node-vm` defines **no serial console** (confirmed in the module + and in the live domain XML -- no serial/console devices), so a node stuck + in `Loading ephemeral` is a sealed box: no console, no agent, no way to + see the boot. This is the identical finding already logged against + `modules/cloudinit-vm` (item 4), now proven to bite a second module and an + actual live diagnosis. **The fix is the same opt-in serial+log pattern + `modules/opnsense-edge` already carries** -- that module's console log is + what made the 2026-07-12 boot bugs legible, and it is exactly what is + missing here. +- NOT DONE, deliberately: adding a console to node-vm means re-applying + domains mid-diagnosis, which is a change to a built surface and belongs to + an operator ruling, not a reflex. +- Current fleet: 3 Ready (commissioned, deployable), 6 Failed/retrying. + Nothing is lost -- failed commissioning is re-runnable at will. +- **Step E (netem) NOT STARTED.** Measured prerequisites for when it is: + the dc0<->dc1 mesh is **virbr5 ON VCLOUD** (note: virbr5 INSIDE the rack + is provider-public -- same number, different host, measured not assumed); + its qdisc is currently default (`noqueue`). D-100 rules the parameters as + a PLACEHOLDER for the rehearsal within a "same-metro dark-fiber, low + single-digit ms" lean. Applying to the dc0<->dc1 leg is both the + semantically correct link for D-100 and the SAFE one -- it carries no + traffic (dc1 does not exist), unlike the transit, which is currently + load-bearing for MAAS/PXE. **`modules/netem-link` cannot run unattended + from here: it shells out to `ssh 'sudo tc ...'` and assumes + PASSWORDLESS SUDO, which vcloud's operator account does not have** -- a + documented module assumption that this host violates. + +## 19. node-vm console SHIPPED; commissioning narrowed but UNRESOLVED -- and two +## of my own experiments were INVALID (disclosed) + +- SHIPPED (operator-ruled): `modules/node-vm` now defines a serial console + + boot log, with the log directory PARAMETERIZED (`serial_log_dir`) rather + than opnsense-edge's hardcoded vcloud literal, which is wrong on any other + libvirt host. Applied to all 9 domains: **0 add / 9 change / 0 destroy**, + in-place, nothing destroyed. validate PASS, repo-lint 0 fail. +- **The console did NOT yield the hoped-for evidence, and that is itself a + finding:** the logs stay 0 bytes even after a clean domain restart. Cause: + SeaBIOS/PXE firmware writes to the VGA console, not serial, and MAAS's + ephemeral kernel is not told `console=ttyS0`. **A serial console alone does + not make a PXE-booting node observable** -- seeing the firmware/ephemeral + stage needs a graphics device + `virsh screenshot` (or a MAAS-side kernel + cmdline change). Queue this as the correction to the node-vm console work; + do not assume the console added today closes the observability gap. + +WHAT IS SOLIDLY ESTABLISHED (all measured): +- PXE works and the ephemeral handoff happens: event log shows `Performing + PXE boot` 18:17:03 -> `Loading ephemeral` 18:17:45 -> silence -> timeout. +- The ephemeral OS BOOTS AND HAS WORKING NETWORKING: tcpdump on metal-admin + caught 5-6 nodes holding DHCP leases (10.12.8.112-.117) and exchanging NTP + with the rack. So node->rack reachability is fine. +- No traffic toward the REGION (10.10.0.20) or the metadata ports (5248/5240) + was seen in the sampled windows. SEC-010 was suspected of blocking the + node->region path, but its rules carry no `counter`, so drops cannot be + read directly -- **hypothesis UNCONFIRMED, not refuted.** +- Ruled out by measurement: memory (385 GiB free), rack boot-image sync + (`synced`, 10 images), DHCP (leases work), and node SHAPE (Ready and + Failed both span 16/64, 12/48 and 8/24). + +**INVALID EXPERIMENTS -- MINE, RECORDED SO NOBODY TRUSTS THEM:** +1. The first "commission a single node ALONE" test was NOT alone: tcpdump + later proved 5-6 other nodes were still up and running ephemeral images. + My conclusion "contention is refuted" was therefore UNSOUND. Contention + remains a live hypothesis. +2. The second attempt was also invalid: MAAS had the machine in + `Commissioning` already, so the new request did not restart the 30-minute + timer (the failure at 19:24:51 belongs to the 18:54 attempt), and I then + destroyed domains mid-commission, killing the in-flight run. + STOPPED HERE rather than iterate again -- this is the reconciliation + circuit-breaker: repeated cycles without convergence, and contaminated + evidence is worse than none. + +CLEANEST NEXT EXPERIMENT (not run): with ALL node domains powered off and +nothing else touching the fleet, commission EXACTLY ONE machine from a clean +`Ready`/`New` state and leave it strictly untouched for 35+ minutes. Only +that isolates contention from a genuine per-node failure. Candidate fixes to +test after it: raise MAAS `node_timeout` (nested I/O is slow and 30 min may +simply be too short), add a graphics device for `virsh screenshot`, add +`counter` to the SEC-010 rules so the node->region hypothesis becomes +readable, and check the kernel cmdline MAAS actually serves. + +## 20. CLEAN commissioning experiment: contention REFUTED (properly this time) + +- Ran the isolated experiment the earlier invalid ones only pretended to: + ALL node domains OFF (verified 0 running), 397 GiB free, load 0.05, canary + in a TERMINAL `Failed commissioning` state (so the commission restarts + MAAS's 30-min timer), commission EXACTLY ONE, poll READ-ONLY only, touch + nothing. +- RESULT: `Failed commissioning after 1770s` (~29.5 min = the 30-min + timeout), a SINGLE node entirely alone. **Contention is refuted** -- it is + not I/O bandwidth, not memory, not the metadata service being swamped. The + failure is systemic and per-node. +- This CORRECTS changelog item 18-19's contention hypothesis (which rested on + contaminated runs). The event trace is consistent every time: `Commissioning` + -> `Performing PXE boot` -> `Loading ephemeral` (~40s later) -> 30 min of + silence -> timeout. The node gets the ephemeral image handoff and then never + reports back. +- REFRAME after further measurement: "contention refuted" is right but too + narrow. The single-node fail refutes CONTENTION (competing for resources) + but is fully CONSISTENT with INHERENTLY SLOW -- the node may have still been + working at 30 min, just not done. And the batch result **3 pass / 6 fail + exactly at the 30-min mark** is the classic signature of a MARGINAL TIMEOUT + over a slow, variable operation: the faster-scheduled nodes finish under the + wire, the rest are still going when the timer fires. 1770s is 29.5 of 30. +- KEY CLUE against a network/metadata block: **3 nodes DID reach Ready on this + exact rack, subnet, and metadata proxy.** So the metadata path is NOT + globally broken; a "node never reaches :5248" theory has to explain why it + works for 3 identical nodes. Infra confirmed UP: rack `:5248` listening (2 + sockets), rack->region 5240 = 301. A `:5248` tcpdump returned 0 packets but + its 220s window likely closed before the ephemeral OS reached the + metadata-fetch stage -- INCONCLUSIVE (a timing race), not evidence. +- **LEADING HYPOTHESIS (operator-facing): MAAS's default 30-min commissioning + timeout is too short for depth-4 nested commissioning I/O.** Cheap, decisive + test that needs no stall-pinpointing: raise `node_timeout` + (`maas admin maas set-config name=node_timeout value=`, e.g. 60), + commission ONE node, see if it completes. If it does, the fix is a config + value (and a Roosevelt note: bare metal commissions faster, so this is a + VR1-nested tuning, not a Roosevelt default). If it still fails well past + 30 min, the timeout is exonerated and the next probe is the metadata-proxy + access log / a console screenshot (needs the graphics-device correction to + the node-vm console -- item 19). +- STOPPED the live diagnosis here per the reconciliation circuit-breaker: + many diagnostic cycles, and the remaining hypotheses need an operator + decision (change a MAAS-wide config) rather than more agent spelunking. + +## 8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review + +- Operator directive (verbatim intent): at the proper time/place, review the + OPNsense 26.7 changelog + documentation for upgraded features and changes; + incorporate beneficial improvements (esp. hardening) into the next + deployment iteration, following project rules. +- Scheduled: STAGE-3 CLOSE (GA-R6 close-out pass, with this stage's queued + DOCFIXes). Deliverable: a findings doc citing the official 26.7 release + notes; actionable items route per GA-R3 -- hardening/profile deltas as + PROPOSED items anchored on D-129 (its 4 open sub-decisions are the likely + surface), operational items as runbook DOCFIXes. NO self-executing + incorporation: each finding is ruled per GA-R5 (one per exchange) before + touching any built surface. Target of incorporation: dc1 standup + + Roosevelt runbooks; live-edge retrofits individually gated. + +## 6. ledger-scan hardening: FAIL-OPEN is terminology, not a status + +- SEC-010's close surfaced a scanner false positive: the OPEN-row detector + (`awk toupper(cell) ~ /OPEN|PENDING/`) matched the substring OPEN inside + "fail-open" in the closed row's disposition -- the row counted OPEN + forever (same class as the GA-F15 wrap trap). Fix: strip FAIL-OPEN from + the cell before matching (`scripts/ledger-scan.sh`); harness gains a + CLOSED-row-with-fail-open fixture + nochk (45/45 PASS). Count re-verified: + 7 open, SEC-010 absent. +- **Revert:** git revert of this commit's ledger-scan.sh + test hunks. + +## 5. Host tooling: libguestfs-tools installed on vcloud (operator-run) + +- `sudo apt-get install -y libguestfs-tools` (1:1.52.0-5ubuntu3) -- enables + offline guest inspection (virt-cat/virt-ls), used for items 4's captures. + Standard KVM-host tooling; Roosevelt-transferable. +- **Revert:** `sudo apt-get remove libguestfs-tools`. diff --git a/docs/archive/changelogs/changelog-20260719-phase5-sweep.md b/docs/archive/changelogs/changelog-20260719-phase5-sweep.md new file mode 100644 index 0000000..c89baf0 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260719-phase5-sweep.md @@ -0,0 +1,292 @@ +# Changelog 2026-07-19 -- GA Phase-5 sweep session (one changelog for the whole +# session per GA-R2 amendment D1; batches append here as they close) + +Executing session for the Phase-5 repair sweep (`docs/audit/phase5-sweep-plan.md`, +reviewed PASS + S1-S5). Every batch is operator-gated at its stated gate point; +every commit names its GA-F/GA-R trace and carries its own revert note. Status +lives in `docs/CURRENT-STATE.md` (GA-R1) -- this file records what CHANGED. + +## Batch 0 -- tooling first (OPENED by operator 2026-07-19; repo-only, no cloud contact) + +1. **0.1 repo-lint L10 status-coupling check (GA-R1 rule 8 / C1)** -- commit `0b991a5`. + - WHAT: `repo_lint.py` gains L10: a change set modifying a state-bearing surface + (anything under `docs/audit/`; a `**Status:` line in design-decisions.md; a + gate row `| G |` in session-ledger.md) without touching CURRENT-STATE.md is + flagged. Pending working-tree set when dirty, HEAD commit when clean. WARN this + batch (`L10_FAIL = False`); the flip to FAIL is a one-constant change due at + Batch 1 open (sweep plan: "WARN for one batch"). + - WHY: amendment C1's Phase-5 implementation clause, verbatim. DIVERGENCE NOTED: + the sweep plan's item 0.1 paraphrase said "CURRENT-STATE gate rows", which is + vacuous (editing a CURRENT-STATE row touches CURRENT-STATE by construction); + the ratified utterance says "ledger gate rows" and the ruling wins. The ledger + carries no gate rows today; the pattern guards their reappearance. + - Harness: tests/repo-lint T34-T41 (git fixtures), 42/42 PASS. + - REVERT: `git revert 0b991a5`. + +2. **0.2 ledger-scan SEC reporting repointed at security-ledger.md (GA-R4/F3)** -- + commit `b504cb6`. + - WHAT: scan emits open COUNT + ID list + register-of-record pointer; row-level + disposition text dropped from scan output. Session-ledger machine-derived SEC + bullet re-seeded to pointer + count form. G14 evidence cell repointed at + `docs/security-ledger.md` same commit. + - WHY: F3 verbatim ("pointer and a scan-derived count, never row-level + dispositions"). MEASUREMENT CORRECTION (GA-R1/C2): G14's count said 9; its own + enumeration and the scan both measure 8 OPEN rows -- corrected to 8, divergence + noted in the cell. + - Harness: tests/ledger-scan +4 checks, 41/41 PASS. + - REVERT: `git revert b504cb6` (restores the 2026-07-18 cell incl. its count error). + +3. **0.3 ledger-scan wrap-aware next-free exclusion (GA-F15)** -- commit `527f1fd`. + - WHAT: the "next-free"/"reproduc*" exclusion now suppresses the matching line + AND the immediately following line (window exactly one line, reset per file), + so a quoted next-free value whose hyphenated token wraps no longer counts as an + assignment. Chose this over assignment-site counting: DOCFIX/BUNDLEFIX have no + single register file to key assignment sites on (unlike D headers). + - WHY: GA-F15 -- the wrap class fired twice live inside the audit itself + (CURRENT-STATE section 9, now marked FIXED). Live counters re-verified + UNCHANGED post-fix: D=130, DOCFIX=197, BUNDLEFIX=052. + - Harness: tests/ledger-scan +3 checks, 44/44 PASS. + - REVERT: `git revert 527f1fd`. + +4. **0.4 record-audit status scan keyed to the legal token set (GA-F10 / vocab H1)** + -- commit `f7580fd`. + - WHAT: extractor knows Option A: legal OPEN/BLOCKED/HELD/CLOSED (OPEN was + previously UNSCANNED -- post-migration OPEN-vs-CLOSED contradictions would have + been invisible); DONE/TO RUN/RUNNING added; new report section 6 classifies + every distinct token and flags illegal tokens + event words (VOID/REOPENED/ + SUPERSEDED) minted in status position. Prose event words stay legitimate + row-history vocabulary. Report-only: 0/1/2 exit contract unchanged. + - WHY: H1 requires the scan before any token migration (Batch 2.5 dependency). + Live sanity run (scratch --out; the committed Phase-1 capture untouched): + 12 distinct tokens, 163 flagged counted occurrences = Batch 2.5's work-list. + Contradiction groups 12 -> 13 on the scratch run: the OPEN-token detection + gain, not drift. + - Harness: tests/record-audit +9 checks, 33/33 PASS. + - REVERT: `git revert f7580fd`. + +Batch 0 verification: all three touched harnesses green (42/42, 44/44, 33/33), +full gauntlet + repo-lint results recorded in the batch completion report and the +ledger entry for this session. + +## Batch 1 -- D-130 verification pair, decision, and the plan gate rebuild +## (OPENED by operator 2026-07-19) + +1. **L10 flip WARN -> FAIL at batch open** -- commit `9683a18`. One constant + (`L10_FAIL = True`) + harness rc updates, 42/42. Per the Batch 0 revert note. + REVERT: `git revert 9683a18`. + +2. **1.1 v8 plan-only verification (GA-F01, G4)** -- captures + `docs/audit/outer-plan-20260719-baseline.txt` (7/2/7, zero drift from + 2026-07-18) and `outer-plan-20260719-v8-ignorechanges.txt` (6/2/6, ZERO + forces-replacement). Suppression CONFIRMED; the operator's contingent lean + survived its verification. + +3. **1.2 v7 throwaway exercise (GA-F01, G4)** -- capture + `docs/audit/throwaway-v7-20260719.txt`. Isolated `throwaway-v7-*` root + (own scratch pool under the AppArmor-covered parent, no networks, no OS): + defect reproduced (seed volume "must be replaced" after staging-ISO loss); + replacement applied UNDER the running domain, rc=0, NO bounce (domid 3->3); + teardown verified zero residue (0 domains, 0 pools, dir gone). New fact + recorded: the bounce path is the autostart/domain Update, not the volume + replace. + +4. **1.3+1.4 D-130 ADOPTED (a) + implemented** -- commit `dac10e0`. + GA-R5 ruling (question + utterance in the D-130 register entry); + `lifecycle { ignore_changes = [create] }` on `libvirt_volume.seed` in + `modules/cloudinit-vm`; NEW harness `tests/cloudinit-vm/run-tests.sh` (6/6: + guard placement, exact attribute, no leak onto the data disk, tofu + validate). G4 + G5 CLOSED. REVERT: `git revert dac10e0`. + +5. **1.5-1.7 G6 reconcile + triple + G7 re-plan** -- commit `5a4f5eb`. + Operator ruled (ii) state surgery (S3/GA-R5): autostart:true injected into + both domain resources (were None), serial 22->23, backup + `opentofu/terraform.tfstate.pre-G6-20260719`; guests untouched (ids 1/2, + running). Re-plan capture `docs/audit/outer-plan-20260719-postG6.txt` = + **6/0/6**, equals the section-5 recorded triple exactly -- G6 + G7 CLOSED. + REVERT: docs via `git revert 5a4f5eb`; state via push of the pre-G6 backup. + +## Out-of-batch: permission posture extension (operator-directed 2026-07-19) + +- WHAT: `.claude/settings.json` `ask` list extended -- `Bash(git merge*)`, + `Bash(git revert*)`, `Bash(git reset*)`, `Bash(git rebase*)`, + `Bash(git branch -D *)`, `Bash(bash scripts/* apply*)`, + `Bash(bash scripts/* install*)`, `Bash(ssh * sudo *)`, `Skill(*)`. + Nothing added to `allow`; `deny` + the PreToolUse guard hook unchanged. +- WHY: operator drives sessions via Remote Control without a second terminal; + remaining auto-mode classifier HARD-DENIALS (walls) on mutation surfaces + matching no ask rule blocked the approve flow entirely (hit live: a Skill + invocation hard-denied 2026-07-19). Extends the 2026-07-18 "ask preempts + classifier" posture; targeted ask rules chosen over a catch-all `Bash(*)` + per doc-cited guidance (ask appears to preempt allow -- a catch-all would + destroy frictionless reads). `Skill(*)` syntax is undocumented; verified + EMPIRICALLY (post-change Skill invocation prompted + ran; pre-change + identical call hard-denied). Remaining walls get their own ask rule when + hit -- the settings edit itself now prompts instead of denying. +- REVERT: `git revert` this commit, or `cp .claude/settings.json.bak` per the + operator backup if the bak is refreshed first. + +## Batch 2 -- record fixes (OPENED by operator 2026-07-19; commits per-GA-F) + +1. **2.4 GA-F05 hostname** (`02af860`): vcloud on CLAUDE.md + SKILL.md + a + discovered fourth surface (environment.md:43). Memory copy = Batch 3. +2. **2.2 GA-F03 tofu pins** (`b5981a9`): four sites demoted to the + CURRENT-STATE section-7 pointer; the one LIVE false claim (workflow :360) + corrected. A fifth site (platform-traps:19) fixed at batch close. +3. **2.3 GA-F04 OPNsense rows** (`6232dd3`): skill pin row -> pointer; trap + sections requalified as entered-at-26.1-persisting; env.md's stale tofu + pin (GA-F03 class) caught on the same surface. +4. **2.1 GA-F02 readiness doc** (`360e8d6`): three contradictory plan-count + forms -> section-5 pointers; falsified deploy-ready blockquote reframed + as history; fresh-session banner -> one-line pointer at the G9 canonical + entry doc (operator scope addition, probe hole H1). +5. **2.5+2.6 H1 migration + GA-R1 demotions** (`47423c6`): mapping record + docs/audit/h1-token-map-20260719.md (no token unmapped); workflow doc 26 + flags -> 0 (8 State lines demoted, tracker model dropped from the + header); runbooks 18 -> 0 with close-out instructions RETARGETED at + CURRENT-STATE; CURRENT-STATE 6 -> 0 (G9 STOPPED -> BLOCKED w/ named + blocker; stale "GA-R1 NOT yet ratified" header fixed per C2); gate rows + gained GA-R6/E1 [V]/[R] type tags; CLAUDE.md expectation line demoted + (S5b). History classes deliberately untouched (Batch 4 / GA-R3). +6. **2.8 GA-F14 banner** on the D-127 changelog (content preserved). +7. **2.9 GA-F11/F12 adjudication**: one real live version row fixed + (env.md:29 -> pin authorities); 192 non-history broken-ref rows + adjudicated ALL false-positive (provider-repo doc paths, glob/range + forms, slash-phrases, wraps) or VR0-era history docs routed to Batch 4. + Spot-checks: as-built:59 "netbox/tailscale" = slash-phrase; workflow:473 + retired-script mention = history; env.md:86 "phase-00..08" = range glob. +8. **2.10 S1 R3-F disposition**: all 13 items accounted -- F01-F07/F09/F10 + closed by the 2026-07-16 Phase A/B + rulings + SEC-010 artifact; + F08 CLOSED-SUPERSEDED (R-3's 4th storage node restored rebuild + headroom); F11 FIXED (supernet-unassigned clause dropped; vr1-dc1 stays + HELD on transit addressing, G12); F12 CLOSED (committed budget model, + 186 GiB headroom covers transient N+1 amphora; Stage-6 drill exercises + the real path); F13 CLOSED as recorded observation. Stale readiness C5 + queue corrected (it queued amendments that landed 2026-07-16). +9. **2.7 GA-R3 D-tags (operator-ruled)**: D-121..125 [ARCH], D-126..129 + [OPS], D-130 [ARCH]; register template gains the class-tag convention; + D-129's ARCH fragment (Roosevelt metal-edge profile) split to a dated + D-122 AMENDMENT per A1 (utterance: "Tags as proposed + D-122 + amendment"). + +**Batch-2 verification + the criterion-mechanics finding.** Extractor +re-run: live-surface vocab flags = 0; the remaining flagged occurrences +(110) and contradiction groups (12) sit ENTIRELY on history-class surfaces +(session ledger, changelogs, the append-only register, VR0-era docs -- +Batch 4's rotation/consolidation) or are extractor MECHANICS: the global +plan key conflates different roots' plans (Stage-1's 13-object plan vs the +DC0 triple), charm channels/step numbers read as juju versions, the +provider (2.7.2) vs MAAS-server (3.7.2) attribution, compatibility +thresholds (LXD >= 6.7), and prose like "Stage-4 concerns" keying a stage +status. FINDING (logged, not silently absorbed): the sweep plan's Batch-2 +exit criterion "contradiction groups on counted non-changelog surfaces +reach ZERO" is not literally computable by the extractor's changelog-only +exclusion -- ledger/register/finding classes are counted non-changelog but +their ruled disposition is Batch 4/GA-R3 retention, and the status +authority itself must narrate history (7/2/7 -> 6/2/6 -> 6/0/6) to comply +with GA-R1 rule 2. The achieved, verifiable state: zero contradiction +among LIVE-surface status claims (mechanical classification script in the +session record); final ZERO-on-counted comes with Batch 4's rotation + +Batch 6.1 as the plan already sequences. + +## Batch 3 -- memory hygiene (OPENED by operator 2026-07-19; edits per-file gated) + +Pre-edit evidence: full memory dir copied to docs/archive/memory-20260718/. +1. **GA-F06**: `review-after-each-step.md` REMOVED. Repo copy of the + provenance ruling: grounding-audit-20260718.md GA-F06 ("It was not a + standing instruction. It was only supposed to be for a one time / very + short sequence of steps."). No durable observation survives it -- the + drift-catch anecdote is repo-recorded in the review changelogs. +2. **GA-F07**: `project-posture-closed-test.md` REMOVED. Repo copy of the + provenance ruling: GA-F07 (stance never ruled; posture returns NEUTRAL; + the closed-test FACT is repo-carried; SEC gates were never deferred). +3. **GA-F05**: hostname -> vcloud in `multi-workstation-remote-control.md` + (the last of the four surfaces). +4. **GA-F08**: MEMORY.md index rebuilt -- two defective lines gone, the two + surviving entries re-described accurately. +5. **GA-R7 rule 6 skill line**: "Claude Code sessions launch from the repo + root, always" added to SKILL.md's session-bootstrap loop with the + GA-F06-origin rationale (subdirectory launch = divergent auto-memory). +6. **G1 full review (rules 1-4)**: `creds-folder-convention.md` found + duplicating a repo-carried convention + linking both removed entries -- + registered as **GA-F16** and reduced to a pointer entry (one + possibly-unduplicated fact retained with a verify-then-graduate note). + `multi-workstation-remote-control.md` reviewed: workflow facts operator- + confirmed (the 2026-07-19 walls-vs-gates entry quotes a same-session + directive; policy copy lives in the repo changelog it points at). + Post-review state: 3 files, zero rule-1..4 violations, zero stale refs + (grep-verified). +REVERT: restore any file from docs/archive/memory-20260718/ (pre-edit +copies); git revert the batch commit for the repo surfaces. +Note on the archive copies: `memory-20260718/MEMORY.md`'s em-dashes were +ASCII-normalized (`--`) to satisfy the repo L1 rule; the transform is +whitespace/punctuation-only and content-faithful (this note is the record +of it). No gauntlet run for this batch: zero script/tooling changes +(memory + docs + one skill prose line); repo-lint 0-fail + fence check OK. + +## Batch 4 -- consolidation + rotation (OPENED by operator 2026-07-19) + +- **C1 ledger rotation (GA-R4 rule 6, GA-F13)**: 1187 -> 131 lines; bodies + verbatim to docs/archive/session-ledger-rotated-20260719.md; lessons/facts + ROUTED first (format=line trap -> appendix-A; guard discipline -> + operating-discipline; VR0 facts + admin/controller -> maas-as-built; + lessons 1-6 verified already covered). F1 cap now enforceable. +- **C2-C5 changelog consolidation (GA-R2, GA-F09)**: 96 files (95 + the + v1-redeploy running record) -> docs/archive/changelogs/, four stage + records under docs/archive/stage-records/ (v1-era; stage0-1 = 45; stage2 + = 38; stage3-partial = 11), each commit message MANIFESTS its sources + (D2). The live session changelog stays top-level until session close. + NOTE: v1-redeploy-changelog's own "completion consolidation" TODO is NOT + discharged by the move -- it remains in the ledger's project-completion + block. +- **C6 docs disposition (D2 scope)**: 24 non-changelog history docs -> + docs/archive/ (v1-era packs/handoffs/findings/drafts; executed plans; + the R3-F register; Model-A fallback; DR seed; ipv6/netem research; + D-068-openbao DRAFT). RETAINED top-level, 16 files each justified: + status authority, registers (D/SEC), ledger, workflow (identity), design + doc, readiness checklist, two as-builts, tenant contract (lint L7 + auto-covered), log convention, live finding doc (CURRENT-STATE-cited), + D-129 review (open sub-decisions), NetBox scope (live apex reference), + D-068 analysis (open-decision evidence), live session changelog. + Live-surface references to every moved file rewritten to archive paths + (18 files touched); history-internal refs left verbatim. + +## Batch 5 -- skill sweep (OPENED by operator 2026-07-19; single commit) + +SKILL.md reconciled against the RATIFIED GA-R texts (not the draft): new +"Record authority and session discipline" section (~35 lines per the +draft's budget); header phase claim + the false "opentofu UNVALIDATED / +no tofu binary" claim DEMOTED to CURRENT-STATE delegation; session +bookends rewritten (OPEN = CURRENT-STATE first; CLOSE = 15-line summary ++ archived body + push); stage-close loop gains the GA-R6 close-out set +(named-check closes, no conditional close, consolidation commit, memory +review); change-delivery loop gains D1 (one session changelog, never +citable); routing rows re-pointed (VR1 status, versions, resume). The +draft's reconciliation circuit-breaker KEPT as discipline (no ruling +quote -- marked as such). Line-by-line checklist: +docs/audit/skill-sweep-checklist-20260719.md. 176 -> 224 lines. + +## Batch 6 -- Phase-6 exit runs (OPENED by operator 2026-07-19) + +- **6.1 ruling + run**: operator ruled the docs/archive/ counting exclusion + IN ("Yes, extend exclusion"); record-audit.py + harness updated (35/35). + Post-run: 9 groups, per-group adjudication = ZERO contradictory pairs + (full verdict table: docs/audit/phase6-exit-runs-20260719.md; group-count + floor 9 recorded as the re-run baseline). Missed-disposition fix: + stage3-review-base.patch archived; CURRENT-STATE known-stale note made + token-free. +- **6.2**: docs/*.md = 16 (< 25), history preserved via git mv + manifests. +- **6.4**: fresh capture outer-plan-20260719-phase6-exit.txt = 6/0/6 == + section-5 triple EXACTLY. +- **6.5 (S2)**: G9 row now names SEC-010's step-B apply+verify dependency + (the only apply-gated SEC row). +- **6.3 + 6.6 PENDING operator execution**: probe trio (3 clean sessions, + auto-memory OFF, 7-question set, 7/7 on 2-of-3) + the top-to-bottom + re-read and re-signature. + +## SESSION CLOSE (2026-07-19) + +Batch 6 completed: probe trio 21/21 (7/7 all three); operator re-signed +CURRENT-STATE ("Reviewed, approved, continue."). ALL exit runs PASS; the +grounding audit is EXITED, the freeze lifted, G9 OPEN. This changelog is +the session's complete WHAT-CHANGED record; it consolidates into the +Stage-3 record at stage close per GA-R2. diff --git a/docs/archive/changelogs/changelog-20260721-close-and-delivery.md b/docs/archive/changelogs/changelog-20260721-close-and-delivery.md new file mode 100644 index 0000000..9ccf6bc --- /dev/null +++ b/docs/archive/changelogs/changelog-20260721-close-and-delivery.md @@ -0,0 +1,140 @@ +# Session changelog 2026-07-21 (successor session) -- disconnect recovery + queued delivery + +One changelog per session (GA-R2/D1). Status lives ONLY in +docs/CURRENT-STATE.md; this file is narrative + reverts. Predecessor +session (ops-commissioning-diag, disconnected before its bookend): +docs/changelog-20260721-commissioning-diag.md. + +## 1. Predecessor session close LANDED (commit 8b3adb3) + +- The ops-commissioning-diag session disconnected before committing its + deliverable. This session re-verified read-only first (9/9 nodes Ready + in MAAS with virsh power; dc0-node-dns active+enabled with the 10.12.8.3 + alias up; dhcpd running on BOTH controllers; diag-serial tag deleted and + serial-console re-attached to hardy-dove = Phase-7 cleanup confirmed + done), then committed the predecessor's 7 files verbatim plus the + bounded session-ledger close entry and a machine-derived block re-seed + from the 2026-07-21 scan (D next-free 132, DOCFIX 197, BUNDLEFIX 052, + 10 open SEC rows). Operator-approved this session. +- **Revert:** git revert 8b3adb3 (returns the record to the + not-yet-durable state; the live cloud is untouched by that commit). + +## 2. MAC pinning in modules/node-vm (queued item 9a of the predecessor) + +- modules/node-vm: new OPTIONAL `interface_macs` variable (list(string), + default [] = unpinned for pre-enlistment use; validations: all-or- + nothing length vs network_names -- partial pinning is rejected -- and + 6-byte hex format). Interfaces comprehension consumes it as + `mac = { address = ... }` -- shape VERIFIED against the provider schema + (dmacvicar/libvirt 0.9.8, `tofu providers schema -json`, this session). +- opentofu/vr1-dc0-substrate: all 9 nodes pinned. 54 MACs MEASURED this + session (`virsh domiflist` on vvr1-dc0, read-only; order = the + vr1_dc0_node_nics plane order) and the 9 metal-admin entries + cross-checked against MAAS boot-interface records 9/9 (machines read). + No value inferred; no value reused from the adjudication file without + re-measurement. +- NEW harness tests/node-vm/run-tests.sh (12 cases: variable shape, + validations, consumption, 9 lists / 54 literals / 0 duplicates in the + inner root, pass-through, tofu validate). Gauntlet 72 -> 73 ALL GREEN; + repo-lint 0 fail; opentofu-validate PASS (both EXTRA roots). +- NOT YET APPLIED: the inner root's state (on voffice1) does not carry + the pins until an operator-gated `tofu apply` there. The plan gate for + that apply: expected NO REPLACE -- pins equal live values, so the plan + must show only benign in-place mac adoption or no-op; ANY replace is a + STOP (present to operator, do not apply). +- **Revert:** git revert of this delivery commit (module + root + harness + together); live domains unaffected until the gated apply. + +## 3. node-vm power-ownership guard (operator-ruled this session) + +- The MAC-pin verification plan on voffice1 (captured: + docs/audit/inner-plan-20260721-macpin.txt -- 0/9/0, 54 mac adoptions, + ZERO replaces) exposed an entanglement: the module hardcodes + `running = true` while MAAS holds all 9 Ready nodes OFF, so the apply + would ALSO have powered on all 9 nodes outside MAAS's control. +- RULING (this session): question = how to proceed with the entangled + apply; operator selection = "ignore_changes on running first + (Recommended)". +- Shipped: `lifecycle { ignore_changes = [running] }` on the node domain + (create still boots for PXE enlistment; after that MAAS owns power -- + virsh here, IPMI on Roosevelt). Harness grew 3 cases (T13-T15: guard + present, exactly one ignore_changes, cites MAAS). 15 cases green. +- **Revert:** remove the lifecycle block + T13-T15; the 9 power-on + changes return to the next plan. + +## 4. MAC-pin apply EXECUTED on voffice1 (operator-approved) + +- Guarded re-plan captured (inner-plan-20260721-macpin-guarded.txt: 54 + mac adoptions, zero power flips, zero replaces), then saved-plan apply + (`tofu plan -out` -> `tofu show` re-verify 0/9/0 -> `tofu apply`): + exact 0/9/0, convergence re-plan "no differences" + (docs/audit/inner-apply-20260721-macpin.txt). Post-apply verified + read-only: all 9 domains still shut off, edge untouched, spot-checked + MACs identical (control-01, storage-04). +- The inner tfstate on voffice1 now carries the pins (state serial + advanced; it remains the state-of-record per CURRENT-STATE section 1). +- **Revert:** git revert the pinning commits, re-plan/apply on voffice1 + (returns MACs to provider-owned -- NOT recommended; reopens the drift + hole that caused the 2026-07-21 incident). + +## 5. dc-rack-net.sh: D-131 sub-1 delivery (forwarder + rack legs, repo-carried) + +- D-131 sub-1 RULED this session (standing per-DC pattern; ruling commit + fad3007), then delivered: NEW scripts/dc-rack-net.sh + -- site-keyed (dc0 rows MEASURED), runs ON the rack via piped + ssh. Owns: -rack-legs.service oneshot (re-adds 10.12.8.2/22 + + 10.12.4.2/22 + the 10.12.8.3/22 forwarder alias on boot -- the D-126 + reboot-loss class), the D-131 dnsmasq forwarder unit+conf (generated, + content-checked by diff in check mode), and a boot helper that + resolves bridges FROM LIBVIRT NET NAMES at runtime (virbrN is + auto-assigned = drifting ID; measured: dc-planes does NOT pin bridge + names). Region-side MAAS subnet setting stays a runbook step (same + split as site-headend-install's DHCP note). +- Delta from the live hand-placed artifacts (intentional, minimal): the + legs move from bare `ip addr` (reboot-lost) into the oneshot unit; the + dns unit gains Requires/After on the legs unit instead of its own + ExecStartPre alias hack; config content otherwise identical. +- NEW tests/dc-rack-net harness (14 cases incl. no-virbrN-literal, + MEASURED-tag discipline, read-only check path). Gauntlet 73 -> 74 ALL + GREEN; repo-lint 0 fail. +- Rack install EXECUTED (operator-approved, same session): EXIT 0, + self-check PASS 10/10 (docs/audit/dc-rack-net-install-20260721.txt); + behavioral probe: `dig @10.12.8.3 maas-internal SOA` answers through + the region. dnsmasq restart blip occurred with all 9 nodes powered + off (zero impact). The interim hand-placed unit/conf are OVERWRITTEN + in place by the generated equivalents (same paths). +- **Revert:** git rm the script + harness; on the rack (if installed): + systemctl disable --now dc0-rack-legs dc0-node-dns; rm the four + installed files; re-add the legs + old unit per changelog-20260721- + commissioning-diag item 4's revert. + +## 6. Incident documentation (queued items 9b/9c/10 of the predecessor) + +- appendix-A: two new entries -- (a) 30-min timeout with NO PXE event + for a known machine -> MAC-record drift (diff domiflist vs MAAS + interfaces; repair + pin); (b) cloud-init "no datasource" on a + rack-only DC -> agent-resolver SERVFAIL (dig SOA check; D-131 + forwarder fix via dc-rack-net.sh). +- platform-traps 1e: second corollary (in-place apply REGENERATES + unpinned NIC MACs; pin rule) + verbatim-index row. +- LP draft authored: docs/audit/lp-draft-20260721-maas-agent-resolver.md + (operator to file; sanitization warning re SEC-014 material included). +- **Revert:** git revert this commit (docs only). + +## 7. Step-E netem sudo mechanism (operator-ruled) + sudoers fragment + +- RULING (this session, OPS -- netem is VR-only, no D-number per GA-R3): + question = sudo mechanism for modules/netem-link on vcloud; operator + selection = "Scoped NOPASSWD for tc (Recommended)". +- Shipped scripts/sudoers.d/netem-tc: exactly the module's two verbs + (qdisc replace ... netem * / qdisc del ... root) per MEASURED mesh + bridge (virbr5=dc0-dc1 [the step-E target], virbr7=dc0-office1, + virbr3=dc1-office1; virsh net-dumpxml this session). Bridge names are + auto-assigned/drifting (mesh-link does not pin them) -- fragment + documents the fail-closed property + mandatory pre-apply re-measure. +- NEW tests/netem-sudoers harness (9 cases incl. exact-shape rule match, + no-broadening, visudo -cf). Gauntlet 74 -> 75 ALL GREEN. +- Install is operator-only (root; visudo-checked): command in the + fragment header. NOT installed at time of this entry. +- **Revert:** git rm fragment + harness; if installed: + sudo rm /etc/sudoers.d/netem-tc && sudo visudo -c. diff --git a/docs/archive/changelogs/changelog-20260721-commissioning-diag.md b/docs/archive/changelogs/changelog-20260721-commissioning-diag.md new file mode 100644 index 0000000..3588f9f --- /dev/null +++ b/docs/archive/changelogs/changelog-20260721-commissioning-diag.md @@ -0,0 +1,132 @@ +# Session changelog 2026-07-21 -- commissioning diagnosis + repair (logged: ops-commissioning-diag) + +One changelog per session (GA-R2/D1). Status lives ONLY in +docs/CURRENT-STATE.md; this file is narrative + reverts. Full adjudication: +docs/audit/commissioning-diag-20260721.txt. As-executed log: +~/as-executed/2026-07-21-ops-commissioning-diag.log. + +## 1. Instrumented commissioning run (committee-prescribed) EXECUTED + +- Phase 1 zero-cost reads, then gated: diag-serial MAAS tag + (kernel_opts console=ttyS0,115200) on canary sw6wqe/hardy-dove only; + serial-console tag temporarily DETACHED from the canary (alphabetical + tag concatenation would have left console=tty0 last, sending userspace + output to VGA). Instruments: MAC+broadcast tcpdump on virbr2, full + tcpdump on enp1s0, live Temporal journal follow -- all 45-min capped. +- One commission, hands-off, full window. Result: timeout at 30 min, + serial 0 bytes -- and the pcap held the answer (item 2). +- FINDING (record divergence): a serial-console tag with console + kernel_opts ALREADY existed (comment cites D-114 debug) contradicting + the committed "ephemeral kernel is not told console=ttyS0" claim. +- **Revert:** diag-serial tag delete + re-attach serial-console to sw6wqe + (Phase 7, item 8). + +## 2. FAULT 1 FOUND + REPAIRED: all 9 node MACs drifted (in-place apply) + +- PXE DHCP in the capture came from a MAC absent from MAAS's records; + fleet audit: 9/9 live metal-admin MACs differ from MAAS boot-interface + records. Boundary: the 2026-07-20 in-place serial-console apply + (tofu 0/9/0) regenerated NIC MACs. Full chain in the adjudication file. +- RULING (GA-R5, 2026-07-21): question = how to repair the records; + operator selection = "Update MACs in place (Recommended)". +- Executed: per-machine boot-interface MAC -> measured live value; + Failed-state machines needed mark-broken -> update -> mark-fixed. + Read-back verified 9/9. +- **Revert:** maas admin interface update + mac_address= per + machine (old values in the as-executed log); machines return to their + pre-session states via mark-broken/mark-fixed as needed. + +## 3. FAULT 2 FOUND: MAAS 3.7 rack-only agent resolver defect + +- Post-repair proof run booted fully; cloud-init found no datasource: + the agent resolver (node DNS via DHCP) SERVFAILs everything on an + isolated rack -- walks public root hints even for authoritative + maas-internal; ignores resolv.conf. Office1 immune (region BIND owns + node DNS there). First-exercised surface in VR1. LP REPORT QUEUED + (item 10). SEC-010 re-verified enforced during diagnosis. +- Interim RULING "Region upstream, runtime": resolvectl dns enp1s0 + 10.10.0.20 + domain ~. on the rack + /etc/resolv.conf relinked to + /run/systemd/resolve/resolv.conf. Did NOT affect the agent (kept -- + gives the rack itself working resolution via region). +- **Revert:** resolvectl revert enp1s0; ln -sf + ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf. + +## 4. FAULT 2 WORKAROUND: dc0-node-dns forwarder (operator-ruled) + +- RULING (GA-R5, 2026-07-21): question = node DNS remedy; operator + selection = "Rack dnsmasq forwarder (Recommended)". +- Shipped to the rack (NOT yet repo-carried -- item 9): + /etc/dnsmasq-dc0-node.conf + /etc/systemd/system/dc0-node-dns.service + (dnsmasq on virbr2 alias 10.12.8.3, no-resolv, server=10.10.0.20; + alias via ExecStartPre ip addr replace; After=libvirtd). Enabled. +- MAAS: subnet 6 (metal-admin) dns_servers=10.12.8.3 + allow_dns=false. + dhcpd still prepends the rack IP (option 6 = "10.12.8.2, 10.12.8.3") + -- works via SERVFAIL failover (proven); template behavior to + investigate at delivery. +- **Revert:** maas admin subnet update 6 dns_servers= allow_dns=true; + on rack: systemctl disable --now dc0-node-dns; rm + /etc/systemd/system/dc0-node-dns.service /etc/dnsmasq-dc0-node.conf; + ip addr del 10.12.8.3/22 dev virbr2. + +## 5. PROOF + fleet re-commission + +- Canary hardy-dove: Ready in ~3 min after seven consecutive 30-min + failures; commissioning scripts visible on serial, rc=0. +- Fleet: remaining 8 commissioned (5 never truly commissioned -- + mark-fixed does not commission; 3 carried stale interface data). + Outcome recorded in CURRENT-STATE (same commit). +- **Revert:** n/a (commissioning is re-runnable). + +## 6. SEC-014 OPENED: rack cluster secret exposed to session context + +- agent.yaml read (diagnosing the resolver) included the `secret` field. + Exposure scope: Claude session context, operator terminal, as-executed + log (0600, jumphost). Row + disposition: docs/security-ledger.md. + Rotation = rack re-enrollment; schedule at operator convenience. +- **Revert:** n/a (exposure event; row tracks rotation). + +## 7. Session tooling: targeted ask rules (classifier walls) + +- .claude/settings.local.json (untracked, global-gitignored): ask rules + added for wrapped/unwrapped `maas admin tags|tag|machine|interface| + subnet` via voffice1 and rack-sudo via the dc0 key -- per the standing + multi-workstation pattern (operator-approved this session). Lesson: + a `sleep N;` prefix breaks prefix-matching -- keep wrapped commands + rule-shaped. +- **Revert:** remove the ask entries from settings.local.json. + +## 8. Phase 7 cleanup (pending fleet-green) + +- diag-serial tag: detach from sw6wqe + delete; re-attach serial-console. + QUEUED OBSERVATION: serial-console's opts end in console=tty0, which + claims /dev/console for VGA -- fine for kernel logs, hides userspace; + revisit the tag's opts when serial observability next matters. +- **Revert:** n/a (is itself the revert). + +## 9. QUEUED delivery work (change-delivery loop, not this session) + +- modules/node-vm: PIN NIC MACs (set to current live values) so no apply + can drift them again; harness case for MAC stability. Roosevelt note: + metal has fixed MACs; the trap is VR-only. +- platform-traps entry: tofu/libvirt "in-place" 0/N/0 domain change can + REGENERATE NIC MACs (measured 2026-07-20/21). +- appendix-A entries: (a) "commissioning times out at exactly 30 min, + NO 'Performing PXE boot' event" -> diff live domain MACs vs MAAS + records; (b) "cloud-init: no datasource, searched classes ()" on + rack-only controllers -> query the rack agent resolver for + maas-internal; if SERVFAIL see D-131/forwarder. +- dc0-node-dns forwarder + rack-legs persistence: repo-carried + script/module + tests (site-keyed, per D-128 register item 20). +- D-131 PROPOSED (authored this session, design-decisions.md): node-facing + DNS strategy for rack-only controllers (Roosevelt-relevant). +- Committee record addendum: authored this session (outcome section). +- Stale MAAS pod vr1-dc0-inner (id 4): stage-close cleanup with SEC-013. + +## 10. LP report (queued for operator) + +- Title: "maas-agent resolver on rack-only controller SERVFAILs + authoritative maas-internal queries on internet-isolated racks (walks + root hints; ignores resolv.conf); breaks commissioning cloud-config-url" + MAAS 3.7.2 / snap 41649. Evidence: adjudication file + agent journal + excerpts in the as-executed log. diff --git a/docs/archive/stage-records/vr1-stage3-partial-record.md b/docs/archive/stage-records/vr1-stage3-partial-record.md deleted file mode 100644 index 886b7c0..0000000 --- a/docs/archive/stage-records/vr1-stage3-partial-record.md +++ /dev/null @@ -1,17 +0,0 @@ -# VR1 Stage 3 changelogs THROUGH 2026-07-18 (partial record -- stage still OPEN) - -Stage 3 is the ACTIVE stage; this record consolidates only its closed sessions (2026-07-16..18: adversarial review sweep, Model B reshape, D-121..D-129 era, grounding audit). The live session changelog (changelog-20260719-phase5-sweep.md) stays top-level until its session closes. Durable content: rulings in design-decisions D-121..D-129 + ga-rulings; status in CURRENT-STATE. - -| file (now docs/archive/changelogs/) | title line | lines | -|---|---|---| -| changelog-20260716-d124-addressing-pin.md | 2026-07-16 -- D-124 addressing pinned: transit supernet 172.31.0.0/24 + ratified rack values | 65 | -| changelog-20260716-d124-rack-addressing.md | 2026-07-16 -- D-124 adopted; vr1-dc0 MAAS rack (vvr1-dc0) wired; rack-mgmt NetBox importer | 92 | -| changelog-20260716-d126-durable-forward.md | 2026-07-16 -- D-126: durable, rootless vcloud->service-VM access (SSH local-forward via systemd --user) | 111 | -| changelog-20260716-review-sweep-phaseAB.md | 2026-07-16 -- Stage-3 review sweep, Phase A (record amendments) + Phase B (whole-host calculator) | 59 | -| changelog-20260716-review-sweep-phaseC.md | 2026-07-16 -- Stage-3 review sweep, Phase C (D-123 Model B OpenTofu reshape + node-host bootstrap) | 64 | -| changelog-20260716-review-sweep-phaseC2-D125.md | 2026-07-16 -- D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3's design gap; egress | 73 | -| changelog-20260717-d126-baseleg-oneshot-d128-operating-model.md | 2026-07-17 -- vcloud post-restart sweep + D-126 base-leg oneshot + D-128 operating model | 75 | -| changelog-20260717-d127-iac-autostart-and-guest-updates.md | 2026-07-17 -- D-127 IaC autostart matrix (baked into the deploy tooling) + Office1 guest updates | 68 | -| changelog-20260717-netbox-apex-recovery-edge-wan-import.md | 2026-07-17 -- office1-netbox apex recovery + restart policy + full D-124/edge-WAN import (Blocks A + B) | 54 | -| changelog-20260718-opnsense-plugins-installer.md | Changelog 2026-07-18 -- OPNsense edge plugin profile-installer (D-129) | 40 | -| changelog-20260718-permission-posture-ask-preempts-classifier.md | Changelog 2026-07-18 -- permission posture: mutations PROMPT, not classifier-deny | 40 | diff --git a/docs/archive/stage-records/vr1-stage3-record.md b/docs/archive/stage-records/vr1-stage3-record.md new file mode 100644 index 0000000..12e77a0 --- /dev/null +++ b/docs/archive/stage-records/vr1-stage3-record.md @@ -0,0 +1,40 @@ +# VR1 Stage 3 record -- CLOSED 2026-07-21 (dc0 substrate; dc1 HELD under G12) + +**Stage close (operator-ruled 2026-07-21, "Close and merge"):** all dc0 gates CLOSED with captures +-- G9 (outer apply), G10 (deploy steps B-E: bootstrap, depth-4 boot, D-125 egress, MAAS + all 9 +nodes Ready, netem step E on virbr5), G16 (office1 channels state surgery, outer plan ZERO DIFF). +The dc1 remainder is split into HELD gate G12 per the stage's own vr1-dc0-first design (E3 +satisfied). Merged to `main` as a merge commit; branch `dc-dc-stage3-phase2-dc-substrate` retired. +Notable en-route: the 2026-07-21 commissioning double-fault (MAC regen + rack agent-resolver +SERVFAIL, both fixed + doc'd), the pod-mechanism refutation (D-103/D-123 amendments; vr1-dc0-maas +retired), and the same-day pod-delete-cascade incident + full recovery +(docs/audit/incident-20260721-*). Durable content: rulings in design-decisions D-121..D-131 + +ga-rulings; traps in platform-traps + appendix-A; status ONLY in CURRENT-STATE. + +Sessions 2026-07-16..18 (adversarial review sweep, Model B reshape, D-121..D-129 era, grounding +audit) were consolidated while the stage was open; the close adds the 2026-07-19..21 execution +sessions below. The closing session's changelog (changelog-20260721-netem-install-verify.md) +stays top-level until that session closes (per this record's own precedent). + +Closing-era sessions (now docs/archive/changelogs/): + +| file | scope | lines | +|---|---|---| +| changelog-20260719-phase5-sweep.md | GA Phase-5 sweep Batches 0-6 + audit exit | 292 | +| changelog-20260719-dc0-deploy-stepB.md | deploy steps A-D: transit, bootstrap, inner apply, edge, enlistment, power | 838 | +| changelog-20260721-commissioning-diag.md | commissioning double-fault diagnosis + repair (9/9 Ready) | 132 | +| changelog-20260721-close-and-delivery.md | MAC pinning + dc-rack-net + incident docs + netem sudo mechanism | 140 | + +| file (now docs/archive/changelogs/) | title line | lines | +|---|---|---| +| changelog-20260716-d124-addressing-pin.md | 2026-07-16 -- D-124 addressing pinned: transit supernet 172.31.0.0/24 + ratified rack values | 65 | +| changelog-20260716-d124-rack-addressing.md | 2026-07-16 -- D-124 adopted; vr1-dc0 MAAS rack (vvr1-dc0) wired; rack-mgmt NetBox importer | 92 | +| changelog-20260716-d126-durable-forward.md | 2026-07-16 -- D-126: durable, rootless vcloud->service-VM access (SSH local-forward via systemd --user) | 111 | +| changelog-20260716-review-sweep-phaseAB.md | 2026-07-16 -- Stage-3 review sweep, Phase A (record amendments) + Phase B (whole-host calculator) | 59 | +| changelog-20260716-review-sweep-phaseC.md | 2026-07-16 -- Stage-3 review sweep, Phase C (D-123 Model B OpenTofu reshape + node-host bootstrap) | 64 | +| changelog-20260716-review-sweep-phaseC2-D125.md | 2026-07-16 -- D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3's design gap; egress | 73 | +| changelog-20260717-d126-baseleg-oneshot-d128-operating-model.md | 2026-07-17 -- vcloud post-restart sweep + D-126 base-leg oneshot + D-128 operating model | 75 | +| changelog-20260717-d127-iac-autostart-and-guest-updates.md | 2026-07-17 -- D-127 IaC autostart matrix (baked into the deploy tooling) + Office1 guest updates | 68 | +| changelog-20260717-netbox-apex-recovery-edge-wan-import.md | 2026-07-17 -- office1-netbox apex recovery + restart policy + full D-124/edge-WAN import (Blocks A + B) | 54 | +| changelog-20260718-opnsense-plugins-installer.md | Changelog 2026-07-18 -- OPNsense edge plugin profile-installer (D-129) | 40 | +| changelog-20260718-permission-posture-ask-preempts-classifier.md | Changelog 2026-07-18 -- permission posture: mutations PROMPT, not classifier-deny | 40 | diff --git a/docs/changelog-20260719-dc0-deploy-stepB.md b/docs/changelog-20260719-dc0-deploy-stepB.md deleted file mode 100644 index c039ff1..0000000 --- a/docs/changelog-20260719-dc0-deploy-stepB.md +++ /dev/null @@ -1,838 +0,0 @@ -# Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap) - -Session changelog (GA-R2/D1: ONE per session). Logged window: `dc0-deploy` -(same-day append, per the A/B-boundary handoff). Status lives ONLY in -`docs/CURRENT-STATE.md`; this file is narrative + reverts. - -## 1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124) - -- **Question as presented (2026-07-19):** "Who should hold 172.31.0.1 (the - Office1 end of the dc0 transit /30), and by what mechanism? This is the - prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS - rack-to-region enrollment path." -- **Operator answer, exact utterance (option selected):** "voffice1 via IaC + - restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to - `module "voffice1"` in the outer root; apply in a planned voffice1 restart - window (graceful shutdown first; region/NetBox/Tailscale return via proven - autostart). -- Classification: OPS mechanism decision implementing D-124's already-ruled - addressing ("region/peer .1" -- `docs/design-decisions.md:3727`); no new - D-number (GA-R3, doubt resolves DOWN). -- **Measured basis (this session, logged):** mesh-vr1-dc0-office1 (virbr7) is - ISOLATED (no ``/``) with exactly ONE attached interface - (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing - holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only - route `10.10.0.0/22 via 172.31.0.1` (d124-rack.auto.tfvars, consumed by - step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached - by `-J voffice1` (D-126 shape) -- NO vcloud host leg / site-baseleg DC row - needed for this net. - -## 2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated) - -- **What:** `opentofu/main.tf` `module "voffice1"`: `network_names` gains - `module.mesh_vr1_dc0_office1.network_name` SECOND (office1-local stays - first -- PCI order preserves enp1s0); `network_config` rewritten from the - single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4; - transit=enp2s0 static `${vr1_dc0_rack_transit_peer_ip}/${prefix}` -- same - D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init- - re-run caveats documented in-block: the seed netplan is REBUILD correctness; - the live transit config is an in-guest netplan drop-in at the gated attach - step. -- **Why:** section 1's measured gap -- step B (ssh reach + rack enrollment) - has no L3 path until the region end of the transit exists. -- **Revert:** restore `network_names = [module.office1_network.network_name]` - and the previous "en*" glob `network_config` (git revert of this commit); - if already applied, a follow-up plan shows the interface removal and the - in-guest drop-in `/etc/netplan/60-transit.yaml` is deleted. - -## 3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation - -- Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply - (1/1/1 exact, `docs/audit/outer-apply-20260719-voffice1-transit.txt`) -> - domain up with both NICs, office1-local index 0. -- **INCIDENT (found+fixed in-window):** the provider REGENERATED both NIC - MACs on the interface-list rebuild (old lan MAC `52:54:00:6a:87:e5` -> - new `52:54:00:89:e1:19`). The Kea reservation (MAC-keyed) missed and - voffice1 leased pool address `.108` (measured via edge ARP). FIX: re-keyed - reservation uuid `b12621e6-63e9-4714-9726-2e130c84e069` to the new MAC via - the D-113 API (`kea/dhcpv4/set_reservation` -> saved; - `kea/service/reconfigure` -> ok), then a detached - `networkctl reconfigure enp1s0` -> voffice1 back on `10.10.0.20`. -- **LOGGED FINDING (not executed):** `modules/cloudinit-vm` does not pin - MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed - DHCP reservations. Roosevelt-relevant; candidate module `mac` var or a - documented ops step. Also: live transit config landed as in-guest netplan - drop-in `/etc/netplan/60-transit.yaml` (narrows the lan glob to enp1s0 + - transit enp2s0 static) -- the drop-in must ALSO narrow `lan`, else the - 50-cloud-init `en*` glob claims enp2s0 (measured: first drop-in attempt - without it left enp2s0 addressless). -- **Revert:** `kea/dhcpv4/set_reservation` back to the old MAC (senseless - unless the NIC change is also reverted); delete the drop-in. - -## 4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix - -- Symptom: `172.31.0.2` unreachable; MEASURED: both rack NICs emitted ZERO - frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive - (ACPI-responsive; cloud-init completed per offline logs). -- Guest is a SEALED BOX: `modules/cloudinit-vm` domains have NO serial - console, NO qga channel, NO graphics; qga is NOT in the noble base image - and its cloud-init install failed (no egress) -- so no console, no agent, - no network. **LOGGED FINDING: add opt-in serial console (+ the D-129-style - qga channel var) to `modules/cloudinit-vm` -- observability gap bites every - future DC VM. NOT executed mid-step.** -- Offline inspection (libguestfs, operator-run, captures - `~/vvr1-dc0-offline-inspect{,2}.txt`): NIC names enp1s0/enp2s0 CORRECT - (naming trap closed); root cause = netplan `match.name` + `set-name`: - boot 1 renames don't apply (.link written after udev add) so the rendered - config matched nothing; after removing set-name, boot 2 STILL failed - because the early-boot netplan generator ran against the OLD config and - renamed the devices to mgmt/uplink before cloud-init wrote the new file - (one-boot lag, measured in syslog). Boot 3 (current config, no renames) - is the verification boot. -- FIX (this commit + applied): `network_config` drops set-name, keeps - MEASURED kernel names; `instance-id` bumped `vvr1-dc0-d123 -> -d123-r2` - (side effects noted in-block); seed replaced via explicit - `-replace=module.vvr1_dc0.libvirt_volume.seed` (the sanctioned override - of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2 - exact (`docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt`). -- CONSEQUENCE for step B: bootstrap runs with - `--transit-if enp1s0 --uplink-if enp2s0` (script defaults mgmt/uplink no - longer match reality); runbook prose naming "mgmt" needs a DOCFIX at - stage close. -- **Revert:** git revert of the netplan/instance-id edit + one more - seed `-replace` apply + reboot. - -## 4b. VERIFICATION (2026-07-20, logged window) - -- Boot 3 with the fixed config: `enp1s0 UP 172.31.0.2/30`, region route - present, `br-vr1-dc0-wan` up. -- voffice1 -> 172.31.0.2 ping 0% loss; ssh - `-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2` - works (D-126 -J shape, per-env key); rack -> region `ping 10.10.0.20` - 0% loss (the MAAS-enrollment path for the G10 bootstrap). -- Outer plan CONVERGED back to ZERO DIFF - (`docs/audit/outer-plan-20260720-postfix-converged.txt`); CURRENT-STATE - section 5 updated in the same commit. -- Follow-items (logged, not executed): D-126 DC ssh-config alias row for - vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key); - cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC - pinning; DOCFIX for runbook "mgmt/uplink" naming. - -## 4c. Step B bootstrap EXECUTED on vvr1-dc0 (2026-07-20, logged window) - -- Snap proxy set to the region maas-proxy (measured serving: HTTP 200 to - archive.ubuntu.com, CONNECT to api.snapcraft.io) -- the D-107-shaped - answer to the rack's no-egress bootstrap; apt used the same proxy via - per-invocation env. Enrollment secret piped region->rack - (`install -m600 /dev/stdin`), verified by byte-count only (32), never - in context. -- Run 1 (`docs/audit/stepB-bootstrap-20260720.txt`): maas snap 3.7/stable - installed, **rack ENROLLED to http://10.10.0.20:5240/MAAS**; FAILED at - qemu apt install on stale image package lists (404s via proxy). - Fix: `apt-get update` (proxied), re-run. -- Run 2 (`docs/audit/stepB-bootstrap-20260720-run2.txt`): EXIT 0 -- - libvirt/qemu + nested KVM on, inner pool + AppArmor, SEC-010 - FORWARD-drop loaded on enp1s0 + `sec010-fw.service` enabled - (boot-persistent), `br-vr1-dc0-wan` verified with enp2s0 enslaved. -- `--check` (`docs/audit/stepB-check-20260720.txt`): all items ok EXCEPT - opnsense base (the remaining step-B item); rackd running, enrolled. -- Region-side residue for the MAAS phase (per script exit text): enable - DHCP on metal-admin at the Office1 region naming this rack primary_rack - (step D territory, NOT done). -- **Revert:** rack enrollment: region-side `maas ... rack-controllers - delete`; SEC-010: disable sec010-fw.service + rm /etc/nftables-sec010.nft; - packages: apt/snap remove; snap proxy: `snap unset system proxy.http - proxy.https`. - -## 4d. RULING RECORD (GA-R5) -- DC edge base version (2026-07-20) - -- **Question as presented:** reuse verified 26.1 artifact vs fresh download - (first exchange); then, sharpened after the operator noted "Office1 is - running OPNsense 26.7": 26.1-base-then-upgrade vs fresh 26.7 nano base. -- **Operator answer, exact utterance (option selected):** "Fresh 26.7 nano - base". -- Consequences implemented: prep runs 26.7 -> `opnsense-26.7-nano.qcow2`; - `vr1-dc0-substrate` `opnsense_base_path` default updated (no inner state - exists; pre-apply edit); step-B `--check` passes `--opnsense-base` for - the 26.7 path. LOGGED follow-items: site-headend-install.sh's 26.1 - default + hint strings (script change + harness, stage close); the - D-112/D-113 serial/API boot path is REVALIDATED on 26.7 at the step-C - boot gate (office1 precedent: 26.1-proven, upgraded in place to 26.7). -- Mirror: `https://mirror.ams1.nl.leaseweb.net/opnsense` -- MEASURED from - the vendor's live download page this session (not memory), HEAD-verified - serving the 26.7 nano (~468 MiB). LOGGED FINDING: opnsense-prep-image.sh - performs no checksum verification of the downloaded image (same property - as the original 26.1 prep; candidate hardening). - -## 4e. Step B CLOSED: 26.7 base staged; final check EXIT 0; SEC-010 CLOSED both ends - -- OPNsense 26.7 nano prepped ON the rack (leaseweb mirror via region proxy, - 468M; python3 bz2 fallback -- bunzip2 never needed, DOCFIX-184; - `docs/audit/stepB-opnsense-prep-20260720.txt`). Inner-root - `opnsense_base_path` default -> 26.7 (validate PASS). -- Final `--check` EXIT 0, ALL items ok - (`docs/audit/stepB-check-20260720-final.txt`). -- SEC-010 voffice1-side pin (the row's "same pin follows" clause, due since - this session wired voffice1's transit leg; voffice1 runs LXD so - ip_forward=1): identical scoped nft artifact + sec010-fw.service on - enp2s0, verified loaded + persistent; rack proxy/region paths re-verified - working post-pin. **SEC-010 row CLOSED (operator ruling, utterance in the - row).** Open SEC count 8 -> 7. -- LOGGED follow-item: site-headend-install.sh region-role support for the - transit-if pin (voffice1's was hand-applied to the script's exact - artifact shape; fold into the script + harness at stage close). -- **Revert:** rm the two files + `systemctl disable sec010-fw` on voffice1; - `nft delete table inet sec010`; ledger row reopens citing this item. - -## 7. Step C: voffice1 becomes the Plane-2 executor; INNER APPLY COMPLETE (28/28) - -- voffice1 setup (gated, logged): tofu 1.12.4 via vcloud's exact apt - source (keyrings byte-copied; version == section-7 pin); repo cloned at - `29cf7bf` (branch); dc0 private key staged to `~/vr1-dc0-creds/` - (0700/0600, piped -- SEC-009 convention now spans hosts); ssh config - entry for 172.31.0.2; known_hosts seeded via ssh-keyscan with the - ED25519 fingerprint cross-checked against the accept-new-trusted entry - (MATCH). -- Inner-root fixes measured at the FIRST REAL dial/apply (each committed): - (i) provider needs `keyfile`+`sshauth=privkey` URI params -- its Go ssh - ignores ~/.ssh/config AND default identities; single-type known_hosts - fails as "key mismatch" (multi-type scan required); - (ii) `modules/wan-bridge`: `` is ILLEGAL in bridge-mode networks -- - removed (MTU belongs to the host bridge; DOCFIX-194 class: parsed, never - applied); - (iii) base-image location premise WRONG under remote provider -- content - UPLOADS from the executing host; 26.7 qcow2 copied rack->voffice1 - (`~/vr1-dc0-images/`), tfvar override + variables.tf description - corrected; - (iv) edge serial-log dir `/var/lib/libvirt/vr1/staging/` is a HARDCODED - vcloud literal in modules/opnsense-edge (:223) -- dir created on the rack - (755); LOGGED findings: parameterize the path; fold staging-dir creation - into site-headend-install --host-nodes; opentofu-validate.sh does not - cover the inner root. -- Applies (captures `docs/audit/inner-{plan,apply}-20260720-stepC*.txt`): - run 1 = 25/28 (nodes+planes+pool; wan-bridge mtu abort); run 2 = wan - bridge (edge disk stat abort); run 3 = edge disk (staging-dir abort; - provider self-cleaned the failed domain -- operator undefine found - nothing); run 4 = edge domain, EXIT 0. CONVERGED: inner plan zero diff - (`docs/audit/inner-converge-20260720-stepC.txt`). -- VERIFIED: 10/10 domains RUNNING inside vvr1-dc0 (9 nodes depth-4 + edge); - edge serial log = FreeBSD login prompt on ttyu0 -- 26.7 boots the D-112 - serial path (first revalidation datapoint; full D-112(c) bootstrap is the - next window). -- NEW STATE OF RECORD: the inner tfstate lives on voffice1 - (`~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/terraform.tfstate`) - -- back it up with the site; follow-item for the backup set. -- **Revert:** `tofu destroy` in the inner root from voffice1 (site-down - alternative: `virsh destroy vvr1-dc0`, D-122); module fixes revert by - commit. - -## 9. DC edge: D-112(c) console bootstrap + API key minted (26.7) -- REPLACEMENT CHAIN PROVEN - -- Per-env edge key minted on vcloud (`~/vr1-dc0-creds/vr1-dc0-edge_ed25519`, - 0600/0644, SEC-009 convention), staged to the rack the same way. -- Rack legs added (operator-ruled "10.12.4.2, manual now + script later"): - `10.12.4.2/22` on the provider-public inner bridge (virbr5, MEASURED) + - the D-124-ruled `10.12.8.2/22` on metal-admin (virbr2). INTERIM: `ip addr - add`, not persistent. **LOGGED FINDING (queued):** site-headend-install.sh - never implemented the rack legs despite main.tf's comment claiming it - consumes `vr1_dc0_rack_metal_admin_ip` -- add rack-leg support + harness - before step D closes. Also interim: `192.168.1.2/24` on virbr5 to reach - the edge's FACTORY LAN; REMOVE once the edge is re-addressed. -- **D-112(c) console bootstrap: DONE** (`~/d112c-console.log` on the rack). - Driver evolved v1->v6 against MEASURED failures, each real and worth - keeping for the runbook: - (i) serial console wraps long lines -> ship payload as <=160-char base64 - chunks (`openssl base64 -d`), never one long line; - (ii) `set-name` in the edge netplan is irrelevant here but the same class - bit the rack (item 4); - (iii) tcsh on the edge (documented trap) breaks nested-quoted `sed` -- - "Unmatched '"'"'" -- so NEVER patch files via quoted one-liners; re-ship - the whole payload instead; - (iv) `write_config()` needs an include beyond config.inc -- MEASURED - (`grep -l shell_safe /usr/local/etc/inc/*.inc` -> auth/console/filter/ - interfaces/rrd/system/util.inc), not guessed; - (v) writing config.xml does NOT materialize `/root/.ssh/authorized_keys` - -- must call the vendor's `local_user_set()` (auth.inc). This was the - actual reason key auth failed after a "successful" config write. -- VERIFIED: `ssh -i vr1-dc0-edge_ed25519 root@192.168.1.1` -> key-only auth - OK, `uname -r` = 15.1-RELEASE-p1, `ifconfig -l` = vtnet0/vtnet1. - INTERFACE MAPPING MEASURED: vtnet0 = LAN = provider-public plane (it - answered on virbr5), vtnet1 = WAN -- same order as the office1 edge. -- **API key minted via the vendor model** (`opnsense-bootstrap-apikey.sh` - + `opnsense-mint-apikey.php`, run ON the rack): key/secret 80 chars -> - `~/vr1-dc0-creds/opnsense-api.txt` (0600, secret never printed). - SMOKE TEST PASS: `GET core/firmware/status` -> HTTP 200, `CORE_ABI 26.7`. - **This is the first proof the D-113(a2) API path works on 26.7** (the - 26.7 revalidation the operator's base-version ruling called for). - Script note: the bootstrap script requires the minter BESIDE it (its - own dir), not in `scripts/` -- worth a usage-line DOCFIX. -- **Revert:** delete the API key (`POST auth/user/del_api_key/`, the - proven negative-tested path) + creds file; `ip addr del` the three - interim addresses; the console bootstrap reverts by re-imaging the edge - (inner `tofu destroy`/re-apply of the edge module). - -## 10. D-125 egress isolation gate: EXECUTED as written -- PARTIAL (bridge-in PROVEN, -## upstream NAT egress FAILS). Gate NOT closed; double-NAT fallback NOT invoked. - -- Method: operator-ruled "throwaway VM, as written" -- noble cloud image - fetched ON the rack via the region proxy, one cloud-init guest, SINGLE - NIC bridged to `br-vr1-dc0-wan`, serial-log probe, then destroyed - (`leftover-domains=0` asserted). Script: scratchpad - `d125-egress-test.sh`; capture `docs/audit/d125-egress-gate-20260720.txt`. -- Run 1 was a TEST ARTIFACT, not a gate failure: the probe used DHCP and - came up address-less. MEASURED cause: `vr1-dc0-uplink` carries an `` - but NO `` block -- there is no DHCP server on that /24 by design - (the edge WAN takes a STATIC .2, D-113/D-125). **DOCFIX QUEUED:** the - gate's wording "must GET a vcloud-ISP address" implies DHCP; it must say - STATIC, or the /24 needs a range. Re-run used static 172.30.2.50/24. -- RESULT (run 2): guest configured 172.30.2.50/24 + default via 172.30.2.1; - **gateway ping 3/3, 0% loss** -- a FOREIGN MAC inside vvr1-dc0 reaches - the vcloud ISP NAT across the nested bridge. That is the D-125 bridge-in - property, and it PASSES. But `ping 1.1.1.1` FAILED (NET-PING-RC=1). -- **CORRECTION (2026-07-20, same-day): the rack-bridge follow-up is - CONFOUNDED and is NOT evidence.** It sourced from 172.30.2.51 on the - rack, but the rack has NO DEFAULT ROUTE (netplan gives it only - `10.10.0.0/22 via 172.31.0.1`; the WAN bridge is IP-less by design), so - traffic to 1.1.1.1 fails in the rack's own routing table before the NAT - is ever consulted. Its ICMP+TCP failures say nothing about virbr4. The - ONLY clean signal is the GUEST's (172.30.2.50) failed egress, and that - guest tested ICMP only -- if the rule dump comes back identical for both - bridges, re-run the guest probe with `curl` before concluding anything. - (Recorded per GA-R1/C2: a wrong measurement is corrected in the record, - not quietly dropped.) -- Controls: vcloud egress 0% loss; the office1 edge - egresses through the SAME-SHAPE `office1-wan` NAT 0% loss. The two - network definitions are byte-equivalent in substance (`net-dumpxml`: - both `forward mode=nat` + ``; only netmask-vs- - prefix notation differs) -- so the delta is in vcloud's LIVE firewall - rules for virbr4, NOT in the design or the config. -- **GATE STATUS: PASS / CLOSED 2026-07-20** (runs 3+4, capture - `docs/audit/d125-egress-gate-20260720-matrix.txt`). The "missing rule" - hypothesis was DISPROVED by measurement first: the operator-run dumps - show virbr4's rule set is structurally IDENTICAL to the WORKING virbr11 - (only counters differ; virbr11's one extra rule is a DHCP checksum - fixup it has because it carries a DHCP range). All three masquerade - rules exist for 172.30.2.0/24 INCLUDING the generic (non-tcp/udp) one, - and it had FIRED (1 packet/84 bytes = one ICMP conntrack entry), with - FORWARD accept counting 136 packets. So `net-destroy/net-start` would - have fixed nothing -- the pre-staged fix was correctly NOT run. -- Decisive re-probe used a protocol/destination MATRIX instead of counter - archaeology. Result, IDENTICAL on two consecutive runs: gateway ping 0, - **internet ping 0**, `curl http://1.1.1.1` -> 301, `curl - archive.ubuntu.com` -> 200. Egress works for ICMP AND TCP to real - hosts, so the D-125 bridge-in chain is PROVEN end to end and the - double-NAT fallback is NOT needed. -- **Run-2 anomaly, recorded UNEXPLAINED (not swept):** the same path - failed ICMP-to-internet once while its gateway ping passed 3/3. Runs 3+4 - passed identically with no intervening change to the uplink or its - rules. Most likely a first-use conntrack/ARP transient on vcloud. It is - not reproducible now; if a DC edge ever shows first-boot egress failure, - START HERE rather than re-deriving. -- **Security-positive side observation** (extra probe, outside the gate): - the guest could NOT reach the Office1 management LAN through the ISP NAT - (`10.10.0.20:8000` -> 000) on all runs, while internet egress worked. - That is the D-122/SEC-010 intent holding: the DC's WAN segment is an ISP - uplink, not a path into Office1. Worth an explicit assertion in the - hardening tests rather than leaving it an incidental observation. -- Prior status line (superseded, kept for the audit trail): gate was held - OPEN pending measurement rather than FAILED. The runbook's FAIL remedy - (revert to - double-NAT) is for a DESIGN failure; invoking it against what is very - likely a host firewall-rule gap would be the wrong remedy and would add - Roosevelt delta. Next step is a read-only measurement needing operator - sudo: FULL dumps of BOTH `nft list ruleset` and `iptables-save` (which - backend is authoritative here is itself unmeasured -- do not assume), - diffing virbr4 against the WORKING virbr11. Two rule families must both - be checked, and a subnet-only grep would MISS the second: POSTROUTING - masquerade (keyed on `172.30.2.0/24`) and FORWARD accept (keyed on the - INTERFACE `virbr4`). Pre-staged fix-forward IF the rules are absent or - malformed vs virbr11: `virsh net-destroy vr1-dc0-uplink && virsh - net-start vr1-dc0-uplink` reinstalls libvirt's rules from scratch -- - cheap and safe (nothing rides that uplink yet but the probe). If instead - the dump shows libvirt never installed them correctly, the restart will - not help and the dig moves to the firewall backend / conntrack. -- **Revert:** nothing to revert -- the probe guest and both temporary - addresses were removed; the only residue is `/tmp/noble.img` on the rack - (kept deliberately: it is the base for any future throwaway probe). - -## 11. NEW TOOLING + edge re-addressed: opnsense-set-interface-v4 pair (53/53) - -- **RULING RECORD (GA-R5), 2026-07-20.** Question as presented: base-iface - IPv4 is not REST-covered on 26.7 (measured); how should LAN 10.12.4.1/22 - and WAN 172.30.2.2/24 + gw 172.30.2.1 be set -- build the v4 setter pair - / console menu now / console now + build at stage close? Operator answer, - exact utterance (option selected): "Build the v4 setter pair - (Recommended)". Classification: OPS tooling under the existing D-113 - amendment; no new D-number (GA-R3). -- **D-113 AMENDMENT RE-MEASURED ON 26.7** (feeds the item-8 review): the - amendment was measured on 26.1 only. On this 26.7 edge: - `interfaces/settings/get` returns ONLY global offload flags, while - `interfaces/overview/list` and `interfaces/lan/get` both return 404 - "Endpoint not found". Base-interface addressing is still legacy - `/interfaces.php`. The amendment now holds on TWO majors, measured. -- SHIPPED: `scripts/opnsense-set-iface-v4.php` + - `scripts/opnsense-set-interface-v4.sh` + `tests/opnsense-set-interface-v4/` - (53/53 PASS first run; gauntlet 70 -> **71 harnesses ALL GREEN**; - repo-lint 0 fail). Mirrors the proven v6 sibling exactly -- vendor Config - singleton (never a config.xml push), dry-by-default, interface-count - guard, fresh read-back verify, `sh -s` for the tcsh trap, KERNEL as - ground truth. Two deliberate improvements over the v6 sibling: - (i) the GATEWAY half (creates/updates at most ONE `gateway_item`, - count-guarded exactly like the interface count, opt-in via a 5th arg); - (ii) the interface's DEVICE is MEASURED from the edge's own config - instead of the v6 sibling's hardcoded `lan->vtnet0 / wan->vtnet1` sed -- - that mapping is per-edge and hardcoding it is a hard-rule-2 violation - waiting to happen. Harness pins BOTH (test 12 fails if the hardcode - returns). **LOGGED FINDING:** backport the measured-device fix to the v6 - sibling at stage close. -- APPLIED to vr1-dc0-opnsense, WAN first per the script's own ordering - warning (re-addressing the LAN drops the session you arrived on): - - WAN: dhcp -> `172.30.2.2/24` + `WAN_GW` = 172.30.2.1 default. Note the - edge had shipped with WAN on **dhcp**, which could NEVER have worked -- - the uplink /24 has no DHCP server (same fact that broke probe run 1). - Verified on the kernel: `inet 172.30.2.2`, `default 172.30.2.1 UGS - vtnet1`, and **the EDGE ITSELF pings 1.1.1.1 at 0% loss** -- the D-125 - chain now proven through the real OPNsense edge, not just a probe. - - LAN: `192.168.1.1/24` -> `10.12.4.1/22` (the D-115/lib-net ruled - provider-public gateway). Session dropped as designed; edge verified at - the new address from the rack: `inet 10.12.4.1 netmask 0xfffffc00`. - - Post-move reconciliation at the NEW address: API `GET - core/firmware/status` -> 200 `CORE_ABI 26.7`; edge egress still 0% loss. -- CLEANUP: the interim `192.168.1.2/24` on the rack's provider-public - bridge is REMOVED (virbr5 now carries only the ruled `10.12.4.2/22`). -- **Revert:** re-run the setter with the previous values - (`--commit lan 192.168.1.1 24`, `--commit wan ...`); the - gateway item is named `WAN_GW` and is the only one this tool created. - -## 12. D-129 edge profile APPLIED on 26.7 + qga channel shipped + a FALSE-SUCCESS bug fixed - -- **RULING RECORD (GA-R5), 2026-07-20.** Question: the qga virtio channel is - absent from the edge domain and unsupported by `modules/opnsense-edge` - (both MEASURED), so installing os-qemu-guest-agent alone yields a package - that can never answer. Options presented: add channel var now / iperf only - + defer / install both and leave qga inert. Operator answer, exact - utterance (option selected): "Add channel var now, then install both - (Recommended)". [OPS under D-129; no new D-number, GA-R3.] -- **SHIPPED `expose_qga_channel`** (opt-in, DEFAULT OFF) in - `modules/opnsense-edge`; `vr1-dc0-substrate` sets it true so NEW DC edges - are BORN with the channel (the runbook's no-retrofit intent). Default OFF - is load-bearing: a device change alters the domain, so defaulting ON would - bounce every existing edge on the next plan. Shape taken from - `tofu providers schema -json` (dmacvicar 0.9.8: - `devices.channels[].target.virt_io{name}` + `source.unix{mode}`), NOT from - a blog; `source.unix` carries no path, letting libvirt allocate the socket - (same reasoning the serial block documents for pty). -- APPLY was SAFE: plan showed **0 add / 1 change / 0 destroy -- an IN-PLACE - domain update**, so `libvirt_volume.disk` (holding the console + API - bootstrap) was never at risk. Verified after: channel present (grep count - 2), domain running, edge still reachable at 10.12.4.1. -- **BUG FOUND ON THE FIRST-EVER LIVE RUN -- `opnsense-plugins.sh apply` had - NEVER installed anything.** `api()` used `bash "$API" ${DRY_RUN:+--dry-run}`: - `${VAR:+word}` expands when VAR is set and NON-EMPTY, and `DRY_RUN=0` is - non-empty -- so every apply silently dry-ran while printing - "OK: profile 'vr1-edge' applied". A textbook false success, and the reason - it survived review is that the harness drove ONLY `--dry-run`, leaving the - live path with zero coverage. FIX tests the VALUE; harness gains two - live-path tests against a stub api client, **verified to FAIL 2/2 on the - buggy version before restoring** (a regression test that cannot fail is - worthless). 20/20. This is exactly the class D-129's own "package present - is not success" warning is about -- here the SCRIPT was the false reporter. -- REAL install then ran: `{"status":"ok","msg_uuid":...}` per plugin (not - dry-run echoes), capture `docs/audit/d129-plugins-20260720.txt` (the - earlier capture of the same name recorded the false dry-run and is - REPLACED). Measured on the edge: `os-qemu-guest-agent-1.3`, - `qemu-guest-agent-11.0.1`, `os-iperf-1.0_2`, `iperf3-3.21`. -- Service was NOT auto-started (the script's own NOTE); started via the - plugin's OWN API (`POST qemuguestagent/service/start` -> `{"response":"OK"}`, - status -> running) rather than by hand-editing rc.conf. -- **GATE MET, to the runbook's exact bar (not package-presence):** - `virsh qemu-agent-command vr1-dc0-opnsense '{"execute":"guest-ping"}'` -> - `{"return":{}}`, and `virsh domifaddr --source agent` reports vtnet0 - 10.12.4.1/22 + vtnet1 172.30.2.2/24. The agent ANSWERS over the channel. -- **SECURITY FINDING (queued for the item-8 hardening review):** the plugin - model exposes `DisabledRPCs`, including **`guest-exec`/`guest-exec-status` - -- arbitrary command execution on the edge from the libvirt host**. On a - commercial multi-tenant cloud the DC edge should almost certainly disable - the exec RPCs (we need guest-ping/fsfreeze/shutdown coordination, not - remote exec). Not changed mid-step; propose with options at the review. -- **Revert:** `POST qemuguestagent/service/stop`; uninstall via - `core/firmware/remove/`; set `expose_qga_channel = false` and re-apply - (in-place domain update again). - -## 13. Step D part 1 DONE (DHCP configured) -- then INCIDENT: MAAS DHCP is DOWN -## REGION-WIDE (Temporal wedged). Not caused by this deploy. - -Step-D work completed before the incident surfaced: -- Both controllers confirmed registered: `voffice1` (region, mtstwf) and - **`vvr1-dc0` (rack, 7chphy, rackd running)**. -- MAAS auto-discovered the inner planes from the rack's interfaces: - metal-admin `10.12.8.0/22` = VLAN 5005/fabric-4, provider-public - `10.12.4.0/22` = VLAN 5006/fabric-5. (Stale `192.168.1.0/24` also sits on - fabric-5 -- the edge's old factory LAN, now vacated; cleanup queued.) -- **RULING RECORD (GA-R5), 2026-07-20.** Question: metal-admin is a /22 but - D-120's band layout is written for a compose /24; what dynamic range? - Operator answer, exact utterance (option selected): "Inherit D-120 band: - 10.12.8.100-10.12.8.200 (Recommended)". Range CREATED and verified on the - correct subnet; VLAN 5005 updated `dhcp_on=true primary_rack=7chphy`, - verified by read-back (`10.12.8.0/22 dhcp=True primary=7chphy`). - -**INCIDENT (found by verification, not by a self-report):** -- Canary `virsh reset vr1-dc0-control-01` did NOT enlist. MEASURED cause - chain: rack service_set shows `dhcpd: off`; no dhcpd process on the rack; - no `/var/snap/maas/common/maas/dhcpd.conf` generated -- while MAAS's own - API reports the VLAN as `dhcp_on=True` with the correct primary rack and - the interface correctly linked (VLAN 5005, static 10.12.8.2). Config - state and running state DISAGREE. -- Root cause, from the region's journal: MAAS 3.7 drives DHCP through - **Temporal workflows**, and Temporal is wedged -- - `"Not enough hosts to serve the request"`, `poll_workflow_task_queue - retried 2807 times`, task queue `mtstwf@region:...`. No worker is polling, - so the workflow that pushes DHCP config to a rack can never run. -- **SCOPE IS WIDER THAN THIS DEPLOY:** `voffice1` itself has **NO dhcpd - process** (`pgrep -c dhcpd` -> 0) and reports no dhcpd service, while - Office1's compose net `10.10.1.0/24` reads `dhcp=True`. So MAAS DHCP has - been DOWN REGION-WIDE -- almost certainly since the 2026-07-17 host reboot - -- and went unnoticed because office1-netbox/office1-tailscale were - already Deployed and kept their addresses. **This is NOT caused by the DC0 - deploy; the deploy is what exposed it.** -- Consequence for the as-built record: any claim that Office1 MAAS DHCP is - "working" is currently FALSE (last positive evidence predates the reboot). -- **Detection gap (queued):** `cloud-assert.sh` and the Office1 health - checks did not catch a dead DHCP server, because MAAS's own service_set - reported the region healthy. A behavioral DHCP assertion belongs in the - health sweep -- this is precisely the "service's own verdict" principle - the repo already applies elsewhere. -- No prior art: appendix-A has no entry for this symptom. NEW incident - material once the fix is known. -**RESOLVED 2026-07-20 (operator-approved: "Go ahead with the restart").** -- `sudo snap restart maas` on voffice1 -> region back (HTTP 301 on - localhost:5240). Recovery verified BEHAVIORALLY, not by service status: - `pgrep -c dhcpd` on voffice1 0 -> **1**, and on the DC rack 0 -> **1**; - MAAS then reported `dhcpd: running` on BOTH controllers (was `off` on the - rack, absent entirely on the region). One restart fixed both sites, which - confirms the single wedged-Temporal root cause rather than two faults. -- **PROOF IT WAS THE REAL BLOCKER:** the identical canary that had failed - before (`virsh reset vr1-dc0-control-01`) enlisted immediately after. -- APPENDIX-A MATERIAL (queued, new incident class): symptom "MAAS says - dhcp_on=True but no dhcpd runs and nothing PXEs"; check `pgrep -c dhcpd` - on each controller and the region journal for Temporal - `"Not enough hosts to serve the request"`; remedy `snap restart maas`; - verify behaviorally on every controller, never by service_set. - -## 14. G10 DEPTH-4 NESTED BOOT GATE: **PASS** -- all 9 DC0 nodes enlisted in MAAS - -- After the DHCP fix, all nine node VMs were reset and **all nine enlisted** - (`docs/audit/stepD-enlistment-20260720.txt`): count 9, status 8 - Commissioning + 1 New, shapes `{16cpu/64GiB: 3, 12cpu/48GiB: 2, - 8cpu/24GiB: 4}` -- an EXACT match to the D-121 Option C / R-3 layout - (3 control + 2 compute + 4 storage). -- **This closes the D-114/D-123 depth-4 boot gate, the flagged-risk item of - Model B.** A node VM inside `vvr1-dc0` (itself a VM on vcloud) PXE-booted - from the Office1 region across the D-124 transit and is running MAAS's - ephemeral commissioning kernel -- i.e. real nested KVM at depth 4, proven - by behavior rather than by inference from `svm` being present. -- Still open in step D: commissioning completes with a power-off, and MAAS - has no power driver for these VMs yet -- the `maas-vm-host` registration - (module BUILT 2026-07-09, still UNVALIDATED) is the next piece, and it - needs `TF_VAR_maas_api_key`. -- **Revert:** `maas admin machines delete` per machine (or delete the pod - once registered); the VMs themselves are inner-root state. - -- PROPOSED REMEDY (superseded by the RESOLVED block above; kept for the - audit trail): restart MAAS on the region - (`sudo snap restart maas` on voffice1) to re-establish Temporal workers, - then re-verify dhcpd on BOTH controllers behaviorally. Blast radius is - low right now (no OpenStack workloads; both Office1 VMs already deployed - and unaffected by a brief DHCP/API outage), which makes this the cheapest - moment to fix it. - -## 15. Step D part 2: NEW root for MAAS registration; and a MEASURED REFUTATION -## of D-123's "power_address = qemu:///system LOCAL to the rack" - -- SHIPPED `opentofu/vr1-dc0-maas/` -- a THIRD root holding the MAAS - registration alone. Rationale is DOCFIX-179's own measured lesson applied - rather than repeated: a `provider "maas"` block forces EVERY plan in its - root to demand `maas_api_url` + the SENSITIVE key, even for plans creating - zero MAAS resources (the DOCFIX-175 plaintext-secret surface). Putting it - in the inner substrate root would make routine substrate work require MAAS - creds. Substrate roots stay credential-free. Validated explicitly -- - **note `opentofu-validate.sh` covers only the outer root + modules, so - neither extra root is gauntlet-covered; extending it stays QUEUED and is - now more load-bearing (two uncovered roots).** -- Secret handling: key supplied ONLY as `TF_VAR_maas_api_key` from the - operator-placed 0600 file; verified by FORMAT alone (71 bytes, 3 - colon-separated parts), never printed. Caveat recorded in the root's - header: the key DOES land in that root's tfstate (unavoidable with this - provider), so that state inherits credential handling. -- Plan was clean (1 to add). **APPLY FAILED:** - `ServerError: 503 ... Failed talking to pod: Failed to login to virsh - console.` -- **ROOT CAUSE, MEASURED (not inferred):** the MAAS snap is strictly - confined. From inside it, `virsh -c qemu:///system` -> - `Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission - denied`. And `snap connections maas` lists **NO libvirt interface at all** - -- there is nothing to `snap connect`. So a LOCAL `qemu:///system` VM host - is ARCHITECTURALLY IMPOSSIBLE with snap-based MAAS. -- **THIS REFUTES A DOCUMENTED DESIGN LINE.** D-123 Model B and - `modules/maas-vm-host`'s header both state the DC pod is registered with - `power_address = qemu:///system` LOCAL to the rack ("vvr1-dc0 is BOTH the - MAAS rack AND the libvirt host, so no cross-fiber virsh dial"). The - no-cross-fiber INTENT survives; the stated MECHANISM cannot work. Both - surfaces need an amendment once the replacement is ruled. Roosevelt- - relevant: the same constraint hits every DC pod and any snap-MAAS metal - build, so this is not a VR1 quirk. -- Feasibility of the replacement, MEASURED: the snap DOES ship an ssh client - (`/snap/maas/41649/usr/bin/ssh`), and `jessea123` on the rack is in the - `libvirt` group (113) -- so `qemu+ssh://jessea123@172.31.0.2/system` - reaches libvirt through sshd (unconfined) instead of the blocked socket. - It needs a key the snap's root can read (`/var/snap/maas/current/root/.ssh/` - does not exist yet) plus that pubkey in the rack user's authorized_keys. -- **Revert:** the failed apply created nothing (state empty); `git rm -r - opentofu/vr1-dc0-maas` removes the root. - -## 16. The pod path, chased to its end -- and a SECOND refutation: MAAS virsh -## pods are incompatible with this repo's node-vm disks - -Sequence, each step measured (operator ruled the DEDICATED key: "Dedicated -MAAS->libvirt key (Recommended)"): -1. Minted `maas-virsh_ed25519` on the rack (0600, SEC-009 folder), pubkey into - the rack user's authorized_keys, private half into the RACK's MAAS snap. - Rack snap could then `virsh -c qemu+ssh://...` fine -- **but the apply still - failed the same way.** -2. MEASURED why: MAAS attempts the pod connection from the **REGION**, not the - rack. voffice1's snap had no key and no known_hosts -> `Host key - verification failed`. Same key + keyscan installed there; region snap then - listed the inner domains. (Worth keeping: for a virsh pod, the credential - belongs wherever MAAS dials FROM, which is not necessarily the rack that - owns the hardware.) -3. Next apply hit `Pod vr1-dc0-inner ... already exist` -- the two earlier 503s - had EACH created a pod server-side before erroring, so MAAS held two broken - `cores=0` pods outside tofu state. Deleted both (own residue), re-applied. -4. Next apply reached the storage inventory and failed: - `Virsh command ['domblkinfo','vr1-dc0-compute-01','vda'] failed: internal - error: missing storage backend for 'volume' storage`. -- **ROOT CAUSE, reproduced LOCALLY on the rack** (so not a snap/SSH artifact): - `virsh domblkinfo vr1-dc0-compute-01 vda` fails identically at the console - with an ACTIVE, autostart pool. libvirtd cannot run domblkinfo against disks - declared as pool+volume references -- which is exactly how `modules/node-vm` - builds them. **MAAS virsh pods are therefore incompatible with this repo's - node-vm disk shape**; making the pod work would mean converting node-vm to - file-path disks and re-applying all nine domains. -- **BUT THE POD IS UNNECESSARY.** Its designed job (D-103) was DISCOVERY, and - discovery already happened via PXE enlistment. All MAAS still needs is - POWER. MEASURED, working: per-machine `power_type=virsh` + - `power_parameters_power_{address,id}` on the canary (`ace-oyster` = - `vr1-dc0-control-01`, matched by MAC) -> `maas admin machine - query-power-state` returns `{"state": "off"}`. No pod, no domblkinfo, no - module change, no touching nine live domains. -- Roosevelt note that makes this more than expedience: on bare metal each - machine carries its OWN power parameters (IPMI/BMC). Per-machine power is - the SHAPE Roosevelt uses; the virsh pod is a VR1-only convenience. So the - per-machine route is the smaller Roosevelt delta as well as the smaller - change here. -- Both routes are recorded for the operator's ruling; nothing further applied. - -## 17. STEP D COMPLETE: per-machine power shipped as tooling; nodes COMMISSIONING to Ready - -- **RULING RECORD (GA-R5), 2026-07-20.** Question: pods are incompatible with - node-vm's pool+volume disks (measured) and the pod's discovery role is - already satisfied by PXE -- per-machine virsh power, or convert node-vm to - file-path disks and keep the pod? Operator answer, exact utterance (option - selected): "Per-machine virsh power (Recommended)". [Amends D-103/D-123's - stated vm-host mechanism; amendment text queued for stage close.] -- SHIPPED `scripts/maas-node-power.sh` + `tests/maas-node-power/` (24/24 PASS - first run; **gauntlet 71 -> 72 harnesses ALL GREEN**; repo-lint 0 fail) -- - delivered as TOOLING rather than nine hand-typed CLI calls so dc1 and any - re-run are repeatable. Design points the harness pins: - (i) matching is **BY MAC, never by name** -- MAAS assigns random hostnames - at enlistment (`ace-oyster`, not `vr1-dc0-control-01`), so a name match - would map nothing or, worse, the wrong node; - (ii) dry by default; - (iii) every write is verified by an actual `query-power-state` -- a stored - power parameter proves nothing about whether MAAS can REACH the hypervisor; - (iv) the measured pod-refutation reasoning travels in the script header. -- Prereqs measured and fixed on the region: it had **no `virsh`** (it is the - MAAS/LXD host, not a libvirt host) -> installed `libvirt-clients`; and the - `maas` CLI profile was root-only -> created a user profile with `maas login - ... -` reading the key from STDIN (never printed, never in argv). -- LIVE RESULT (`docs/audit/stepD-power-20260720.txt`): 10 domains found, **9 - mapped and set, each confirmed `power state: off` by query**. The 10th - (the edge) is correctly unmapped -- it is not a MAAS machine. -- **COMMISSIONING NOW WORKS END TO END:** all 9 commissioned; at the time of - writing **3 Ready, 6 Commissioning**, shapes still exactly D-121 Option C. - `Ready` means MAAS commissioned the node and it is deployable -- the whole - point of step D. The remaining six finish unattended. -- **Revert:** `maas admin machine update power_type=` (clears power) per - machine; the script itself reverts by commit. - -## 18. OPEN: 6 of 9 nodes fail commissioning ("timed out"); NOT contention; blocked -## on node observability. Step E not started. - -- After power was wired, all 9 were commissioned at once: **3 reached Ready, - 6 hit MAAS's 30-minute timeout** (`Marking node failed - Node operation - 'Commissioning' timed out after 30 minutes`). -- FIRST HYPOTHESIS (resource contention -- 9 nodes x depth-4 nested, 384 GiB - of 416 GiB, all pulling ephemeral images through one /30 transit) was - TESTED AND REFUTED: a single failed node re-commissioned ALONE sat in - `Loading ephemeral` for 20+ minutes with the rack at **385 GiB free and - load 0.04**. It is not contention and not memory. -- Also RULED OUT by measurement: rack boot-image sync (`list-boot-images` - -> `status: synced`, 10 images), and DHCP itself (these same nodes - enlisted fine over PXE earlier, which is what proved the depth-4 gate). -- **BLOCKER IS OBSERVABILITY, and it is the same gap twice:** - `modules/node-vm` defines **no serial console** (confirmed in the module - and in the live domain XML -- no serial/console devices), so a node stuck - in `Loading ephemeral` is a sealed box: no console, no agent, no way to - see the boot. This is the identical finding already logged against - `modules/cloudinit-vm` (item 4), now proven to bite a second module and an - actual live diagnosis. **The fix is the same opt-in serial+log pattern - `modules/opnsense-edge` already carries** -- that module's console log is - what made the 2026-07-12 boot bugs legible, and it is exactly what is - missing here. -- NOT DONE, deliberately: adding a console to node-vm means re-applying - domains mid-diagnosis, which is a change to a built surface and belongs to - an operator ruling, not a reflex. -- Current fleet: 3 Ready (commissioned, deployable), 6 Failed/retrying. - Nothing is lost -- failed commissioning is re-runnable at will. -- **Step E (netem) NOT STARTED.** Measured prerequisites for when it is: - the dc0<->dc1 mesh is **virbr5 ON VCLOUD** (note: virbr5 INSIDE the rack - is provider-public -- same number, different host, measured not assumed); - its qdisc is currently default (`noqueue`). D-100 rules the parameters as - a PLACEHOLDER for the rehearsal within a "same-metro dark-fiber, low - single-digit ms" lean. Applying to the dc0<->dc1 leg is both the - semantically correct link for D-100 and the SAFE one -- it carries no - traffic (dc1 does not exist), unlike the transit, which is currently - load-bearing for MAAS/PXE. **`modules/netem-link` cannot run unattended - from here: it shells out to `ssh 'sudo tc ...'` and assumes - PASSWORDLESS SUDO, which vcloud's operator account does not have** -- a - documented module assumption that this host violates. - -## 19. node-vm console SHIPPED; commissioning narrowed but UNRESOLVED -- and two -## of my own experiments were INVALID (disclosed) - -- SHIPPED (operator-ruled): `modules/node-vm` now defines a serial console + - boot log, with the log directory PARAMETERIZED (`serial_log_dir`) rather - than opnsense-edge's hardcoded vcloud literal, which is wrong on any other - libvirt host. Applied to all 9 domains: **0 add / 9 change / 0 destroy**, - in-place, nothing destroyed. validate PASS, repo-lint 0 fail. -- **The console did NOT yield the hoped-for evidence, and that is itself a - finding:** the logs stay 0 bytes even after a clean domain restart. Cause: - SeaBIOS/PXE firmware writes to the VGA console, not serial, and MAAS's - ephemeral kernel is not told `console=ttyS0`. **A serial console alone does - not make a PXE-booting node observable** -- seeing the firmware/ephemeral - stage needs a graphics device + `virsh screenshot` (or a MAAS-side kernel - cmdline change). Queue this as the correction to the node-vm console work; - do not assume the console added today closes the observability gap. - -WHAT IS SOLIDLY ESTABLISHED (all measured): -- PXE works and the ephemeral handoff happens: event log shows `Performing - PXE boot` 18:17:03 -> `Loading ephemeral` 18:17:45 -> silence -> timeout. -- The ephemeral OS BOOTS AND HAS WORKING NETWORKING: tcpdump on metal-admin - caught 5-6 nodes holding DHCP leases (10.12.8.112-.117) and exchanging NTP - with the rack. So node->rack reachability is fine. -- No traffic toward the REGION (10.10.0.20) or the metadata ports (5248/5240) - was seen in the sampled windows. SEC-010 was suspected of blocking the - node->region path, but its rules carry no `counter`, so drops cannot be - read directly -- **hypothesis UNCONFIRMED, not refuted.** -- Ruled out by measurement: memory (385 GiB free), rack boot-image sync - (`synced`, 10 images), DHCP (leases work), and node SHAPE (Ready and - Failed both span 16/64, 12/48 and 8/24). - -**INVALID EXPERIMENTS -- MINE, RECORDED SO NOBODY TRUSTS THEM:** -1. The first "commission a single node ALONE" test was NOT alone: tcpdump - later proved 5-6 other nodes were still up and running ephemeral images. - My conclusion "contention is refuted" was therefore UNSOUND. Contention - remains a live hypothesis. -2. The second attempt was also invalid: MAAS had the machine in - `Commissioning` already, so the new request did not restart the 30-minute - timer (the failure at 19:24:51 belongs to the 18:54 attempt), and I then - destroyed domains mid-commission, killing the in-flight run. - STOPPED HERE rather than iterate again -- this is the reconciliation - circuit-breaker: repeated cycles without convergence, and contaminated - evidence is worse than none. - -CLEANEST NEXT EXPERIMENT (not run): with ALL node domains powered off and -nothing else touching the fleet, commission EXACTLY ONE machine from a clean -`Ready`/`New` state and leave it strictly untouched for 35+ minutes. Only -that isolates contention from a genuine per-node failure. Candidate fixes to -test after it: raise MAAS `node_timeout` (nested I/O is slow and 30 min may -simply be too short), add a graphics device for `virsh screenshot`, add -`counter` to the SEC-010 rules so the node->region hypothesis becomes -readable, and check the kernel cmdline MAAS actually serves. - -## 20. CLEAN commissioning experiment: contention REFUTED (properly this time) - -- Ran the isolated experiment the earlier invalid ones only pretended to: - ALL node domains OFF (verified 0 running), 397 GiB free, load 0.05, canary - in a TERMINAL `Failed commissioning` state (so the commission restarts - MAAS's 30-min timer), commission EXACTLY ONE, poll READ-ONLY only, touch - nothing. -- RESULT: `Failed commissioning after 1770s` (~29.5 min = the 30-min - timeout), a SINGLE node entirely alone. **Contention is refuted** -- it is - not I/O bandwidth, not memory, not the metadata service being swamped. The - failure is systemic and per-node. -- This CORRECTS changelog item 18-19's contention hypothesis (which rested on - contaminated runs). The event trace is consistent every time: `Commissioning` - -> `Performing PXE boot` -> `Loading ephemeral` (~40s later) -> 30 min of - silence -> timeout. The node gets the ephemeral image handoff and then never - reports back. -- REFRAME after further measurement: "contention refuted" is right but too - narrow. The single-node fail refutes CONTENTION (competing for resources) - but is fully CONSISTENT with INHERENTLY SLOW -- the node may have still been - working at 30 min, just not done. And the batch result **3 pass / 6 fail - exactly at the 30-min mark** is the classic signature of a MARGINAL TIMEOUT - over a slow, variable operation: the faster-scheduled nodes finish under the - wire, the rest are still going when the timer fires. 1770s is 29.5 of 30. -- KEY CLUE against a network/metadata block: **3 nodes DID reach Ready on this - exact rack, subnet, and metadata proxy.** So the metadata path is NOT - globally broken; a "node never reaches :5248" theory has to explain why it - works for 3 identical nodes. Infra confirmed UP: rack `:5248` listening (2 - sockets), rack->region 5240 = 301. A `:5248` tcpdump returned 0 packets but - its 220s window likely closed before the ephemeral OS reached the - metadata-fetch stage -- INCONCLUSIVE (a timing race), not evidence. -- **LEADING HYPOTHESIS (operator-facing): MAAS's default 30-min commissioning - timeout is too short for depth-4 nested commissioning I/O.** Cheap, decisive - test that needs no stall-pinpointing: raise `node_timeout` - (`maas admin maas set-config name=node_timeout value=`, e.g. 60), - commission ONE node, see if it completes. If it does, the fix is a config - value (and a Roosevelt note: bare metal commissions faster, so this is a - VR1-nested tuning, not a Roosevelt default). If it still fails well past - 30 min, the timeout is exonerated and the next probe is the metadata-proxy - access log / a console screenshot (needs the graphics-device correction to - the node-vm console -- item 19). -- STOPPED the live diagnosis here per the reconciliation circuit-breaker: - many diagnostic cycles, and the remaining hypotheses need an operator - decision (change a MAAS-wide config) rather than more agent spelunking. - -## 8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review - -- Operator directive (verbatim intent): at the proper time/place, review the - OPNsense 26.7 changelog + documentation for upgraded features and changes; - incorporate beneficial improvements (esp. hardening) into the next - deployment iteration, following project rules. -- Scheduled: STAGE-3 CLOSE (GA-R6 close-out pass, with this stage's queued - DOCFIXes). Deliverable: a findings doc citing the official 26.7 release - notes; actionable items route per GA-R3 -- hardening/profile deltas as - PROPOSED items anchored on D-129 (its 4 open sub-decisions are the likely - surface), operational items as runbook DOCFIXes. NO self-executing - incorporation: each finding is ruled per GA-R5 (one per exchange) before - touching any built surface. Target of incorporation: dc1 standup + - Roosevelt runbooks; live-edge retrofits individually gated. - -## 6. ledger-scan hardening: FAIL-OPEN is terminology, not a status - -- SEC-010's close surfaced a scanner false positive: the OPEN-row detector - (`awk toupper(cell) ~ /OPEN|PENDING/`) matched the substring OPEN inside - "fail-open" in the closed row's disposition -- the row counted OPEN - forever (same class as the GA-F15 wrap trap). Fix: strip FAIL-OPEN from - the cell before matching (`scripts/ledger-scan.sh`); harness gains a - CLOSED-row-with-fail-open fixture + nochk (45/45 PASS). Count re-verified: - 7 open, SEC-010 absent. -- **Revert:** git revert of this commit's ledger-scan.sh + test hunks. - -## 5. Host tooling: libguestfs-tools installed on vcloud (operator-run) - -- `sudo apt-get install -y libguestfs-tools` (1:1.52.0-5ubuntu3) -- enables - offline guest inspection (virt-cat/virt-ls), used for items 4's captures. - Standard KVM-host tooling; Roosevelt-transferable. -- **Revert:** `sudo apt-get remove libguestfs-tools`. diff --git a/docs/changelog-20260719-phase5-sweep.md b/docs/changelog-20260719-phase5-sweep.md deleted file mode 100644 index c89baf0..0000000 --- a/docs/changelog-20260719-phase5-sweep.md +++ /dev/null @@ -1,292 +0,0 @@ -# Changelog 2026-07-19 -- GA Phase-5 sweep session (one changelog for the whole -# session per GA-R2 amendment D1; batches append here as they close) - -Executing session for the Phase-5 repair sweep (`docs/audit/phase5-sweep-plan.md`, -reviewed PASS + S1-S5). Every batch is operator-gated at its stated gate point; -every commit names its GA-F/GA-R trace and carries its own revert note. Status -lives in `docs/CURRENT-STATE.md` (GA-R1) -- this file records what CHANGED. - -## Batch 0 -- tooling first (OPENED by operator 2026-07-19; repo-only, no cloud contact) - -1. **0.1 repo-lint L10 status-coupling check (GA-R1 rule 8 / C1)** -- commit `0b991a5`. - - WHAT: `repo_lint.py` gains L10: a change set modifying a state-bearing surface - (anything under `docs/audit/`; a `**Status:` line in design-decisions.md; a - gate row `| G |` in session-ledger.md) without touching CURRENT-STATE.md is - flagged. Pending working-tree set when dirty, HEAD commit when clean. WARN this - batch (`L10_FAIL = False`); the flip to FAIL is a one-constant change due at - Batch 1 open (sweep plan: "WARN for one batch"). - - WHY: amendment C1's Phase-5 implementation clause, verbatim. DIVERGENCE NOTED: - the sweep plan's item 0.1 paraphrase said "CURRENT-STATE gate rows", which is - vacuous (editing a CURRENT-STATE row touches CURRENT-STATE by construction); - the ratified utterance says "ledger gate rows" and the ruling wins. The ledger - carries no gate rows today; the pattern guards their reappearance. - - Harness: tests/repo-lint T34-T41 (git fixtures), 42/42 PASS. - - REVERT: `git revert 0b991a5`. - -2. **0.2 ledger-scan SEC reporting repointed at security-ledger.md (GA-R4/F3)** -- - commit `b504cb6`. - - WHAT: scan emits open COUNT + ID list + register-of-record pointer; row-level - disposition text dropped from scan output. Session-ledger machine-derived SEC - bullet re-seeded to pointer + count form. G14 evidence cell repointed at - `docs/security-ledger.md` same commit. - - WHY: F3 verbatim ("pointer and a scan-derived count, never row-level - dispositions"). MEASUREMENT CORRECTION (GA-R1/C2): G14's count said 9; its own - enumeration and the scan both measure 8 OPEN rows -- corrected to 8, divergence - noted in the cell. - - Harness: tests/ledger-scan +4 checks, 41/41 PASS. - - REVERT: `git revert b504cb6` (restores the 2026-07-18 cell incl. its count error). - -3. **0.3 ledger-scan wrap-aware next-free exclusion (GA-F15)** -- commit `527f1fd`. - - WHAT: the "next-free"/"reproduc*" exclusion now suppresses the matching line - AND the immediately following line (window exactly one line, reset per file), - so a quoted next-free value whose hyphenated token wraps no longer counts as an - assignment. Chose this over assignment-site counting: DOCFIX/BUNDLEFIX have no - single register file to key assignment sites on (unlike D headers). - - WHY: GA-F15 -- the wrap class fired twice live inside the audit itself - (CURRENT-STATE section 9, now marked FIXED). Live counters re-verified - UNCHANGED post-fix: D=130, DOCFIX=197, BUNDLEFIX=052. - - Harness: tests/ledger-scan +3 checks, 44/44 PASS. - - REVERT: `git revert 527f1fd`. - -4. **0.4 record-audit status scan keyed to the legal token set (GA-F10 / vocab H1)** - -- commit `f7580fd`. - - WHAT: extractor knows Option A: legal OPEN/BLOCKED/HELD/CLOSED (OPEN was - previously UNSCANNED -- post-migration OPEN-vs-CLOSED contradictions would have - been invisible); DONE/TO RUN/RUNNING added; new report section 6 classifies - every distinct token and flags illegal tokens + event words (VOID/REOPENED/ - SUPERSEDED) minted in status position. Prose event words stay legitimate - row-history vocabulary. Report-only: 0/1/2 exit contract unchanged. - - WHY: H1 requires the scan before any token migration (Batch 2.5 dependency). - Live sanity run (scratch --out; the committed Phase-1 capture untouched): - 12 distinct tokens, 163 flagged counted occurrences = Batch 2.5's work-list. - Contradiction groups 12 -> 13 on the scratch run: the OPEN-token detection - gain, not drift. - - Harness: tests/record-audit +9 checks, 33/33 PASS. - - REVERT: `git revert f7580fd`. - -Batch 0 verification: all three touched harnesses green (42/42, 44/44, 33/33), -full gauntlet + repo-lint results recorded in the batch completion report and the -ledger entry for this session. - -## Batch 1 -- D-130 verification pair, decision, and the plan gate rebuild -## (OPENED by operator 2026-07-19) - -1. **L10 flip WARN -> FAIL at batch open** -- commit `9683a18`. One constant - (`L10_FAIL = True`) + harness rc updates, 42/42. Per the Batch 0 revert note. - REVERT: `git revert 9683a18`. - -2. **1.1 v8 plan-only verification (GA-F01, G4)** -- captures - `docs/audit/outer-plan-20260719-baseline.txt` (7/2/7, zero drift from - 2026-07-18) and `outer-plan-20260719-v8-ignorechanges.txt` (6/2/6, ZERO - forces-replacement). Suppression CONFIRMED; the operator's contingent lean - survived its verification. - -3. **1.2 v7 throwaway exercise (GA-F01, G4)** -- capture - `docs/audit/throwaway-v7-20260719.txt`. Isolated `throwaway-v7-*` root - (own scratch pool under the AppArmor-covered parent, no networks, no OS): - defect reproduced (seed volume "must be replaced" after staging-ISO loss); - replacement applied UNDER the running domain, rc=0, NO bounce (domid 3->3); - teardown verified zero residue (0 domains, 0 pools, dir gone). New fact - recorded: the bounce path is the autostart/domain Update, not the volume - replace. - -4. **1.3+1.4 D-130 ADOPTED (a) + implemented** -- commit `dac10e0`. - GA-R5 ruling (question + utterance in the D-130 register entry); - `lifecycle { ignore_changes = [create] }` on `libvirt_volume.seed` in - `modules/cloudinit-vm`; NEW harness `tests/cloudinit-vm/run-tests.sh` (6/6: - guard placement, exact attribute, no leak onto the data disk, tofu - validate). G4 + G5 CLOSED. REVERT: `git revert dac10e0`. - -5. **1.5-1.7 G6 reconcile + triple + G7 re-plan** -- commit `5a4f5eb`. - Operator ruled (ii) state surgery (S3/GA-R5): autostart:true injected into - both domain resources (were None), serial 22->23, backup - `opentofu/terraform.tfstate.pre-G6-20260719`; guests untouched (ids 1/2, - running). Re-plan capture `docs/audit/outer-plan-20260719-postG6.txt` = - **6/0/6**, equals the section-5 recorded triple exactly -- G6 + G7 CLOSED. - REVERT: docs via `git revert 5a4f5eb`; state via push of the pre-G6 backup. - -## Out-of-batch: permission posture extension (operator-directed 2026-07-19) - -- WHAT: `.claude/settings.json` `ask` list extended -- `Bash(git merge*)`, - `Bash(git revert*)`, `Bash(git reset*)`, `Bash(git rebase*)`, - `Bash(git branch -D *)`, `Bash(bash scripts/* apply*)`, - `Bash(bash scripts/* install*)`, `Bash(ssh * sudo *)`, `Skill(*)`. - Nothing added to `allow`; `deny` + the PreToolUse guard hook unchanged. -- WHY: operator drives sessions via Remote Control without a second terminal; - remaining auto-mode classifier HARD-DENIALS (walls) on mutation surfaces - matching no ask rule blocked the approve flow entirely (hit live: a Skill - invocation hard-denied 2026-07-19). Extends the 2026-07-18 "ask preempts - classifier" posture; targeted ask rules chosen over a catch-all `Bash(*)` - per doc-cited guidance (ask appears to preempt allow -- a catch-all would - destroy frictionless reads). `Skill(*)` syntax is undocumented; verified - EMPIRICALLY (post-change Skill invocation prompted + ran; pre-change - identical call hard-denied). Remaining walls get their own ask rule when - hit -- the settings edit itself now prompts instead of denying. -- REVERT: `git revert` this commit, or `cp .claude/settings.json.bak` per the - operator backup if the bak is refreshed first. - -## Batch 2 -- record fixes (OPENED by operator 2026-07-19; commits per-GA-F) - -1. **2.4 GA-F05 hostname** (`02af860`): vcloud on CLAUDE.md + SKILL.md + a - discovered fourth surface (environment.md:43). Memory copy = Batch 3. -2. **2.2 GA-F03 tofu pins** (`b5981a9`): four sites demoted to the - CURRENT-STATE section-7 pointer; the one LIVE false claim (workflow :360) - corrected. A fifth site (platform-traps:19) fixed at batch close. -3. **2.3 GA-F04 OPNsense rows** (`6232dd3`): skill pin row -> pointer; trap - sections requalified as entered-at-26.1-persisting; env.md's stale tofu - pin (GA-F03 class) caught on the same surface. -4. **2.1 GA-F02 readiness doc** (`360e8d6`): three contradictory plan-count - forms -> section-5 pointers; falsified deploy-ready blockquote reframed - as history; fresh-session banner -> one-line pointer at the G9 canonical - entry doc (operator scope addition, probe hole H1). -5. **2.5+2.6 H1 migration + GA-R1 demotions** (`47423c6`): mapping record - docs/audit/h1-token-map-20260719.md (no token unmapped); workflow doc 26 - flags -> 0 (8 State lines demoted, tracker model dropped from the - header); runbooks 18 -> 0 with close-out instructions RETARGETED at - CURRENT-STATE; CURRENT-STATE 6 -> 0 (G9 STOPPED -> BLOCKED w/ named - blocker; stale "GA-R1 NOT yet ratified" header fixed per C2); gate rows - gained GA-R6/E1 [V]/[R] type tags; CLAUDE.md expectation line demoted - (S5b). History classes deliberately untouched (Batch 4 / GA-R3). -6. **2.8 GA-F14 banner** on the D-127 changelog (content preserved). -7. **2.9 GA-F11/F12 adjudication**: one real live version row fixed - (env.md:29 -> pin authorities); 192 non-history broken-ref rows - adjudicated ALL false-positive (provider-repo doc paths, glob/range - forms, slash-phrases, wraps) or VR0-era history docs routed to Batch 4. - Spot-checks: as-built:59 "netbox/tailscale" = slash-phrase; workflow:473 - retired-script mention = history; env.md:86 "phase-00..08" = range glob. -8. **2.10 S1 R3-F disposition**: all 13 items accounted -- F01-F07/F09/F10 - closed by the 2026-07-16 Phase A/B + rulings + SEC-010 artifact; - F08 CLOSED-SUPERSEDED (R-3's 4th storage node restored rebuild - headroom); F11 FIXED (supernet-unassigned clause dropped; vr1-dc1 stays - HELD on transit addressing, G12); F12 CLOSED (committed budget model, - 186 GiB headroom covers transient N+1 amphora; Stage-6 drill exercises - the real path); F13 CLOSED as recorded observation. Stale readiness C5 - queue corrected (it queued amendments that landed 2026-07-16). -9. **2.7 GA-R3 D-tags (operator-ruled)**: D-121..125 [ARCH], D-126..129 - [OPS], D-130 [ARCH]; register template gains the class-tag convention; - D-129's ARCH fragment (Roosevelt metal-edge profile) split to a dated - D-122 AMENDMENT per A1 (utterance: "Tags as proposed + D-122 - amendment"). - -**Batch-2 verification + the criterion-mechanics finding.** Extractor -re-run: live-surface vocab flags = 0; the remaining flagged occurrences -(110) and contradiction groups (12) sit ENTIRELY on history-class surfaces -(session ledger, changelogs, the append-only register, VR0-era docs -- -Batch 4's rotation/consolidation) or are extractor MECHANICS: the global -plan key conflates different roots' plans (Stage-1's 13-object plan vs the -DC0 triple), charm channels/step numbers read as juju versions, the -provider (2.7.2) vs MAAS-server (3.7.2) attribution, compatibility -thresholds (LXD >= 6.7), and prose like "Stage-4 concerns" keying a stage -status. FINDING (logged, not silently absorbed): the sweep plan's Batch-2 -exit criterion "contradiction groups on counted non-changelog surfaces -reach ZERO" is not literally computable by the extractor's changelog-only -exclusion -- ledger/register/finding classes are counted non-changelog but -their ruled disposition is Batch 4/GA-R3 retention, and the status -authority itself must narrate history (7/2/7 -> 6/2/6 -> 6/0/6) to comply -with GA-R1 rule 2. The achieved, verifiable state: zero contradiction -among LIVE-surface status claims (mechanical classification script in the -session record); final ZERO-on-counted comes with Batch 4's rotation + -Batch 6.1 as the plan already sequences. - -## Batch 3 -- memory hygiene (OPENED by operator 2026-07-19; edits per-file gated) - -Pre-edit evidence: full memory dir copied to docs/archive/memory-20260718/. -1. **GA-F06**: `review-after-each-step.md` REMOVED. Repo copy of the - provenance ruling: grounding-audit-20260718.md GA-F06 ("It was not a - standing instruction. It was only supposed to be for a one time / very - short sequence of steps."). No durable observation survives it -- the - drift-catch anecdote is repo-recorded in the review changelogs. -2. **GA-F07**: `project-posture-closed-test.md` REMOVED. Repo copy of the - provenance ruling: GA-F07 (stance never ruled; posture returns NEUTRAL; - the closed-test FACT is repo-carried; SEC gates were never deferred). -3. **GA-F05**: hostname -> vcloud in `multi-workstation-remote-control.md` - (the last of the four surfaces). -4. **GA-F08**: MEMORY.md index rebuilt -- two defective lines gone, the two - surviving entries re-described accurately. -5. **GA-R7 rule 6 skill line**: "Claude Code sessions launch from the repo - root, always" added to SKILL.md's session-bootstrap loop with the - GA-F06-origin rationale (subdirectory launch = divergent auto-memory). -6. **G1 full review (rules 1-4)**: `creds-folder-convention.md` found - duplicating a repo-carried convention + linking both removed entries -- - registered as **GA-F16** and reduced to a pointer entry (one - possibly-unduplicated fact retained with a verify-then-graduate note). - `multi-workstation-remote-control.md` reviewed: workflow facts operator- - confirmed (the 2026-07-19 walls-vs-gates entry quotes a same-session - directive; policy copy lives in the repo changelog it points at). - Post-review state: 3 files, zero rule-1..4 violations, zero stale refs - (grep-verified). -REVERT: restore any file from docs/archive/memory-20260718/ (pre-edit -copies); git revert the batch commit for the repo surfaces. -Note on the archive copies: `memory-20260718/MEMORY.md`'s em-dashes were -ASCII-normalized (`--`) to satisfy the repo L1 rule; the transform is -whitespace/punctuation-only and content-faithful (this note is the record -of it). No gauntlet run for this batch: zero script/tooling changes -(memory + docs + one skill prose line); repo-lint 0-fail + fence check OK. - -## Batch 4 -- consolidation + rotation (OPENED by operator 2026-07-19) - -- **C1 ledger rotation (GA-R4 rule 6, GA-F13)**: 1187 -> 131 lines; bodies - verbatim to docs/archive/session-ledger-rotated-20260719.md; lessons/facts - ROUTED first (format=line trap -> appendix-A; guard discipline -> - operating-discipline; VR0 facts + admin/controller -> maas-as-built; - lessons 1-6 verified already covered). F1 cap now enforceable. -- **C2-C5 changelog consolidation (GA-R2, GA-F09)**: 96 files (95 + the - v1-redeploy running record) -> docs/archive/changelogs/, four stage - records under docs/archive/stage-records/ (v1-era; stage0-1 = 45; stage2 - = 38; stage3-partial = 11), each commit message MANIFESTS its sources - (D2). The live session changelog stays top-level until session close. - NOTE: v1-redeploy-changelog's own "completion consolidation" TODO is NOT - discharged by the move -- it remains in the ledger's project-completion - block. -- **C6 docs disposition (D2 scope)**: 24 non-changelog history docs -> - docs/archive/ (v1-era packs/handoffs/findings/drafts; executed plans; - the R3-F register; Model-A fallback; DR seed; ipv6/netem research; - D-068-openbao DRAFT). RETAINED top-level, 16 files each justified: - status authority, registers (D/SEC), ledger, workflow (identity), design - doc, readiness checklist, two as-builts, tenant contract (lint L7 - auto-covered), log convention, live finding doc (CURRENT-STATE-cited), - D-129 review (open sub-decisions), NetBox scope (live apex reference), - D-068 analysis (open-decision evidence), live session changelog. - Live-surface references to every moved file rewritten to archive paths - (18 files touched); history-internal refs left verbatim. - -## Batch 5 -- skill sweep (OPENED by operator 2026-07-19; single commit) - -SKILL.md reconciled against the RATIFIED GA-R texts (not the draft): new -"Record authority and session discipline" section (~35 lines per the -draft's budget); header phase claim + the false "opentofu UNVALIDATED / -no tofu binary" claim DEMOTED to CURRENT-STATE delegation; session -bookends rewritten (OPEN = CURRENT-STATE first; CLOSE = 15-line summary -+ archived body + push); stage-close loop gains the GA-R6 close-out set -(named-check closes, no conditional close, consolidation commit, memory -review); change-delivery loop gains D1 (one session changelog, never -citable); routing rows re-pointed (VR1 status, versions, resume). The -draft's reconciliation circuit-breaker KEPT as discipline (no ruling -quote -- marked as such). Line-by-line checklist: -docs/audit/skill-sweep-checklist-20260719.md. 176 -> 224 lines. - -## Batch 6 -- Phase-6 exit runs (OPENED by operator 2026-07-19) - -- **6.1 ruling + run**: operator ruled the docs/archive/ counting exclusion - IN ("Yes, extend exclusion"); record-audit.py + harness updated (35/35). - Post-run: 9 groups, per-group adjudication = ZERO contradictory pairs - (full verdict table: docs/audit/phase6-exit-runs-20260719.md; group-count - floor 9 recorded as the re-run baseline). Missed-disposition fix: - stage3-review-base.patch archived; CURRENT-STATE known-stale note made - token-free. -- **6.2**: docs/*.md = 16 (< 25), history preserved via git mv + manifests. -- **6.4**: fresh capture outer-plan-20260719-phase6-exit.txt = 6/0/6 == - section-5 triple EXACTLY. -- **6.5 (S2)**: G9 row now names SEC-010's step-B apply+verify dependency - (the only apply-gated SEC row). -- **6.3 + 6.6 PENDING operator execution**: probe trio (3 clean sessions, - auto-memory OFF, 7-question set, 7/7 on 2-of-3) + the top-to-bottom - re-read and re-signature. - -## SESSION CLOSE (2026-07-19) - -Batch 6 completed: probe trio 21/21 (7/7 all three); operator re-signed -CURRENT-STATE ("Reviewed, approved, continue."). ALL exit runs PASS; the -grounding audit is EXITED, the freeze lifted, G9 OPEN. This changelog is -the session's complete WHAT-CHANGED record; it consolidates into the -Stage-3 record at stage close per GA-R2. diff --git a/docs/changelog-20260721-close-and-delivery.md b/docs/changelog-20260721-close-and-delivery.md deleted file mode 100644 index 9ccf6bc..0000000 --- a/docs/changelog-20260721-close-and-delivery.md +++ /dev/null @@ -1,140 +0,0 @@ -# Session changelog 2026-07-21 (successor session) -- disconnect recovery + queued delivery - -One changelog per session (GA-R2/D1). Status lives ONLY in -docs/CURRENT-STATE.md; this file is narrative + reverts. Predecessor -session (ops-commissioning-diag, disconnected before its bookend): -docs/changelog-20260721-commissioning-diag.md. - -## 1. Predecessor session close LANDED (commit 8b3adb3) - -- The ops-commissioning-diag session disconnected before committing its - deliverable. This session re-verified read-only first (9/9 nodes Ready - in MAAS with virsh power; dc0-node-dns active+enabled with the 10.12.8.3 - alias up; dhcpd running on BOTH controllers; diag-serial tag deleted and - serial-console re-attached to hardy-dove = Phase-7 cleanup confirmed - done), then committed the predecessor's 7 files verbatim plus the - bounded session-ledger close entry and a machine-derived block re-seed - from the 2026-07-21 scan (D next-free 132, DOCFIX 197, BUNDLEFIX 052, - 10 open SEC rows). Operator-approved this session. -- **Revert:** git revert 8b3adb3 (returns the record to the - not-yet-durable state; the live cloud is untouched by that commit). - -## 2. MAC pinning in modules/node-vm (queued item 9a of the predecessor) - -- modules/node-vm: new OPTIONAL `interface_macs` variable (list(string), - default [] = unpinned for pre-enlistment use; validations: all-or- - nothing length vs network_names -- partial pinning is rejected -- and - 6-byte hex format). Interfaces comprehension consumes it as - `mac = { address = ... }` -- shape VERIFIED against the provider schema - (dmacvicar/libvirt 0.9.8, `tofu providers schema -json`, this session). -- opentofu/vr1-dc0-substrate: all 9 nodes pinned. 54 MACs MEASURED this - session (`virsh domiflist` on vvr1-dc0, read-only; order = the - vr1_dc0_node_nics plane order) and the 9 metal-admin entries - cross-checked against MAAS boot-interface records 9/9 (machines read). - No value inferred; no value reused from the adjudication file without - re-measurement. -- NEW harness tests/node-vm/run-tests.sh (12 cases: variable shape, - validations, consumption, 9 lists / 54 literals / 0 duplicates in the - inner root, pass-through, tofu validate). Gauntlet 72 -> 73 ALL GREEN; - repo-lint 0 fail; opentofu-validate PASS (both EXTRA roots). -- NOT YET APPLIED: the inner root's state (on voffice1) does not carry - the pins until an operator-gated `tofu apply` there. The plan gate for - that apply: expected NO REPLACE -- pins equal live values, so the plan - must show only benign in-place mac adoption or no-op; ANY replace is a - STOP (present to operator, do not apply). -- **Revert:** git revert of this delivery commit (module + root + harness - together); live domains unaffected until the gated apply. - -## 3. node-vm power-ownership guard (operator-ruled this session) - -- The MAC-pin verification plan on voffice1 (captured: - docs/audit/inner-plan-20260721-macpin.txt -- 0/9/0, 54 mac adoptions, - ZERO replaces) exposed an entanglement: the module hardcodes - `running = true` while MAAS holds all 9 Ready nodes OFF, so the apply - would ALSO have powered on all 9 nodes outside MAAS's control. -- RULING (this session): question = how to proceed with the entangled - apply; operator selection = "ignore_changes on running first - (Recommended)". -- Shipped: `lifecycle { ignore_changes = [running] }` on the node domain - (create still boots for PXE enlistment; after that MAAS owns power -- - virsh here, IPMI on Roosevelt). Harness grew 3 cases (T13-T15: guard - present, exactly one ignore_changes, cites MAAS). 15 cases green. -- **Revert:** remove the lifecycle block + T13-T15; the 9 power-on - changes return to the next plan. - -## 4. MAC-pin apply EXECUTED on voffice1 (operator-approved) - -- Guarded re-plan captured (inner-plan-20260721-macpin-guarded.txt: 54 - mac adoptions, zero power flips, zero replaces), then saved-plan apply - (`tofu plan -out` -> `tofu show` re-verify 0/9/0 -> `tofu apply`): - exact 0/9/0, convergence re-plan "no differences" - (docs/audit/inner-apply-20260721-macpin.txt). Post-apply verified - read-only: all 9 domains still shut off, edge untouched, spot-checked - MACs identical (control-01, storage-04). -- The inner tfstate on voffice1 now carries the pins (state serial - advanced; it remains the state-of-record per CURRENT-STATE section 1). -- **Revert:** git revert the pinning commits, re-plan/apply on voffice1 - (returns MACs to provider-owned -- NOT recommended; reopens the drift - hole that caused the 2026-07-21 incident). - -## 5. dc-rack-net.sh: D-131 sub-1 delivery (forwarder + rack legs, repo-carried) - -- D-131 sub-1 RULED this session (standing per-DC pattern; ruling commit - fad3007), then delivered: NEW scripts/dc-rack-net.sh - -- site-keyed (dc0 rows MEASURED), runs ON the rack via piped - ssh. Owns: -rack-legs.service oneshot (re-adds 10.12.8.2/22 + - 10.12.4.2/22 + the 10.12.8.3/22 forwarder alias on boot -- the D-126 - reboot-loss class), the D-131 dnsmasq forwarder unit+conf (generated, - content-checked by diff in check mode), and a boot helper that - resolves bridges FROM LIBVIRT NET NAMES at runtime (virbrN is - auto-assigned = drifting ID; measured: dc-planes does NOT pin bridge - names). Region-side MAAS subnet setting stays a runbook step (same - split as site-headend-install's DHCP note). -- Delta from the live hand-placed artifacts (intentional, minimal): the - legs move from bare `ip addr` (reboot-lost) into the oneshot unit; the - dns unit gains Requires/After on the legs unit instead of its own - ExecStartPre alias hack; config content otherwise identical. -- NEW tests/dc-rack-net harness (14 cases incl. no-virbrN-literal, - MEASURED-tag discipline, read-only check path). Gauntlet 73 -> 74 ALL - GREEN; repo-lint 0 fail. -- Rack install EXECUTED (operator-approved, same session): EXIT 0, - self-check PASS 10/10 (docs/audit/dc-rack-net-install-20260721.txt); - behavioral probe: `dig @10.12.8.3 maas-internal SOA` answers through - the region. dnsmasq restart blip occurred with all 9 nodes powered - off (zero impact). The interim hand-placed unit/conf are OVERWRITTEN - in place by the generated equivalents (same paths). -- **Revert:** git rm the script + harness; on the rack (if installed): - systemctl disable --now dc0-rack-legs dc0-node-dns; rm the four - installed files; re-add the legs + old unit per changelog-20260721- - commissioning-diag item 4's revert. - -## 6. Incident documentation (queued items 9b/9c/10 of the predecessor) - -- appendix-A: two new entries -- (a) 30-min timeout with NO PXE event - for a known machine -> MAC-record drift (diff domiflist vs MAAS - interfaces; repair + pin); (b) cloud-init "no datasource" on a - rack-only DC -> agent-resolver SERVFAIL (dig SOA check; D-131 - forwarder fix via dc-rack-net.sh). -- platform-traps 1e: second corollary (in-place apply REGENERATES - unpinned NIC MACs; pin rule) + verbatim-index row. -- LP draft authored: docs/audit/lp-draft-20260721-maas-agent-resolver.md - (operator to file; sanitization warning re SEC-014 material included). -- **Revert:** git revert this commit (docs only). - -## 7. Step-E netem sudo mechanism (operator-ruled) + sudoers fragment - -- RULING (this session, OPS -- netem is VR-only, no D-number per GA-R3): - question = sudo mechanism for modules/netem-link on vcloud; operator - selection = "Scoped NOPASSWD for tc (Recommended)". -- Shipped scripts/sudoers.d/netem-tc: exactly the module's two verbs - (qdisc replace ... netem * / qdisc del ... root) per MEASURED mesh - bridge (virbr5=dc0-dc1 [the step-E target], virbr7=dc0-office1, - virbr3=dc1-office1; virsh net-dumpxml this session). Bridge names are - auto-assigned/drifting (mesh-link does not pin them) -- fragment - documents the fail-closed property + mandatory pre-apply re-measure. -- NEW tests/netem-sudoers harness (9 cases incl. exact-shape rule match, - no-broadening, visudo -cf). Gauntlet 74 -> 75 ALL GREEN. -- Install is operator-only (root; visudo-checked): command in the - fragment header. NOT installed at time of this entry. -- **Revert:** git rm fragment + harness; if installed: - sudo rm /etc/sudoers.d/netem-tc && sudo visudo -c. diff --git a/docs/changelog-20260721-commissioning-diag.md b/docs/changelog-20260721-commissioning-diag.md deleted file mode 100644 index 3588f9f..0000000 --- a/docs/changelog-20260721-commissioning-diag.md +++ /dev/null @@ -1,132 +0,0 @@ -# Session changelog 2026-07-21 -- commissioning diagnosis + repair (logged: ops-commissioning-diag) - -One changelog per session (GA-R2/D1). Status lives ONLY in -docs/CURRENT-STATE.md; this file is narrative + reverts. Full adjudication: -docs/audit/commissioning-diag-20260721.txt. As-executed log: -~/as-executed/2026-07-21-ops-commissioning-diag.log. - -## 1. Instrumented commissioning run (committee-prescribed) EXECUTED - -- Phase 1 zero-cost reads, then gated: diag-serial MAAS tag - (kernel_opts console=ttyS0,115200) on canary sw6wqe/hardy-dove only; - serial-console tag temporarily DETACHED from the canary (alphabetical - tag concatenation would have left console=tty0 last, sending userspace - output to VGA). Instruments: MAC+broadcast tcpdump on virbr2, full - tcpdump on enp1s0, live Temporal journal follow -- all 45-min capped. -- One commission, hands-off, full window. Result: timeout at 30 min, - serial 0 bytes -- and the pcap held the answer (item 2). -- FINDING (record divergence): a serial-console tag with console - kernel_opts ALREADY existed (comment cites D-114 debug) contradicting - the committed "ephemeral kernel is not told console=ttyS0" claim. -- **Revert:** diag-serial tag delete + re-attach serial-console to sw6wqe - (Phase 7, item 8). - -## 2. FAULT 1 FOUND + REPAIRED: all 9 node MACs drifted (in-place apply) - -- PXE DHCP in the capture came from a MAC absent from MAAS's records; - fleet audit: 9/9 live metal-admin MACs differ from MAAS boot-interface - records. Boundary: the 2026-07-20 in-place serial-console apply - (tofu 0/9/0) regenerated NIC MACs. Full chain in the adjudication file. -- RULING (GA-R5, 2026-07-21): question = how to repair the records; - operator selection = "Update MACs in place (Recommended)". -- Executed: per-machine boot-interface MAC -> measured live value; - Failed-state machines needed mark-broken -> update -> mark-fixed. - Read-back verified 9/9. -- **Revert:** maas admin interface update - mac_address= per - machine (old values in the as-executed log); machines return to their - pre-session states via mark-broken/mark-fixed as needed. - -## 3. FAULT 2 FOUND: MAAS 3.7 rack-only agent resolver defect - -- Post-repair proof run booted fully; cloud-init found no datasource: - the agent resolver (node DNS via DHCP) SERVFAILs everything on an - isolated rack -- walks public root hints even for authoritative - maas-internal; ignores resolv.conf. Office1 immune (region BIND owns - node DNS there). First-exercised surface in VR1. LP REPORT QUEUED - (item 10). SEC-010 re-verified enforced during diagnosis. -- Interim RULING "Region upstream, runtime": resolvectl dns enp1s0 - 10.10.0.20 + domain ~. on the rack + /etc/resolv.conf relinked to - /run/systemd/resolve/resolv.conf. Did NOT affect the agent (kept -- - gives the rack itself working resolution via region). -- **Revert:** resolvectl revert enp1s0; ln -sf - ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf. - -## 4. FAULT 2 WORKAROUND: dc0-node-dns forwarder (operator-ruled) - -- RULING (GA-R5, 2026-07-21): question = node DNS remedy; operator - selection = "Rack dnsmasq forwarder (Recommended)". -- Shipped to the rack (NOT yet repo-carried -- item 9): - /etc/dnsmasq-dc0-node.conf + /etc/systemd/system/dc0-node-dns.service - (dnsmasq on virbr2 alias 10.12.8.3, no-resolv, server=10.10.0.20; - alias via ExecStartPre ip addr replace; After=libvirtd). Enabled. -- MAAS: subnet 6 (metal-admin) dns_servers=10.12.8.3 + allow_dns=false. - dhcpd still prepends the rack IP (option 6 = "10.12.8.2, 10.12.8.3") - -- works via SERVFAIL failover (proven); template behavior to - investigate at delivery. -- **Revert:** maas admin subnet update 6 dns_servers= allow_dns=true; - on rack: systemctl disable --now dc0-node-dns; rm - /etc/systemd/system/dc0-node-dns.service /etc/dnsmasq-dc0-node.conf; - ip addr del 10.12.8.3/22 dev virbr2. - -## 5. PROOF + fleet re-commission - -- Canary hardy-dove: Ready in ~3 min after seven consecutive 30-min - failures; commissioning scripts visible on serial, rc=0. -- Fleet: remaining 8 commissioned (5 never truly commissioned -- - mark-fixed does not commission; 3 carried stale interface data). - Outcome recorded in CURRENT-STATE (same commit). -- **Revert:** n/a (commissioning is re-runnable). - -## 6. SEC-014 OPENED: rack cluster secret exposed to session context - -- agent.yaml read (diagnosing the resolver) included the `secret` field. - Exposure scope: Claude session context, operator terminal, as-executed - log (0600, jumphost). Row + disposition: docs/security-ledger.md. - Rotation = rack re-enrollment; schedule at operator convenience. -- **Revert:** n/a (exposure event; row tracks rotation). - -## 7. Session tooling: targeted ask rules (classifier walls) - -- .claude/settings.local.json (untracked, global-gitignored): ask rules - added for wrapped/unwrapped `maas admin tags|tag|machine|interface| - subnet` via voffice1 and rack-sudo via the dc0 key -- per the standing - multi-workstation pattern (operator-approved this session). Lesson: - a `sleep N;` prefix breaks prefix-matching -- keep wrapped commands - rule-shaped. -- **Revert:** remove the ask entries from settings.local.json. - -## 8. Phase 7 cleanup (pending fleet-green) - -- diag-serial tag: detach from sw6wqe + delete; re-attach serial-console. - QUEUED OBSERVATION: serial-console's opts end in console=tty0, which - claims /dev/console for VGA -- fine for kernel logs, hides userspace; - revisit the tag's opts when serial observability next matters. -- **Revert:** n/a (is itself the revert). - -## 9. QUEUED delivery work (change-delivery loop, not this session) - -- modules/node-vm: PIN NIC MACs (set to current live values) so no apply - can drift them again; harness case for MAC stability. Roosevelt note: - metal has fixed MACs; the trap is VR-only. -- platform-traps entry: tofu/libvirt "in-place" 0/N/0 domain change can - REGENERATE NIC MACs (measured 2026-07-20/21). -- appendix-A entries: (a) "commissioning times out at exactly 30 min, - NO 'Performing PXE boot' event" -> diff live domain MACs vs MAAS - records; (b) "cloud-init: no datasource, searched classes ()" on - rack-only controllers -> query the rack agent resolver for - maas-internal; if SERVFAIL see D-131/forwarder. -- dc0-node-dns forwarder + rack-legs persistence: repo-carried - script/module + tests (site-keyed, per D-128 register item 20). -- D-131 PROPOSED (authored this session, design-decisions.md): node-facing - DNS strategy for rack-only controllers (Roosevelt-relevant). -- Committee record addendum: authored this session (outcome section). -- Stale MAAS pod vr1-dc0-inner (id 4): stage-close cleanup with SEC-013. - -## 10. LP report (queued for operator) - -- Title: "maas-agent resolver on rack-only controller SERVFAILs - authoritative maas-internal queries on internet-isolated racks (walks - root hints; ignores resolv.conf); breaks commissioning cloud-config-url" - MAAS 3.7.2 / snap 41649. Evidence: adjudication file + agent journal - excerpts in the as-executed log. diff --git a/docs/changelog-20260721-netem-install-verify.md b/docs/changelog-20260721-netem-install-verify.md index 709963b..bf11a50 100644 --- a/docs/changelog-20260721-netem-install-verify.md +++ b/docs/changelog-20260721-netem-install-verify.md @@ -264,3 +264,29 @@ section 5 re-recorded to ZERO DIFF + G16 row CLOSED, same commit. - **Revert:** `tofu state push opentofu/terraform.tfstate.pre-G16-20260721` (returns serial-29 state; the 0/1/0 residual and gate reopen). + +## 17. STAGE 3 CLOSE (stage-close ruling 3) -- consolidation + skill sweep + +- RULING (GA-R5). Question as presented: "Stage-close ruling 3: close + Stage 3 now?" (full close-out set enumerated in the question). + Operator selection, exact: "Close and merge (Recommended)". +- GA-R2 consolidation: the four closed execution-session changelogs + (phase5-sweep, dc0-deploy-stepB, commissioning-diag, + close-and-delivery -- 1,402 lines) moved to docs/archive/changelogs/; + vr1-stage3-partial-record.md finalized as vr1-stage3-record.md with + the close summary + closing-era session table. THIS changelog stays + top-level until its session closes (record precedent). +- Status surfaces updated same commit (GA-R1 C1): CURRENT-STATE stage + line (CLOSED for vr1-dc0 scope; dc1 = designed HELD remainder G12), + workflow doc Stage-3 State row (+ note that its Gate line predates + the pod refutation + DOCFIX-197), runbooks/README row 3 (vm_host + language corrected) + the stale NONE-EXECUTED banner demoted to a + CURRENT-STATE pointer. +- Skill sweep (GA-R6 close item): folded the D-131 sub-1 standing + rack-forwarder invariant, the two MAAS standing traps (pods refuted + -> per-machine virsh; pod delete cascades -- check machine list + first), and the Stage-3 merge precedent. No stale facts found (the + skill delegates facts to the repo by design). +- **Revert:** git revert the consolidation commit (moves + record + + status lines return); the merge itself is reverted only by operator + decision (git revert -m 1 ). diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md index 54ce425..dcf2ad5 100644 --- a/docs/dc-dc-deployment-workflow.md +++ b/docs/dc-dc-deployment-workflow.md @@ -157,9 +157,13 @@ | **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. | | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** vr1-dc0-first (D-119 naming); vr1-dc1 explicitly hard-gated (HELD, gate G12: transit/rack addressing unruled -- its v4 supernet 10.12.64.0/19 IS assigned by D-115, R3-F11 fix) rather than run with placeholders. Originally walked a set of STOP-gate decisions; most are now RULED (see the rulings summary below): node sizing = D-121/R-3, edge/WAN shape = D-122, site-down/placement = D-123 Model B, rack transit = D-124, ISP egress = D-125, edge-config delivery = D-113(a2) REST (config.xml/ISO retired -- no more `config.xml` tokens). STILL open at execution: MAAS zone/pool + `power_address` (Stage-4 concerns), boot-measured `WAN_IF`/`LAN_IF` (real OPNsense boot), and netem params (D-100 gap #11). NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`; under Model B it targets `vvr1-dc0`'s inner virsh -> Office1 region, DOCFIX-179-deferred) all now exist AND are instantiated across the two Model B roots (D-123): the outer root (`opentofu/`) + the inner root (`opentofu/vr1-dc0-substrate/`), plus the new `modules/wan-bridge` (D-125). Both roots `tofu validate`. Remaining un-instantiated is deploy-time (MAAS zone/pool, `power_address`) not authoring. `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | -**State:** THE ACTIVE STAGE (identity: branch `dc-dc-stage3-phase2-dc-substrate`); status -authority: docs/CURRENT-STATE.md sections 1 and 6. History: runbook written + reshaped to -Model B; authoring effectively complete; execution position and gates live in CURRENT-STATE. +**State:** CLOSED 2026-07-21 for the vr1-dc0 scope (operator-ruled; status authority: +docs/CURRENT-STATE.md sections 1 and 6; stage record +docs/archive/stage-records/vr1-stage3-record.md). vr1-dc1 remains the stage's designed HELD +remainder (gate G12) and returns when its addressing is ruled. Branch merged to `main` and +retired. NOTE the Gate line above predates two recorded corrections: `maas-vm-host` +registration was REFUTED for DCs (per-machine virsh power replaced it -- D-103/D-123 +amendments 2026-07-20) and the netem target is the dc0<->dc1 leg (DOCFIX-197). **Rulings landed (through 2026-07-16; supersedes the earlier "two blocking rulings landed 2026-07-15" note):** - **D-121 (ADOPTED) + R-3** -- node layout = **Option C** = 3 control + 2 compute + **4 storage = 9 diff --git a/runbooks/README.md b/runbooks/README.md index 7770a69..b4b87f7 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -65,17 +65,19 @@ | 0 | (no runbook -- `docs/design-decisions.md`) | Decision ratification (D-100..D-110) | D-100..D-110 | | 1 | dc-dc-phase0-vcloud-prep.md | vcloud host prep + first real `tofu apply` (DC1 planes/Office1 pool/mesh links) | D-101/D-102 | | 2 | dc-dc-phase1-office1-standup.md | Office1 headend: MAAS region, OpenTofu, NetBox, GitBucket, Tailscale, Office1's own OPNsense edge | D-103/D-106 | -| 3 | dc-dc-phase2-tofu-dc-substrate.md | OpenTofu builds each DC substrate (node-VM domains, OPNsense edge, MAAS `vm_host`) | D-103 | +| 3 | dc-dc-phase2-tofu-dc-substrate.md | OpenTofu builds each DC substrate (node-VM domains, OPNsense edge; power = per-machine virsh, the `vm_host` pod route was REFUTED -- D-103/D-123 amendments 2026-07-20) | D-103 | | 4 | dc-dc-phase3-maas-enlist-deploy.md | MAAS enlist / commission / deploy, per DC | | | 5 | dc-dc-phase4-juju-bundle-per-dc.md | Juju controller + OpenStack bundle, per DC (high reuse from VR0's bundle.yaml) | D-106/D-108 | | 6 | dc-dc-phase5-dr-failover-drill.md | DR wiring (radosgw multisite, rbd-mirror) + failover drill | D-108 | | 7 | dc-dc-phase6-designate-cos-magnum.md | Designate, COS, Magnum/CAPI, per DC | | | -- | dc-dc-teardown-rollback.md | Teardown/rollback for this track's OpenTofu-provisioned infrastructure specifically -- NOT the same as `phase-00-teardown-*.sh` (D-061), which covers VR0's juju/MAAS-machine layer | -- | -**As of 2026-07-10: all seven Stage runbooks + the teardown/rollback runbook -are WRITTEN, NONE EXECUTED.** `docs/dc-dc-deployment-workflow.md`'s tooling -gap register tracks what's still open before each stage can run for real -- -read it before starting Stage 1, not just this index. +All seven Stage runbooks + the teardown/rollback runbook are WRITTEN +(2026-07-10). Which stages have EXECUTED, and where the live position is, is +a STATUS question -- `docs/CURRENT-STATE.md` only (GA-R1). The +`docs/dc-dc-deployment-workflow.md` tooling gap register tracks what's still +open before each stage can run for real -- read both before starting a +stage, not just this index. ## Appendices