diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 909bbea..cadec2e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -333,6 +333,35 @@ `scripts/dc-dc-whole-host-budget.py` before applying (the D-104 gate passed at RAM 83% for the 10-node shape and this adds a 12th domain per rack); the substrate caller is `for_each`-keyed so it plans `1 add / 0 change / 0 destroy`. + **^ THE `1 add` FIGURE IS WRONG, CORRECTED BY MEASUREMENT: it is `2 to add`** -- + `modules/node-vm` creates a `libvirt_domain` AND a `libvirt_volume` per node. Carried in + from the D-104 amendment's prose, which is wrong at resource level for the same reason. The + load-bearing half is `0 change / 0 destroy`, and that held. + **BUILD STARTED 2026-07-30 -- dc0 region VM APPLIED, dc1 AUTHORED ONLY.** + `vr1-dc0-maas-01` / `vr1-dc1-maas-01` authored into both substrate roots at + 4 vCPU / 8 GiB / 150 GiB (disk 150 not 100: this VM holds the region's PostgreSQL AND its + boot-image set). `opentofu-validate` PASS (root + 12 modules + both extra roots). + **CAPACITY GATE PASS, re-measured:** `dc-dc-whole-host-budget.py + --containment-overhead-mem-gib 32 --containment-overhead-vcpu 12` -> RAM **870/1024 = 85%, + FIT, 154 GiB headroom** (a 16 GiB variant also fits, 886/1024 = 87%). **dc0 APPLIED:** + plan asserted on CONTENT (`tofu show | grep "will be (created|destroyed|updated| + replaced)"` -> exactly two lines, both `vr1-dc0-maas-01`), applied `2 added, 0 changed, + 0 destroyed`; MACs measured and pinned; converged **ZERO DIFF** + (`tofu plan -detailed-exitcode` -> 0) with live `virsh domiflist` byte-identical to the + pin; virsh power set via `maas-node-power.sh --commit` (dry run first showed exactly one + machine needing it). Enlisted as **`hot-kid` / `tw7ptw`**, machine count 22 -> 23. + **ORDERING TRAP MEASURED AND RECORDED: pinning MACs bounces the guest and interrupts the + auto-commission the first apply triggered.** The first apply sets `running = true`, so the + VM PXE-enlists and starts commissioning within ~90s; the pin is an in-place + `libvirt_domain` update (`Still modifying... 32s`) which bounced it -- domain `shut off`, + MAAS stalled at `Loading ephemeral`. Recovery is cheap and deliberate: `machine abort` -> + `New`, then `machine commission` -> powers on via the virsh type just set. **Standing rule + for every future DC standup: after pinning MACs on a freshly-applied VM, ASSUME + commissioning was interrupted and re-commission; do not read a stalled `Commissioning` as a + fault.** Pinning before first boot is impossible while the module creates domains + `running = true`. **STILL OWED for dc0:** carve metal-admin `10.12.8.6` + provider-public + `10.12.4.6` (gw `10.12.4.1`) once Ready, deploy an OS, then install regiond + PostgreSQL. + **dc1's VM is authored but NOT applied.** - Project: Omega Cloud, VR1 DC-DC rehearsal -- a two-DC + Office1-headend virtual rehearsal on KVM (vcloud host), rehearsing the future bare-metal diff --git a/docs/changelog-20260730-stage5-open.md b/docs/changelog-20260730-stage5-open.md index 343520b..1a04948 100644 --- a/docs/changelog-20260730-stage5-open.md +++ b/docs/changelog-20260730-stage5-open.md @@ -210,3 +210,52 @@ **Revert.** `git revert ` removes D-138, SEC-026 and the CURRENT-STATE paragraph together. Nothing was executed on the cloud under this ruling yet, so no live state depends on it at the time of writing. + +### 7. Per-DC MAAS region VM AUTHORED and APPLIED on dc0 (D-132 addendum, first build step) + +**What.** `vr1-dc0-maas-01` and `vr1-dc1-maas-01` added to their substrate roots +(4 vCPU / 8 GiB / 150 GiB). dc0 APPLIED: `2 added, 0 changed, 0 destroyed`; MACs pinned +from measurement and applied; converged **zero diff** (`tofu plan -detailed-exitcode` -> 0); +virsh power set; enlisted as `hot-kid` / `tw7ptw` and commissioning. dc1 is AUTHORED ONLY, +not applied. + +**Why (evidence).** D-132's amendment + addendum, both RULED 2026-07-30. Sizing is the shape +the capacity gate actually modelled -- `dc-dc-whole-host-budget.py +--containment-overhead-mem-gib 32 --containment-overhead-vcpu 12` -> RAM **870/1024 = 85%, +FIT, 154 GiB headroom**; a 16 GiB variant also fits (886/1024 = 87%). Disk is 150 GiB rather +than the controller's 100 because this VM holds the region's PostgreSQL AND its boot-image +set. The plan was asserted on CONTENT, not on its summary line: `tofu show | grep +"will be (created|destroyed|updated|replaced)"` returned exactly two lines, both +`vr1-dc0-maas-01`. + +**A count correction, made against measurement rather than inherited prose.** Both new +entries' comments initially claimed the addition "plans 1 add / 0 change / 0 destroy", +carried from the D-104 amendment's text. MEASURED: it is **2 to add** -- `modules/node-vm` +creates a `libvirt_domain` AND a `libvirt_volume` per node. The load-bearing half of that +figure is `0 change / 0 destroy`, which held. D-104's "1 add" figure appears to be wrong at +resource level for the same reason; flagged, not edited, since it is not this item's surface. + +**THE MAC-PIN BOUNCE INTERRUPTED COMMISSIONING -- a real ordering trap, recorded because the +runbook rule as written cannot avoid it.** The standing rule is: apply with `macs = []`, then +pin from measurement "immediately after first apply, before enlistment". But the first apply +sets `running = true`, so the VM PXE-boots and BEGINS ENLISTING AND COMMISSIONING on its own +within ~90 seconds. The pin is therefore always racing an auto-commission, and applying it is +an in-place `libvirt_domain` update -- which bounces the guest (`Still modifying... 32s +elapsed`; platform-traps already records "in-place bounces the guest"). Measured result: +domain `shut off`, MAAS stuck at `Loading ephemeral`, commissioning stalled. +**Recovery, clean and cheap:** `maas admin machine abort` -> `New`, then +`maas admin machine commission` -> MAAS powers it on via the virsh power type just set. +**The generalisable rule: after pinning MACs on a freshly-applied VM, ASSUME commissioning +was interrupted and re-commission deliberately.** Do not read a stalled `Commissioning` as a +fault. The alternative -- pinning before the VM ever boots -- is impossible while the module +creates domains `running = true`. + +**Verification that the pin is real, not just written.** Live `virsh domiflist` after the +in-place update returns all six MACs byte-identical to the pinned values, and the root +re-plans to ZERO DIFF. Both were checked because an in-place domain update is precisely where +this project regenerated MACs and stranded nine nodes on 2026-07-20. + +**Revert.** `tofu destroy -target='module.vr1_dc0_node["vr1-dc0-maas-01"]'` in +`opentofu/vr1-dc0-substrate` removes the VM and its volume; then delete the MAAS machine +record (`maas admin machine delete tw7ptw`) and `git revert` this commit to drop both roots' +entries. Nothing else in either root is touched -- the plan proved `0 change / 0 destroy`.