|
Both DC VMs 416 -> 480 GiB; 128 GiB swap; new preflight gate P8 (substrate drift)
POST-BOOKEND: the GA-R4 bookend (4b8ba3c) was committed BEFORE this work, so the ledger close summary and the 08-01 sweep do not cover it. The ledger line is AMENDED in this commit rather than left stale. Operator: "Option 2 look sgood me. I approve the sequencing, process as autonomously as possible" -- +64 GiB to EACH DC with swap first, over my recommendation of +48. The deciding arithmetic, put up before the ruling: at +64 each, 994 GiB allocated against a 1007.4 GiB host = 13.4 GiB residual while the host's OWN measured footprint is 18.0 GiB, with 1 GiB swap free. Swap is what makes +64 safe, hence the sequencing. PROBLEM FIXED, MEASURED: dc0's rack ran 402 GiB of inner guests in a 409 GiB host (98.3%), leaving 7 GiB for the rack OS, mirror, snap proxy and page cache. Each rack now has ~71 GiB. SWAP: operator-run -- sudo -n is NOT available on vcloud so that half could not be automated. /swap2.img 128 GiB, 600 root:root, live + fstab. 135 GiB total. CHANGED THROUGH TOFU, the only correct place: a virsh setmaxmem would have been reverted by the next apply. The derivation now lives IN the variable comment. PLAN ASSERTED ON CONTENT (2026-07-20: an in-place apply silently regenerated 9 node MACs): per DC 1 in-place, 0 create/destroy/replace, 0 MAC changes, memory the only changed attribute. A THIRD resource appeared and was resolved BEFORE applying: module.office1_opnsense ~ id = 2 -> 11 sits under "has changed outside of OpenTofu" (drift OBSERVED), not "will be updated in-place" (action PLANNED) -- libvirt's domain id is a runtime value. It was not touched. dc1 FIRST AS CANARY, and it answered the open question: the in-place update BOUNCES the guest (domain id 9 -> 12, uptime 0 min). Guest sees 472.2 GiB. >>> FINDING: vr1-dc0-maas-01 (MAAS region) and vr1-dc0-juju-01 (juju controller) have autostart=disable. <<< Only the edge auto-recovers. Checking first is the only reason the dc0 bounce did not come back looking catastrophic. Both started by hand and verified. LOGGED NOT FIXED: a host reboot leaves dc0 with no region and no controller -- a standing exposure independent of this change. RESULT: host used 809 -> 63 GiB, available 197 -> 944 GiB; the restarts also released the stranded RSS (vvr1-dc1 354 -> 12 GiB), discharging 08-01 sweep F1 as a side effect. Recovery verified end to end: MAAS region API 000 -> 502 -> 200 (a real boot progression), snap proxy listening, mirror 200, juju controller model "Last connection: just now". GATE P8 -- substrate drift. Built because answering "does tofu need updates logged" produced a measurement: preflight.sh and pre-flight-checks.sh contained ZERO tofu plan checks and the opnsense drift had sat unseen. DESIGN POINT: -detailed-exitcode returns 2 for BOTH a pending change and a harmless observation, so P8 asserts on CONTENT -- pending ACTION FAILS, observed-only drift WARNS, unrecognised shape REFUSES. A gate permanently red on benign drift gets ignored. It WARNS rather than fails when it cannot look, since the outer root lives on vcloud and "not the substrate host" is a legitimate state. Harness 33 -> 38 (T34-T38). Baseline proven by stashing: my first draft's 7 failures were MINE, not pre-existing. Three mutations each killed a NAMED test, script restored sha256-identical. T38 exercises the absence by removing the fixture STATE, never the fake tofu from fakebin -- the real tofu is on the system PATH and a "closed" PATH would fall through to the LIVE substrate. PROVEN LIVE: tofu exit=0, 0 pending, 0 drift -> [ok] zero diff. SCOPE: tofu owns the SUBSTRATE only. MAAS carves, rack services and the v6 node carve are script-driven by design (Model B / D-123); P8 does not check them. gauntlet ALL GREEN (96); repo-lint 0 fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf |
|---|
|
|
| docs/CURRENT-STATE.md |
|---|
| docs/changelog-20260731-snap-proxy-apply-ipv6.md |
|---|
| docs/session-ledger.md |
|---|
| opentofu/variables.tf |
|---|
| scripts/preflight.sh |
|---|
| tests/preflight/run-tests.sh |
|---|