Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260719-dc0-deploy-stepB.md
@JANeumatrix JANeumatrix 18 hours ago 52 KB Clean commissioning experiment: contention refuted; leading hypothesis = marginal 30-min timeout over slow nested I/O
Raw Blame History

Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap)

Session changelog (GA-R2/D1: ONE per session). Logged window: dc0-deploy (same-day append, per the A/B-boundary handoff). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts.

1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124)

  • Question as presented (2026-07-19): "Who should hold 172.31.0.1 (the Office1 end of the dc0 transit /30), and by what mechanism? This is the prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS rack-to-region enrollment path."
  • Operator answer, exact utterance (option selected): "voffice1 via IaC + restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to module "voffice1" in the outer root; apply in a planned voffice1 restart window (graceful shutdown first; region/NetBox/Tailscale return via proven autostart).
  • Classification: OPS mechanism decision implementing D-124's already-ruled addressing ("region/peer .1" -- docs/design-decisions.md:3727); no new D-number (GA-R3, doubt resolves DOWN).
  • Measured basis (this session, logged): mesh-vr1-dc0-office1 (virbr7) is ISOLATED (no <forward>/<ip>) with exactly ONE attached interface (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only route 10.10.0.0/22 via 172.31.0.1 (d124-rack.auto.tfvars, consumed by step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached by -J voffice1 (D-126 shape) -- NO vcloud host leg / site-baseleg DC row needed for this net.

2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated)

  • What: opentofu/main.tf module "voffice1": network_names gains module.mesh_vr1_dc0_office1.network_name SECOND (office1-local stays first -- PCI order preserves enp1s0); network_config rewritten from the single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4; transit=enp2s0 static ${vr1_dc0_rack_transit_peer_ip}/${prefix} -- same D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init- re-run caveats documented in-block: the seed netplan is REBUILD correctness; the live transit config is an in-guest netplan drop-in at the gated attach step.
  • Why: section 1's measured gap -- step B (ssh reach + rack enrollment) has no L3 path until the region end of the transit exists.
  • Revert: restore network_names = [module.office1_network.network_name] and the previous "en*" glob network_config (git revert of this commit); if already applied, a follow-up plan shows the interface removal and the in-guest drop-in /etc/netplan/60-transit.yaml is deleted.

3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation

  • Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply (1/1/1 exact, docs/audit/outer-apply-20260719-voffice1-transit.txt) -> domain up with both NICs, office1-local index 0.
  • INCIDENT (found+fixed in-window): the provider REGENERATED both NIC MACs on the interface-list rebuild (old lan MAC 52:54:00:6a:87:e5 -> new 52:54:00:89:e1:19). The Kea reservation (MAC-keyed) missed and voffice1 leased pool address .108 (measured via edge ARP). FIX: re-keyed reservation uuid b12621e6-63e9-4714-9726-2e130c84e069 to the new MAC via the D-113 API (kea/dhcpv4/set_reservation -> saved; kea/service/reconfigure -> ok), then a detached networkctl reconfigure enp1s0 -> voffice1 back on 10.10.0.20.
  • LOGGED FINDING (not executed): modules/cloudinit-vm does not pin MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed DHCP reservations. Roosevelt-relevant; candidate module mac var or a documented ops step. Also: live transit config landed as in-guest netplan drop-in /etc/netplan/60-transit.yaml (narrows the lan glob to enp1s0 + transit enp2s0 static) -- the drop-in must ALSO narrow lan, else the 50-cloud-init en* glob claims enp2s0 (measured: first drop-in attempt without it left enp2s0 addressless).
  • Revert: kea/dhcpv4/set_reservation back to the old MAC (senseless unless the NIC change is also reverted); delete the drop-in.

4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix

  • Symptom: 172.31.0.2 unreachable; MEASURED: both rack NICs emitted ZERO frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive (ACPI-responsive; cloud-init completed per offline logs).
  • Guest is a SEALED BOX: modules/cloudinit-vm domains have NO serial console, NO qga channel, NO graphics; qga is NOT in the noble base image and its cloud-init install failed (no egress) -- so no console, no agent, no network. LOGGED FINDING: add opt-in serial console (+ the D-129-style qga channel var) to modules/cloudinit-vm -- observability gap bites every future DC VM. NOT executed mid-step.
  • Offline inspection (libguestfs, operator-run, captures ~/vvr1-dc0-offline-inspect{,2}.txt): NIC names enp1s0/enp2s0 CORRECT (naming trap closed); root cause = netplan match.name + set-name: boot 1 renames don't apply (.link written after udev add) so the rendered config matched nothing; after removing set-name, boot 2 STILL failed because the early-boot netplan generator ran against the OLD config and renamed the devices to mgmt/uplink before cloud-init wrote the new file (one-boot lag, measured in syslog). Boot 3 (current config, no renames) is the verification boot.
  • FIX (this commit + applied): network_config drops set-name, keeps MEASURED kernel names; instance-id bumped vvr1-dc0-d123 -> -d123-r2 (side effects noted in-block); seed replaced via explicit -replace=module.vvr1_dc0.libvirt_volume.seed (the sanctioned override of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2 exact (docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt).
  • CONSEQUENCE for step B: bootstrap runs with --transit-if enp1s0 --uplink-if enp2s0 (script defaults mgmt/uplink no longer match reality); runbook prose naming "mgmt" needs a DOCFIX at stage close.
  • Revert: git revert of the netplan/instance-id edit + one more seed -replace apply + reboot.

4b. VERIFICATION (2026-07-20, logged window)

  • Boot 3 with the fixed config: enp1s0 UP 172.31.0.2/30, region route present, br-vr1-dc0-wan up.
  • voffice1 -> 172.31.0.2 ping 0% loss; ssh -i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2 works (D-126 -J shape, per-env key); rack -> region ping 10.10.0.20 0% loss (the MAAS-enrollment path for the G10 bootstrap).
  • Outer plan CONVERGED back to ZERO DIFF (docs/audit/outer-plan-20260720-postfix-converged.txt); CURRENT-STATE section 5 updated in the same commit.
  • Follow-items (logged, not executed): D-126 DC ssh-config alias row for vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key); cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC pinning; DOCFIX for runbook "mgmt/uplink" naming.

4c. Step B bootstrap EXECUTED on vvr1-dc0 (2026-07-20, logged window)

  • Snap proxy set to the region maas-proxy (measured serving: HTTP 200 to archive.ubuntu.com, CONNECT to api.snapcraft.io) -- the D-107-shaped answer to the rack's no-egress bootstrap; apt used the same proxy via per-invocation env. Enrollment secret piped region->rack (install -m600 /dev/stdin), verified by byte-count only (32), never in context.
  • Run 1 (docs/audit/stepB-bootstrap-20260720.txt): maas snap 3.7/stable installed, rack ENROLLED to http://10.10.0.20:5240/MAAS; FAILED at qemu apt install on stale image package lists (404s via proxy). Fix: apt-get update (proxied), re-run.
  • Run 2 (docs/audit/stepB-bootstrap-20260720-run2.txt): EXIT 0 -- libvirt/qemu + nested KVM on, inner pool + AppArmor, SEC-010 FORWARD-drop loaded on enp1s0 + sec010-fw.service enabled (boot-persistent), br-vr1-dc0-wan verified with enp2s0 enslaved.
  • --check (docs/audit/stepB-check-20260720.txt): all items ok EXCEPT opnsense base (the remaining step-B item); rackd running, enrolled.
  • Region-side residue for the MAAS phase (per script exit text): enable DHCP on metal-admin at the Office1 region naming this rack primary_rack (step D territory, NOT done).
  • Revert: rack enrollment: region-side maas ... rack-controllers delete; SEC-010: disable sec010-fw.service + rm /etc/nftables-sec010.nft; packages: apt/snap remove; snap proxy: snap unset system proxy.http proxy.https.

4d. RULING RECORD (GA-R5) -- DC edge base version (2026-07-20)

  • Question as presented: reuse verified 26.1 artifact vs fresh download (first exchange); then, sharpened after the operator noted "Office1 is running OPNsense 26.7": 26.1-base-then-upgrade vs fresh 26.7 nano base.
  • Operator answer, exact utterance (option selected): "Fresh 26.7 nano base".
  • Consequences implemented: prep runs 26.7 -> opnsense-26.7-nano.qcow2; vr1-dc0-substrate opnsense_base_path default updated (no inner state exists; pre-apply edit); step-B --check passes --opnsense-base for the 26.7 path. LOGGED follow-items: site-headend-install.sh's 26.1 default + hint strings (script change + harness, stage close); the D-112/D-113 serial/API boot path is REVALIDATED on 26.7 at the step-C boot gate (office1 precedent: 26.1-proven, upgraded in place to 26.7).
  • Mirror: https://mirror.ams1.nl.leaseweb.net/opnsense -- MEASURED from the vendor's live download page this session (not memory), HEAD-verified serving the 26.7 nano (~468 MiB). LOGGED FINDING: opnsense-prep-image.sh performs no checksum verification of the downloaded image (same property as the original 26.1 prep; candidate hardening).

4e. Step B CLOSED: 26.7 base staged; final check EXIT 0; SEC-010 CLOSED both ends

  • OPNsense 26.7 nano prepped ON the rack (leaseweb mirror via region proxy, 468M; python3 bz2 fallback -- bunzip2 never needed, DOCFIX-184; docs/audit/stepB-opnsense-prep-20260720.txt). Inner-root opnsense_base_path default -> 26.7 (validate PASS).
  • Final --check EXIT 0, ALL items ok (docs/audit/stepB-check-20260720-final.txt).
  • SEC-010 voffice1-side pin (the row's "same pin follows" clause, due since this session wired voffice1's transit leg; voffice1 runs LXD so ip_forward=1): identical scoped nft artifact + sec010-fw.service on enp2s0, verified loaded + persistent; rack proxy/region paths re-verified working post-pin. SEC-010 row CLOSED (operator ruling, utterance in the row). Open SEC count 8 -> 7.
  • LOGGED follow-item: site-headend-install.sh region-role support for the transit-if pin (voffice1's was hand-applied to the script's exact artifact shape; fold into the script + harness at stage close).
  • Revert: rm the two files + systemctl disable sec010-fw on voffice1; nft delete table inet sec010; ledger row reopens citing this item.

7. Step C: voffice1 becomes the Plane-2 executor; INNER APPLY COMPLETE (28/28)

  • voffice1 setup (gated, logged): tofu 1.12.4 via vcloud's exact apt source (keyrings byte-copied; version == section-7 pin); repo cloned at 29cf7bf (branch); dc0 private key staged to ~/vr1-dc0-creds/ (0700/0600, piped -- SEC-009 convention now spans hosts); ssh config entry for 172.31.0.2; known_hosts seeded via ssh-keyscan with the ED25519 fingerprint cross-checked against the accept-new-trusted entry (MATCH).
  • Inner-root fixes measured at the FIRST REAL dial/apply (each committed): (i) provider needs keyfile+sshauth=privkey URI params -- its Go ssh ignores ~/.ssh/config AND default identities; single-type known_hosts fails as "key mismatch" (multi-type scan required); (ii) modules/wan-bridge: <mtu> is ILLEGAL in bridge-mode networks -- removed (MTU belongs to the host bridge; DOCFIX-194 class: parsed, never applied); (iii) base-image location premise WRONG under remote provider -- content UPLOADS from the executing host; 26.7 qcow2 copied rack->voffice1 (~/vr1-dc0-images/), tfvar override + variables.tf description corrected; (iv) edge serial-log dir /var/lib/libvirt/vr1/staging/ is a HARDCODED vcloud literal in modules/opnsense-edge (:223) -- dir created on the rack (755); LOGGED findings: parameterize the path; fold staging-dir creation into site-headend-install --host-nodes; opentofu-validate.sh does not cover the inner root.
  • Applies (captures docs/audit/inner-{plan,apply}-20260720-stepC*.txt): run 1 = 25/28 (nodes+planes+pool; wan-bridge mtu abort); run 2 = wan bridge (edge disk stat abort); run 3 = edge disk (staging-dir abort; provider self-cleaned the failed domain -- operator undefine found nothing); run 4 = edge domain, EXIT 0. CONVERGED: inner plan zero diff (docs/audit/inner-converge-20260720-stepC.txt).
  • VERIFIED: 10/10 domains RUNNING inside vvr1-dc0 (9 nodes depth-4 + edge); edge serial log = FreeBSD login prompt on ttyu0 -- 26.7 boots the D-112 serial path (first revalidation datapoint; full D-112(c) bootstrap is the next window).
  • NEW STATE OF RECORD: the inner tfstate lives on voffice1 (~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/terraform.tfstate) -- back it up with the site; follow-item for the backup set.
  • Revert: tofu destroy in the inner root from voffice1 (site-down alternative: virsh destroy vvr1-dc0, D-122); module fixes revert by commit.

9. DC edge: D-112(c) console bootstrap + API key minted (26.7) -- REPLACEMENT CHAIN PROVEN

  • Per-env edge key minted on vcloud (~/vr1-dc0-creds/vr1-dc0-edge_ed25519, 0600/0644, SEC-009 convention), staged to the rack the same way.
  • Rack legs added (operator-ruled "10.12.4.2, manual now + script later"): 10.12.4.2/22 on the provider-public inner bridge (virbr5, MEASURED) + the D-124-ruled 10.12.8.2/22 on metal-admin (virbr2). INTERIM: ip addr add, not persistent. LOGGED FINDING (queued): site-headend-install.sh never implemented the rack legs despite main.tf's comment claiming it consumes vr1_dc0_rack_metal_admin_ip -- add rack-leg support + harness before step D closes. Also interim: 192.168.1.2/24 on virbr5 to reach the edge's FACTORY LAN; REMOVE once the edge is re-addressed.
  • D-112(c) console bootstrap: DONE (~/d112c-console.log on the rack). Driver evolved v1->v6 against MEASURED failures, each real and worth keeping for the runbook: (i) serial console wraps long lines -> ship payload as <=160-char base64 chunks (openssl base64 -d), never one long line; (ii) set-name in the edge netplan is irrelevant here but the same class bit the rack (item 4); (iii) tcsh on the edge (documented trap) breaks nested-quoted sed -- "Unmatched '"'"'" -- so NEVER patch files via quoted one-liners; re-ship the whole payload instead; (iv) write_config() needs an include beyond config.inc -- MEASURED (grep -l shell_safe /usr/local/etc/inc/*.inc -> auth/console/filter/ interfaces/rrd/system/util.inc), not guessed; (v) writing config.xml does NOT materialize /root/.ssh/authorized_keys -- must call the vendor's local_user_set() (auth.inc). This was the actual reason key auth failed after a "successful" config write.
  • VERIFIED: ssh -i vr1-dc0-edge_ed25519 root@192.168.1.1 -> key-only auth OK, uname -r = 15.1-RELEASE-p1, ifconfig -l = vtnet0/vtnet1. INTERFACE MAPPING MEASURED: vtnet0 = LAN = provider-public plane (it answered on virbr5), vtnet1 = WAN -- same order as the office1 edge.
  • API key minted via the vendor model (opnsense-bootstrap-apikey.sh
    • opnsense-mint-apikey.php, run ON the rack): key/secret 80 chars -> ~/vr1-dc0-creds/opnsense-api.txt (0600, secret never printed). SMOKE TEST PASS: GET core/firmware/status -> HTTP 200, CORE_ABI 26.7. This is the first proof the D-113(a2) API path works on 26.7 (the 26.7 revalidation the operator's base-version ruling called for). Script note: the bootstrap script requires the minter BESIDE it (its own dir), not in scripts/ -- worth a usage-line DOCFIX.
  • Revert: delete the API key (POST auth/user/del_api_key/<id>, the proven negative-tested path) + creds file; ip addr del the three interim addresses; the console bootstrap reverts by re-imaging the edge (inner tofu destroy/re-apply of the edge module).

10. D-125 egress isolation gate: EXECUTED as written -- PARTIAL (bridge-in PROVEN,

upstream NAT egress FAILS). Gate NOT closed; double-NAT fallback NOT invoked.

  • Method: operator-ruled "throwaway VM, as written" -- noble cloud image fetched ON the rack via the region proxy, one cloud-init guest, SINGLE NIC bridged to br-vr1-dc0-wan, serial-log probe, then destroyed (leftover-domains=0 asserted). Script: scratchpad d125-egress-test.sh; capture docs/audit/d125-egress-gate-20260720.txt.
  • Run 1 was a TEST ARTIFACT, not a gate failure: the probe used DHCP and came up address-less. MEASURED cause: vr1-dc0-uplink carries an <ip> but NO <dhcp> block -- there is no DHCP server on that /24 by design (the edge WAN takes a STATIC .2, D-113/D-125). DOCFIX QUEUED: the gate's wording "must GET a vcloud-ISP address" implies DHCP; it must say STATIC, or the /24 needs a range. Re-run used static 172.30.2.50/24.
  • RESULT (run 2): guest configured 172.30.2.50/24 + default via 172.30.2.1; gateway ping 3/3, 0% loss -- a FOREIGN MAC inside vvr1-dc0 reaches the vcloud ISP NAT across the nested bridge. That is the D-125 bridge-in property, and it PASSES. But ping 1.1.1.1 FAILED (NET-PING-RC=1).
  • CORRECTION (2026-07-20, same-day): the rack-bridge follow-up is CONFOUNDED and is NOT evidence. It sourced from 172.30.2.51 on the rack, but the rack has NO DEFAULT ROUTE (netplan gives it only 10.10.0.0/22 via 172.31.0.1; the WAN bridge is IP-less by design), so traffic to 1.1.1.1 fails in the rack's own routing table before the NAT is ever consulted. Its ICMP+TCP failures say nothing about virbr4. The ONLY clean signal is the GUEST's (172.30.2.50) failed egress, and that guest tested ICMP only -- if the rule dump comes back identical for both bridges, re-run the guest probe with curl before concluding anything. (Recorded per GA-R1/C2: a wrong measurement is corrected in the record, not quietly dropped.)
  • Controls: vcloud egress 0% loss; the office1 edge egresses through the SAME-SHAPE office1-wan NAT 0% loss. The two network definitions are byte-equivalent in substance (net-dumpxml: both forward mode=nat + <nat><port 1024-65535>; only netmask-vs- prefix notation differs) -- so the delta is in vcloud's LIVE firewall rules for virbr4, NOT in the design or the config.
  • GATE STATUS: PASS / CLOSED 2026-07-20 (runs 3+4, capture docs/audit/d125-egress-gate-20260720-matrix.txt). The "missing rule" hypothesis was DISPROVED by measurement first: the operator-run dumps show virbr4's rule set is structurally IDENTICAL to the WORKING virbr11 (only counters differ; virbr11's one extra rule is a DHCP checksum fixup it has because it carries a DHCP range). All three masquerade rules exist for 172.30.2.0/24 INCLUDING the generic (non-tcp/udp) one, and it had FIRED (1 packet/84 bytes = one ICMP conntrack entry), with FORWARD accept counting 136 packets. So net-destroy/net-start would have fixed nothing -- the pre-staged fix was correctly NOT run.
  • Decisive re-probe used a protocol/destination MATRIX instead of counter archaeology. Result, IDENTICAL on two consecutive runs: gateway ping 0, internet ping 0, curl http://1.1.1.1 -> 301, curl archive.ubuntu.com -> 200. Egress works for ICMP AND TCP to real hosts, so the D-125 bridge-in chain is PROVEN end to end and the double-NAT fallback is NOT needed.
  • Run-2 anomaly, recorded UNEXPLAINED (not swept): the same path failed ICMP-to-internet once while its gateway ping passed 3/3. Runs 3+4 passed identically with no intervening change to the uplink or its rules. Most likely a first-use conntrack/ARP transient on vcloud. It is not reproducible now; if a DC edge ever shows first-boot egress failure, START HERE rather than re-deriving.
  • Security-positive side observation (extra probe, outside the gate): the guest could NOT reach the Office1 management LAN through the ISP NAT (10.10.0.20:8000 -> 000) on all runs, while internet egress worked. That is the D-122/SEC-010 intent holding: the DC's WAN segment is an ISP uplink, not a path into Office1. Worth an explicit assertion in the hardening tests rather than leaving it an incidental observation.
  • Prior status line (superseded, kept for the audit trail): gate was held OPEN pending measurement rather than FAILED. The runbook's FAIL remedy (revert to double-NAT) is for a DESIGN failure; invoking it against what is very likely a host firewall-rule gap would be the wrong remedy and would add Roosevelt delta. Next step is a read-only measurement needing operator sudo: FULL dumps of BOTH nft list ruleset and iptables-save (which backend is authoritative here is itself unmeasured -- do not assume), diffing virbr4 against the WORKING virbr11. Two rule families must both be checked, and a subnet-only grep would MISS the second: POSTROUTING masquerade (keyed on 172.30.2.0/24) and FORWARD accept (keyed on the INTERFACE virbr4). Pre-staged fix-forward IF the rules are absent or malformed vs virbr11: virsh net-destroy vr1-dc0-uplink && virsh net-start vr1-dc0-uplink reinstalls libvirt's rules from scratch -- cheap and safe (nothing rides that uplink yet but the probe). If instead the dump shows libvirt never installed them correctly, the restart will not help and the dig moves to the firewall backend / conntrack.
  • Revert: nothing to revert -- the probe guest and both temporary addresses were removed; the only residue is /tmp/noble.img on the rack (kept deliberately: it is the base for any future throwaway probe).

11. NEW TOOLING + edge re-addressed: opnsense-set-interface-v4 pair (53/53)

  • RULING RECORD (GA-R5), 2026-07-20. Question as presented: base-iface IPv4 is not REST-covered on 26.7 (measured); how should LAN 10.12.4.1/22 and WAN 172.30.2.2/24 + gw 172.30.2.1 be set -- build the v4 setter pair / console menu now / console now + build at stage close? Operator answer, exact utterance (option selected): "Build the v4 setter pair (Recommended)". Classification: OPS tooling under the existing D-113 amendment; no new D-number (GA-R3).
  • D-113 AMENDMENT RE-MEASURED ON 26.7 (feeds the item-8 review): the amendment was measured on 26.1 only. On this 26.7 edge: interfaces/settings/get returns ONLY global offload flags, while interfaces/overview/list and interfaces/lan/get both return 404 "Endpoint not found". Base-interface addressing is still legacy /interfaces.php. The amendment now holds on TWO majors, measured.
  • SHIPPED: scripts/opnsense-set-iface-v4.php + scripts/opnsense-set-interface-v4.sh + tests/opnsense-set-interface-v4/ (53/53 PASS first run; gauntlet 70 -> 71 harnesses ALL GREEN; repo-lint 0 fail). Mirrors the proven v6 sibling exactly -- vendor Config singleton (never a config.xml push), dry-by-default, interface-count guard, fresh read-back verify, sh -s for the tcsh trap, KERNEL as ground truth. Two deliberate improvements over the v6 sibling: (i) the GATEWAY half (creates/updates at most ONE gateway_item, count-guarded exactly like the interface count, opt-in via a 5th arg); (ii) the interface's DEVICE is MEASURED from the edge's own config instead of the v6 sibling's hardcoded lan->vtnet0 / wan->vtnet1 sed -- that mapping is per-edge and hardcoding it is a hard-rule-2 violation waiting to happen. Harness pins BOTH (test 12 fails if the hardcode returns). LOGGED FINDING: backport the measured-device fix to the v6 sibling at stage close.
  • APPLIED to vr1-dc0-opnsense, WAN first per the script's own ordering warning (re-addressing the LAN drops the session you arrived on):
    • WAN: dhcp -> 172.30.2.2/24 + WAN_GW = 172.30.2.1 default. Note the edge had shipped with WAN on dhcp, which could NEVER have worked -- the uplink /24 has no DHCP server (same fact that broke probe run 1). Verified on the kernel: inet 172.30.2.2, default 172.30.2.1 UGS vtnet1, and the EDGE ITSELF pings 1.1.1.1 at 0% loss -- the D-125 chain now proven through the real OPNsense edge, not just a probe.
    • LAN: 192.168.1.1/24 -> 10.12.4.1/22 (the D-115/lib-net ruled provider-public gateway). Session dropped as designed; edge verified at the new address from the rack: inet 10.12.4.1 netmask 0xfffffc00.
    • Post-move reconciliation at the NEW address: API GET core/firmware/status -> 200 CORE_ABI 26.7; edge egress still 0% loss.
  • CLEANUP: the interim 192.168.1.2/24 on the rack's provider-public bridge is REMOVED (virbr5 now carries only the ruled 10.12.4.2/22).
  • Revert: re-run the setter with the previous values (--commit <edge> lan 192.168.1.1 24, --commit <edge> wan ...); the gateway item is named WAN_GW and is the only one this tool created.

12. D-129 edge profile APPLIED on 26.7 + qga channel shipped + a FALSE-SUCCESS bug fixed

  • RULING RECORD (GA-R5), 2026-07-20. Question: the qga virtio channel is absent from the edge domain and unsupported by modules/opnsense-edge (both MEASURED), so installing os-qemu-guest-agent alone yields a package that can never answer. Options presented: add channel var now / iperf only
    • defer / install both and leave qga inert. Operator answer, exact utterance (option selected): "Add channel var now, then install both (Recommended)". [OPS under D-129; no new D-number, GA-R3.]
  • SHIPPED expose_qga_channel (opt-in, DEFAULT OFF) in modules/opnsense-edge; vr1-dc0-substrate sets it true so NEW DC edges are BORN with the channel (the runbook's no-retrofit intent). Default OFF is load-bearing: a device change alters the domain, so defaulting ON would bounce every existing edge on the next plan. Shape taken from tofu providers schema -json (dmacvicar 0.9.8: devices.channels[].target.virt_io{name} + source.unix{mode}), NOT from a blog; source.unix carries no path, letting libvirt allocate the socket (same reasoning the serial block documents for pty).
  • APPLY was SAFE: plan showed 0 add / 1 change / 0 destroy -- an IN-PLACE domain update, so libvirt_volume.disk (holding the console + API bootstrap) was never at risk. Verified after: channel present (grep count 2), domain running, edge still reachable at 10.12.4.1.
  • BUG FOUND ON THE FIRST-EVER LIVE RUN -- opnsense-plugins.sh apply had NEVER installed anything. api() used bash "$API" ${DRY_RUN:+--dry-run}: ${VAR:+word} expands when VAR is set and NON-EMPTY, and DRY_RUN=0 is non-empty -- so every apply silently dry-ran while printing "OK: profile 'vr1-edge' applied". A textbook false success, and the reason it survived review is that the harness drove ONLY --dry-run, leaving the live path with zero coverage. FIX tests the VALUE; harness gains two live-path tests against a stub api client, verified to FAIL 2/2 on the buggy version before restoring (a regression test that cannot fail is worthless). 20/20. This is exactly the class D-129's own "package present is not success" warning is about -- here the SCRIPT was the false reporter.
  • REAL install then ran: {"status":"ok","msg_uuid":...} per plugin (not dry-run echoes), capture docs/audit/d129-plugins-20260720.txt (the earlier capture of the same name recorded the false dry-run and is REPLACED). Measured on the edge: os-qemu-guest-agent-1.3, qemu-guest-agent-11.0.1, os-iperf-1.0_2, iperf3-3.21.
  • Service was NOT auto-started (the script's own NOTE); started via the plugin's OWN API (POST qemuguestagent/service/start -> {"response":"OK"}, status -> running) rather than by hand-editing rc.conf.
  • GATE MET, to the runbook's exact bar (not package-presence): virsh qemu-agent-command vr1-dc0-opnsense '{"execute":"guest-ping"}' -> {"return":{}}, and virsh domifaddr --source agent reports vtnet0 10.12.4.1/22 + vtnet1 172.30.2.2/24. The agent ANSWERS over the channel.
  • SECURITY FINDING (queued for the item-8 hardening review): the plugin model exposes DisabledRPCs, including guest-exec/guest-exec-status -- arbitrary command execution on the edge from the libvirt host. On a commercial multi-tenant cloud the DC edge should almost certainly disable the exec RPCs (we need guest-ping/fsfreeze/shutdown coordination, not remote exec). Not changed mid-step; propose with options at the review.
  • Revert: POST qemuguestagent/service/stop; uninstall via core/firmware/remove/<pkg>; set expose_qga_channel = false and re-apply (in-place domain update again).

13. Step D part 1 DONE (DHCP configured) -- then INCIDENT: MAAS DHCP is DOWN

REGION-WIDE (Temporal wedged). Not caused by this deploy.

Step-D work completed before the incident surfaced:

  • Both controllers confirmed registered: voffice1 (region, mtstwf) and vvr1-dc0 (rack, 7chphy, rackd running).
  • MAAS auto-discovered the inner planes from the rack's interfaces: metal-admin 10.12.8.0/22 = VLAN 5005/fabric-4, provider-public 10.12.4.0/22 = VLAN 5006/fabric-5. (Stale 192.168.1.0/24 also sits on fabric-5 -- the edge's old factory LAN, now vacated; cleanup queued.)
  • RULING RECORD (GA-R5), 2026-07-20. Question: metal-admin is a /22 but D-120's band layout is written for a compose /24; what dynamic range? Operator answer, exact utterance (option selected): "Inherit D-120 band: 10.12.8.100-10.12.8.200 (Recommended)". Range CREATED and verified on the correct subnet; VLAN 5005 updated dhcp_on=true primary_rack=7chphy, verified by read-back (10.12.8.0/22 dhcp=True primary=7chphy).

INCIDENT (found by verification, not by a self-report):

  • Canary virsh reset vr1-dc0-control-01 did NOT enlist. MEASURED cause chain: rack service_set shows dhcpd: off; no dhcpd process on the rack; no /var/snap/maas/common/maas/dhcpd.conf generated -- while MAAS's own API reports the VLAN as dhcp_on=True with the correct primary rack and the interface correctly linked (VLAN 5005, static 10.12.8.2). Config state and running state DISAGREE.
  • Root cause, from the region's journal: MAAS 3.7 drives DHCP through Temporal workflows, and Temporal is wedged -- "Not enough hosts to serve the request", poll_workflow_task_queue retried 2807 times, task queue mtstwf@region:.... No worker is polling, so the workflow that pushes DHCP config to a rack can never run.
  • SCOPE IS WIDER THAN THIS DEPLOY: voffice1 itself has NO dhcpd process (pgrep -c dhcpd -> 0) and reports no dhcpd service, while Office1's compose net 10.10.1.0/24 reads dhcp=True. So MAAS DHCP has been DOWN REGION-WIDE -- almost certainly since the 2026-07-17 host reboot -- and went unnoticed because office1-netbox/office1-tailscale were already Deployed and kept their addresses. This is NOT caused by the DC0 deploy; the deploy is what exposed it.
  • Consequence for the as-built record: any claim that Office1 MAAS DHCP is "working" is currently FALSE (last positive evidence predates the reboot).
  • Detection gap (queued): cloud-assert.sh and the Office1 health checks did not catch a dead DHCP server, because MAAS's own service_set reported the region healthy. A behavioral DHCP assertion belongs in the health sweep -- this is precisely the "service's own verdict" principle the repo already applies elsewhere.
  • No prior art: appendix-A has no entry for this symptom. NEW incident material once the fix is known. RESOLVED 2026-07-20 (operator-approved: "Go ahead with the restart").
  • sudo snap restart maas on voffice1 -> region back (HTTP 301 on localhost:5240). Recovery verified BEHAVIORALLY, not by service status: pgrep -c dhcpd on voffice1 0 -> 1, and on the DC rack 0 -> 1; MAAS then reported dhcpd: running on BOTH controllers (was off on the rack, absent entirely on the region). One restart fixed both sites, which confirms the single wedged-Temporal root cause rather than two faults.
  • PROOF IT WAS THE REAL BLOCKER: the identical canary that had failed before (virsh reset vr1-dc0-control-01) enlisted immediately after.
  • APPENDIX-A MATERIAL (queued, new incident class): symptom "MAAS says dhcp_on=True but no dhcpd runs and nothing PXEs"; check pgrep -c dhcpd on each controller and the region journal for Temporal "Not enough hosts to serve the request"; remedy snap restart maas; verify behaviorally on every controller, never by service_set.

14. G10 DEPTH-4 NESTED BOOT GATE: PASS -- all 9 DC0 nodes enlisted in MAAS

  • After the DHCP fix, all nine node VMs were reset and all nine enlisted (docs/audit/stepD-enlistment-20260720.txt): count 9, status 8 Commissioning + 1 New, shapes {16cpu/64GiB: 3, 12cpu/48GiB: 2, 8cpu/24GiB: 4} -- an EXACT match to the D-121 Option C / R-3 layout (3 control + 2 compute + 4 storage).
  • This closes the D-114/D-123 depth-4 boot gate, the flagged-risk item of Model B. A node VM inside vvr1-dc0 (itself a VM on vcloud) PXE-booted from the Office1 region across the D-124 transit and is running MAAS's ephemeral commissioning kernel -- i.e. real nested KVM at depth 4, proven by behavior rather than by inference from svm being present.
  • Still open in step D: commissioning completes with a power-off, and MAAS has no power driver for these VMs yet -- the maas-vm-host registration (module BUILT 2026-07-09, still UNVALIDATED) is the next piece, and it needs TF_VAR_maas_api_key.

  • Revert: maas admin machines delete per machine (or delete the pod once registered); the VMs themselves are inner-root state.

  • PROPOSED REMEDY (superseded by the RESOLVED block above; kept for the audit trail): restart MAAS on the region (sudo snap restart maas on voffice1) to re-establish Temporal workers, then re-verify dhcpd on BOTH controllers behaviorally. Blast radius is low right now (no OpenStack workloads; both Office1 VMs already deployed and unaffected by a brief DHCP/API outage), which makes this the cheapest moment to fix it.

15. Step D part 2: NEW root for MAAS registration; and a MEASURED REFUTATION

of D-123's "power_address = qemu:///system LOCAL to the rack"

  • SHIPPED opentofu/vr1-dc0-maas/ -- a THIRD root holding the MAAS registration alone. Rationale is DOCFIX-179's own measured lesson applied rather than repeated: a provider "maas" block forces EVERY plan in its root to demand maas_api_url + the SENSITIVE key, even for plans creating zero MAAS resources (the DOCFIX-175 plaintext-secret surface). Putting it in the inner substrate root would make routine substrate work require MAAS creds. Substrate roots stay credential-free. Validated explicitly -- note opentofu-validate.sh covers only the outer root + modules, so neither extra root is gauntlet-covered; extending it stays QUEUED and is now more load-bearing (two uncovered roots).
  • Secret handling: key supplied ONLY as TF_VAR_maas_api_key from the operator-placed 0600 file; verified by FORMAT alone (71 bytes, 3 colon-separated parts), never printed. Caveat recorded in the root's header: the key DOES land in that root's tfstate (unavoidable with this provider), so that state inherits credential handling.
  • Plan was clean (1 to add). APPLY FAILED: ServerError: 503 ... Failed talking to pod: Failed to login to virsh console.
  • ROOT CAUSE, MEASURED (not inferred): the MAAS snap is strictly confined. From inside it, virsh -c qemu:///system -> Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied. And snap connections maas lists NO libvirt interface at all -- there is nothing to snap connect. So a LOCAL qemu:///system VM host is ARCHITECTURALLY IMPOSSIBLE with snap-based MAAS.
  • THIS REFUTES A DOCUMENTED DESIGN LINE. D-123 Model B and modules/maas-vm-host's header both state the DC pod is registered with power_address = qemu:///system LOCAL to the rack ("vvr1-dc0 is BOTH the MAAS rack AND the libvirt host, so no cross-fiber virsh dial"). The no-cross-fiber INTENT survives; the stated MECHANISM cannot work. Both surfaces need an amendment once the replacement is ruled. Roosevelt- relevant: the same constraint hits every DC pod and any snap-MAAS metal build, so this is not a VR1 quirk.
  • Feasibility of the replacement, MEASURED: the snap DOES ship an ssh client (/snap/maas/41649/usr/bin/ssh), and jessea123 on the rack is in the libvirt group (113) -- so qemu+ssh://jessea123@172.31.0.2/system reaches libvirt through sshd (unconfined) instead of the blocked socket. It needs a key the snap's root can read (/var/snap/maas/current/root/.ssh/ does not exist yet) plus that pubkey in the rack user's authorized_keys.
  • Revert: the failed apply created nothing (state empty); git rm -r opentofu/vr1-dc0-maas removes the root.

16. The pod path, chased to its end -- and a SECOND refutation: MAAS virsh

pods are incompatible with this repo's node-vm disks

Sequence, each step measured (operator ruled the DEDICATED key: "Dedicated MAAS->libvirt key (Recommended)"):

  1. Minted maas-virsh_ed25519 on the rack (0600, SEC-009 folder), pubkey into the rack user's authorized_keys, private half into the RACK's MAAS snap. Rack snap could then virsh -c qemu+ssh://... fine -- but the apply still failed the same way.
  2. MEASURED why: MAAS attempts the pod connection from the REGION, not the rack. voffice1's snap had no key and no known_hosts -> Host key verification failed. Same key + keyscan installed there; region snap then listed the inner domains. (Worth keeping: for a virsh pod, the credential belongs wherever MAAS dials FROM, which is not necessarily the rack that owns the hardware.)
  3. Next apply hit Pod vr1-dc0-inner ... already exist -- the two earlier 503s had EACH created a pod server-side before erroring, so MAAS held two broken cores=0 pods outside tofu state. Deleted both (own residue), re-applied.
  4. Next apply reached the storage inventory and failed: Virsh command ['domblkinfo','vr1-dc0-compute-01','vda'] failed: internal error: missing storage backend for 'volume' storage.
  5. ROOT CAUSE, reproduced LOCALLY on the rack (so not a snap/SSH artifact): virsh domblkinfo vr1-dc0-compute-01 vda fails identically at the console with an ACTIVE, autostart pool. libvirtd cannot run domblkinfo against disks declared as pool+volume references -- which is exactly how modules/node-vm builds them. MAAS virsh pods are therefore incompatible with this repo's node-vm disk shape; making the pod work would mean converting node-vm to file-path disks and re-applying all nine domains.
  6. BUT THE POD IS UNNECESSARY. Its designed job (D-103) was DISCOVERY, and discovery already happened via PXE enlistment. All MAAS still needs is POWER. MEASURED, working: per-machine power_type=virsh + power_parameters_power_{address,id} on the canary (ace-oyster = vr1-dc0-control-01, matched by MAC) -> maas admin machine query-power-state returns {"state": "off"}. No pod, no domblkinfo, no module change, no touching nine live domains.
  7. Roosevelt note that makes this more than expedience: on bare metal each machine carries its OWN power parameters (IPMI/BMC). Per-machine power is the SHAPE Roosevelt uses; the virsh pod is a VR1-only convenience. So the per-machine route is the smaller Roosevelt delta as well as the smaller change here.
  8. Both routes are recorded for the operator's ruling; nothing further applied.

17. STEP D COMPLETE: per-machine power shipped as tooling; nodes COMMISSIONING to Ready

  • RULING RECORD (GA-R5), 2026-07-20. Question: pods are incompatible with node-vm's pool+volume disks (measured) and the pod's discovery role is already satisfied by PXE -- per-machine virsh power, or convert node-vm to file-path disks and keep the pod? Operator answer, exact utterance (option selected): "Per-machine virsh power (Recommended)". [Amends D-103/D-123's stated vm-host mechanism; amendment text queued for stage close.]
  • SHIPPED scripts/maas-node-power.sh + tests/maas-node-power/ (24/24 PASS first run; gauntlet 71 -> 72 harnesses ALL GREEN; repo-lint 0 fail) -- delivered as TOOLING rather than nine hand-typed CLI calls so dc1 and any re-run are repeatable. Design points the harness pins: (i) matching is BY MAC, never by name -- MAAS assigns random hostnames at enlistment (ace-oyster, not vr1-dc0-control-01), so a name match would map nothing or, worse, the wrong node; (ii) dry by default; (iii) every write is verified by an actual query-power-state -- a stored power parameter proves nothing about whether MAAS can REACH the hypervisor; (iv) the measured pod-refutation reasoning travels in the script header.
  • Prereqs measured and fixed on the region: it had no virsh (it is the MAAS/LXD host, not a libvirt host) -> installed libvirt-clients; and the maas CLI profile was root-only -> created a user profile with maas login ... - reading the key from STDIN (never printed, never in argv).
  • LIVE RESULT (docs/audit/stepD-power-20260720.txt): 10 domains found, 9 mapped and set, each confirmed power state: off by query. The 10th (the edge) is correctly unmapped -- it is not a MAAS machine.
  • COMMISSIONING NOW WORKS END TO END: all 9 commissioned; at the time of writing 3 Ready, 6 Commissioning, shapes still exactly D-121 Option C. Ready means MAAS commissioned the node and it is deployable -- the whole point of step D. The remaining six finish unattended.
  • Revert: maas admin machine update <id> power_type= (clears power) per machine; the script itself reverts by commit.

18. OPEN: 6 of 9 nodes fail commissioning ("timed out"); NOT contention; blocked

on node observability. Step E not started.

  • After power was wired, all 9 were commissioned at once: 3 reached Ready, 6 hit MAAS's 30-minute timeout (Marking node failed - Node operation 'Commissioning' timed out after 30 minutes).
  • FIRST HYPOTHESIS (resource contention -- 9 nodes x depth-4 nested, 384 GiB of 416 GiB, all pulling ephemeral images through one /30 transit) was TESTED AND REFUTED: a single failed node re-commissioned ALONE sat in Loading ephemeral for 20+ minutes with the rack at 385 GiB free and load 0.04. It is not contention and not memory.
  • Also RULED OUT by measurement: rack boot-image sync (list-boot-images -> status: synced, 10 images), and DHCP itself (these same nodes enlisted fine over PXE earlier, which is what proved the depth-4 gate).
  • BLOCKER IS OBSERVABILITY, and it is the same gap twice: modules/node-vm defines no serial console (confirmed in the module and in the live domain XML -- no serial/console devices), so a node stuck in Loading ephemeral is a sealed box: no console, no agent, no way to see the boot. This is the identical finding already logged against modules/cloudinit-vm (item 4), now proven to bite a second module and an actual live diagnosis. The fix is the same opt-in serial+log pattern modules/opnsense-edge already carries -- that module's console log is what made the 2026-07-12 boot bugs legible, and it is exactly what is missing here.
  • NOT DONE, deliberately: adding a console to node-vm means re-applying domains mid-diagnosis, which is a change to a built surface and belongs to an operator ruling, not a reflex.
  • Current fleet: 3 Ready (commissioned, deployable), 6 Failed/retrying. Nothing is lost -- failed commissioning is re-runnable at will.
  • Step E (netem) NOT STARTED. Measured prerequisites for when it is: the dc0<->dc1 mesh is virbr5 ON VCLOUD (note: virbr5 INSIDE the rack is provider-public -- same number, different host, measured not assumed); its qdisc is currently default (noqueue). D-100 rules the parameters as a PLACEHOLDER for the rehearsal within a "same-metro dark-fiber, low single-digit ms" lean. Applying to the dc0<->dc1 leg is both the semantically correct link for D-100 and the SAFE one -- it carries no traffic (dc1 does not exist), unlike the transit, which is currently load-bearing for MAAS/PXE. modules/netem-link cannot run unattended from here: it shells out to ssh <host> 'sudo tc ...' and assumes PASSWORDLESS SUDO, which vcloud's operator account does not have -- a documented module assumption that this host violates.

19. node-vm console SHIPPED; commissioning narrowed but UNRESOLVED -- and two

of my own experiments were INVALID (disclosed)

  • SHIPPED (operator-ruled): modules/node-vm now defines a serial console + boot log, with the log directory PARAMETERIZED (serial_log_dir) rather than opnsense-edge's hardcoded vcloud literal, which is wrong on any other libvirt host. Applied to all 9 domains: 0 add / 9 change / 0 destroy, in-place, nothing destroyed. validate PASS, repo-lint 0 fail.
  • The console did NOT yield the hoped-for evidence, and that is itself a finding: the logs stay 0 bytes even after a clean domain restart. Cause: SeaBIOS/PXE firmware writes to the VGA console, not serial, and MAAS's ephemeral kernel is not told console=ttyS0. A serial console alone does not make a PXE-booting node observable -- seeing the firmware/ephemeral stage needs a graphics device + virsh screenshot (or a MAAS-side kernel cmdline change). Queue this as the correction to the node-vm console work; do not assume the console added today closes the observability gap.

WHAT IS SOLIDLY ESTABLISHED (all measured):

  • PXE works and the ephemeral handoff happens: event log shows Performing PXE boot 18:17:03 -> Loading ephemeral 18:17:45 -> silence -> timeout.
  • The ephemeral OS BOOTS AND HAS WORKING NETWORKING: tcpdump on metal-admin caught 5-6 nodes holding DHCP leases (10.12.8.112-.117) and exchanging NTP with the rack. So node->rack reachability is fine.
  • No traffic toward the REGION (10.10.0.20) or the metadata ports (5248/5240) was seen in the sampled windows. SEC-010 was suspected of blocking the node->region path, but its rules carry no counter, so drops cannot be read directly -- hypothesis UNCONFIRMED, not refuted.
  • Ruled out by measurement: memory (385 GiB free), rack boot-image sync (synced, 10 images), DHCP (leases work), and node SHAPE (Ready and Failed both span 16/64, 12/48 and 8/24).

INVALID EXPERIMENTS -- MINE, RECORDED SO NOBODY TRUSTS THEM:

  1. The first "commission a single node ALONE" test was NOT alone: tcpdump later proved 5-6 other nodes were still up and running ephemeral images. My conclusion "contention is refuted" was therefore UNSOUND. Contention remains a live hypothesis.
  2. The second attempt was also invalid: MAAS had the machine in Commissioning already, so the new request did not restart the 30-minute timer (the failure at 19:24:51 belongs to the 18:54 attempt), and I then destroyed domains mid-commission, killing the in-flight run. STOPPED HERE rather than iterate again -- this is the reconciliation circuit-breaker: repeated cycles without convergence, and contaminated evidence is worse than none.

CLEANEST NEXT EXPERIMENT (not run): with ALL node domains powered off and nothing else touching the fleet, commission EXACTLY ONE machine from a clean Ready/New state and leave it strictly untouched for 35+ minutes. Only that isolates contention from a genuine per-node failure. Candidate fixes to test after it: raise MAAS node_timeout (nested I/O is slow and 30 min may simply be too short), add a graphics device for virsh screenshot, add counter to the SEC-010 rules so the node->region hypothesis becomes readable, and check the kernel cmdline MAAS actually serves.

20. CLEAN commissioning experiment: contention REFUTED (properly this time)

  • Ran the isolated experiment the earlier invalid ones only pretended to: ALL node domains OFF (verified 0 running), 397 GiB free, load 0.05, canary in a TERMINAL Failed commissioning state (so the commission restarts MAAS's 30-min timer), commission EXACTLY ONE, poll READ-ONLY only, touch nothing.
  • RESULT: Failed commissioning after 1770s (~29.5 min = the 30-min timeout), a SINGLE node entirely alone. Contention is refuted -- it is not I/O bandwidth, not memory, not the metadata service being swamped. The failure is systemic and per-node.
  • This CORRECTS changelog item 18-19's contention hypothesis (which rested on contaminated runs). The event trace is consistent every time: Commissioning -> Performing PXE boot -> Loading ephemeral (~40s later) -> 30 min of silence -> timeout. The node gets the ephemeral image handoff and then never reports back.
  • REFRAME after further measurement: "contention refuted" is right but too narrow. The single-node fail refutes CONTENTION (competing for resources) but is fully CONSISTENT with INHERENTLY SLOW -- the node may have still been working at 30 min, just not done. And the batch result 3 pass / 6 fail exactly at the 30-min mark is the classic signature of a MARGINAL TIMEOUT over a slow, variable operation: the faster-scheduled nodes finish under the wire, the rest are still going when the timer fires. 1770s is 29.5 of 30.
  • KEY CLUE against a network/metadata block: 3 nodes DID reach Ready on this exact rack, subnet, and metadata proxy. So the metadata path is NOT globally broken; a "node never reaches :5248" theory has to explain why it works for 3 identical nodes. Infra confirmed UP: rack :5248 listening (2 sockets), rack->region 5240 = 301. A :5248 tcpdump returned 0 packets but its 220s window likely closed before the ephemeral OS reached the metadata-fetch stage -- INCONCLUSIVE (a timing race), not evidence.
  • LEADING HYPOTHESIS (operator-facing): MAAS's default 30-min commissioning timeout is too short for depth-4 nested commissioning I/O. Cheap, decisive test that needs no stall-pinpointing: raise node_timeout (maas admin maas set-config name=node_timeout value=<min>, e.g. 60), commission ONE node, see if it completes. If it does, the fix is a config value (and a Roosevelt note: bare metal commissions faster, so this is a VR1-nested tuning, not a Roosevelt default). If it still fails well past 30 min, the timeout is exonerated and the next probe is the metadata-proxy access log / a console screenshot (needs the graphics-device correction to the node-vm console -- item 19).
  • STOPPED the live diagnosis here per the reconciliation circuit-breaker: many diagnostic cycles, and the remaining hypotheses need an operator decision (change a MAAS-wide config) rather than more agent spelunking.

8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review

  • Operator directive (verbatim intent): at the proper time/place, review the OPNsense 26.7 changelog + documentation for upgraded features and changes; incorporate beneficial improvements (esp. hardening) into the next deployment iteration, following project rules.
  • Scheduled: STAGE-3 CLOSE (GA-R6 close-out pass, with this stage's queued DOCFIXes). Deliverable: a findings doc citing the official 26.7 release notes; actionable items route per GA-R3 -- hardening/profile deltas as PROPOSED items anchored on D-129 (its 4 open sub-decisions are the likely surface), operational items as runbook DOCFIXes. NO self-executing incorporation: each finding is ruled per GA-R5 (one per exchange) before touching any built surface. Target of incorporation: dc1 standup + Roosevelt runbooks; live-edge retrofits individually gated.

6. ledger-scan hardening: FAIL-OPEN is terminology, not a status

  • SEC-010's close surfaced a scanner false positive: the OPEN-row detector (awk toupper(cell) ~ /OPEN|PENDING/) matched the substring OPEN inside "fail-open" in the closed row's disposition -- the row counted OPEN forever (same class as the GA-F15 wrap trap). Fix: strip FAIL-OPEN from the cell before matching (scripts/ledger-scan.sh); harness gains a CLOSED-row-with-fail-open fixture + nochk (45/45 PASS). Count re-verified: 7 open, SEC-010 absent.
  • Revert: git revert of this commit's ledger-scan.sh + test hunks.

5. Host tooling: libguestfs-tools installed on vcloud (operator-run)

  • sudo apt-get install -y libguestfs-tools (1:1.52.0-5ubuntu3) -- enables offline guest inspection (virt-cat/virt-ls), used for items 4's captures. Standard KVM-host tooling; Roosevelt-transferable.
  • Revert: sudo apt-get remove libguestfs-tools.