# Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap)

Session changelog (GA-R2/D1: ONE per session). Logged window: `dc0-deploy`
(same-day append, per the A/B-boundary handoff). Status lives ONLY in
`docs/CURRENT-STATE.md`; this file is narrative + reverts.

## 1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124)

- **Question as presented (2026-07-19):** "Who should hold 172.31.0.1 (the
  Office1 end of the dc0 transit /30), and by what mechanism? This is the
  prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS
  rack-to-region enrollment path."
- **Operator answer, exact utterance (option selected):** "voffice1 via IaC +
  restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to
  `module "voffice1"` in the outer root; apply in a planned voffice1 restart
  window (graceful shutdown first; region/NetBox/Tailscale return via proven
  autostart).
- Classification: OPS mechanism decision implementing D-124's already-ruled
  addressing ("region/peer .1" -- `docs/design-decisions.md:3727`); no new
  D-number (GA-R3, doubt resolves DOWN).
- **Measured basis (this session, logged):** mesh-vr1-dc0-office1 (virbr7) is
  ISOLATED (no `<forward>`/`<ip>`) with exactly ONE attached interface
  (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing
  holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only
  route `10.10.0.0/22 via 172.31.0.1` (d124-rack.auto.tfvars, consumed by
  step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached
  by `-J voffice1` (D-126 shape) -- NO vcloud host leg / site-baseleg DC row
  needed for this net.

## 2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated)

- **What:** `opentofu/main.tf` `module "voffice1"`: `network_names` gains
  `module.mesh_vr1_dc0_office1.network_name` SECOND (office1-local stays
  first -- PCI order preserves enp1s0); `network_config` rewritten from the
  single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4;
  transit=enp2s0 static `${vr1_dc0_rack_transit_peer_ip}/${prefix}` -- same
  D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init-
  re-run caveats documented in-block: the seed netplan is REBUILD correctness;
  the live transit config is an in-guest netplan drop-in at the gated attach
  step.
- **Why:** section 1's measured gap -- step B (ssh reach + rack enrollment)
  has no L3 path until the region end of the transit exists.
- **Revert:** restore `network_names = [module.office1_network.network_name]`
  and the previous "en*" glob `network_config` (git revert of this commit);
  if already applied, a follow-up plan shows the interface removal and the
  in-guest drop-in `/etc/netplan/60-transit.yaml` is deleted.

## 3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation

- Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply
  (1/1/1 exact, `docs/audit/outer-apply-20260719-voffice1-transit.txt`) ->
  domain up with both NICs, office1-local index 0.
- **INCIDENT (found+fixed in-window):** the provider REGENERATED both NIC
  MACs on the interface-list rebuild (old lan MAC `52:54:00:6a:87:e5` ->
  new `52:54:00:89:e1:19`). The Kea reservation (MAC-keyed) missed and
  voffice1 leased pool address `.108` (measured via edge ARP). FIX: re-keyed
  reservation uuid `b12621e6-63e9-4714-9726-2e130c84e069` to the new MAC via
  the D-113 API (`kea/dhcpv4/set_reservation` -> saved;
  `kea/service/reconfigure` -> ok), then a detached
  `networkctl reconfigure enp1s0` -> voffice1 back on `10.10.0.20`.
- **LOGGED FINDING (not executed):** `modules/cloudinit-vm` does not pin
  MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed
  DHCP reservations. Roosevelt-relevant; candidate module `mac` var or a
  documented ops step. Also: live transit config landed as in-guest netplan
  drop-in `/etc/netplan/60-transit.yaml` (narrows the lan glob to enp1s0 +
  transit enp2s0 static) -- the drop-in must ALSO narrow `lan`, else the
  50-cloud-init `en*` glob claims enp2s0 (measured: first drop-in attempt
  without it left enp2s0 addressless).
- **Revert:** `kea/dhcpv4/set_reservation` back to the old MAC (senseless
  unless the NIC change is also reverted); delete the drop-in.

## 4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix

- Symptom: `172.31.0.2` unreachable; MEASURED: both rack NICs emitted ZERO
  frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive
  (ACPI-responsive; cloud-init completed per offline logs).
- Guest is a SEALED BOX: `modules/cloudinit-vm` domains have NO serial
  console, NO qga channel, NO graphics; qga is NOT in the noble base image
  and its cloud-init install failed (no egress) -- so no console, no agent,
  no network. **LOGGED FINDING: add opt-in serial console (+ the D-129-style
  qga channel var) to `modules/cloudinit-vm` -- observability gap bites every
  future DC VM. NOT executed mid-step.**
- Offline inspection (libguestfs, operator-run, captures
  `~/vvr1-dc0-offline-inspect{,2}.txt`): NIC names enp1s0/enp2s0 CORRECT
  (naming trap closed); root cause = netplan `match.name` + `set-name`:
  boot 1 renames don't apply (.link written after udev add) so the rendered
  config matched nothing; after removing set-name, boot 2 STILL failed
  because the early-boot netplan generator ran against the OLD config and
  renamed the devices to mgmt/uplink before cloud-init wrote the new file
  (one-boot lag, measured in syslog). Boot 3 (current config, no renames)
  is the verification boot.
- FIX (this commit + applied): `network_config` drops set-name, keeps
  MEASURED kernel names; `instance-id` bumped `vvr1-dc0-d123 -> -d123-r2`
  (side effects noted in-block); seed replaced via explicit
  `-replace=module.vvr1_dc0.libvirt_volume.seed` (the sanctioned override
  of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2
  exact (`docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt`).
- CONSEQUENCE for step B: bootstrap runs with
  `--transit-if enp1s0 --uplink-if enp2s0` (script defaults mgmt/uplink no
  longer match reality); runbook prose naming "mgmt" needs a DOCFIX at
  stage close.
- **Revert:** git revert of the netplan/instance-id edit + one more
  seed `-replace` apply + reboot.

## 4b. VERIFICATION (2026-07-20, logged window)

- Boot 3 with the fixed config: `enp1s0 UP 172.31.0.2/30`, region route
  present, `br-vr1-dc0-wan` up.
- voffice1 -> 172.31.0.2 ping 0% loss; ssh
  `-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2`
  works (D-126 -J shape, per-env key); rack -> region `ping 10.10.0.20`
  0% loss (the MAAS-enrollment path for the G10 bootstrap).
- Outer plan CONVERGED back to ZERO DIFF
  (`docs/audit/outer-plan-20260720-postfix-converged.txt`); CURRENT-STATE
  section 5 updated in the same commit.
- Follow-items (logged, not executed): D-126 DC ssh-config alias row for
  vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key);
  cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC
  pinning; DOCFIX for runbook "mgmt/uplink" naming.

## 4c. Step B bootstrap EXECUTED on vvr1-dc0 (2026-07-20, logged window)

- Snap proxy set to the region maas-proxy (measured serving: HTTP 200 to
  archive.ubuntu.com, CONNECT to api.snapcraft.io) -- the D-107-shaped
  answer to the rack's no-egress bootstrap; apt used the same proxy via
  per-invocation env. Enrollment secret piped region->rack
  (`install -m600 /dev/stdin`), verified by byte-count only (32), never
  in context.
- Run 1 (`docs/audit/stepB-bootstrap-20260720.txt`): maas snap 3.7/stable
  installed, **rack ENROLLED to http://10.10.0.20:5240/MAAS**; FAILED at
  qemu apt install on stale image package lists (404s via proxy).
  Fix: `apt-get update` (proxied), re-run.
- Run 2 (`docs/audit/stepB-bootstrap-20260720-run2.txt`): EXIT 0 --
  libvirt/qemu + nested KVM on, inner pool + AppArmor, SEC-010
  FORWARD-drop loaded on enp1s0 + `sec010-fw.service` enabled
  (boot-persistent), `br-vr1-dc0-wan` verified with enp2s0 enslaved.
- `--check` (`docs/audit/stepB-check-20260720.txt`): all items ok EXCEPT
  opnsense base (the remaining step-B item); rackd running, enrolled.
- Region-side residue for the MAAS phase (per script exit text): enable
  DHCP on metal-admin at the Office1 region naming this rack primary_rack
  (step D territory, NOT done).
- **Revert:** rack enrollment: region-side `maas ... rack-controllers
  delete`; SEC-010: disable sec010-fw.service + rm /etc/nftables-sec010.nft;
  packages: apt/snap remove; snap proxy: `snap unset system proxy.http
  proxy.https`.

## 4d. RULING RECORD (GA-R5) -- DC edge base version (2026-07-20)

- **Question as presented:** reuse verified 26.1 artifact vs fresh download
  (first exchange); then, sharpened after the operator noted "Office1 is
  running OPNsense 26.7": 26.1-base-then-upgrade vs fresh 26.7 nano base.
- **Operator answer, exact utterance (option selected):** "Fresh 26.7 nano
  base".
- Consequences implemented: prep runs 26.7 -> `opnsense-26.7-nano.qcow2`;
  `vr1-dc0-substrate` `opnsense_base_path` default updated (no inner state
  exists; pre-apply edit); step-B `--check` passes `--opnsense-base` for
  the 26.7 path. LOGGED follow-items: site-headend-install.sh's 26.1
  default + hint strings (script change + harness, stage close); the
  D-112/D-113 serial/API boot path is REVALIDATED on 26.7 at the step-C
  boot gate (office1 precedent: 26.1-proven, upgraded in place to 26.7).
- Mirror: `https://mirror.ams1.nl.leaseweb.net/opnsense` -- MEASURED from
  the vendor's live download page this session (not memory), HEAD-verified
  serving the 26.7 nano (~468 MiB). LOGGED FINDING: opnsense-prep-image.sh
  performs no checksum verification of the downloaded image (same property
  as the original 26.1 prep; candidate hardening).

## 4e. Step B CLOSED: 26.7 base staged; final check EXIT 0; SEC-010 CLOSED both ends

- OPNsense 26.7 nano prepped ON the rack (leaseweb mirror via region proxy,
  468M; python3 bz2 fallback -- bunzip2 never needed, DOCFIX-184;
  `docs/audit/stepB-opnsense-prep-20260720.txt`). Inner-root
  `opnsense_base_path` default -> 26.7 (validate PASS).
- Final `--check` EXIT 0, ALL items ok
  (`docs/audit/stepB-check-20260720-final.txt`).
- SEC-010 voffice1-side pin (the row's "same pin follows" clause, due since
  this session wired voffice1's transit leg; voffice1 runs LXD so
  ip_forward=1): identical scoped nft artifact + sec010-fw.service on
  enp2s0, verified loaded + persistent; rack proxy/region paths re-verified
  working post-pin. **SEC-010 row CLOSED (operator ruling, utterance in the
  row).** Open SEC count 8 -> 7.
- LOGGED follow-item: site-headend-install.sh region-role support for the
  transit-if pin (voffice1's was hand-applied to the script's exact
  artifact shape; fold into the script + harness at stage close).
- **Revert:** rm the two files + `systemctl disable sec010-fw` on voffice1;
  `nft delete table inet sec010`; ledger row reopens citing this item.

## 7. Step C: voffice1 becomes the Plane-2 executor; INNER APPLY COMPLETE (28/28)

- voffice1 setup (gated, logged): tofu 1.12.4 via vcloud's exact apt
  source (keyrings byte-copied; version == section-7 pin); repo cloned at
  `29cf7bf` (branch); dc0 private key staged to `~/vr1-dc0-creds/`
  (0700/0600, piped -- SEC-009 convention now spans hosts); ssh config
  entry for 172.31.0.2; known_hosts seeded via ssh-keyscan with the
  ED25519 fingerprint cross-checked against the accept-new-trusted entry
  (MATCH).
- Inner-root fixes measured at the FIRST REAL dial/apply (each committed):
  (i) provider needs `keyfile`+`sshauth=privkey` URI params -- its Go ssh
  ignores ~/.ssh/config AND default identities; single-type known_hosts
  fails as "key mismatch" (multi-type scan required);
  (ii) `modules/wan-bridge`: `<mtu>` is ILLEGAL in bridge-mode networks --
  removed (MTU belongs to the host bridge; DOCFIX-194 class: parsed, never
  applied);
  (iii) base-image location premise WRONG under remote provider -- content
  UPLOADS from the executing host; 26.7 qcow2 copied rack->voffice1
  (`~/vr1-dc0-images/`), tfvar override + variables.tf description
  corrected;
  (iv) edge serial-log dir `/var/lib/libvirt/vr1/staging/` is a HARDCODED
  vcloud literal in modules/opnsense-edge (:223) -- dir created on the rack
  (755); LOGGED findings: parameterize the path; fold staging-dir creation
  into site-headend-install --host-nodes; opentofu-validate.sh does not
  cover the inner root.
- Applies (captures `docs/audit/inner-{plan,apply}-20260720-stepC*.txt`):
  run 1 = 25/28 (nodes+planes+pool; wan-bridge mtu abort); run 2 = wan
  bridge (edge disk stat abort); run 3 = edge disk (staging-dir abort;
  provider self-cleaned the failed domain -- operator undefine found
  nothing); run 4 = edge domain, EXIT 0. CONVERGED: inner plan zero diff
  (`docs/audit/inner-converge-20260720-stepC.txt`).
- VERIFIED: 10/10 domains RUNNING inside vvr1-dc0 (9 nodes depth-4 + edge);
  edge serial log = FreeBSD login prompt on ttyu0 -- 26.7 boots the D-112
  serial path (first revalidation datapoint; full D-112(c) bootstrap is the
  next window).
- NEW STATE OF RECORD: the inner tfstate lives on voffice1
  (`~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/terraform.tfstate`)
  -- back it up with the site; follow-item for the backup set.
- **Revert:** `tofu destroy` in the inner root from voffice1 (site-down
  alternative: `virsh destroy vvr1-dc0`, D-122); module fixes revert by
  commit.

## 9. DC edge: D-112(c) console bootstrap + API key minted (26.7) -- REPLACEMENT CHAIN PROVEN

- Per-env edge key minted on vcloud (`~/vr1-dc0-creds/vr1-dc0-edge_ed25519`,
  0600/0644, SEC-009 convention), staged to the rack the same way.
- Rack legs added (operator-ruled "10.12.4.2, manual now + script later"):
  `10.12.4.2/22` on the provider-public inner bridge (virbr5, MEASURED) +
  the D-124-ruled `10.12.8.2/22` on metal-admin (virbr2). INTERIM: `ip addr
  add`, not persistent. **LOGGED FINDING (queued):** site-headend-install.sh
  never implemented the rack legs despite main.tf's comment claiming it
  consumes `vr1_dc0_rack_metal_admin_ip` -- add rack-leg support + harness
  before step D closes. Also interim: `192.168.1.2/24` on virbr5 to reach
  the edge's FACTORY LAN; REMOVE once the edge is re-addressed.
- **D-112(c) console bootstrap: DONE** (`~/d112c-console.log` on the rack).
  Driver evolved v1->v6 against MEASURED failures, each real and worth
  keeping for the runbook:
  (i) serial console wraps long lines -> ship payload as <=160-char base64
  chunks (`openssl base64 -d`), never one long line;
  (ii) `set-name` in the edge netplan is irrelevant here but the same class
  bit the rack (item 4);
  (iii) tcsh on the edge (documented trap) breaks nested-quoted `sed` --
  "Unmatched '"'"'" -- so NEVER patch files via quoted one-liners; re-ship
  the whole payload instead;
  (iv) `write_config()` needs an include beyond config.inc -- MEASURED
  (`grep -l shell_safe /usr/local/etc/inc/*.inc` -> auth/console/filter/
  interfaces/rrd/system/util.inc), not guessed;
  (v) writing config.xml does NOT materialize `/root/.ssh/authorized_keys`
  -- must call the vendor's `local_user_set()` (auth.inc). This was the
  actual reason key auth failed after a "successful" config write.
- VERIFIED: `ssh -i vr1-dc0-edge_ed25519 root@192.168.1.1` -> key-only auth
  OK, `uname -r` = 15.1-RELEASE-p1, `ifconfig -l` = vtnet0/vtnet1.
  INTERFACE MAPPING MEASURED: vtnet0 = LAN = provider-public plane (it
  answered on virbr5), vtnet1 = WAN -- same order as the office1 edge.
- **API key minted via the vendor model** (`opnsense-bootstrap-apikey.sh`
  + `opnsense-mint-apikey.php`, run ON the rack): key/secret 80 chars ->
  `~/vr1-dc0-creds/opnsense-api.txt` (0600, secret never printed).
  SMOKE TEST PASS: `GET core/firmware/status` -> HTTP 200, `CORE_ABI 26.7`.
  **This is the first proof the D-113(a2) API path works on 26.7** (the
  26.7 revalidation the operator's base-version ruling called for).
  Script note: the bootstrap script requires the minter BESIDE it (its
  own dir), not in `scripts/` -- worth a usage-line DOCFIX.
- **Revert:** delete the API key (`POST auth/user/del_api_key/<id>`, the
  proven negative-tested path) + creds file; `ip addr del` the three
  interim addresses; the console bootstrap reverts by re-imaging the edge
  (inner `tofu destroy`/re-apply of the edge module).

## 10. D-125 egress isolation gate: EXECUTED as written -- PARTIAL (bridge-in PROVEN,
## upstream NAT egress FAILS). Gate NOT closed; double-NAT fallback NOT invoked.

- Method: operator-ruled "throwaway VM, as written" -- noble cloud image
  fetched ON the rack via the region proxy, one cloud-init guest, SINGLE
  NIC bridged to `br-vr1-dc0-wan`, serial-log probe, then destroyed
  (`leftover-domains=0` asserted). Script: scratchpad
  `d125-egress-test.sh`; capture `docs/audit/d125-egress-gate-20260720.txt`.
- Run 1 was a TEST ARTIFACT, not a gate failure: the probe used DHCP and
  came up address-less. MEASURED cause: `vr1-dc0-uplink` carries an `<ip>`
  but NO `<dhcp>` block -- there is no DHCP server on that /24 by design
  (the edge WAN takes a STATIC .2, D-113/D-125). **DOCFIX QUEUED:** the
  gate's wording "must GET a vcloud-ISP address" implies DHCP; it must say
  STATIC, or the /24 needs a range. Re-run used static 172.30.2.50/24.
- RESULT (run 2): guest configured 172.30.2.50/24 + default via 172.30.2.1;
  **gateway ping 3/3, 0% loss** -- a FOREIGN MAC inside vvr1-dc0 reaches
  the vcloud ISP NAT across the nested bridge. That is the D-125 bridge-in
  property, and it PASSES. But `ping 1.1.1.1` FAILED (NET-PING-RC=1).
- **CORRECTION (2026-07-20, same-day): the rack-bridge follow-up is
  CONFOUNDED and is NOT evidence.** It sourced from 172.30.2.51 on the
  rack, but the rack has NO DEFAULT ROUTE (netplan gives it only
  `10.10.0.0/22 via 172.31.0.1`; the WAN bridge is IP-less by design), so
  traffic to 1.1.1.1 fails in the rack's own routing table before the NAT
  is ever consulted. Its ICMP+TCP failures say nothing about virbr4. The
  ONLY clean signal is the GUEST's (172.30.2.50) failed egress, and that
  guest tested ICMP only -- if the rule dump comes back identical for both
  bridges, re-run the guest probe with `curl` before concluding anything.
  (Recorded per GA-R1/C2: a wrong measurement is corrected in the record,
  not quietly dropped.)
- Controls: vcloud egress 0% loss; the office1 edge
  egresses through the SAME-SHAPE `office1-wan` NAT 0% loss. The two
  network definitions are byte-equivalent in substance (`net-dumpxml`:
  both `forward mode=nat` + `<nat><port 1024-65535>`; only netmask-vs-
  prefix notation differs) -- so the delta is in vcloud's LIVE firewall
  rules for virbr4, NOT in the design or the config.
- **GATE STATUS: PASS / CLOSED 2026-07-20** (runs 3+4, capture
  `docs/audit/d125-egress-gate-20260720-matrix.txt`). The "missing rule"
  hypothesis was DISPROVED by measurement first: the operator-run dumps
  show virbr4's rule set is structurally IDENTICAL to the WORKING virbr11
  (only counters differ; virbr11's one extra rule is a DHCP checksum
  fixup it has because it carries a DHCP range). All three masquerade
  rules exist for 172.30.2.0/24 INCLUDING the generic (non-tcp/udp) one,
  and it had FIRED (1 packet/84 bytes = one ICMP conntrack entry), with
  FORWARD accept counting 136 packets. So `net-destroy/net-start` would
  have fixed nothing -- the pre-staged fix was correctly NOT run.
- Decisive re-probe used a protocol/destination MATRIX instead of counter
  archaeology. Result, IDENTICAL on two consecutive runs: gateway ping 0,
  **internet ping 0**, `curl http://1.1.1.1` -> 301, `curl
  archive.ubuntu.com` -> 200. Egress works for ICMP AND TCP to real
  hosts, so the D-125 bridge-in chain is PROVEN end to end and the
  double-NAT fallback is NOT needed.
- **Run-2 anomaly, recorded UNEXPLAINED (not swept):** the same path
  failed ICMP-to-internet once while its gateway ping passed 3/3. Runs 3+4
  passed identically with no intervening change to the uplink or its
  rules. Most likely a first-use conntrack/ARP transient on vcloud. It is
  not reproducible now; if a DC edge ever shows first-boot egress failure,
  START HERE rather than re-deriving.
- **Security-positive side observation** (extra probe, outside the gate):
  the guest could NOT reach the Office1 management LAN through the ISP NAT
  (`10.10.0.20:8000` -> 000) on all runs, while internet egress worked.
  That is the D-122/SEC-010 intent holding: the DC's WAN segment is an ISP
  uplink, not a path into Office1. Worth an explicit assertion in the
  hardening tests rather than leaving it an incidental observation.
- Prior status line (superseded, kept for the audit trail): gate was held
  OPEN pending measurement rather than FAILED. The runbook's FAIL remedy
  (revert to
  double-NAT) is for a DESIGN failure; invoking it against what is very
  likely a host firewall-rule gap would be the wrong remedy and would add
  Roosevelt delta. Next step is a read-only measurement needing operator
  sudo: FULL dumps of BOTH `nft list ruleset` and `iptables-save` (which
  backend is authoritative here is itself unmeasured -- do not assume),
  diffing virbr4 against the WORKING virbr11. Two rule families must both
  be checked, and a subnet-only grep would MISS the second: POSTROUTING
  masquerade (keyed on `172.30.2.0/24`) and FORWARD accept (keyed on the
  INTERFACE `virbr4`). Pre-staged fix-forward IF the rules are absent or
  malformed vs virbr11: `virsh net-destroy vr1-dc0-uplink && virsh
  net-start vr1-dc0-uplink` reinstalls libvirt's rules from scratch --
  cheap and safe (nothing rides that uplink yet but the probe). If instead
  the dump shows libvirt never installed them correctly, the restart will
  not help and the dig moves to the firewall backend / conntrack.
- **Revert:** nothing to revert -- the probe guest and both temporary
  addresses were removed; the only residue is `/tmp/noble.img` on the rack
  (kept deliberately: it is the base for any future throwaway probe).

## 11. NEW TOOLING + edge re-addressed: opnsense-set-interface-v4 pair (53/53)

- **RULING RECORD (GA-R5), 2026-07-20.** Question as presented: base-iface
  IPv4 is not REST-covered on 26.7 (measured); how should LAN 10.12.4.1/22
  and WAN 172.30.2.2/24 + gw 172.30.2.1 be set -- build the v4 setter pair
  / console menu now / console now + build at stage close? Operator answer,
  exact utterance (option selected): "Build the v4 setter pair
  (Recommended)". Classification: OPS tooling under the existing D-113
  amendment; no new D-number (GA-R3).
- **D-113 AMENDMENT RE-MEASURED ON 26.7** (feeds the item-8 review): the
  amendment was measured on 26.1 only. On this 26.7 edge:
  `interfaces/settings/get` returns ONLY global offload flags, while
  `interfaces/overview/list` and `interfaces/lan/get` both return 404
  "Endpoint not found". Base-interface addressing is still legacy
  `/interfaces.php`. The amendment now holds on TWO majors, measured.
- SHIPPED: `scripts/opnsense-set-iface-v4.php` +
  `scripts/opnsense-set-interface-v4.sh` + `tests/opnsense-set-interface-v4/`
  (53/53 PASS first run; gauntlet 70 -> **71 harnesses ALL GREEN**;
  repo-lint 0 fail). Mirrors the proven v6 sibling exactly -- vendor Config
  singleton (never a config.xml push), dry-by-default, interface-count
  guard, fresh read-back verify, `sh -s` for the tcsh trap, KERNEL as
  ground truth. Two deliberate improvements over the v6 sibling:
  (i) the GATEWAY half (creates/updates at most ONE `gateway_item`,
  count-guarded exactly like the interface count, opt-in via a 5th arg);
  (ii) the interface's DEVICE is MEASURED from the edge's own config
  instead of the v6 sibling's hardcoded `lan->vtnet0 / wan->vtnet1` sed --
  that mapping is per-edge and hardcoding it is a hard-rule-2 violation
  waiting to happen. Harness pins BOTH (test 12 fails if the hardcode
  returns). **LOGGED FINDING:** backport the measured-device fix to the v6
  sibling at stage close.
- APPLIED to vr1-dc0-opnsense, WAN first per the script's own ordering
  warning (re-addressing the LAN drops the session you arrived on):
  - WAN: dhcp -> `172.30.2.2/24` + `WAN_GW` = 172.30.2.1 default. Note the
    edge had shipped with WAN on **dhcp**, which could NEVER have worked --
    the uplink /24 has no DHCP server (same fact that broke probe run 1).
    Verified on the kernel: `inet 172.30.2.2`, `default 172.30.2.1 UGS
    vtnet1`, and **the EDGE ITSELF pings 1.1.1.1 at 0% loss** -- the D-125
    chain now proven through the real OPNsense edge, not just a probe.
  - LAN: `192.168.1.1/24` -> `10.12.4.1/22` (the D-115/lib-net ruled
    provider-public gateway). Session dropped as designed; edge verified at
    the new address from the rack: `inet 10.12.4.1 netmask 0xfffffc00`.
  - Post-move reconciliation at the NEW address: API `GET
    core/firmware/status` -> 200 `CORE_ABI 26.7`; edge egress still 0% loss.
- CLEANUP: the interim `192.168.1.2/24` on the rack's provider-public
  bridge is REMOVED (virbr5 now carries only the ruled `10.12.4.2/22`).
- **Revert:** re-run the setter with the previous values
  (`--commit <edge> lan 192.168.1.1 24`, `--commit <edge> wan ...`); the
  gateway item is named `WAN_GW` and is the only one this tool created.

## 12. D-129 edge profile APPLIED on 26.7 + qga channel shipped + a FALSE-SUCCESS bug fixed

- **RULING RECORD (GA-R5), 2026-07-20.** Question: the qga virtio channel is
  absent from the edge domain and unsupported by `modules/opnsense-edge`
  (both MEASURED), so installing os-qemu-guest-agent alone yields a package
  that can never answer. Options presented: add channel var now / iperf only
  + defer / install both and leave qga inert. Operator answer, exact
  utterance (option selected): "Add channel var now, then install both
  (Recommended)". [OPS under D-129; no new D-number, GA-R3.]
- **SHIPPED `expose_qga_channel`** (opt-in, DEFAULT OFF) in
  `modules/opnsense-edge`; `vr1-dc0-substrate` sets it true so NEW DC edges
  are BORN with the channel (the runbook's no-retrofit intent). Default OFF
  is load-bearing: a device change alters the domain, so defaulting ON would
  bounce every existing edge on the next plan. Shape taken from
  `tofu providers schema -json` (dmacvicar 0.9.8:
  `devices.channels[].target.virt_io{name}` + `source.unix{mode}`), NOT from
  a blog; `source.unix` carries no path, letting libvirt allocate the socket
  (same reasoning the serial block documents for pty).
- APPLY was SAFE: plan showed **0 add / 1 change / 0 destroy -- an IN-PLACE
  domain update**, so `libvirt_volume.disk` (holding the console + API
  bootstrap) was never at risk. Verified after: channel present (grep count
  2), domain running, edge still reachable at 10.12.4.1.
- **BUG FOUND ON THE FIRST-EVER LIVE RUN -- `opnsense-plugins.sh apply` had
  NEVER installed anything.** `api()` used `bash "$API" ${DRY_RUN:+--dry-run}`:
  `${VAR:+word}` expands when VAR is set and NON-EMPTY, and `DRY_RUN=0` is
  non-empty -- so every apply silently dry-ran while printing
  "OK: profile 'vr1-edge' applied". A textbook false success, and the reason
  it survived review is that the harness drove ONLY `--dry-run`, leaving the
  live path with zero coverage. FIX tests the VALUE; harness gains two
  live-path tests against a stub api client, **verified to FAIL 2/2 on the
  buggy version before restoring** (a regression test that cannot fail is
  worthless). 20/20. This is exactly the class D-129's own "package present
  is not success" warning is about -- here the SCRIPT was the false reporter.
- REAL install then ran: `{"status":"ok","msg_uuid":...}` per plugin (not
  dry-run echoes), capture `docs/audit/d129-plugins-20260720.txt` (the
  earlier capture of the same name recorded the false dry-run and is
  REPLACED). Measured on the edge: `os-qemu-guest-agent-1.3`,
  `qemu-guest-agent-11.0.1`, `os-iperf-1.0_2`, `iperf3-3.21`.
- Service was NOT auto-started (the script's own NOTE); started via the
  plugin's OWN API (`POST qemuguestagent/service/start` -> `{"response":"OK"}`,
  status -> running) rather than by hand-editing rc.conf.
- **GATE MET, to the runbook's exact bar (not package-presence):**
  `virsh qemu-agent-command vr1-dc0-opnsense '{"execute":"guest-ping"}'` ->
  `{"return":{}}`, and `virsh domifaddr --source agent` reports vtnet0
  10.12.4.1/22 + vtnet1 172.30.2.2/24. The agent ANSWERS over the channel.
- **SECURITY FINDING (queued for the item-8 hardening review):** the plugin
  model exposes `DisabledRPCs`, including **`guest-exec`/`guest-exec-status`
  -- arbitrary command execution on the edge from the libvirt host**. On a
  commercial multi-tenant cloud the DC edge should almost certainly disable
  the exec RPCs (we need guest-ping/fsfreeze/shutdown coordination, not
  remote exec). Not changed mid-step; propose with options at the review.
- **Revert:** `POST qemuguestagent/service/stop`; uninstall via
  `core/firmware/remove/<pkg>`; set `expose_qga_channel = false` and re-apply
  (in-place domain update again).

## 13. Step D part 1 DONE (DHCP configured) -- then INCIDENT: MAAS DHCP is DOWN
## REGION-WIDE (Temporal wedged). Not caused by this deploy.

Step-D work completed before the incident surfaced:
- Both controllers confirmed registered: `voffice1` (region, mtstwf) and
  **`vvr1-dc0` (rack, 7chphy, rackd running)**.
- MAAS auto-discovered the inner planes from the rack's interfaces:
  metal-admin `10.12.8.0/22` = VLAN 5005/fabric-4, provider-public
  `10.12.4.0/22` = VLAN 5006/fabric-5. (Stale `192.168.1.0/24` also sits on
  fabric-5 -- the edge's old factory LAN, now vacated; cleanup queued.)
- **RULING RECORD (GA-R5), 2026-07-20.** Question: metal-admin is a /22 but
  D-120's band layout is written for a compose /24; what dynamic range?
  Operator answer, exact utterance (option selected): "Inherit D-120 band:
  10.12.8.100-10.12.8.200 (Recommended)". Range CREATED and verified on the
  correct subnet; VLAN 5005 updated `dhcp_on=true primary_rack=7chphy`,
  verified by read-back (`10.12.8.0/22 dhcp=True primary=7chphy`).

**INCIDENT (found by verification, not by a self-report):**
- Canary `virsh reset vr1-dc0-control-01` did NOT enlist. MEASURED cause
  chain: rack service_set shows `dhcpd: off`; no dhcpd process on the rack;
  no `/var/snap/maas/common/maas/dhcpd.conf` generated -- while MAAS's own
  API reports the VLAN as `dhcp_on=True` with the correct primary rack and
  the interface correctly linked (VLAN 5005, static 10.12.8.2). Config
  state and running state DISAGREE.
- Root cause, from the region's journal: MAAS 3.7 drives DHCP through
  **Temporal workflows**, and Temporal is wedged --
  `"Not enough hosts to serve the request"`, `poll_workflow_task_queue
  retried 2807 times`, task queue `mtstwf@region:...`. No worker is polling,
  so the workflow that pushes DHCP config to a rack can never run.
- **SCOPE IS WIDER THAN THIS DEPLOY:** `voffice1` itself has **NO dhcpd
  process** (`pgrep -c dhcpd` -> 0) and reports no dhcpd service, while
  Office1's compose net `10.10.1.0/24` reads `dhcp=True`. So MAAS DHCP has
  been DOWN REGION-WIDE -- almost certainly since the 2026-07-17 host reboot
  -- and went unnoticed because office1-netbox/office1-tailscale were
  already Deployed and kept their addresses. **This is NOT caused by the DC0
  deploy; the deploy is what exposed it.**
- Consequence for the as-built record: any claim that Office1 MAAS DHCP is
  "working" is currently FALSE (last positive evidence predates the reboot).
- **Detection gap (queued):** `cloud-assert.sh` and the Office1 health
  checks did not catch a dead DHCP server, because MAAS's own service_set
  reported the region healthy. A behavioral DHCP assertion belongs in the
  health sweep -- this is precisely the "service's own verdict" principle
  the repo already applies elsewhere.
- No prior art: appendix-A has no entry for this symptom. NEW incident
  material once the fix is known.
**RESOLVED 2026-07-20 (operator-approved: "Go ahead with the restart").**
- `sudo snap restart maas` on voffice1 -> region back (HTTP 301 on
  localhost:5240). Recovery verified BEHAVIORALLY, not by service status:
  `pgrep -c dhcpd` on voffice1 0 -> **1**, and on the DC rack 0 -> **1**;
  MAAS then reported `dhcpd: running` on BOTH controllers (was `off` on the
  rack, absent entirely on the region). One restart fixed both sites, which
  confirms the single wedged-Temporal root cause rather than two faults.
- **PROOF IT WAS THE REAL BLOCKER:** the identical canary that had failed
  before (`virsh reset vr1-dc0-control-01`) enlisted immediately after.
- APPENDIX-A MATERIAL (queued, new incident class): symptom "MAAS says
  dhcp_on=True but no dhcpd runs and nothing PXEs"; check `pgrep -c dhcpd`
  on each controller and the region journal for Temporal
  `"Not enough hosts to serve the request"`; remedy `snap restart maas`;
  verify behaviorally on every controller, never by service_set.

## 14. G10 DEPTH-4 NESTED BOOT GATE: **PASS** -- all 9 DC0 nodes enlisted in MAAS

- After the DHCP fix, all nine node VMs were reset and **all nine enlisted**
  (`docs/audit/stepD-enlistment-20260720.txt`): count 9, status 8
  Commissioning + 1 New, shapes `{16cpu/64GiB: 3, 12cpu/48GiB: 2,
  8cpu/24GiB: 4}` -- an EXACT match to the D-121 Option C / R-3 layout
  (3 control + 2 compute + 4 storage).
- **This closes the D-114/D-123 depth-4 boot gate, the flagged-risk item of
  Model B.** A node VM inside `vvr1-dc0` (itself a VM on vcloud) PXE-booted
  from the Office1 region across the D-124 transit and is running MAAS's
  ephemeral commissioning kernel -- i.e. real nested KVM at depth 4, proven
  by behavior rather than by inference from `svm` being present.
- Still open in step D: commissioning completes with a power-off, and MAAS
  has no power driver for these VMs yet -- the `maas-vm-host` registration
  (module BUILT 2026-07-09, still UNVALIDATED) is the next piece, and it
  needs `TF_VAR_maas_api_key`.
- **Revert:** `maas admin machines delete` per machine (or delete the pod
  once registered); the VMs themselves are inner-root state.

- PROPOSED REMEDY (superseded by the RESOLVED block above; kept for the
  audit trail): restart MAAS on the region
  (`sudo snap restart maas` on voffice1) to re-establish Temporal workers,
  then re-verify dhcpd on BOTH controllers behaviorally. Blast radius is
  low right now (no OpenStack workloads; both Office1 VMs already deployed
  and unaffected by a brief DHCP/API outage), which makes this the cheapest
  moment to fix it.

## 15. Step D part 2: NEW root for MAAS registration; and a MEASURED REFUTATION
## of D-123's "power_address = qemu:///system LOCAL to the rack"

- SHIPPED `opentofu/vr1-dc0-maas/` -- a THIRD root holding the MAAS
  registration alone. Rationale is DOCFIX-179's own measured lesson applied
  rather than repeated: a `provider "maas"` block forces EVERY plan in its
  root to demand `maas_api_url` + the SENSITIVE key, even for plans creating
  zero MAAS resources (the DOCFIX-175 plaintext-secret surface). Putting it
  in the inner substrate root would make routine substrate work require MAAS
  creds. Substrate roots stay credential-free. Validated explicitly --
  **note `opentofu-validate.sh` covers only the outer root + modules, so
  neither extra root is gauntlet-covered; extending it stays QUEUED and is
  now more load-bearing (two uncovered roots).**
- Secret handling: key supplied ONLY as `TF_VAR_maas_api_key` from the
  operator-placed 0600 file; verified by FORMAT alone (71 bytes, 3
  colon-separated parts), never printed. Caveat recorded in the root's
  header: the key DOES land in that root's tfstate (unavoidable with this
  provider), so that state inherits credential handling.
- Plan was clean (1 to add). **APPLY FAILED:**
  `ServerError: 503 ... Failed talking to pod: Failed to login to virsh
  console.`
- **ROOT CAUSE, MEASURED (not inferred):** the MAAS snap is strictly
  confined. From inside it, `virsh -c qemu:///system` ->
  `Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission
  denied`. And `snap connections maas` lists **NO libvirt interface at all**
  -- there is nothing to `snap connect`. So a LOCAL `qemu:///system` VM host
  is ARCHITECTURALLY IMPOSSIBLE with snap-based MAAS.
- **THIS REFUTES A DOCUMENTED DESIGN LINE.** D-123 Model B and
  `modules/maas-vm-host`'s header both state the DC pod is registered with
  `power_address = qemu:///system` LOCAL to the rack ("vvr1-dc0 is BOTH the
  MAAS rack AND the libvirt host, so no cross-fiber virsh dial"). The
  no-cross-fiber INTENT survives; the stated MECHANISM cannot work. Both
  surfaces need an amendment once the replacement is ruled. Roosevelt-
  relevant: the same constraint hits every DC pod and any snap-MAAS metal
  build, so this is not a VR1 quirk.
- Feasibility of the replacement, MEASURED: the snap DOES ship an ssh client
  (`/snap/maas/41649/usr/bin/ssh`), and `jessea123` on the rack is in the
  `libvirt` group (113) -- so `qemu+ssh://jessea123@172.31.0.2/system`
  reaches libvirt through sshd (unconfined) instead of the blocked socket.
  It needs a key the snap's root can read (`/var/snap/maas/current/root/.ssh/`
  does not exist yet) plus that pubkey in the rack user's authorized_keys.
- **Revert:** the failed apply created nothing (state empty); `git rm -r
  opentofu/vr1-dc0-maas` removes the root.

## 16. The pod path, chased to its end -- and a SECOND refutation: MAAS virsh
## pods are incompatible with this repo's node-vm disks

Sequence, each step measured (operator ruled the DEDICATED key: "Dedicated
MAAS->libvirt key (Recommended)"):
1. Minted `maas-virsh_ed25519` on the rack (0600, SEC-009 folder), pubkey into
   the rack user's authorized_keys, private half into the RACK's MAAS snap.
   Rack snap could then `virsh -c qemu+ssh://...` fine -- **but the apply still
   failed the same way.**
2. MEASURED why: MAAS attempts the pod connection from the **REGION**, not the
   rack. voffice1's snap had no key and no known_hosts -> `Host key
   verification failed`. Same key + keyscan installed there; region snap then
   listed the inner domains. (Worth keeping: for a virsh pod, the credential
   belongs wherever MAAS dials FROM, which is not necessarily the rack that
   owns the hardware.)
3. Next apply hit `Pod vr1-dc0-inner ... already exist` -- the two earlier 503s
   had EACH created a pod server-side before erroring, so MAAS held two broken
   `cores=0` pods outside tofu state. Deleted both (own residue), re-applied.
4. Next apply reached the storage inventory and failed:
   `Virsh command ['domblkinfo','vr1-dc0-compute-01','vda'] failed: internal
   error: missing storage backend for 'volume' storage`.
- **ROOT CAUSE, reproduced LOCALLY on the rack** (so not a snap/SSH artifact):
  `virsh domblkinfo vr1-dc0-compute-01 vda` fails identically at the console
  with an ACTIVE, autostart pool. libvirtd cannot run domblkinfo against disks
  declared as pool+volume references -- which is exactly how `modules/node-vm`
  builds them. **MAAS virsh pods are therefore incompatible with this repo's
  node-vm disk shape**; making the pod work would mean converting node-vm to
  file-path disks and re-applying all nine domains.
- **BUT THE POD IS UNNECESSARY.** Its designed job (D-103) was DISCOVERY, and
  discovery already happened via PXE enlistment. All MAAS still needs is
  POWER. MEASURED, working: per-machine `power_type=virsh` +
  `power_parameters_power_{address,id}` on the canary (`ace-oyster` =
  `vr1-dc0-control-01`, matched by MAC) -> `maas admin machine
  query-power-state` returns `{"state": "off"}`. No pod, no domblkinfo, no
  module change, no touching nine live domains.
- Roosevelt note that makes this more than expedience: on bare metal each
  machine carries its OWN power parameters (IPMI/BMC). Per-machine power is
  the SHAPE Roosevelt uses; the virsh pod is a VR1-only convenience. So the
  per-machine route is the smaller Roosevelt delta as well as the smaller
  change here.
- Both routes are recorded for the operator's ruling; nothing further applied.

## 17. STEP D COMPLETE: per-machine power shipped as tooling; nodes COMMISSIONING to Ready

- **RULING RECORD (GA-R5), 2026-07-20.** Question: pods are incompatible with
  node-vm's pool+volume disks (measured) and the pod's discovery role is
  already satisfied by PXE -- per-machine virsh power, or convert node-vm to
  file-path disks and keep the pod? Operator answer, exact utterance (option
  selected): "Per-machine virsh power (Recommended)". [Amends D-103/D-123's
  stated vm-host mechanism; amendment text queued for stage close.]
- SHIPPED `scripts/maas-node-power.sh` + `tests/maas-node-power/` (24/24 PASS
  first run; **gauntlet 71 -> 72 harnesses ALL GREEN**; repo-lint 0 fail) --
  delivered as TOOLING rather than nine hand-typed CLI calls so dc1 and any
  re-run are repeatable. Design points the harness pins:
  (i) matching is **BY MAC, never by name** -- MAAS assigns random hostnames
  at enlistment (`ace-oyster`, not `vr1-dc0-control-01`), so a name match
  would map nothing or, worse, the wrong node;
  (ii) dry by default;
  (iii) every write is verified by an actual `query-power-state` -- a stored
  power parameter proves nothing about whether MAAS can REACH the hypervisor;
  (iv) the measured pod-refutation reasoning travels in the script header.
- Prereqs measured and fixed on the region: it had **no `virsh`** (it is the
  MAAS/LXD host, not a libvirt host) -> installed `libvirt-clients`; and the
  `maas` CLI profile was root-only -> created a user profile with `maas login
  ... -` reading the key from STDIN (never printed, never in argv).
- LIVE RESULT (`docs/audit/stepD-power-20260720.txt`): 10 domains found, **9
  mapped and set, each confirmed `power state: off` by query**. The 10th
  (the edge) is correctly unmapped -- it is not a MAAS machine.
- **COMMISSIONING NOW WORKS END TO END:** all 9 commissioned; at the time of
  writing **3 Ready, 6 Commissioning**, shapes still exactly D-121 Option C.
  `Ready` means MAAS commissioned the node and it is deployable -- the whole
  point of step D. The remaining six finish unattended.
- **Revert:** `maas admin machine update <id> power_type=` (clears power) per
  machine; the script itself reverts by commit.

## 18. OPEN: 6 of 9 nodes fail commissioning ("timed out"); NOT contention; blocked
## on node observability. Step E not started.

- After power was wired, all 9 were commissioned at once: **3 reached Ready,
  6 hit MAAS's 30-minute timeout** (`Marking node failed - Node operation
  'Commissioning' timed out after 30 minutes`).
- FIRST HYPOTHESIS (resource contention -- 9 nodes x depth-4 nested, 384 GiB
  of 416 GiB, all pulling ephemeral images through one /30 transit) was
  TESTED AND REFUTED: a single failed node re-commissioned ALONE sat in
  `Loading ephemeral` for 20+ minutes with the rack at **385 GiB free and
  load 0.04**. It is not contention and not memory.
- Also RULED OUT by measurement: rack boot-image sync (`list-boot-images`
  -> `status: synced`, 10 images), and DHCP itself (these same nodes
  enlisted fine over PXE earlier, which is what proved the depth-4 gate).
- **BLOCKER IS OBSERVABILITY, and it is the same gap twice:**
  `modules/node-vm` defines **no serial console** (confirmed in the module
  and in the live domain XML -- no serial/console devices), so a node stuck
  in `Loading ephemeral` is a sealed box: no console, no agent, no way to
  see the boot. This is the identical finding already logged against
  `modules/cloudinit-vm` (item 4), now proven to bite a second module and an
  actual live diagnosis. **The fix is the same opt-in serial+log pattern
  `modules/opnsense-edge` already carries** -- that module's console log is
  what made the 2026-07-12 boot bugs legible, and it is exactly what is
  missing here.
- NOT DONE, deliberately: adding a console to node-vm means re-applying
  domains mid-diagnosis, which is a change to a built surface and belongs to
  an operator ruling, not a reflex.
- Current fleet: 3 Ready (commissioned, deployable), 6 Failed/retrying.
  Nothing is lost -- failed commissioning is re-runnable at will.
- **Step E (netem) NOT STARTED.** Measured prerequisites for when it is:
  the dc0<->dc1 mesh is **virbr5 ON VCLOUD** (note: virbr5 INSIDE the rack
  is provider-public -- same number, different host, measured not assumed);
  its qdisc is currently default (`noqueue`). D-100 rules the parameters as
  a PLACEHOLDER for the rehearsal within a "same-metro dark-fiber, low
  single-digit ms" lean. Applying to the dc0<->dc1 leg is both the
  semantically correct link for D-100 and the SAFE one -- it carries no
  traffic (dc1 does not exist), unlike the transit, which is currently
  load-bearing for MAAS/PXE. **`modules/netem-link` cannot run unattended
  from here: it shells out to `ssh <host> 'sudo tc ...'` and assumes
  PASSWORDLESS SUDO, which vcloud's operator account does not have** -- a
  documented module assumption that this host violates.

## 19. node-vm console SHIPPED; commissioning narrowed but UNRESOLVED -- and two
## of my own experiments were INVALID (disclosed)

- SHIPPED (operator-ruled): `modules/node-vm` now defines a serial console +
  boot log, with the log directory PARAMETERIZED (`serial_log_dir`) rather
  than opnsense-edge's hardcoded vcloud literal, which is wrong on any other
  libvirt host. Applied to all 9 domains: **0 add / 9 change / 0 destroy**,
  in-place, nothing destroyed. validate PASS, repo-lint 0 fail.
- **The console did NOT yield the hoped-for evidence, and that is itself a
  finding:** the logs stay 0 bytes even after a clean domain restart. Cause:
  SeaBIOS/PXE firmware writes to the VGA console, not serial, and MAAS's
  ephemeral kernel is not told `console=ttyS0`. **A serial console alone does
  not make a PXE-booting node observable** -- seeing the firmware/ephemeral
  stage needs a graphics device + `virsh screenshot` (or a MAAS-side kernel
  cmdline change). Queue this as the correction to the node-vm console work;
  do not assume the console added today closes the observability gap.

WHAT IS SOLIDLY ESTABLISHED (all measured):
- PXE works and the ephemeral handoff happens: event log shows `Performing
  PXE boot` 18:17:03 -> `Loading ephemeral` 18:17:45 -> silence -> timeout.
- The ephemeral OS BOOTS AND HAS WORKING NETWORKING: tcpdump on metal-admin
  caught 5-6 nodes holding DHCP leases (10.12.8.112-.117) and exchanging NTP
  with the rack. So node->rack reachability is fine.
- No traffic toward the REGION (10.10.0.20) or the metadata ports (5248/5240)
  was seen in the sampled windows. SEC-010 was suspected of blocking the
  node->region path, but its rules carry no `counter`, so drops cannot be
  read directly -- **hypothesis UNCONFIRMED, not refuted.**
- Ruled out by measurement: memory (385 GiB free), rack boot-image sync
  (`synced`, 10 images), DHCP (leases work), and node SHAPE (Ready and
  Failed both span 16/64, 12/48 and 8/24).

**INVALID EXPERIMENTS -- MINE, RECORDED SO NOBODY TRUSTS THEM:**
1. The first "commission a single node ALONE" test was NOT alone: tcpdump
   later proved 5-6 other nodes were still up and running ephemeral images.
   My conclusion "contention is refuted" was therefore UNSOUND. Contention
   remains a live hypothesis.
2. The second attempt was also invalid: MAAS had the machine in
   `Commissioning` already, so the new request did not restart the 30-minute
   timer (the failure at 19:24:51 belongs to the 18:54 attempt), and I then
   destroyed domains mid-commission, killing the in-flight run.
   STOPPED HERE rather than iterate again -- this is the reconciliation
   circuit-breaker: repeated cycles without convergence, and contaminated
   evidence is worse than none.

CLEANEST NEXT EXPERIMENT (not run): with ALL node domains powered off and
nothing else touching the fleet, commission EXACTLY ONE machine from a clean
`Ready`/`New` state and leave it strictly untouched for 35+ minutes. Only
that isolates contention from a genuine per-node failure. Candidate fixes to
test after it: raise MAAS `node_timeout` (nested I/O is slow and 30 min may
simply be too short), add a graphics device for `virsh screenshot`, add
`counter` to the SEC-010 rules so the node->region hypothesis becomes
readable, and check the kernel cmdline MAAS actually serves.

## 20. CLEAN commissioning experiment: contention REFUTED (properly this time)

- Ran the isolated experiment the earlier invalid ones only pretended to:
  ALL node domains OFF (verified 0 running), 397 GiB free, load 0.05, canary
  in a TERMINAL `Failed commissioning` state (so the commission restarts
  MAAS's 30-min timer), commission EXACTLY ONE, poll READ-ONLY only, touch
  nothing.
- RESULT: `Failed commissioning after 1770s` (~29.5 min = the 30-min
  timeout), a SINGLE node entirely alone. **Contention is refuted** -- it is
  not I/O bandwidth, not memory, not the metadata service being swamped. The
  failure is systemic and per-node.
- This CORRECTS changelog item 18-19's contention hypothesis (which rested on
  contaminated runs). The event trace is consistent every time: `Commissioning`
  -> `Performing PXE boot` -> `Loading ephemeral` (~40s later) -> 30 min of
  silence -> timeout. The node gets the ephemeral image handoff and then never
  reports back.
- REFRAME after further measurement: "contention refuted" is right but too
  narrow. The single-node fail refutes CONTENTION (competing for resources)
  but is fully CONSISTENT with INHERENTLY SLOW -- the node may have still been
  working at 30 min, just not done. And the batch result **3 pass / 6 fail
  exactly at the 30-min mark** is the classic signature of a MARGINAL TIMEOUT
  over a slow, variable operation: the faster-scheduled nodes finish under the
  wire, the rest are still going when the timer fires. 1770s is 29.5 of 30.
- KEY CLUE against a network/metadata block: **3 nodes DID reach Ready on this
  exact rack, subnet, and metadata proxy.** So the metadata path is NOT
  globally broken; a "node never reaches :5248" theory has to explain why it
  works for 3 identical nodes. Infra confirmed UP: rack `:5248` listening (2
  sockets), rack->region 5240 = 301. A `:5248` tcpdump returned 0 packets but
  its 220s window likely closed before the ephemeral OS reached the
  metadata-fetch stage -- INCONCLUSIVE (a timing race), not evidence.
- **LEADING HYPOTHESIS (operator-facing): MAAS's default 30-min commissioning
  timeout is too short for depth-4 nested commissioning I/O.** Cheap, decisive
  test that needs no stall-pinpointing: raise `node_timeout`
  (`maas admin maas set-config name=node_timeout value=<min>`, e.g. 60),
  commission ONE node, see if it completes. If it does, the fix is a config
  value (and a Roosevelt note: bare metal commissions faster, so this is a
  VR1-nested tuning, not a Roosevelt default). If it still fails well past
  30 min, the timeout is exonerated and the next probe is the metadata-proxy
  access log / a console screenshot (needs the graphics-device correction to
  the node-vm console -- item 19).
- STOPPED the live diagnosis here per the reconciliation circuit-breaker:
  many diagnostic cycles, and the remaining hypotheses need an operator
  decision (change a MAAS-wide config) rather than more agent spelunking.

## 8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review

- Operator directive (verbatim intent): at the proper time/place, review the
  OPNsense 26.7 changelog + documentation for upgraded features and changes;
  incorporate beneficial improvements (esp. hardening) into the next
  deployment iteration, following project rules.
- Scheduled: STAGE-3 CLOSE (GA-R6 close-out pass, with this stage's queued
  DOCFIXes). Deliverable: a findings doc citing the official 26.7 release
  notes; actionable items route per GA-R3 -- hardening/profile deltas as
  PROPOSED items anchored on D-129 (its 4 open sub-decisions are the likely
  surface), operational items as runbook DOCFIXes. NO self-executing
  incorporation: each finding is ruled per GA-R5 (one per exchange) before
  touching any built surface. Target of incorporation: dc1 standup +
  Roosevelt runbooks; live-edge retrofits individually gated.

## 6. ledger-scan hardening: FAIL-OPEN is terminology, not a status

- SEC-010's close surfaced a scanner false positive: the OPEN-row detector
  (`awk toupper(cell) ~ /OPEN|PENDING/`) matched the substring OPEN inside
  "fail-open" in the closed row's disposition -- the row counted OPEN
  forever (same class as the GA-F15 wrap trap). Fix: strip FAIL-OPEN from
  the cell before matching (`scripts/ledger-scan.sh`); harness gains a
  CLOSED-row-with-fail-open fixture + nochk (45/45 PASS). Count re-verified:
  7 open, SEC-010 absent.
- **Revert:** git revert of this commit's ledger-scan.sh + test hunks.

## 5. Host tooling: libguestfs-tools installed on vcloud (operator-run)

- `sudo apt-get install -y libguestfs-tools` (1:1.52.0-5ubuntu3) -- enables
  offline guest inspection (virt-cat/virt-ls), used for items 4's captures.
  Standard KVM-host tooling; Roosevelt-transferable.
- **Revert:** `sudo apt-get remove libguestfs-tools`.
