modules/node-vm: add serial console + boot log (parameterized dir) ...

Its absence blocked a live diagnosis: 6/9 dc0 nodes stall in MAAS 'Loading
ephemeral' and, with no console and no agent, a stuck node is unobservable.
Same pattern opnsense-edge already carries, for the same reason -- a
PXE-booting node emits everything interesting before anything sshable
exists. Unconditional (MAAS-managed nodes are rebuilt at will; no
live-router bounce risk like the D-129 qga channel). Log dir is a variable
rather than opnsense-edge's hardcoded vcloud literal, which is wrong on any
other libvirt host -- backport queued.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

OPEN: 6/9 nodes fail commissioning; contention refuted; blocked on node-vm having no console ...

3 Ready, 6 timed out at 30 min. Hypotheses tested and REFUTED by
measurement: contention (single node alone stuck in Loading ephemeral 20+
min with 385GiB free, load 0.04), rack boot-image sync (status synced, 10
images), DHCP (same nodes enlisted fine earlier). Real blocker is
observability: modules/node-vm defines no serial console, so a stuck node
is a sealed box -- the same gap already logged for cloudinit-vm, now biting
a live diagnosis. Fix is the opt-in serial+log pattern opnsense-edge
already carries; NOT applied, since it re-applies domains mid-diagnosis and
is an operator ruling. Step E prerequisites measured and recorded (dc0-dc1
mesh = virbr5 ON VCLOUD; netem-link assumes passwordless sudo vcloud lacks).
CURRENT-STATE updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

STEP D COMPLETE: per-machine power tooling shipped; commissioning works, nodes reaching Ready ...

Operator-ruled per-machine virsh power. maas-node-power.sh + harness (24/24,
gauntlet 72 ALL GREEN): MAC-matched because MAAS renames machines at
enlistment, dry-by-default, every write verified by a real query-power-state.
Region prereqs fixed (no virsh there; maas profile was root-only -- created
via stdin login, key never printed). All 9 nodes powered and commissioning
end-to-end: 3 Ready / 6 Commissioning, shapes exact to D-121 Option C.
vr1-dc0-maas root retained but unused pending the stage-close D-103/D-123
amendment. G10 remainder: netem only.
CURRENT-STATE updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

NEW: maas-node-power.sh (+harness 24/24) -- per-machine virsh power, MAC-matched ...

Operator-ruled 2026-07-20 over the pod route. Sets power_type=virsh +
power_address/power_id on already-enlisted machines, matching domains BY MAC
(MAAS renames machines at enlistment, so name matching would map nothing or
the wrong node). Dry by default; every write verified by an actual
query-power-state, since a stored parameter proves nothing about reachability.
Carries the measured reasoning: pods fail on node-vm's pool+volume disks
(domblkinfo 'missing storage backend', reproduced locally), the pod's
discovery role was already done by PXE, and per-node power is the Roosevelt
(IPMI) shape. Gauntlet 72 harnesses ALL GREEN.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

CURRENT-STATE: record step-D pod findings + pending ruling (clears L10 on prior commit) ...

Prior commit landed an audit capture without the same-commit CURRENT-STATE
update that GA-R1/C1 requires; L10 caught it. Position now records: pods
incompatible with node-vm volume-ref disks (reproduced locally), pod
unnecessary since PXE already discovered the nodes, per-machine virsh power
measured working, ruling pending.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Pod path chased to root cause: MAAS virsh pods incompatible with node-vm volume-ref disks; per-machine power PROVEN working ...

Dedicated MAAS->libvirt key minted+wired (operator-ruled). Two further
measured findings: (1) MAAS dials the pod from the REGION, not the rack, so
the credential belongs where MAAS dials from; (2) each 503 created a broken
pod server-side outside tofu state (both deleted). Final blocker: domblkinfo
fails on pool+volume disk refs -- reproduced LOCALLY on the rack with an
active pool, so it is libvirtd, not snap/SSH. Making pods work would mean
converting node-vm to file-path disks + re-applying 9 domains. Not needed:
the pod's job was discovery, which PXE already did, and per-machine
power_type=virsh query-power-state returns state=off on the canary. Per
machine power is also the Roosevelt shape (IPMI per node). Awaiting ruling.
CURRENT-STATE untouched pending the ruling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Step D part 2: MAAS root shipped; apply refutes D-123's local qemu:///system pod mechanism ...

New opentofu/vr1-dc0-maas root (isolates MAAS creds from substrate roots
per DOCFIX-179's lesson); plan clean, apply failed 'Failed to login to
virsh console'. MEASURED root cause: confined MAAS snap gets Permission
denied on libvirt-sock and ships NO libvirt interface to connect -- a local
qemu:///system pod is architecturally impossible with snap MAAS. D-123
Model B and modules/maas-vm-host both state that mechanism; intent (no
cross-fiber dial) survives, mechanism does not. Amendment pending a ruling.
qemu+ssh replacement measured feasible (snap ships ssh; rack user in
libvirt group). Nothing was created by the failed apply.
CURRENT-STATE updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

NEW root opentofu/vr1-dc0-maas: MAAS VM-host registration, isolated from substrate roots ...

Third root by design, applying DOCFIX-179's own lesson rather than
repeating it: a provider 'maas' block forces EVERY plan in its root to
demand maas_api_url + the sensitive key, so coupling it to the substrate
roots would make routine substrate plans require MAAS creds. This root
touches MAAS only; the substrate roots stay credential-free. Registers
the pod via modules/maas-vm-host (discover, never compose -- D-103) with
power_address local to the rack per Model B. Validated explicitly (the
validator covers only the outer root + modules -- extending it to all
roots stays queued).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

INCIDENT RESOLVED + G10 depth-4 boot gate PASS: all 9 DC0 nodes enlisted in MAAS ...

Region restart recovered Temporal; dhcpd verified by PROCESS on both
controllers (0->1 each), not by service_set. One restart fixed both sites,
confirming the single wedged-Temporal root cause. The identical canary that
had failed then enlisted immediately -- proof it was the real blocker.
All 9 nodes enlisted with shapes exactly matching D-121 Option C
(3x16/64, 2x12/48, 4x8/24): node VMs inside vvr1-dc0 PXE-booted from the
Office1 region across the transit and run the ephemeral kernel = real
nested KVM at depth 4, proven behaviorally. Appendix-A entry queued for
this new incident class.
CURRENT-STATE updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Step D part 1 done (metal-admin DHCP configured); INCIDENT: MAAS DHCP down region-wide (Temporal wedged) ...

Rack registered + ruled dynamic range 10.12.8.100-.200 + VLAN 5005
dhcp_on/primary_rack verified by read-back. Canary node did NOT enlist:
measured dhcpd off, no process, no generated config on the rack while the
API reports dhcp_on=True -- config and running state disagree. Region
journal shows Temporal wedged ('Not enough hosts to serve the request',
2807 retries), and MAAS 3.7 drives DHCP through Temporal workflows.
Scope is wider than this deploy: voffice1 has NO dhcpd process either, so
Office1's own DHCP has been down since ~the 2026-07-17 reboot, masked by
both service VMs already being Deployed. Remedy proposed, NOT run (gated).
Detection gap queued: cloud-assert trusts MAAS's self-report.
CURRENT-STATE updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

D-129 edge profile APPLIED on 26.7: qga channel shipped, plugins installed, agent ANSWERS guest-ping ...

expose_qga_channel (opt-in, default OFF; dc0 true) applied as an in-place
domain update -- disk with the console/API bootstrap never at risk. Real
install (status ok + msg_uuid, not dry-run echoes); service started via the
plugin's own API; gate met to the runbook's bar: guest-ping -> {'return':{}}
and domifaddr --source agent reports both legs. Prior capture of the same
name recorded the false dry-run and is replaced. Security finding queued:
the plugin can disable guest-exec RPCs (remote command execution) -- propose
at the 26.7 hardening review.
CURRENT-STATE updated in this commit (GA-R1/C1, L10 satisfied).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

FIX false-success: opnsense-plugins.sh apply ALWAYS dry-ran (DRY_RUN:+ expands on 0) ...

Found on the first-ever live run (vr1-dc0 edge, 2026-07-20): ${DRY_RUN:+--dry-run}
expands whenever the var is set and NON-EMPTY, and DRY_RUN=0 is non-empty -- so
every apply silently dry-ran while printing 'OK: profile applied'. Nothing was
ever installed by this script. The harness missed it because every test drove
--dry-run, leaving the live path with zero coverage. Fix tests the VALUE; harness
gains two live-path tests via a stub api client (verified to FAIL 2/2 against the
buggy version before restoring). 20/20.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

modules/opnsense-edge: opt-in qga virtio channel (D-129); dc0 edge sets it true ...

Shape taken from the provider schema (devices.channels[].target.virt_io /
source.unix on dmacvicar 0.9.8), not inferred. Default OFF so existing
edges are never bounced by a plan; the new dc0 edge boots with it per the
runbook's no-retrofit intent for NEW DC edges. Without the channel the
os-qemu-guest-agent package is inert-but-present -- the false success the
D-129 review flagged -- so package and channel ship as one decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

DC edge ADDRESSED: WAN 172.30.2.2/24 + default gw, LAN -> 10.12.4.1/22; edge egresses 0% loss ...

Applied via the new operator-ruled v4 setter pair, WAN first per its own
ordering warning. Kernel-verified both legs; 3 interfaces intact; API
answers at the new LAN address (CORE_ABI 26.7). The edge had shipped with
WAN on dhcp -- unworkable on a /24 with no DHCP server (same fact behind
the earlier probe artifact). Interim bootstrap address removed.
CURRENT-STATE position + G10 row updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

NEW: opnsense-set-interface-v4 pair (+harness 53/53) -- IPv4 + WAN gateway via the vendor config model ...

Operator-ruled mechanism 2026-07-20 ('Build the v4 setter pair'). Mirrors
the proven v6 sibling: vendor Config singleton (never a config.xml push),
dry-by-default, interface-count guard, fresh read-back verify, sh -s for
the tcsh trap, kernel ground truth. Adds the gateway half (at most ONE
gateway_item, count-guarded) and MEASURES the interface's device from the
edge's own config instead of hardcoding lan->vtnet0 like the v6 sibling.
D-113 amendment RE-MEASURED on 26.7: interfaces/settings/get carries only
offload flags; interfaces/overview/list and interfaces/<if>/get both 404 --
so the amendment holds on two majors. Gauntlet 71 harnesses ALL GREEN.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

D-125 egress isolation gate: PASS / CLOSED -- bridge-in proven end to end (2 identical runs) ...

Measurement disproved the missing-rule hypothesis first: operator-run nft
dumps show virbr4 structurally identical to the working virbr11, with all
three masquerade rules present INCLUDING the generic one, already fired --
so the pre-staged net-destroy/net-start was correctly not run. Matrix
re-probe: gateway ping 0, internet ping 0, curl 1.1.1.1 301, curl
archive.ubuntu.com 200, twice. Run-2's one-off ICMP failure recorded
UNEXPLAINED rather than swept. Side observation kept: the WAN segment
cannot reach the Office1 LAN through the ISP NAT (D-122/SEC-010 intent
holding) -- queued as an explicit hardening assertion.
CURRENT-STATE position + G10 row updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

CORRECTION (GA-R1/C2): D-125 rack-bridge follow-up was CONFOUNDED, not evidence ...

The rack has no default route (transit-only netplan; WAN bridge IP-less),
so 172.30.2.51-sourced traffic to 1.1.1.1 fails in the rack's own routing
table before the NAT is consulted. Its ICMP/TCP failures say nothing about
virbr4; the only clean signal is the guest's ICMP failure, and a guest TCP
datapoint is still missing. Measurement request widened accordingly: FULL
nft AND iptables-save dumps (backend unmeasured), diffing virbr4 vs
virbr11 across BOTH rule families -- POSTROUTING masquerade (subnet-keyed)
and FORWARD accept (interface-keyed, which a subnet grep would miss).
Fix-forward pre-staged: net-destroy/net-start reinstalls libvirt's rules.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

D-125 egress gate executed (throwaway VM, as ruled): bridge-in PROVEN, upstream NAT egress fails -- gate OPEN pending one sudo measurement ...

Run 1 was a test artifact (probe used DHCP; uplink /24 has no <dhcp> by
design -- DOCFIX queued for the gate wording). Run 2 static: gateway
ping 3/3 across the nested bridge = the D-125 property PASSES; 1.1.1.1
fails for ICMP and TCP alike. Controls pass and net-dumpxml shows the
working/failing NATs are substantively identical, so the suspect is
vcloud's live virbr4 rules, not the design -- double-NAT fallback
deliberately NOT invoked. Probe guest + temp addresses removed.
CURRENT-STATE updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

DC edge bootstrap: D-112(c) console chain complete + API key minted; D-113(a2) PROVEN on 26.7 ...

Key-only root SSH to vr1-dc0-opnsense; GET core/firmware/status 200
(CORE_ABI 26.7) -- first 26.7 revalidation datapoint for the API path.
Five measured console-bootstrap findings recorded (base64 chunking for
serial wrap; tcsh quoting; measured shell_safe include; local_user_set
required to materialize authorized_keys). Rack legs 10.12.4.2/10.12.8.2
added interim; script support queued as a finding.
CURRENT-STATE position updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Permissions: targeted ask rules for scp / pipe-to-ssh / ssh-virsh (classifier walls -> gates, per 2026-07-19 ruling) ...

Deploy-window surfaces that hard-denied and stalled work: scp staging to
the dc0 rack, cat|ssh file staging, nested virsh over ssh. Per the ruled
convention these become ASK (approval prompts, Remote-Control friendly),
never allow -- hard rule 3 mutation gating is unchanged, and read-only
ssh probes stay frictionless.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Stage-close queue: operator-directed OPNsense 26.7 changelog/docs review (findings->proposals->rulings; D-129 anchor) ...

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Deploy step C COMPLETE: inner apply 28/28 from voffice1, converged zero diff; 9 nodes + 26.7 edge RUNNING at depth-4; three first-real-apply fixes ...

voffice1 = Plane-2 executor (tofu 1.12.4, repo @29cf7bf, dc0 key,
keyscan'd known_hosts). Fixed forward, measured at first real use:
provider keyfile/sshauth URI params; wan-bridge illegal <mtu> in
bridge mode; base image uploads from the EXECUTING host (copied to
voffice1, description corrected); staging-dir literal created on
rack. Captures: inner-{plan,apply,converge}-20260720-stepC*.
Edge serial log at FreeBSD login prompt (26.7 D-112 first datapoint).
CURRENT-STATE position updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

wan-bridge: drop illegal <mtu> on bridge-mode network (measured at first real apply; MTU is the host bridge's) ...

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Inner root: provider keyfile+sshauth URI params (measured: provider ssh ignores ssh config/default identities); known_hosts multi-type note ...

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Deploy step B COMPLETE: rack bootstrapped + enrolled, SEC-010 CLOSED both ends (operator-ruled), OPNsense 26.7 base staged (operator-ruled); ledger-scan fail-open hardening ...

Bootstrap runs 1+2 (stale-lists 404 fixed by proxied apt update):
maas 3.7/stable, rack enrolled, libvirt+nested KVM, SEC-010 loaded
+ persistent on enp1s0; final --check EXIT 0 ALL items
(docs/audit/stepB-check-20260720-final.txt). voffice1-side SEC-010
pin applied per the row's follow-on clause. 26.7 nano prepped on the
rack (measured leaseweb mirror via region proxy); inner-root base
default -> 26.7. ledger-scan: FAIL-OPEN no longer counts as OPEN
(harness 45/45). CURRENT-STATE position/G10/G14 + section-5 updated
in this commit (GA-R1/C1). Open SEC count 8 -> 7.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Step B transit reach ESTABLISHED: voffice1 .1/30 applied (Kea re-key incident fixed); vvr1-dc0 netplan set-name root cause fixed; zero-diff converged ...

voffice1 transit NIC applied 1/1/1 exact; MAC-regen broke the Kea
reservation (measured .108) -- re-keyed via edge API, .20 restored.
vvr1-dc0 both-NICs-silent root cause: netplan match.name+set-name
(one-boot rename lag, measured offline via libguestfs); fix drops
set-name, keeps measured enp1s0/enp2s0, instance-id bump, seed
-replace 2/0/2 exact. Verified: ssh -J voffice1 w/ dc0 key,
rack->region 0% loss. Outer plan ZERO DIFF converged.
CURRENT-STATE sections 3+5 updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Step B prereq: voffice1 transit NIC authored (172.31.0.1/30 region end, D-124); ruling recorded; expected plan 1/1/1 ...

Measured: transit net isolated, only vvr1-dc0 attached; nothing held .1.
Operator-ruled mechanism 2026-07-19: voffice1 via IaC + restart.
Capture: docs/audit/outer-plan-20260719-voffice1-transit.txt (1/1/1).
CURRENT-STATE sections 3+5 updated in this commit (GA-R1/C1).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Deploy paused at A/B boundary; fresh-session handoff note (step B facts measured) ...

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi

DC0 deploy STEP A EXECUTED: outer apply 6/0/6 exact; G8+G9 CLOSED; convergence zero ...

Logged window dc0-deploy (as-executed log + index convention). G8 passed
in the apply session (six planes 0 leases, 0 attachments); validate
PASS; pre-apply capture equaled the recorded triple; the SAVED plan was
applied: 6 added / 0 changed / 6 destroyed. vvr1-dc0 RUNNING (108 vCPU,
416 GiB, autostart disable per D-127); vr1-dc0-uplink active; six vcloud
planes destroyed (Model B relocation); voffice1 + edge untouched (ids
1/2). Convergence re-plan: no differences (captured). CURRENT-STATE:
section 1 position, section 5 expected plan = ZERO DIFF (any diff is a
STOP), G8 + G9 rows CLOSED; active gate G10. Captures committed same
commit (C1/C2).

Revert: infra revert = runbooks/dc-dc-teardown-rollback.md decision tree
(NOT a blind destroy); docs revert = git revert this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi

AUDIT EXIT: operator re-signed CURRENT-STATE; G3+G11 CLOSED, G9 OPEN; session close (GA-R4) ...

Batch 6 item 6: signature utterance "Reviewed, approved, continue."
recorded into section 11 (replaces 2026-07-18 per GA-R1 rule 7). All six
exit runs PASS. G3 CLOSED (all six batches); G9 OPEN (audit criteria
MET; remaining closer = the operator-gated logged apply with G8 first);
FREEZE LIFTED. Session GA-R4 close: bounded 14-line summary in the
ledger, full body rotated to docs/archive/session-20260719-phase5-sweep.md.
Repo HEAD at signing = this commit's parent (437d9d1).

Revert: git revert this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi