Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260719-dc0-deploy-stepB.md
@JANeumatrix JANeumatrix 1 day ago 26 KB DC edge ADDRESSED: WAN 172.30.2.2/24 + default gw, LAN -> 10.12.4.1/22; edge egresses 0% loss
Raw Blame History

Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap)

Session changelog (GA-R2/D1: ONE per session). Logged window: dc0-deploy (same-day append, per the A/B-boundary handoff). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts.

1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124)

  • Question as presented (2026-07-19): "Who should hold 172.31.0.1 (the Office1 end of the dc0 transit /30), and by what mechanism? This is the prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS rack-to-region enrollment path."
  • Operator answer, exact utterance (option selected): "voffice1 via IaC + restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to module "voffice1" in the outer root; apply in a planned voffice1 restart window (graceful shutdown first; region/NetBox/Tailscale return via proven autostart).
  • Classification: OPS mechanism decision implementing D-124's already-ruled addressing ("region/peer .1" -- docs/design-decisions.md:3727); no new D-number (GA-R3, doubt resolves DOWN).
  • Measured basis (this session, logged): mesh-vr1-dc0-office1 (virbr7) is ISOLATED (no <forward>/<ip>) with exactly ONE attached interface (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only route 10.10.0.0/22 via 172.31.0.1 (d124-rack.auto.tfvars, consumed by step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached by -J voffice1 (D-126 shape) -- NO vcloud host leg / site-baseleg DC row needed for this net.

2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated)

  • What: opentofu/main.tf module "voffice1": network_names gains module.mesh_vr1_dc0_office1.network_name SECOND (office1-local stays first -- PCI order preserves enp1s0); network_config rewritten from the single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4; transit=enp2s0 static ${vr1_dc0_rack_transit_peer_ip}/${prefix} -- same D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init- re-run caveats documented in-block: the seed netplan is REBUILD correctness; the live transit config is an in-guest netplan drop-in at the gated attach step.
  • Why: section 1's measured gap -- step B (ssh reach + rack enrollment) has no L3 path until the region end of the transit exists.
  • Revert: restore network_names = [module.office1_network.network_name] and the previous "en*" glob network_config (git revert of this commit); if already applied, a follow-up plan shows the interface removal and the in-guest drop-in /etc/netplan/60-transit.yaml is deleted.

3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation

  • Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply (1/1/1 exact, docs/audit/outer-apply-20260719-voffice1-transit.txt) -> domain up with both NICs, office1-local index 0.
  • INCIDENT (found+fixed in-window): the provider REGENERATED both NIC MACs on the interface-list rebuild (old lan MAC 52:54:00:6a:87:e5 -> new 52:54:00:89:e1:19). The Kea reservation (MAC-keyed) missed and voffice1 leased pool address .108 (measured via edge ARP). FIX: re-keyed reservation uuid b12621e6-63e9-4714-9726-2e130c84e069 to the new MAC via the D-113 API (kea/dhcpv4/set_reservation -> saved; kea/service/reconfigure -> ok), then a detached networkctl reconfigure enp1s0 -> voffice1 back on 10.10.0.20.
  • LOGGED FINDING (not executed): modules/cloudinit-vm does not pin MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed DHCP reservations. Roosevelt-relevant; candidate module mac var or a documented ops step. Also: live transit config landed as in-guest netplan drop-in /etc/netplan/60-transit.yaml (narrows the lan glob to enp1s0 + transit enp2s0 static) -- the drop-in must ALSO narrow lan, else the 50-cloud-init en* glob claims enp2s0 (measured: first drop-in attempt without it left enp2s0 addressless).
  • Revert: kea/dhcpv4/set_reservation back to the old MAC (senseless unless the NIC change is also reverted); delete the drop-in.

4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix

  • Symptom: 172.31.0.2 unreachable; MEASURED: both rack NICs emitted ZERO frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive (ACPI-responsive; cloud-init completed per offline logs).
  • Guest is a SEALED BOX: modules/cloudinit-vm domains have NO serial console, NO qga channel, NO graphics; qga is NOT in the noble base image and its cloud-init install failed (no egress) -- so no console, no agent, no network. LOGGED FINDING: add opt-in serial console (+ the D-129-style qga channel var) to modules/cloudinit-vm -- observability gap bites every future DC VM. NOT executed mid-step.
  • Offline inspection (libguestfs, operator-run, captures ~/vvr1-dc0-offline-inspect{,2}.txt): NIC names enp1s0/enp2s0 CORRECT (naming trap closed); root cause = netplan match.name + set-name: boot 1 renames don't apply (.link written after udev add) so the rendered config matched nothing; after removing set-name, boot 2 STILL failed because the early-boot netplan generator ran against the OLD config and renamed the devices to mgmt/uplink before cloud-init wrote the new file (one-boot lag, measured in syslog). Boot 3 (current config, no renames) is the verification boot.
  • FIX (this commit + applied): network_config drops set-name, keeps MEASURED kernel names; instance-id bumped vvr1-dc0-d123 -> -d123-r2 (side effects noted in-block); seed replaced via explicit -replace=module.vvr1_dc0.libvirt_volume.seed (the sanctioned override of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2 exact (docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt).
  • CONSEQUENCE for step B: bootstrap runs with --transit-if enp1s0 --uplink-if enp2s0 (script defaults mgmt/uplink no longer match reality); runbook prose naming "mgmt" needs a DOCFIX at stage close.
  • Revert: git revert of the netplan/instance-id edit + one more seed -replace apply + reboot.

4b. VERIFICATION (2026-07-20, logged window)

  • Boot 3 with the fixed config: enp1s0 UP 172.31.0.2/30, region route present, br-vr1-dc0-wan up.
  • voffice1 -> 172.31.0.2 ping 0% loss; ssh -i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2 works (D-126 -J shape, per-env key); rack -> region ping 10.10.0.20 0% loss (the MAAS-enrollment path for the G10 bootstrap).
  • Outer plan CONVERGED back to ZERO DIFF (docs/audit/outer-plan-20260720-postfix-converged.txt); CURRENT-STATE section 5 updated in the same commit.
  • Follow-items (logged, not executed): D-126 DC ssh-config alias row for vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key); cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC pinning; DOCFIX for runbook "mgmt/uplink" naming.

4c. Step B bootstrap EXECUTED on vvr1-dc0 (2026-07-20, logged window)

  • Snap proxy set to the region maas-proxy (measured serving: HTTP 200 to archive.ubuntu.com, CONNECT to api.snapcraft.io) -- the D-107-shaped answer to the rack's no-egress bootstrap; apt used the same proxy via per-invocation env. Enrollment secret piped region->rack (install -m600 /dev/stdin), verified by byte-count only (32), never in context.
  • Run 1 (docs/audit/stepB-bootstrap-20260720.txt): maas snap 3.7/stable installed, rack ENROLLED to http://10.10.0.20:5240/MAAS; FAILED at qemu apt install on stale image package lists (404s via proxy). Fix: apt-get update (proxied), re-run.
  • Run 2 (docs/audit/stepB-bootstrap-20260720-run2.txt): EXIT 0 -- libvirt/qemu + nested KVM on, inner pool + AppArmor, SEC-010 FORWARD-drop loaded on enp1s0 + sec010-fw.service enabled (boot-persistent), br-vr1-dc0-wan verified with enp2s0 enslaved.
  • --check (docs/audit/stepB-check-20260720.txt): all items ok EXCEPT opnsense base (the remaining step-B item); rackd running, enrolled.
  • Region-side residue for the MAAS phase (per script exit text): enable DHCP on metal-admin at the Office1 region naming this rack primary_rack (step D territory, NOT done).
  • Revert: rack enrollment: region-side maas ... rack-controllers delete; SEC-010: disable sec010-fw.service + rm /etc/nftables-sec010.nft; packages: apt/snap remove; snap proxy: snap unset system proxy.http proxy.https.

4d. RULING RECORD (GA-R5) -- DC edge base version (2026-07-20)

  • Question as presented: reuse verified 26.1 artifact vs fresh download (first exchange); then, sharpened after the operator noted "Office1 is running OPNsense 26.7": 26.1-base-then-upgrade vs fresh 26.7 nano base.
  • Operator answer, exact utterance (option selected): "Fresh 26.7 nano base".
  • Consequences implemented: prep runs 26.7 -> opnsense-26.7-nano.qcow2; vr1-dc0-substrate opnsense_base_path default updated (no inner state exists; pre-apply edit); step-B --check passes --opnsense-base for the 26.7 path. LOGGED follow-items: site-headend-install.sh's 26.1 default + hint strings (script change + harness, stage close); the D-112/D-113 serial/API boot path is REVALIDATED on 26.7 at the step-C boot gate (office1 precedent: 26.1-proven, upgraded in place to 26.7).
  • Mirror: https://mirror.ams1.nl.leaseweb.net/opnsense -- MEASURED from the vendor's live download page this session (not memory), HEAD-verified serving the 26.7 nano (~468 MiB). LOGGED FINDING: opnsense-prep-image.sh performs no checksum verification of the downloaded image (same property as the original 26.1 prep; candidate hardening).

4e. Step B CLOSED: 26.7 base staged; final check EXIT 0; SEC-010 CLOSED both ends

  • OPNsense 26.7 nano prepped ON the rack (leaseweb mirror via region proxy, 468M; python3 bz2 fallback -- bunzip2 never needed, DOCFIX-184; docs/audit/stepB-opnsense-prep-20260720.txt). Inner-root opnsense_base_path default -> 26.7 (validate PASS).
  • Final --check EXIT 0, ALL items ok (docs/audit/stepB-check-20260720-final.txt).
  • SEC-010 voffice1-side pin (the row's "same pin follows" clause, due since this session wired voffice1's transit leg; voffice1 runs LXD so ip_forward=1): identical scoped nft artifact + sec010-fw.service on enp2s0, verified loaded + persistent; rack proxy/region paths re-verified working post-pin. SEC-010 row CLOSED (operator ruling, utterance in the row). Open SEC count 8 -> 7.
  • LOGGED follow-item: site-headend-install.sh region-role support for the transit-if pin (voffice1's was hand-applied to the script's exact artifact shape; fold into the script + harness at stage close).
  • Revert: rm the two files + systemctl disable sec010-fw on voffice1; nft delete table inet sec010; ledger row reopens citing this item.

7. Step C: voffice1 becomes the Plane-2 executor; INNER APPLY COMPLETE (28/28)

  • voffice1 setup (gated, logged): tofu 1.12.4 via vcloud's exact apt source (keyrings byte-copied; version == section-7 pin); repo cloned at 29cf7bf (branch); dc0 private key staged to ~/vr1-dc0-creds/ (0700/0600, piped -- SEC-009 convention now spans hosts); ssh config entry for 172.31.0.2; known_hosts seeded via ssh-keyscan with the ED25519 fingerprint cross-checked against the accept-new-trusted entry (MATCH).
  • Inner-root fixes measured at the FIRST REAL dial/apply (each committed): (i) provider needs keyfile+sshauth=privkey URI params -- its Go ssh ignores ~/.ssh/config AND default identities; single-type known_hosts fails as "key mismatch" (multi-type scan required); (ii) modules/wan-bridge: <mtu> is ILLEGAL in bridge-mode networks -- removed (MTU belongs to the host bridge; DOCFIX-194 class: parsed, never applied); (iii) base-image location premise WRONG under remote provider -- content UPLOADS from the executing host; 26.7 qcow2 copied rack->voffice1 (~/vr1-dc0-images/), tfvar override + variables.tf description corrected; (iv) edge serial-log dir /var/lib/libvirt/vr1/staging/ is a HARDCODED vcloud literal in modules/opnsense-edge (:223) -- dir created on the rack (755); LOGGED findings: parameterize the path; fold staging-dir creation into site-headend-install --host-nodes; opentofu-validate.sh does not cover the inner root.
  • Applies (captures docs/audit/inner-{plan,apply}-20260720-stepC*.txt): run 1 = 25/28 (nodes+planes+pool; wan-bridge mtu abort); run 2 = wan bridge (edge disk stat abort); run 3 = edge disk (staging-dir abort; provider self-cleaned the failed domain -- operator undefine found nothing); run 4 = edge domain, EXIT 0. CONVERGED: inner plan zero diff (docs/audit/inner-converge-20260720-stepC.txt).
  • VERIFIED: 10/10 domains RUNNING inside vvr1-dc0 (9 nodes depth-4 + edge); edge serial log = FreeBSD login prompt on ttyu0 -- 26.7 boots the D-112 serial path (first revalidation datapoint; full D-112(c) bootstrap is the next window).
  • NEW STATE OF RECORD: the inner tfstate lives on voffice1 (~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/terraform.tfstate) -- back it up with the site; follow-item for the backup set.
  • Revert: tofu destroy in the inner root from voffice1 (site-down alternative: virsh destroy vvr1-dc0, D-122); module fixes revert by commit.

9. DC edge: D-112(c) console bootstrap + API key minted (26.7) -- REPLACEMENT CHAIN PROVEN

  • Per-env edge key minted on vcloud (~/vr1-dc0-creds/vr1-dc0-edge_ed25519, 0600/0644, SEC-009 convention), staged to the rack the same way.
  • Rack legs added (operator-ruled "10.12.4.2, manual now + script later"): 10.12.4.2/22 on the provider-public inner bridge (virbr5, MEASURED) + the D-124-ruled 10.12.8.2/22 on metal-admin (virbr2). INTERIM: ip addr add, not persistent. LOGGED FINDING (queued): site-headend-install.sh never implemented the rack legs despite main.tf's comment claiming it consumes vr1_dc0_rack_metal_admin_ip -- add rack-leg support + harness before step D closes. Also interim: 192.168.1.2/24 on virbr5 to reach the edge's FACTORY LAN; REMOVE once the edge is re-addressed.
  • D-112(c) console bootstrap: DONE (~/d112c-console.log on the rack). Driver evolved v1->v6 against MEASURED failures, each real and worth keeping for the runbook: (i) serial console wraps long lines -> ship payload as <=160-char base64 chunks (openssl base64 -d), never one long line; (ii) set-name in the edge netplan is irrelevant here but the same class bit the rack (item 4); (iii) tcsh on the edge (documented trap) breaks nested-quoted sed -- "Unmatched '"'"'" -- so NEVER patch files via quoted one-liners; re-ship the whole payload instead; (iv) write_config() needs an include beyond config.inc -- MEASURED (grep -l shell_safe /usr/local/etc/inc/*.inc -> auth/console/filter/ interfaces/rrd/system/util.inc), not guessed; (v) writing config.xml does NOT materialize /root/.ssh/authorized_keys -- must call the vendor's local_user_set() (auth.inc). This was the actual reason key auth failed after a "successful" config write.
  • VERIFIED: ssh -i vr1-dc0-edge_ed25519 root@192.168.1.1 -> key-only auth OK, uname -r = 15.1-RELEASE-p1, ifconfig -l = vtnet0/vtnet1. INTERFACE MAPPING MEASURED: vtnet0 = LAN = provider-public plane (it answered on virbr5), vtnet1 = WAN -- same order as the office1 edge.
  • API key minted via the vendor model (opnsense-bootstrap-apikey.sh
    • opnsense-mint-apikey.php, run ON the rack): key/secret 80 chars -> ~/vr1-dc0-creds/opnsense-api.txt (0600, secret never printed). SMOKE TEST PASS: GET core/firmware/status -> HTTP 200, CORE_ABI 26.7. This is the first proof the D-113(a2) API path works on 26.7 (the 26.7 revalidation the operator's base-version ruling called for). Script note: the bootstrap script requires the minter BESIDE it (its own dir), not in scripts/ -- worth a usage-line DOCFIX.
  • Revert: delete the API key (POST auth/user/del_api_key/<id>, the proven negative-tested path) + creds file; ip addr del the three interim addresses; the console bootstrap reverts by re-imaging the edge (inner tofu destroy/re-apply of the edge module).

10. D-125 egress isolation gate: EXECUTED as written -- PARTIAL (bridge-in PROVEN,

upstream NAT egress FAILS). Gate NOT closed; double-NAT fallback NOT invoked.

  • Method: operator-ruled "throwaway VM, as written" -- noble cloud image fetched ON the rack via the region proxy, one cloud-init guest, SINGLE NIC bridged to br-vr1-dc0-wan, serial-log probe, then destroyed (leftover-domains=0 asserted). Script: scratchpad d125-egress-test.sh; capture docs/audit/d125-egress-gate-20260720.txt.
  • Run 1 was a TEST ARTIFACT, not a gate failure: the probe used DHCP and came up address-less. MEASURED cause: vr1-dc0-uplink carries an <ip> but NO <dhcp> block -- there is no DHCP server on that /24 by design (the edge WAN takes a STATIC .2, D-113/D-125). DOCFIX QUEUED: the gate's wording "must GET a vcloud-ISP address" implies DHCP; it must say STATIC, or the /24 needs a range. Re-run used static 172.30.2.50/24.
  • RESULT (run 2): guest configured 172.30.2.50/24 + default via 172.30.2.1; gateway ping 3/3, 0% loss -- a FOREIGN MAC inside vvr1-dc0 reaches the vcloud ISP NAT across the nested bridge. That is the D-125 bridge-in property, and it PASSES. But ping 1.1.1.1 FAILED (NET-PING-RC=1).
  • CORRECTION (2026-07-20, same-day): the rack-bridge follow-up is CONFOUNDED and is NOT evidence. It sourced from 172.30.2.51 on the rack, but the rack has NO DEFAULT ROUTE (netplan gives it only 10.10.0.0/22 via 172.31.0.1; the WAN bridge is IP-less by design), so traffic to 1.1.1.1 fails in the rack's own routing table before the NAT is ever consulted. Its ICMP+TCP failures say nothing about virbr4. The ONLY clean signal is the GUEST's (172.30.2.50) failed egress, and that guest tested ICMP only -- if the rule dump comes back identical for both bridges, re-run the guest probe with curl before concluding anything. (Recorded per GA-R1/C2: a wrong measurement is corrected in the record, not quietly dropped.)
  • Controls: vcloud egress 0% loss; the office1 edge egresses through the SAME-SHAPE office1-wan NAT 0% loss. The two network definitions are byte-equivalent in substance (net-dumpxml: both forward mode=nat + <nat><port 1024-65535>; only netmask-vs- prefix notation differs) -- so the delta is in vcloud's LIVE firewall rules for virbr4, NOT in the design or the config.
  • GATE STATUS: PASS / CLOSED 2026-07-20 (runs 3+4, capture docs/audit/d125-egress-gate-20260720-matrix.txt). The "missing rule" hypothesis was DISPROVED by measurement first: the operator-run dumps show virbr4's rule set is structurally IDENTICAL to the WORKING virbr11 (only counters differ; virbr11's one extra rule is a DHCP checksum fixup it has because it carries a DHCP range). All three masquerade rules exist for 172.30.2.0/24 INCLUDING the generic (non-tcp/udp) one, and it had FIRED (1 packet/84 bytes = one ICMP conntrack entry), with FORWARD accept counting 136 packets. So net-destroy/net-start would have fixed nothing -- the pre-staged fix was correctly NOT run.
  • Decisive re-probe used a protocol/destination MATRIX instead of counter archaeology. Result, IDENTICAL on two consecutive runs: gateway ping 0, internet ping 0, curl http://1.1.1.1 -> 301, curl archive.ubuntu.com -> 200. Egress works for ICMP AND TCP to real hosts, so the D-125 bridge-in chain is PROVEN end to end and the double-NAT fallback is NOT needed.
  • Run-2 anomaly, recorded UNEXPLAINED (not swept): the same path failed ICMP-to-internet once while its gateway ping passed 3/3. Runs 3+4 passed identically with no intervening change to the uplink or its rules. Most likely a first-use conntrack/ARP transient on vcloud. It is not reproducible now; if a DC edge ever shows first-boot egress failure, START HERE rather than re-deriving.
  • Security-positive side observation (extra probe, outside the gate): the guest could NOT reach the Office1 management LAN through the ISP NAT (10.10.0.20:8000 -> 000) on all runs, while internet egress worked. That is the D-122/SEC-010 intent holding: the DC's WAN segment is an ISP uplink, not a path into Office1. Worth an explicit assertion in the hardening tests rather than leaving it an incidental observation.
  • Prior status line (superseded, kept for the audit trail): gate was held OPEN pending measurement rather than FAILED. The runbook's FAIL remedy (revert to double-NAT) is for a DESIGN failure; invoking it against what is very likely a host firewall-rule gap would be the wrong remedy and would add Roosevelt delta. Next step is a read-only measurement needing operator sudo: FULL dumps of BOTH nft list ruleset and iptables-save (which backend is authoritative here is itself unmeasured -- do not assume), diffing virbr4 against the WORKING virbr11. Two rule families must both be checked, and a subnet-only grep would MISS the second: POSTROUTING masquerade (keyed on 172.30.2.0/24) and FORWARD accept (keyed on the INTERFACE virbr4). Pre-staged fix-forward IF the rules are absent or malformed vs virbr11: virsh net-destroy vr1-dc0-uplink && virsh net-start vr1-dc0-uplink reinstalls libvirt's rules from scratch -- cheap and safe (nothing rides that uplink yet but the probe). If instead the dump shows libvirt never installed them correctly, the restart will not help and the dig moves to the firewall backend / conntrack.
  • Revert: nothing to revert -- the probe guest and both temporary addresses were removed; the only residue is /tmp/noble.img on the rack (kept deliberately: it is the base for any future throwaway probe).

11. NEW TOOLING + edge re-addressed: opnsense-set-interface-v4 pair (53/53)

  • RULING RECORD (GA-R5), 2026-07-20. Question as presented: base-iface IPv4 is not REST-covered on 26.7 (measured); how should LAN 10.12.4.1/22 and WAN 172.30.2.2/24 + gw 172.30.2.1 be set -- build the v4 setter pair / console menu now / console now + build at stage close? Operator answer, exact utterance (option selected): "Build the v4 setter pair (Recommended)". Classification: OPS tooling under the existing D-113 amendment; no new D-number (GA-R3).
  • D-113 AMENDMENT RE-MEASURED ON 26.7 (feeds the item-8 review): the amendment was measured on 26.1 only. On this 26.7 edge: interfaces/settings/get returns ONLY global offload flags, while interfaces/overview/list and interfaces/lan/get both return 404 "Endpoint not found". Base-interface addressing is still legacy /interfaces.php. The amendment now holds on TWO majors, measured.
  • SHIPPED: scripts/opnsense-set-iface-v4.php + scripts/opnsense-set-interface-v4.sh + tests/opnsense-set-interface-v4/ (53/53 PASS first run; gauntlet 70 -> 71 harnesses ALL GREEN; repo-lint 0 fail). Mirrors the proven v6 sibling exactly -- vendor Config singleton (never a config.xml push), dry-by-default, interface-count guard, fresh read-back verify, sh -s for the tcsh trap, KERNEL as ground truth. Two deliberate improvements over the v6 sibling: (i) the GATEWAY half (creates/updates at most ONE gateway_item, count-guarded exactly like the interface count, opt-in via a 5th arg); (ii) the interface's DEVICE is MEASURED from the edge's own config instead of the v6 sibling's hardcoded lan->vtnet0 / wan->vtnet1 sed -- that mapping is per-edge and hardcoding it is a hard-rule-2 violation waiting to happen. Harness pins BOTH (test 12 fails if the hardcode returns). LOGGED FINDING: backport the measured-device fix to the v6 sibling at stage close.
  • APPLIED to vr1-dc0-opnsense, WAN first per the script's own ordering warning (re-addressing the LAN drops the session you arrived on):
    • WAN: dhcp -> 172.30.2.2/24 + WAN_GW = 172.30.2.1 default. Note the edge had shipped with WAN on dhcp, which could NEVER have worked -- the uplink /24 has no DHCP server (same fact that broke probe run 1). Verified on the kernel: inet 172.30.2.2, default 172.30.2.1 UGS vtnet1, and the EDGE ITSELF pings 1.1.1.1 at 0% loss -- the D-125 chain now proven through the real OPNsense edge, not just a probe.
    • LAN: 192.168.1.1/24 -> 10.12.4.1/22 (the D-115/lib-net ruled provider-public gateway). Session dropped as designed; edge verified at the new address from the rack: inet 10.12.4.1 netmask 0xfffffc00.
    • Post-move reconciliation at the NEW address: API GET core/firmware/status -> 200 CORE_ABI 26.7; edge egress still 0% loss.
  • CLEANUP: the interim 192.168.1.2/24 on the rack's provider-public bridge is REMOVED (virbr5 now carries only the ruled 10.12.4.2/22).
  • Revert: re-run the setter with the previous values (--commit <edge> lan 192.168.1.1 24, --commit <edge> wan ...); the gateway item is named WAN_GW and is the only one this tool created.

8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review

  • Operator directive (verbatim intent): at the proper time/place, review the OPNsense 26.7 changelog + documentation for upgraded features and changes; incorporate beneficial improvements (esp. hardening) into the next deployment iteration, following project rules.
  • Scheduled: STAGE-3 CLOSE (GA-R6 close-out pass, with this stage's queued DOCFIXes). Deliverable: a findings doc citing the official 26.7 release notes; actionable items route per GA-R3 -- hardening/profile deltas as PROPOSED items anchored on D-129 (its 4 open sub-decisions are the likely surface), operational items as runbook DOCFIXes. NO self-executing incorporation: each finding is ruled per GA-R5 (one per exchange) before touching any built surface. Target of incorporation: dc1 standup + Roosevelt runbooks; live-edge retrofits individually gated.

6. ledger-scan hardening: FAIL-OPEN is terminology, not a status

  • SEC-010's close surfaced a scanner false positive: the OPEN-row detector (awk toupper(cell) ~ /OPEN|PENDING/) matched the substring OPEN inside "fail-open" in the closed row's disposition -- the row counted OPEN forever (same class as the GA-F15 wrap trap). Fix: strip FAIL-OPEN from the cell before matching (scripts/ledger-scan.sh); harness gains a CLOSED-row-with-fail-open fixture + nochk (45/45 PASS). Count re-verified: 7 open, SEC-010 absent.
  • Revert: git revert of this commit's ledger-scan.sh + test hunks.

5. Host tooling: libguestfs-tools installed on vcloud (operator-run)

  • sudo apt-get install -y libguestfs-tools (1:1.52.0-5ubuntu3) -- enables offline guest inspection (virt-cat/virt-ls), used for items 4's captures. Standard KVM-host tooling; Roosevelt-transferable.
  • Revert: sudo apt-get remove libguestfs-tools.