2026-07-22 -- G12 [V] leg: dc1 apex write + substrate build step A (session changelog)

Session scope: continue the G12 [V] leg from the reconciled tree (predecessor delivery landed as changelog-20260721-g12-dc1-authoring.md items 3-4, commit d2bf743): the one owed apex write, tfvars, and the gated outer build. Branch: dc-dc-g12-dc1-substrate. One changelog for the session (GA-R2/D1); every item carries its revert.

Item 1 -- apex --commit EXECUTED (the one owed dc1 apex write; operator-gated)

Same-session read-only preflight FIRST (verify-before-mutate): dry-run re-run against the live apex = would-create exactly 2, already-present 0 -- docs/audit/dc1-rack-import-dryrun-20260722.txt, identical to the 07-21 preflight. Then the gated write (piped to office1-netbox over ssh; token sourced on-host from /root/netbox-secrets/api.token per creds-manifests/vr1-office1.manifest, value never printed or brought into context):

• CREATED 172.31.0.4/30 (prefix id=139) role=transit scope=dcim.site:vr1-dc1
• CREATED 10.12.68.2/22 (ip id=4) dns=vvr1-dc1

Post-commit idempotency read-back: would-create 0 / already-present 2. Both runs captured in docs/audit/dc1-rack-import-commit-20260722.txt. Values verbatim from the D-124 amendment (ruled 2026-07-21); exactly the two objects the dc0 precedent registered.

Revert: delete the two apex objects by id (prefix 139, ip-address 4) via the NetBox UI/API on office1-netbox; delete the capture files.

Item 2 -- vr1-dc1 service keypair MINTED (manifest-prescribed at-deploy step)

~/vr1-dc1-creds/vr1-dc1_svc_ed25519{,.pub} generated on vcloud (ed25519, modes 600/644) per creds-manifests/vr1-dc1.manifest ("MINTED AT DC DEPLOY"). bash scripts/creds-audit.sh vr1-dc1 = CLEAN. D-126 option (a) per-env key; pubkey feeds vvr1-dc1 cloud-init (item 3), private half jumphost-local.

Revert: remove both key files (couples to item 4 -- vvr1-dc1's cloud-init authorizes this pubkey; re-mint requires a seed rebuild).

Item 3 -- dc1 tfvars authored (LOCAL, gitignored -- recorded here, not committed)

opentofu/d124-rack.auto.tfvars gained the five vr1dc1* values, verbatim from the D-124 amendment's own tfvars line (rack_metal_admin_ip=10.12.68.2, rack_transit_ip=172.31.0.6, rack_transit_prefix=30, rack_transit_peer_ip=172.31.0.5) + vr1_dc1_ssh_pubkey_path to the item-2 pubkey. Stale "dc1 DEFERRED" header comment replaced. tofu fmt applied; scripts/opentofu-validate.sh PASS (all roots).

Revert: remove the vr1dc1* block from the local file.

Item 4 -- outer apply: dc1 substrate step A (operator-gated, logged, saved-plan exact)

Preconditions measured in-session: host RAM 1007 GiB with 450 GiB committed (voffice1 32 + edge 2 + vvr1-dc0 416) -> +416 GiB fits with ~140 GiB headroom; vCPU 234/256 post-apply; dc-dc-whole-host-budget 13/13 PASS.

Saved plan tfplan-dc1-20260722 = 5/0/0 exact (vr1-dc1-uplink network + vvr1-dc1 domain/disk/seed/cloudinit; ZERO touches to live resources) -- capture docs/audit/outer-plan-20260722-dc1-substrate.txt. Applied via the saved plan under a per-command as-executed wrap (~/as-executed/2026-07-22-dc1-deploy.log; index row added): 5 added, 0 changed, 0 destroyed. Convergence re-plan = zero diff (docs/audit/outer-plan-20260722-postdc1-converged.txt). Live verify: vvr1-dc1 RUNNING (Id 8), voffice1/office1-opnsense/vvr1-dc0 untouched and running; vr1-dc1-uplink + both dc1 mesh legs active.

Revert: runbooks/dc-dc-teardown-rollback.md decision tree; the targeted destroy set is the five applied resources (module.vvr1_dc1. + module.vr1_dc1_uplink.); delete the plan/capture files and the tfplan.

CORRECTION (same session): commit 0bd6342 accidentally included the spent binary saved plan (repo-lint L1 x2 -- the lint's exit code was masked by a pipeline; owned). The file deviated from the .gitignore naming convention (opentofu/**/*.tfplan) -- saved plans must be named *.tfplan. Removed in the follow-up commit; lint back to 0-fail. The applied-plan record remains the two dated captures.

Item 5 -- cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (gated apply)

What: modules/cloudinit-vm gained interface_macs (ported verbatim-in-intent from modules/node-vm, same validations; harness +4 cases, cloudinit-vm 10/10) -- the 2026-07-20 voffice1 Kea/MAC-regen incident class, closed at the module layer. Root wiring: module.voffice1 NIC3 -> mesh-vr1-dc1-office1 with all three MACs pinned (NIC1/NIC2 = measured live values; NIC3 = pre-pinned 52:54:01:d1:fe:01, dc1 scheme, fe = region-side); module.vvr1_dc1 pin-adopts its two measured step-A MACs (standup DoD invariant). Saved plan voffice1-dc1nic-20260722.tfplan = 0/2/0, zero replaces, all five MAC values verified in the diff (docs/audit/outer-plan-20260722-voffice1-dc1nic.txt); applied logged (trap 1e: BOTH domains bounced -- presented as such). Post-bounce battery ALL PASS: 4/4 domains running, voffice1 MACs exactly as pinned, Kea lease 10.10.0.20 intact, region dhcpd up, netbox 302 + tailscale up (nested LXD VMs self-recovered, ~3 min), dc0 rack: dhcpd + dc0-node-dns + dc0-rack-legs active, forwarder SOA answers. Convergence re-plan ZERO DIFF.

Revert: remove NIC3 + interface_macs from main.tf, revert the module + harness (git checkout the four files), re-plan/apply (bounces again).

Item 6 -- voffice1 dc1 transit leg (in-guest netplan, gated)

/etc/netplan/61-transit-dc1.yaml (0600) on voffice1: enp3s0 static 172.31.0.5/30 (per-DC drop-in; dc0's 60-transit.yaml untouched). Verified: enp3s0 UP with .5/30, ping 172.31.0.6 = 0% loss, first ssh into vvr1-dc1 over the transit with the dc1 key OK (hostname vvr1-dc1, nested-KVM module present, region route + 10.10.0.20 reach pre-staged by step-A cloud-init). Revert: remove the drop-in + netplan apply.

Item 7 -- rack bootstrap EXECUTED (site-headend-install, dc1-parameterized)

Enrollment secret staged region->rack as a host-to-host pipe (0600 /root/region-enroll.secret, 32 bytes verified by count, value never in context). Snap system proxy set to http://10.10.0.20:8000 (dc0-measured mechanism; env proxy for apt). Dry-run FIRST exposed that the script's host-nodes DEFAULTS are dc0-flavored -- all overridable by existing flags; run with --wan-bridge br-vr1-dc1-wan --inner-pool-path /var/lib/libvirt/vr1-dc1-inner --opnsense-base ...26.7.... First real run FAILED exit 4: stale base-image apt index -> 404s on superseded debs via the proxy; fixed with apt-get update, idempotent re-run exit 0: rack ENROLLED, nested KVM on, inner pool + AppArmor, SEC-010 rack end, WAN bridge verified with enp2s0 enslaved. Region-side verify: maas admin rack-controllers read lists vvr1-dc1 (nmpcq4). SEC-010 REGION end: voffice1's /etc/nftables-sec010.nft extended with the enp3s0 drop pair (dc0 idiom), table reloaded clean, ruleset = both legs dropped. --check capture: docs/audit/dc1-stepB-check-20260722.txt (sole [--] = opnsense base, item 8).

QUEUED findings (logged, not fixed mid-step): (a) site-headend-install NOTE/hint prose still says "26.1" and "vr1-dc0-substrate" even when parameterized for dc1 (cosmetic, misleads operators); (b) nftables-sec010.nft reload is NOT idempotent -- nft -f on a live table appends, so a service restart duplicates rules (observed; cleaned via nft delete table + restart; the file should flush first); (c) opnsense-prep-image.sh dies on BASH_SOURCE[0]: unbound variable when piped via bash -s under set -u (ran into it before the mirror guard).

Revert: rack side -- snap remove maas, remove nftables-sec010 + sec010-fw + kvm-nested modprobe + inner pool dir on vvr1-dc1; region side -- remove the enp3s0 pair from voffice1's nft file + restart unit; delete the staged secret file.

Item 8 -- OPNsense 26.7 base staged on dc1 (proven-artifact copy)

opnsense-prep-image.sh requires OPNSENSE_MIRROR_BASE (deliberately not repo-recorded; mirrors change). Instead of choosing a mirror, the dc0 rack's OPERATOR-RULED and boot-PROVEN 26.7 base was streamed rack->rack through the jump path (direct rack-to-rack is SEC-010-dropped, correctly) and sha256-verified on arrival against the dc0 source hash (3981821e3a3c...476627d). Same bits that passed the D-112 boot path on dc0. Revert: delete /var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2.

Item 9 -- inner apply: dc1 substrate step C EXECUTED (from voffice1, D-128 Plane 2)

Staging on voffice1 (dc0-precedent set): repo clone fetched + switched from the retired stage-3 branch to dc-dc-g12-dc1-substrate @ 61c416e (dc0 inner tfstate untouched, verified); dc1 private key piped in (0600, 432 bytes by count); 172.31.0.6 host keys scanned into known_hosts (all 3 types -- dc0 trap); the 26.7 base copied rack->voffice1 to ~/vr1-dc1-images/ and sha256-verified (provider streams volume content itself -- the dc0 measured fix; rack-side copy satisfies only the bootstrap check). Inner tfvars d124-inner.auto.tfvars written on voffice1, every value sourced (measured transit .6, outer-committed planes map verbatim, Stage-1 mtu 9000, D-106 suffix, keyfile + local base paths).

Plan dc1-inner-20260722.tfplan = 28/0/0 (dc0's exact step-C count), verified in-capture: 54 pinned 52:54:01:d1:* MACs, 9 Option-C nodes + edge, qga channel present (docs/audit/inner-plan-20260722-dc1-stepC.txt). First apply FAILED at 18/28: the edge domain's serial-log dir /var/lib/libvirt/vr1/staging/ did not exist inside vvr1-dc1 (hand-created on dc0 during the 07-20 serial work; QUEUED finding -- belongs in the bootstrap/standup DoD so dc2+ does not repeat it). Fix-forward per the rollback tree: dir created (root:root 0755, dc0 mirror), resume plan 10/0/0, apply exit 0. Final: 28/28 in state, convergence ZERO DIFF (docs/audit/inner-converge-20260722-dc1-stepC.txt), 10/10 domains RUNNING inside vvr1-dc1, edge serial log at the FreeBSD login prompt -- OPNsense 26.7, factory LAN 192.168.1.1/24, WAN unaddressed (second D-112 boot-path datapoint on 26.7; exactly dc0's post-step-C state). The dc1 INNER tfstate lives ON voffice1 (opentofu/vr1-dc1-substrate/terraform.tfstate -- add to the site backup set, same as dc0's).

Revert: dc-dc-teardown-rollback.md tree; inner root destroy from voffice1 tears down the 28 (containment VM unaffected).

Next (gated, not run here)

D-125 egress isolation gate on br-vr1-dc1-wan (throwaway guest, two identical runs); edge bootstrap (D-112(c) console -> key-only SSH -> D-113(a2) API key on 26.7); edge addressing via opnsense-set-interface-v4 (WAN 172.30.3.2/24 gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); rack standup DoD (dc-rack-net.sh install dc1 + forwarder 10.12.68.3, region-side DHCP on metal-admin naming nmpcq4 primary_rack, dynamic range 10.12.68.100-.200 per D-120, maas-node-power dc1 arm). Runbook + CURRENT-STATE govern.