# 2026-07-22 -- G12 [V] leg: dc1 apex write + substrate build step A (session changelog)

Session scope: continue the G12 [V] leg from the reconciled tree (predecessor
delivery landed as changelog-20260721-g12-dc1-authoring.md items 3-4, commit
`d2bf743`): the one owed apex write, tfvars, and the gated outer build.
Branch: `dc-dc-g12-dc1-substrate`. One changelog for the session (GA-R2/D1);
every item carries its revert.

## Item 1 -- apex `--commit` EXECUTED (the one owed dc1 apex write; operator-gated)

Same-session read-only preflight FIRST (verify-before-mutate): dry-run re-run
against the live apex = would-create exactly 2, already-present 0 --
`docs/audit/dc1-rack-import-dryrun-20260722.txt`, identical to the 07-21
preflight. Then the gated write (piped to office1-netbox over ssh; token
sourced on-host from `/root/netbox-secrets/api.token` per
`creds-manifests/vr1-office1.manifest`, value never printed or brought into
context):

- CREATED `172.31.0.4/30` (prefix id=139) role=transit scope=dcim.site:vr1-dc1
- CREATED `10.12.68.2/22` (ip id=4) dns=vvr1-dc1

Post-commit idempotency read-back: would-create 0 / already-present 2. Both
runs captured in `docs/audit/dc1-rack-import-commit-20260722.txt`. Values
verbatim from the D-124 amendment (ruled 2026-07-21); exactly the two objects
the dc0 precedent registered.

Revert: delete the two apex objects by id (prefix 139, ip-address 4) via the
NetBox UI/API on office1-netbox; delete the capture files.

## Item 2 -- vr1-dc1 service keypair MINTED (manifest-prescribed at-deploy step)

`~/vr1-dc1-creds/vr1-dc1_svc_ed25519{,.pub}` generated on vcloud (ed25519,
modes 600/644) per `creds-manifests/vr1-dc1.manifest` ("MINTED AT DC DEPLOY").
`bash scripts/creds-audit.sh vr1-dc1` = CLEAN. D-126 option (a) per-env key;
pubkey feeds vvr1-dc1 cloud-init (item 3), private half jumphost-local.

Revert: remove both key files (couples to item 4 -- vvr1-dc1's cloud-init
authorizes this pubkey; re-mint requires a seed rebuild).

## Item 3 -- dc1 tfvars authored (LOCAL, gitignored -- recorded here, not committed)

`opentofu/d124-rack.auto.tfvars` gained the five vr1_dc1_* values, verbatim
from the D-124 amendment's own tfvars line (rack_metal_admin_ip=10.12.68.2,
rack_transit_ip=172.31.0.6, rack_transit_prefix=30,
rack_transit_peer_ip=172.31.0.5) + `vr1_dc1_ssh_pubkey_path` to the item-2
pubkey. Stale "dc1 DEFERRED" header comment replaced. `tofu fmt` applied;
`scripts/opentofu-validate.sh` PASS (all roots).

Revert: remove the vr1_dc1_* block from the local file.

## Item 4 -- outer apply: dc1 substrate step A (operator-gated, logged, saved-plan exact)

Preconditions measured in-session: host RAM 1007 GiB with 450 GiB committed
(voffice1 32 + edge 2 + vvr1-dc0 416) -> +416 GiB fits with ~140 GiB headroom;
vCPU 234/256 post-apply; dc-dc-whole-host-budget 13/13 PASS.

Saved plan `tfplan-dc1-20260722` = **5/0/0 exact** (vr1-dc1-uplink network +
vvr1-dc1 domain/disk/seed/cloudinit; ZERO touches to live resources) --
capture `docs/audit/outer-plan-20260722-dc1-substrate.txt`. Applied via the
saved plan under a per-command as-executed wrap
(`~/as-executed/2026-07-22-dc1-deploy.log`; index row added): **5 added, 0
changed, 0 destroyed**. Convergence re-plan = **zero diff**
(`docs/audit/outer-plan-20260722-postdc1-converged.txt`). Live verify:
vvr1-dc1 RUNNING (Id 8), voffice1/office1-opnsense/vvr1-dc0 untouched and
running; vr1-dc1-uplink + both dc1 mesh legs active.

Revert: `runbooks/dc-dc-teardown-rollback.md` decision tree; the targeted
destroy set is the five applied resources (module.vvr1_dc1.* +
module.vr1_dc1_uplink.*); delete the plan/capture files and the tfplan.

CORRECTION (same session): commit `0bd6342` accidentally included the spent
binary saved plan (repo-lint L1 x2 -- the lint's exit code was masked by a
pipeline; owned). The file deviated from the `.gitignore` naming convention
(`opentofu/**/*.tfplan`) -- saved plans must be named `*.tfplan`. Removed in
the follow-up commit; lint back to 0-fail. The applied-plan record remains
the two dated captures.

## Item 5 -- cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (gated apply)

What: `modules/cloudinit-vm` gained `interface_macs` (ported verbatim-in-intent
from `modules/node-vm`, same validations; harness +4 cases, cloudinit-vm 10/10)
-- the 2026-07-20 voffice1 Kea/MAC-regen incident class, closed at the module
layer. Root wiring: `module.voffice1` NIC3 -> `mesh-vr1-dc1-office1` with all
three MACs pinned (NIC1/NIC2 = measured live values; NIC3 = pre-pinned
`52:54:01:d1:fe:01`, dc1 scheme, fe = region-side); `module.vvr1_dc1` pin-adopts
its two measured step-A MACs (standup DoD invariant). Saved plan
`voffice1-dc1nic-20260722.tfplan` = **0/2/0, zero replaces**, all five MAC
values verified in the diff (`docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`);
applied logged (trap 1e: BOTH domains bounced -- presented as such). Post-bounce
battery ALL PASS: 4/4 domains running, voffice1 MACs exactly as pinned, Kea
lease 10.10.0.20 intact, region dhcpd up, netbox 302 + tailscale up (nested LXD
VMs self-recovered, ~3 min), dc0 rack: dhcpd + dc0-node-dns + dc0-rack-legs
active, forwarder SOA answers. Convergence re-plan ZERO DIFF.

Revert: remove NIC3 + interface_macs from main.tf, revert the module +
harness (`git checkout` the four files), re-plan/apply (bounces again).

## Item 6 -- voffice1 dc1 transit leg (in-guest netplan, gated)

`/etc/netplan/61-transit-dc1.yaml` (0600) on voffice1: enp3s0 static
`172.31.0.5/30` (per-DC drop-in; dc0's `60-transit.yaml` untouched). Verified:
enp3s0 UP with .5/30, ping 172.31.0.6 = 0% loss, first ssh into vvr1-dc1 over
the transit with the dc1 key OK (hostname vvr1-dc1, nested-KVM module
present, region route + 10.10.0.20 reach pre-staged by step-A cloud-init).
Revert: remove the drop-in + `netplan apply`.

## Item 7 -- rack bootstrap EXECUTED (site-headend-install, dc1-parameterized)

Enrollment secret staged region->rack as a host-to-host pipe (0600
`/root/region-enroll.secret`, 32 bytes verified by count, value never in
context). Snap system proxy set to `http://10.10.0.20:8000` (dc0-measured
mechanism; env proxy for apt). Dry-run FIRST exposed that the script's
host-nodes DEFAULTS are dc0-flavored -- all overridable by existing flags;
run with `--wan-bridge br-vr1-dc1-wan --inner-pool-path
/var/lib/libvirt/vr1-dc1-inner --opnsense-base ...26.7...`. First real run
FAILED exit 4: stale base-image apt index -> 404s on superseded debs via the
proxy; fixed with `apt-get update`, idempotent re-run **exit 0**: rack
ENROLLED, nested KVM on, inner pool + AppArmor, SEC-010 rack end, WAN bridge
verified with enp2s0 enslaved. Region-side verify: `maas admin
rack-controllers read` lists **vvr1-dc1 (nmpcq4)**. SEC-010 REGION end:
voffice1's `/etc/nftables-sec010.nft` extended with the enp3s0 drop pair
(dc0 idiom), table reloaded clean, ruleset = both legs dropped.
`--check` capture: `docs/audit/dc1-stepB-check-20260722.txt` (sole [--] =
opnsense base, item 8).

QUEUED findings (logged, not fixed mid-step): (a) site-headend-install
NOTE/hint prose still says "26.1" and "vr1-dc0-substrate" even when
parameterized for dc1 (cosmetic, misleads operators); (b)
`nftables-sec010.nft` reload is NOT idempotent -- `nft -f` on a live table
appends, so a service restart duplicates rules (observed; cleaned via
`nft delete table` + restart; the file should flush first); (c)
`opnsense-prep-image.sh` dies on `BASH_SOURCE[0]: unbound variable` when
piped via `bash -s` under `set -u` (ran into it before the mirror guard).

Revert: rack side -- `snap remove maas`, remove nftables-sec010 +
sec010-fw + kvm-nested modprobe + inner pool dir on vvr1-dc1; region side --
remove the enp3s0 pair from voffice1's nft file + restart unit; delete
the staged secret file.

## Item 8 -- OPNsense 26.7 base staged on dc1 (proven-artifact copy)

`opnsense-prep-image.sh` requires `OPNSENSE_MIRROR_BASE` (deliberately not
repo-recorded; mirrors change). Instead of choosing a mirror, the dc0 rack's
OPERATOR-RULED and boot-PROVEN 26.7 base was streamed rack->rack through the
jump path (direct rack-to-rack is SEC-010-dropped, correctly) and
sha256-verified on arrival against the dc0 source hash
(`3981821e3a3c...476627d`). Same bits that passed the D-112 boot path on dc0.
Revert: delete `/var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2`.

## Item 9 -- inner apply: dc1 substrate step C EXECUTED (from voffice1, D-128 Plane 2)

Staging on voffice1 (dc0-precedent set): repo clone fetched + switched from the
retired stage-3 branch to `dc-dc-g12-dc1-substrate` @ 61c416e (dc0 inner
tfstate untouched, verified); dc1 private key piped in (0600, 432 bytes by
count); 172.31.0.6 host keys scanned into known_hosts (all 3 types -- dc0
trap); the 26.7 base copied rack->voffice1 to `~/vr1-dc1-images/` and
sha256-verified (provider streams volume content itself -- the dc0 measured
fix; rack-side copy satisfies only the bootstrap check). Inner tfvars
`d124-inner.auto.tfvars` written on voffice1, every value sourced (measured
transit .6, outer-committed planes map verbatim, Stage-1 mtu 9000, D-106
suffix, keyfile + local base paths).

Plan `dc1-inner-20260722.tfplan` = **28/0/0** (dc0's exact step-C count),
verified in-capture: 54 pinned `52:54:01:d1:*` MACs, 9 Option-C nodes + edge,
qga channel present (`docs/audit/inner-plan-20260722-dc1-stepC.txt`). First
apply FAILED at 18/28: the edge domain's serial-log dir
`/var/lib/libvirt/vr1/staging/` did not exist inside vvr1-dc1 (hand-created
on dc0 during the 07-20 serial work; QUEUED finding -- belongs in the
bootstrap/standup DoD so dc2+ does not repeat it). Fix-forward per the
rollback tree: dir created (root:root 0755, dc0 mirror), resume plan 10/0/0,
**apply exit 0**. Final: 28/28 in state, convergence ZERO DIFF
(`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING
inside vvr1-dc1**, edge serial log at the FreeBSD login prompt -- OPNsense
26.7, factory LAN 192.168.1.1/24, WAN unaddressed (second D-112 boot-path
datapoint on 26.7; exactly dc0's post-step-C state). The dc1 INNER tfstate
lives ON voffice1 (`opentofu/vr1-dc1-substrate/terraform.tfstate` -- add to
the site backup set, same as dc0's).

Revert: `dc-dc-teardown-rollback.md` tree; inner root destroy from voffice1
tears down the 28 (containment VM unaffected).

## Item 10 -- D-125 egress isolation gate: PASS (vr1-dc1)

Procedure of record = dc0's `d125-egress-test.sh` (recovered from the dc0
rack's home), adapted per-DC (pool `vr1-dc1-inner`, bridge `br-vr1-dc1-wan`,
probe 172.30.3.50/24 via .1 -- D-124 amendment /24) and staged to the dc1
rack. Prereqs installed via the region proxy (virtinst, cloud-image-utils);
`/tmp/noble.img` streamed dc0->dc1. **Two identical consecutive runs**:
GW-PING-RC=0, NET-PING-RC=0, NET-TCP-http=301, NET-TCP-ubuntu=200,
LAN-TCP-http=000 (the region-isolation check), leftover-domains=0 --
`docs/audit/d125-egress-gate-20260722-dc1.txt`. Bridge-in egress PROVEN
end-to-end for dc1; the double-NAT fallback is NOT needed. dc0's
one-time unexplained first-run ICMP failure did NOT recur.

Revert: none needed (throwaway fully torn down, verified); delete the
capture + the staged script + /tmp/noble.img on the rack if desired.

## Next (gated, not run here)

Edge bootstrap (D-112(c) console -> key-only SSH -> D-113(a2) API key on
26.7); edge addressing via `opnsense-set-interface-v4` (WAN 172.30.3.2/24
gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); rack standup DoD
(dc-rack-net.sh install dc1 + forwarder 10.12.68.3, region-side DHCP on
metal-admin naming nmpcq4 primary_rack, dynamic range 10.12.68.100-.200
per D-120, maas-node-power dc1 arm). Runbook + CURRENT-STATE govern.
