# Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap) Session changelog (GA-R2/D1: ONE per session). Logged window: `dc0-deploy` (same-day append, per the A/B-boundary handoff). Status lives ONLY in `docs/CURRENT-STATE.md`; this file is narrative + reverts. ## 1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124) - **Question as presented (2026-07-19):** "Who should hold 172.31.0.1 (the Office1 end of the dc0 transit /30), and by what mechanism? This is the prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS rack-to-region enrollment path." - **Operator answer, exact utterance (option selected):** "voffice1 via IaC + restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to `module "voffice1"` in the outer root; apply in a planned voffice1 restart window (graceful shutdown first; region/NetBox/Tailscale return via proven autostart). - Classification: OPS mechanism decision implementing D-124's already-ruled addressing ("region/peer .1" -- `docs/design-decisions.md:3727`); no new D-number (GA-R3, doubt resolves DOWN). - **Measured basis (this session, logged):** mesh-vr1-dc0-office1 (virbr7) is ISOLATED (no ``/``) with exactly ONE attached interface (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only route `10.10.0.0/22 via 172.31.0.1` (d124-rack.auto.tfvars, consumed by step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached by `-J voffice1` (D-126 shape) -- NO vcloud host leg / site-baseleg DC row needed for this net. ## 2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated) - **What:** `opentofu/main.tf` `module "voffice1"`: `network_names` gains `module.mesh_vr1_dc0_office1.network_name` SECOND (office1-local stays first -- PCI order preserves enp1s0); `network_config` rewritten from the single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4; transit=enp2s0 static `${vr1_dc0_rack_transit_peer_ip}/${prefix}` -- same D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init- re-run caveats documented in-block: the seed netplan is REBUILD correctness; the live transit config is an in-guest netplan drop-in at the gated attach step. - **Why:** section 1's measured gap -- step B (ssh reach + rack enrollment) has no L3 path until the region end of the transit exists. - **Revert:** restore `network_names = [module.office1_network.network_name]` and the previous "en*" glob `network_config` (git revert of this commit); if already applied, a follow-up plan shows the interface removal and the in-guest drop-in `/etc/netplan/60-transit.yaml` is deleted. ## 3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation - Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply (1/1/1 exact, `docs/audit/outer-apply-20260719-voffice1-transit.txt`) -> domain up with both NICs, office1-local index 0. - **INCIDENT (found+fixed in-window):** the provider REGENERATED both NIC MACs on the interface-list rebuild (old lan MAC `52:54:00:6a:87:e5` -> new `52:54:00:89:e1:19`). The Kea reservation (MAC-keyed) missed and voffice1 leased pool address `.108` (measured via edge ARP). FIX: re-keyed reservation uuid `b12621e6-63e9-4714-9726-2e130c84e069` to the new MAC via the D-113 API (`kea/dhcpv4/set_reservation` -> saved; `kea/service/reconfigure` -> ok), then a detached `networkctl reconfigure enp1s0` -> voffice1 back on `10.10.0.20`. - **LOGGED FINDING (not executed):** `modules/cloudinit-vm` does not pin MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed DHCP reservations. Roosevelt-relevant; candidate module `mac` var or a documented ops step. Also: live transit config landed as in-guest netplan drop-in `/etc/netplan/60-transit.yaml` (narrows the lan glob to enp1s0 + transit enp2s0 static) -- the drop-in must ALSO narrow `lan`, else the 50-cloud-init `en*` glob claims enp2s0 (measured: first drop-in attempt without it left enp2s0 addressless). - **Revert:** `kea/dhcpv4/set_reservation` back to the old MAC (senseless unless the NIC change is also reverted); delete the drop-in. ## 4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix - Symptom: `172.31.0.2` unreachable; MEASURED: both rack NICs emitted ZERO frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive (ACPI-responsive; cloud-init completed per offline logs). - Guest is a SEALED BOX: `modules/cloudinit-vm` domains have NO serial console, NO qga channel, NO graphics; qga is NOT in the noble base image and its cloud-init install failed (no egress) -- so no console, no agent, no network. **LOGGED FINDING: add opt-in serial console (+ the D-129-style qga channel var) to `modules/cloudinit-vm` -- observability gap bites every future DC VM. NOT executed mid-step.** - Offline inspection (libguestfs, operator-run, captures `~/vvr1-dc0-offline-inspect{,2}.txt`): NIC names enp1s0/enp2s0 CORRECT (naming trap closed); root cause = netplan `match.name` + `set-name`: boot 1 renames don't apply (.link written after udev add) so the rendered config matched nothing; after removing set-name, boot 2 STILL failed because the early-boot netplan generator ran against the OLD config and renamed the devices to mgmt/uplink before cloud-init wrote the new file (one-boot lag, measured in syslog). Boot 3 (current config, no renames) is the verification boot. - FIX (this commit + applied): `network_config` drops set-name, keeps MEASURED kernel names; `instance-id` bumped `vvr1-dc0-d123 -> -d123-r2` (side effects noted in-block); seed replaced via explicit `-replace=module.vvr1_dc0.libvirt_volume.seed` (the sanctioned override of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2 exact (`docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt`). - CONSEQUENCE for step B: bootstrap runs with `--transit-if enp1s0 --uplink-if enp2s0` (script defaults mgmt/uplink no longer match reality); runbook prose naming "mgmt" needs a DOCFIX at stage close. - **Revert:** git revert of the netplan/instance-id edit + one more seed `-replace` apply + reboot. ## 4b. VERIFICATION (2026-07-20, logged window) - Boot 3 with the fixed config: `enp1s0 UP 172.31.0.2/30`, region route present, `br-vr1-dc0-wan` up. - voffice1 -> 172.31.0.2 ping 0% loss; ssh `-i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2` works (D-126 -J shape, per-env key); rack -> region `ping 10.10.0.20` 0% loss (the MAAS-enrollment path for the G10 bootstrap). - Outer plan CONVERGED back to ZERO DIFF (`docs/audit/outer-plan-20260720-postfix-converged.txt`); CURRENT-STATE section 5 updated in the same commit. - Follow-items (logged, not executed): D-126 DC ssh-config alias row for vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key); cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC pinning; DOCFIX for runbook "mgmt/uplink" naming. ## 5. Host tooling: libguestfs-tools installed on vcloud (operator-run) - `sudo apt-get install -y libguestfs-tools` (1:1.52.0-5ubuntu3) -- enables offline guest inspection (virt-cat/virt-ls), used for items 4's captures. Standard KVM-host tooling; Roosevelt-transferable. - **Revert:** `sudo apt-get remove libguestfs-tools`.