Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260719-dc0-deploy-stepB.md
@JANeumatrix JANeumatrix 1 day ago 7 KB Step B transit reach ESTABLISHED: voffice1 .1/30 applied (Kea re-key incident fixed); vvr1-dc0 netplan set-name root cause fixed; zero-diff converged
Raw Blame History

Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap)

Session changelog (GA-R2/D1: ONE per session). Logged window: dc0-deploy (same-day append, per the A/B-boundary handoff). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts.

1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124)

  • Question as presented (2026-07-19): "Who should hold 172.31.0.1 (the Office1 end of the dc0 transit /30), and by what mechanism? This is the prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS rack-to-region enrollment path."
  • Operator answer, exact utterance (option selected): "voffice1 via IaC + restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to module "voffice1" in the outer root; apply in a planned voffice1 restart window (graceful shutdown first; region/NetBox/Tailscale return via proven autostart).
  • Classification: OPS mechanism decision implementing D-124's already-ruled addressing ("region/peer .1" -- docs/design-decisions.md:3727); no new D-number (GA-R3, doubt resolves DOWN).
  • Measured basis (this session, logged): mesh-vr1-dc0-office1 (virbr7) is ISOLATED (no <forward>/<ip>) with exactly ONE attached interface (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only route 10.10.0.0/22 via 172.31.0.1 (d124-rack.auto.tfvars, consumed by step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached by -J voffice1 (D-126 shape) -- NO vcloud host leg / site-baseleg DC row needed for this net.

2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated)

  • What: opentofu/main.tf module "voffice1": network_names gains module.mesh_vr1_dc0_office1.network_name SECOND (office1-local stays first -- PCI order preserves enp1s0); network_config rewritten from the single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4; transit=enp2s0 static ${vr1_dc0_rack_transit_peer_ip}/${prefix} -- same D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init- re-run caveats documented in-block: the seed netplan is REBUILD correctness; the live transit config is an in-guest netplan drop-in at the gated attach step.
  • Why: section 1's measured gap -- step B (ssh reach + rack enrollment) has no L3 path until the region end of the transit exists.
  • Revert: restore network_names = [module.office1_network.network_name] and the previous "en*" glob network_config (git revert of this commit); if already applied, a follow-up plan shows the interface removal and the in-guest drop-in /etc/netplan/60-transit.yaml is deleted.

3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation

  • Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply (1/1/1 exact, docs/audit/outer-apply-20260719-voffice1-transit.txt) -> domain up with both NICs, office1-local index 0.
  • INCIDENT (found+fixed in-window): the provider REGENERATED both NIC MACs on the interface-list rebuild (old lan MAC 52:54:00:6a:87:e5 -> new 52:54:00:89:e1:19). The Kea reservation (MAC-keyed) missed and voffice1 leased pool address .108 (measured via edge ARP). FIX: re-keyed reservation uuid b12621e6-63e9-4714-9726-2e130c84e069 to the new MAC via the D-113 API (kea/dhcpv4/set_reservation -> saved; kea/service/reconfigure -> ok), then a detached networkctl reconfigure enp1s0 -> voffice1 back on 10.10.0.20.
  • LOGGED FINDING (not executed): modules/cloudinit-vm does not pin MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed DHCP reservations. Roosevelt-relevant; candidate module mac var or a documented ops step. Also: live transit config landed as in-guest netplan drop-in /etc/netplan/60-transit.yaml (narrows the lan glob to enp1s0 + transit enp2s0 static) -- the drop-in must ALSO narrow lan, else the 50-cloud-init en* glob claims enp2s0 (measured: first drop-in attempt without it left enp2s0 addressless).
  • Revert: kea/dhcpv4/set_reservation back to the old MAC (senseless unless the NIC change is also reverted); delete the drop-in.

4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix

  • Symptom: 172.31.0.2 unreachable; MEASURED: both rack NICs emitted ZERO frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive (ACPI-responsive; cloud-init completed per offline logs).
  • Guest is a SEALED BOX: modules/cloudinit-vm domains have NO serial console, NO qga channel, NO graphics; qga is NOT in the noble base image and its cloud-init install failed (no egress) -- so no console, no agent, no network. LOGGED FINDING: add opt-in serial console (+ the D-129-style qga channel var) to modules/cloudinit-vm -- observability gap bites every future DC VM. NOT executed mid-step.
  • Offline inspection (libguestfs, operator-run, captures ~/vvr1-dc0-offline-inspect{,2}.txt): NIC names enp1s0/enp2s0 CORRECT (naming trap closed); root cause = netplan match.name + set-name: boot 1 renames don't apply (.link written after udev add) so the rendered config matched nothing; after removing set-name, boot 2 STILL failed because the early-boot netplan generator ran against the OLD config and renamed the devices to mgmt/uplink before cloud-init wrote the new file (one-boot lag, measured in syslog). Boot 3 (current config, no renames) is the verification boot.
  • FIX (this commit + applied): network_config drops set-name, keeps MEASURED kernel names; instance-id bumped vvr1-dc0-d123 -> -d123-r2 (side effects noted in-block); seed replaced via explicit -replace=module.vvr1_dc0.libvirt_volume.seed (the sanctioned override of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2 exact (docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt).
  • CONSEQUENCE for step B: bootstrap runs with --transit-if enp1s0 --uplink-if enp2s0 (script defaults mgmt/uplink no longer match reality); runbook prose naming "mgmt" needs a DOCFIX at stage close.
  • Revert: git revert of the netplan/instance-id edit + one more seed -replace apply + reboot.

4b. VERIFICATION (2026-07-20, logged window)

  • Boot 3 with the fixed config: enp1s0 UP 172.31.0.2/30, region route present, br-vr1-dc0-wan up.
  • voffice1 -> 172.31.0.2 ping 0% loss; ssh -i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2 works (D-126 -J shape, per-env key); rack -> region ping 10.10.0.20 0% loss (the MAAS-enrollment path for the G10 bootstrap).
  • Outer plan CONVERGED back to ZERO DIFF (docs/audit/outer-plan-20260720-postfix-converged.txt); CURRENT-STATE section 5 updated in the same commit.
  • Follow-items (logged, not executed): D-126 DC ssh-config alias row for vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key); cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC pinning; DOCFIX for runbook "mgmt/uplink" naming.

5. Host tooling: libguestfs-tools installed on vcloud (operator-run)

  • sudo apt-get install -y libguestfs-tools (1:1.52.0-5ubuntu3) -- enables offline guest inspection (virt-cat/virt-ls), used for items 4's captures. Standard KVM-host tooling; Roosevelt-transferable.
  • Revert: sudo apt-get remove libguestfs-tools.