Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260719-dc0-deploy-stepB.md
@JANeumatrix JANeumatrix 1 day ago 20 KB D-125 egress gate executed (throwaway VM, as ruled): bridge-in PROVEN, upstream NAT egress fails -- gate OPEN pending one sudo measurement
Raw Blame History

Changelog -- 2026-07-19 session: DC0 deploy step B (transit reach + bootstrap)

Session changelog (GA-R2/D1: ONE per session). Logged window: dc0-deploy (same-day append, per the A/B-boundary handoff). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts.

1. RULING RECORD (GA-R5) -- Office1 transit-end mechanism (OPS, under D-124)

  • Question as presented (2026-07-19): "Who should hold 172.31.0.1 (the Office1 end of the dc0 transit /30), and by what mechanism? This is the prerequisite for all of step B -- both ssh reach to vvr1-dc0 and the MAAS rack-to-region enrollment path."
  • Operator answer, exact utterance (option selected): "voffice1 via IaC + restart (Recommended)" -- add the mesh NIC + 172.31.0.1/30 netplan to module "voffice1" in the outer root; apply in a planned voffice1 restart window (graceful shutdown first; region/NetBox/Tailscale return via proven autostart).
  • Classification: OPS mechanism decision implementing D-124's already-ruled addressing ("region/peer .1" -- docs/design-decisions.md:3727); no new D-number (GA-R3, doubt resolves DOWN).
  • Measured basis (this session, logged): mesh-vr1-dc0-office1 (virbr7) is ISOLATED (no <forward>/<ip>) with exactly ONE attached interface (vvr1-dc0 vnet3); voffice1 + office1-opnsense have NO mesh NIC; nothing holds 172.31.0.1; vvr1-dc0's applied transit leg = 172.31.0.2/30 with only route 10.10.0.0/22 via 172.31.0.1 (d124-rack.auto.tfvars, consumed by step A). Item-20 disposition: with voffice1 holding .1, vvr1-dc0 is reached by -J voffice1 (D-126 shape) -- NO vcloud host leg / site-baseleg DC row needed for this net.

2. Outer root: voffice1 grows the transit NIC (AUTHORED; apply gated)

  • What: opentofu/main.tf module "voffice1": network_names gains module.mesh_vr1_dc0_office1.network_name SECOND (office1-local stays first -- PCI order preserves enp1s0); network_config rewritten from the single-NIC "en*" glob to per-NIC name matches (lan=enp1s0 dhcp4; transit=enp2s0 static ${vr1_dc0_rack_transit_peer_ip}/${prefix} -- same D-124 tfvars the rack consumes, no new literal). Naming-trap + no-cloud-init- re-run caveats documented in-block: the seed netplan is REBUILD correctness; the live transit config is an in-guest netplan drop-in at the gated attach step.
  • Why: section 1's measured gap -- step B (ssh reach + rack enrollment) has no L3 path until the region end of the transit exists.
  • Revert: restore network_names = [module.office1_network.network_name] and the previous "en*" glob network_config (git revert of this commit); if already applied, a follow-up plan shows the interface removal and the in-guest drop-in /etc/netplan/60-transit.yaml is deleted.

3. APPLIED: voffice1 transit NIC + incident: MAC regen broke the Kea reservation

  • Gated window executed 2026-07-19/20 (logged): graceful shutdown -> apply (1/1/1 exact, docs/audit/outer-apply-20260719-voffice1-transit.txt) -> domain up with both NICs, office1-local index 0.
  • INCIDENT (found+fixed in-window): the provider REGENERATED both NIC MACs on the interface-list rebuild (old lan MAC 52:54:00:6a:87:e5 -> new 52:54:00:89:e1:19). The Kea reservation (MAC-keyed) missed and voffice1 leased pool address .108 (measured via edge ARP). FIX: re-keyed reservation uuid b12621e6-63e9-4714-9726-2e130c84e069 to the new MAC via the D-113 API (kea/dhcpv4/set_reservation -> saved; kea/service/reconfigure -> ok), then a detached networkctl reconfigure enp1s0 -> voffice1 back on 10.10.0.20.
  • LOGGED FINDING (not executed): modules/cloudinit-vm does not pin MACs, so ANY interface-list change re-rolls them and breaks MAC-keyed DHCP reservations. Roosevelt-relevant; candidate module mac var or a documented ops step. Also: live transit config landed as in-guest netplan drop-in /etc/netplan/60-transit.yaml (narrows the lan glob to enp1s0 + transit enp2s0 static) -- the drop-in must ALSO narrow lan, else the 50-cloud-init en* glob claims enp2s0 (measured: first drop-in attempt without it left enp2s0 addressless).
  • Revert: kea/dhcpv4/set_reservation back to the old MAC (senseless unless the NIC change is also reverted); delete the drop-in.

4. vvr1-dc0 transit dead on arrival -- ROOT CAUSE (netplan set-name) + fix

  • Symptom: 172.31.0.2 unreachable; MEASURED: both rack NICs emitted ZERO frames ever (host FDB virbr7/virbr4, voffice1 ARP FAILED); guest OS alive (ACPI-responsive; cloud-init completed per offline logs).
  • Guest is a SEALED BOX: modules/cloudinit-vm domains have NO serial console, NO qga channel, NO graphics; qga is NOT in the noble base image and its cloud-init install failed (no egress) -- so no console, no agent, no network. LOGGED FINDING: add opt-in serial console (+ the D-129-style qga channel var) to modules/cloudinit-vm -- observability gap bites every future DC VM. NOT executed mid-step.
  • Offline inspection (libguestfs, operator-run, captures ~/vvr1-dc0-offline-inspect{,2}.txt): NIC names enp1s0/enp2s0 CORRECT (naming trap closed); root cause = netplan match.name + set-name: boot 1 renames don't apply (.link written after udev add) so the rendered config matched nothing; after removing set-name, boot 2 STILL failed because the early-boot netplan generator ran against the OLD config and renamed the devices to mgmt/uplink before cloud-init wrote the new file (one-boot lag, measured in syslog). Boot 3 (current config, no renames) is the verification boot.
  • FIX (this commit + applied): network_config drops set-name, keeps MEASURED kernel names; instance-id bumped vvr1-dc0-d123 -> -d123-r2 (side effects noted in-block); seed replaced via explicit -replace=module.vvr1_dc0.libvirt_volume.seed (the sanctioned override of D-130 ignore_changes for INTENDED seed changes) -- plan/apply 2/0/2 exact (docs/audit/outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt).
  • CONSEQUENCE for step B: bootstrap runs with --transit-if enp1s0 --uplink-if enp2s0 (script defaults mgmt/uplink no longer match reality); runbook prose naming "mgmt" needs a DOCFIX at stage close.
  • Revert: git revert of the netplan/instance-id edit + one more seed -replace apply + reboot.

4b. VERIFICATION (2026-07-20, logged window)

  • Boot 3 with the fixed config: enp1s0 UP 172.31.0.2/30, region route present, br-vr1-dc0-wan up.
  • voffice1 -> 172.31.0.2 ping 0% loss; ssh -i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@172.31.0.2 works (D-126 -J shape, per-env key); rack -> region ping 10.10.0.20 0% loss (the MAAS-enrollment path for the G10 bootstrap).
  • Outer plan CONVERGED back to ZERO DIFF (docs/audit/outer-plan-20260720-postfix-converged.txt); CURRENT-STATE section 5 updated in the same commit.
  • Follow-items (logged, not executed): D-126 DC ssh-config alias row for vvr1-dc0 (now measurable: 172.31.0.2 via -J voffice1, dc0 key); cloudinit-vm serial-console/qga observability vars; cloudinit-vm MAC pinning; DOCFIX for runbook "mgmt/uplink" naming.

4c. Step B bootstrap EXECUTED on vvr1-dc0 (2026-07-20, logged window)

  • Snap proxy set to the region maas-proxy (measured serving: HTTP 200 to archive.ubuntu.com, CONNECT to api.snapcraft.io) -- the D-107-shaped answer to the rack's no-egress bootstrap; apt used the same proxy via per-invocation env. Enrollment secret piped region->rack (install -m600 /dev/stdin), verified by byte-count only (32), never in context.
  • Run 1 (docs/audit/stepB-bootstrap-20260720.txt): maas snap 3.7/stable installed, rack ENROLLED to http://10.10.0.20:5240/MAAS; FAILED at qemu apt install on stale image package lists (404s via proxy). Fix: apt-get update (proxied), re-run.
  • Run 2 (docs/audit/stepB-bootstrap-20260720-run2.txt): EXIT 0 -- libvirt/qemu + nested KVM on, inner pool + AppArmor, SEC-010 FORWARD-drop loaded on enp1s0 + sec010-fw.service enabled (boot-persistent), br-vr1-dc0-wan verified with enp2s0 enslaved.
  • --check (docs/audit/stepB-check-20260720.txt): all items ok EXCEPT opnsense base (the remaining step-B item); rackd running, enrolled.
  • Region-side residue for the MAAS phase (per script exit text): enable DHCP on metal-admin at the Office1 region naming this rack primary_rack (step D territory, NOT done).
  • Revert: rack enrollment: region-side maas ... rack-controllers delete; SEC-010: disable sec010-fw.service + rm /etc/nftables-sec010.nft; packages: apt/snap remove; snap proxy: snap unset system proxy.http proxy.https.

4d. RULING RECORD (GA-R5) -- DC edge base version (2026-07-20)

  • Question as presented: reuse verified 26.1 artifact vs fresh download (first exchange); then, sharpened after the operator noted "Office1 is running OPNsense 26.7": 26.1-base-then-upgrade vs fresh 26.7 nano base.
  • Operator answer, exact utterance (option selected): "Fresh 26.7 nano base".
  • Consequences implemented: prep runs 26.7 -> opnsense-26.7-nano.qcow2; vr1-dc0-substrate opnsense_base_path default updated (no inner state exists; pre-apply edit); step-B --check passes --opnsense-base for the 26.7 path. LOGGED follow-items: site-headend-install.sh's 26.1 default + hint strings (script change + harness, stage close); the D-112/D-113 serial/API boot path is REVALIDATED on 26.7 at the step-C boot gate (office1 precedent: 26.1-proven, upgraded in place to 26.7).
  • Mirror: https://mirror.ams1.nl.leaseweb.net/opnsense -- MEASURED from the vendor's live download page this session (not memory), HEAD-verified serving the 26.7 nano (~468 MiB). LOGGED FINDING: opnsense-prep-image.sh performs no checksum verification of the downloaded image (same property as the original 26.1 prep; candidate hardening).

4e. Step B CLOSED: 26.7 base staged; final check EXIT 0; SEC-010 CLOSED both ends

  • OPNsense 26.7 nano prepped ON the rack (leaseweb mirror via region proxy, 468M; python3 bz2 fallback -- bunzip2 never needed, DOCFIX-184; docs/audit/stepB-opnsense-prep-20260720.txt). Inner-root opnsense_base_path default -> 26.7 (validate PASS).
  • Final --check EXIT 0, ALL items ok (docs/audit/stepB-check-20260720-final.txt).
  • SEC-010 voffice1-side pin (the row's "same pin follows" clause, due since this session wired voffice1's transit leg; voffice1 runs LXD so ip_forward=1): identical scoped nft artifact + sec010-fw.service on enp2s0, verified loaded + persistent; rack proxy/region paths re-verified working post-pin. SEC-010 row CLOSED (operator ruling, utterance in the row). Open SEC count 8 -> 7.
  • LOGGED follow-item: site-headend-install.sh region-role support for the transit-if pin (voffice1's was hand-applied to the script's exact artifact shape; fold into the script + harness at stage close).
  • Revert: rm the two files + systemctl disable sec010-fw on voffice1; nft delete table inet sec010; ledger row reopens citing this item.

7. Step C: voffice1 becomes the Plane-2 executor; INNER APPLY COMPLETE (28/28)

  • voffice1 setup (gated, logged): tofu 1.12.4 via vcloud's exact apt source (keyrings byte-copied; version == section-7 pin); repo cloned at 29cf7bf (branch); dc0 private key staged to ~/vr1-dc0-creds/ (0700/0600, piped -- SEC-009 convention now spans hosts); ssh config entry for 172.31.0.2; known_hosts seeded via ssh-keyscan with the ED25519 fingerprint cross-checked against the accept-new-trusted entry (MATCH).
  • Inner-root fixes measured at the FIRST REAL dial/apply (each committed): (i) provider needs keyfile+sshauth=privkey URI params -- its Go ssh ignores ~/.ssh/config AND default identities; single-type known_hosts fails as "key mismatch" (multi-type scan required); (ii) modules/wan-bridge: <mtu> is ILLEGAL in bridge-mode networks -- removed (MTU belongs to the host bridge; DOCFIX-194 class: parsed, never applied); (iii) base-image location premise WRONG under remote provider -- content UPLOADS from the executing host; 26.7 qcow2 copied rack->voffice1 (~/vr1-dc0-images/), tfvar override + variables.tf description corrected; (iv) edge serial-log dir /var/lib/libvirt/vr1/staging/ is a HARDCODED vcloud literal in modules/opnsense-edge (:223) -- dir created on the rack (755); LOGGED findings: parameterize the path; fold staging-dir creation into site-headend-install --host-nodes; opentofu-validate.sh does not cover the inner root.
  • Applies (captures docs/audit/inner-{plan,apply}-20260720-stepC*.txt): run 1 = 25/28 (nodes+planes+pool; wan-bridge mtu abort); run 2 = wan bridge (edge disk stat abort); run 3 = edge disk (staging-dir abort; provider self-cleaned the failed domain -- operator undefine found nothing); run 4 = edge domain, EXIT 0. CONVERGED: inner plan zero diff (docs/audit/inner-converge-20260720-stepC.txt).
  • VERIFIED: 10/10 domains RUNNING inside vvr1-dc0 (9 nodes depth-4 + edge); edge serial log = FreeBSD login prompt on ttyu0 -- 26.7 boots the D-112 serial path (first revalidation datapoint; full D-112(c) bootstrap is the next window).
  • NEW STATE OF RECORD: the inner tfstate lives on voffice1 (~/openstack-caracal-dc-dc/opentofu/vr1-dc0-substrate/terraform.tfstate) -- back it up with the site; follow-item for the backup set.
  • Revert: tofu destroy in the inner root from voffice1 (site-down alternative: virsh destroy vvr1-dc0, D-122); module fixes revert by commit.

9. DC edge: D-112(c) console bootstrap + API key minted (26.7) -- REPLACEMENT CHAIN PROVEN

  • Per-env edge key minted on vcloud (~/vr1-dc0-creds/vr1-dc0-edge_ed25519, 0600/0644, SEC-009 convention), staged to the rack the same way.
  • Rack legs added (operator-ruled "10.12.4.2, manual now + script later"): 10.12.4.2/22 on the provider-public inner bridge (virbr5, MEASURED) + the D-124-ruled 10.12.8.2/22 on metal-admin (virbr2). INTERIM: ip addr add, not persistent. LOGGED FINDING (queued): site-headend-install.sh never implemented the rack legs despite main.tf's comment claiming it consumes vr1_dc0_rack_metal_admin_ip -- add rack-leg support + harness before step D closes. Also interim: 192.168.1.2/24 on virbr5 to reach the edge's FACTORY LAN; REMOVE once the edge is re-addressed.
  • D-112(c) console bootstrap: DONE (~/d112c-console.log on the rack). Driver evolved v1->v6 against MEASURED failures, each real and worth keeping for the runbook: (i) serial console wraps long lines -> ship payload as <=160-char base64 chunks (openssl base64 -d), never one long line; (ii) set-name in the edge netplan is irrelevant here but the same class bit the rack (item 4); (iii) tcsh on the edge (documented trap) breaks nested-quoted sed -- "Unmatched '"'"'" -- so NEVER patch files via quoted one-liners; re-ship the whole payload instead; (iv) write_config() needs an include beyond config.inc -- MEASURED (grep -l shell_safe /usr/local/etc/inc/*.inc -> auth/console/filter/ interfaces/rrd/system/util.inc), not guessed; (v) writing config.xml does NOT materialize /root/.ssh/authorized_keys -- must call the vendor's local_user_set() (auth.inc). This was the actual reason key auth failed after a "successful" config write.
  • VERIFIED: ssh -i vr1-dc0-edge_ed25519 root@192.168.1.1 -> key-only auth OK, uname -r = 15.1-RELEASE-p1, ifconfig -l = vtnet0/vtnet1. INTERFACE MAPPING MEASURED: vtnet0 = LAN = provider-public plane (it answered on virbr5), vtnet1 = WAN -- same order as the office1 edge.
  • API key minted via the vendor model (opnsense-bootstrap-apikey.sh
    • opnsense-mint-apikey.php, run ON the rack): key/secret 80 chars -> ~/vr1-dc0-creds/opnsense-api.txt (0600, secret never printed). SMOKE TEST PASS: GET core/firmware/status -> HTTP 200, CORE_ABI 26.7. This is the first proof the D-113(a2) API path works on 26.7 (the 26.7 revalidation the operator's base-version ruling called for). Script note: the bootstrap script requires the minter BESIDE it (its own dir), not in scripts/ -- worth a usage-line DOCFIX.
  • Revert: delete the API key (POST auth/user/del_api_key/<id>, the proven negative-tested path) + creds file; ip addr del the three interim addresses; the console bootstrap reverts by re-imaging the edge (inner tofu destroy/re-apply of the edge module).

10. D-125 egress isolation gate: EXECUTED as written -- PARTIAL (bridge-in PROVEN,

upstream NAT egress FAILS). Gate NOT closed; double-NAT fallback NOT invoked.

  • Method: operator-ruled "throwaway VM, as written" -- noble cloud image fetched ON the rack via the region proxy, one cloud-init guest, SINGLE NIC bridged to br-vr1-dc0-wan, serial-log probe, then destroyed (leftover-domains=0 asserted). Script: scratchpad d125-egress-test.sh; capture docs/audit/d125-egress-gate-20260720.txt.
  • Run 1 was a TEST ARTIFACT, not a gate failure: the probe used DHCP and came up address-less. MEASURED cause: vr1-dc0-uplink carries an <ip> but NO <dhcp> block -- there is no DHCP server on that /24 by design (the edge WAN takes a STATIC .2, D-113/D-125). DOCFIX QUEUED: the gate's wording "must GET a vcloud-ISP address" implies DHCP; it must say STATIC, or the /24 needs a range. Re-run used static 172.30.2.50/24.
  • RESULT (run 2): guest configured 172.30.2.50/24 + default via 172.30.2.1; gateway ping 3/3, 0% loss -- a FOREIGN MAC inside vvr1-dc0 reaches the vcloud ISP NAT across the nested bridge. That is the D-125 bridge-in property, and it PASSES. But ping 1.1.1.1 FAILED (NET-PING-RC=1).
  • Follow-up measurement (rack bridge, 172.30.2.51, self-cleaning): ICMP AND TCP (curl http://1.1.1.1 -> 000) both fail past the gateway, so it is not ICMP-specific. Controls: vcloud egress 0% loss; the office1 edge egresses through the SAME-SHAPE office1-wan NAT 0% loss. The two network definitions are byte-equivalent in substance (net-dumpxml: both forward mode=nat + <nat><port 1024-65535>; only netmask-vs- prefix notation differs) -- so the delta is in vcloud's LIVE firewall rules for virbr4, NOT in the design or the config.
  • Gate status: OPEN (not FAIL). The runbook's FAIL remedy (revert to double-NAT) is for a DESIGN failure; invoking it against what is very likely a host firewall-rule gap would be the wrong remedy and would add Roosevelt delta. Next step is a read-only measurement needing operator sudo: sudo nft list ruleset | grep -nE "172\.30\.(1|2)\.0/24" (or sudo iptables -t nat -S | grep 172.30) comparing the WORKING virbr11 rules against virbr4's.
  • Revert: nothing to revert -- the probe guest and both temporary addresses were removed; the only residue is /tmp/noble.img on the rack (kept deliberately: it is the base for any future throwaway probe).

8. STAGE-CLOSE QUEUE ITEM (operator directive, 2026-07-20): OPNsense 26.7 review

  • Operator directive (verbatim intent): at the proper time/place, review the OPNsense 26.7 changelog + documentation for upgraded features and changes; incorporate beneficial improvements (esp. hardening) into the next deployment iteration, following project rules.
  • Scheduled: STAGE-3 CLOSE (GA-R6 close-out pass, with this stage's queued DOCFIXes). Deliverable: a findings doc citing the official 26.7 release notes; actionable items route per GA-R3 -- hardening/profile deltas as PROPOSED items anchored on D-129 (its 4 open sub-decisions are the likely surface), operational items as runbook DOCFIXes. NO self-executing incorporation: each finding is ruled per GA-R5 (one per exchange) before touching any built surface. Target of incorporation: dc1 standup + Roosevelt runbooks; live-edge retrofits individually gated.

6. ledger-scan hardening: FAIL-OPEN is terminology, not a status

  • SEC-010's close surfaced a scanner false positive: the OPEN-row detector (awk toupper(cell) ~ /OPEN|PENDING/) matched the substring OPEN inside "fail-open" in the closed row's disposition -- the row counted OPEN forever (same class as the GA-F15 wrap trap). Fix: strip FAIL-OPEN from the cell before matching (scripts/ledger-scan.sh); harness gains a CLOSED-row-with-fail-open fixture + nochk (45/45 PASS). Count re-verified: 7 open, SEC-010 absent.
  • Revert: git revert of this commit's ledger-scan.sh + test hunks.

5. Host tooling: libguestfs-tools installed on vcloud (operator-run)

  • sudo apt-get install -y libguestfs-tools (1:1.52.0-5ubuntu3) -- enables offline guest inspection (virt-cat/virt-ls), used for items 4's captures. Standard KVM-host tooling; Roosevelt-transferable.
  • Revert: sudo apt-get remove libguestfs-tools.