Newer
Older
openstack-caracal-dc-dc / docs / archive / changelogs / changelog-20260721-close-and-delivery.md

Session changelog 2026-07-21 (successor session) -- disconnect recovery + queued delivery

One changelog per session (GA-R2/D1). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts. Predecessor session (ops-commissioning-diag, disconnected before its bookend): docs/changelog-20260721-commissioning-diag.md.

1. Predecessor session close LANDED (commit 8b3adb3)

  • The ops-commissioning-diag session disconnected before committing its deliverable. This session re-verified read-only first (9/9 nodes Ready in MAAS with virsh power; dc0-node-dns active+enabled with the 10.12.8.3 alias up; dhcpd running on BOTH controllers; diag-serial tag deleted and serial-console re-attached to hardy-dove = Phase-7 cleanup confirmed done), then committed the predecessor's 7 files verbatim plus the bounded session-ledger close entry and a machine-derived block re-seed from the 2026-07-21 scan (D next-free 132, DOCFIX 197, BUNDLEFIX 052, 10 open SEC rows). Operator-approved this session.
  • Revert: git revert 8b3adb3 (returns the record to the not-yet-durable state; the live cloud is untouched by that commit).

2. MAC pinning in modules/node-vm (queued item 9a of the predecessor)

  • modules/node-vm: new OPTIONAL interface_macs variable (list(string), default [] = unpinned for pre-enlistment use; validations: all-or- nothing length vs network_names -- partial pinning is rejected -- and 6-byte hex format). Interfaces comprehension consumes it as mac = { address = ... } -- shape VERIFIED against the provider schema (dmacvicar/libvirt 0.9.8, tofu providers schema -json, this session).
  • opentofu/vr1-dc0-substrate: all 9 nodes pinned. 54 MACs MEASURED this session (virsh domiflist on vvr1-dc0, read-only; order = the vr1_dc0_node_nics plane order) and the 9 metal-admin entries cross-checked against MAAS boot-interface records 9/9 (machines read). No value inferred; no value reused from the adjudication file without re-measurement.
  • NEW harness tests/node-vm/run-tests.sh (12 cases: variable shape, validations, consumption, 9 lists / 54 literals / 0 duplicates in the inner root, pass-through, tofu validate). Gauntlet 72 -> 73 ALL GREEN; repo-lint 0 fail; opentofu-validate PASS (both EXTRA roots).
  • NOT YET APPLIED: the inner root's state (on voffice1) does not carry the pins until an operator-gated tofu apply there. The plan gate for that apply: expected NO REPLACE -- pins equal live values, so the plan must show only benign in-place mac adoption or no-op; ANY replace is a STOP (present to operator, do not apply).
  • Revert: git revert of this delivery commit (module + root + harness together); live domains unaffected until the gated apply.

3. node-vm power-ownership guard (operator-ruled this session)

  • The MAC-pin verification plan on voffice1 (captured: docs/audit/inner-plan-20260721-macpin.txt -- 0/9/0, 54 mac adoptions, ZERO replaces) exposed an entanglement: the module hardcodes running = true while MAAS holds all 9 Ready nodes OFF, so the apply would ALSO have powered on all 9 nodes outside MAAS's control.
  • RULING (this session): question = how to proceed with the entangled apply; operator selection = "ignore_changes on running first (Recommended)".
  • Shipped: lifecycle { ignore_changes = [running] } on the node domain (create still boots for PXE enlistment; after that MAAS owns power -- virsh here, IPMI on Roosevelt). Harness grew 3 cases (T13-T15: guard present, exactly one ignore_changes, cites MAAS). 15 cases green.
  • Revert: remove the lifecycle block + T13-T15; the 9 power-on changes return to the next plan.

4. MAC-pin apply EXECUTED on voffice1 (operator-approved)

  • Guarded re-plan captured (inner-plan-20260721-macpin-guarded.txt: 54 mac adoptions, zero power flips, zero replaces), then saved-plan apply (tofu plan -out -> tofu show re-verify 0/9/0 -> tofu apply): exact 0/9/0, convergence re-plan "no differences" (docs/audit/inner-apply-20260721-macpin.txt). Post-apply verified read-only: all 9 domains still shut off, edge untouched, spot-checked MACs identical (control-01, storage-04).
  • The inner tfstate on voffice1 now carries the pins (state serial advanced; it remains the state-of-record per CURRENT-STATE section 1).
  • Revert: git revert the pinning commits, re-plan/apply on voffice1 (returns MACs to provider-owned -- NOT recommended; reopens the drift hole that caused the 2026-07-21 incident).

5. dc-rack-net.sh: D-131 sub-1 delivery (forwarder + rack legs, repo-carried)

  • D-131 sub-1 RULED this session (standing per-DC pattern; ruling commit fad3007), then delivered: NEW scripts/dc-rack-net.sh -- site-keyed (dc0 rows MEASURED), runs ON the rack via piped ssh. Owns: -rack-legs.service oneshot (re-adds 10.12.8.2/22 + 10.12.4.2/22 + the 10.12.8.3/22 forwarder alias on boot -- the D-126 reboot-loss class), the D-131 dnsmasq forwarder unit+conf (generated, content-checked by diff in check mode), and a boot helper that resolves bridges FROM LIBVIRT NET NAMES at runtime (virbrN is auto-assigned = drifting ID; measured: dc-planes does NOT pin bridge names). Region-side MAAS subnet setting stays a runbook step (same split as site-headend-install's DHCP note).
  • Delta from the live hand-placed artifacts (intentional, minimal): the legs move from bare ip addr (reboot-lost) into the oneshot unit; the dns unit gains Requires/After on the legs unit instead of its own ExecStartPre alias hack; config content otherwise identical.
  • NEW tests/dc-rack-net harness (14 cases incl. no-virbrN-literal, MEASURED-tag discipline, read-only check path). Gauntlet 73 -> 74 ALL GREEN; repo-lint 0 fail.
  • Rack install EXECUTED (operator-approved, same session): EXIT 0, self-check PASS 10/10 (docs/audit/dc-rack-net-install-20260721.txt); behavioral probe: dig @10.12.8.3 maas-internal SOA answers through the region. dnsmasq restart blip occurred with all 9 nodes powered off (zero impact). The interim hand-placed unit/conf are OVERWRITTEN in place by the generated equivalents (same paths).
  • Revert: git rm the script + harness; on the rack (if installed): systemctl disable --now dc0-rack-legs dc0-node-dns; rm the four installed files; re-add the legs + old unit per changelog-20260721- commissioning-diag item 4's revert.

6. Incident documentation (queued items 9b/9c/10 of the predecessor)

  • appendix-A: two new entries -- (a) 30-min timeout with NO PXE event for a known machine -> MAC-record drift (diff domiflist vs MAAS interfaces; repair + pin); (b) cloud-init "no datasource" on a rack-only DC -> agent-resolver SERVFAIL (dig SOA check; D-131 forwarder fix via dc-rack-net.sh).
  • platform-traps 1e: second corollary (in-place apply REGENERATES unpinned NIC MACs; pin rule) + verbatim-index row.
  • LP draft authored: docs/audit/lp-draft-20260721-maas-agent-resolver.md (operator to file; sanitization warning re SEC-014 material included).
  • Revert: git revert this commit (docs only).

7. Step-E netem sudo mechanism (operator-ruled) + sudoers fragment

  • RULING (this session, OPS -- netem is VR-only, no D-number per GA-R3): question = sudo mechanism for modules/netem-link on vcloud; operator selection = "Scoped NOPASSWD for tc (Recommended)".
  • Shipped scripts/sudoers.d/netem-tc: exactly the module's two verbs (qdisc replace ... netem * / qdisc del ... root) per MEASURED mesh bridge (virbr5=dc0-dc1 [the step-E target], virbr7=dc0-office1, virbr3=dc1-office1; virsh net-dumpxml this session). Bridge names are auto-assigned/drifting (mesh-link does not pin them) -- fragment documents the fail-closed property + mandatory pre-apply re-measure.
  • NEW tests/netem-sudoers harness (9 cases incl. exact-shape rule match, no-broadening, visudo -cf). Gauntlet 74 -> 75 ALL GREEN.
  • Install is operator-only (root; visudo-checked): command in the fragment header. NOT installed at time of this entry.
  • Revert: git rm fragment + harness; if installed: sudo rm /etc/sudoers.d/netem-tc && sudo visudo -c.