Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260721-netem-install-verify.md
@JANeumatrix JANeumatrix 2 hours ago 16 KB STAGE 3 CLOSE (operator-ruled): GA-R2 consolidation + skill sweep; gauntlet 76 ALL GREEN
Raw Blame History

Session changelog 2026-07-21 (third session) -- disconnect collapse + netem install verification

One changelog per session (GA-R2/D1). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts. Predecessor (close-and-delivery session, disconnected after the operator ran the netem-tc install): docs/changelog-20260721-close-and-delivery.md.

1. Lingering-session sweep (read-only; nothing to collapse but records)

  • Verified: working tree clean at pushed HEAD 8a9c04c; zero unpushed commits; single worktree; no stray run-logged windows or tofu processes; branches = stage branch + main only, both synced. The disconnected session's seven deliverable commits were ALL already durable -- the only outstanding items were its ledger bookend and the record of the operator-run netem-tc install.
  • One divergence FLAGGED (logged, not fixed -- queued finding): ledger-scan.sh omits D-131 from its PROPOSED/OPEN list because the Status line begins "PARTIALLY RULED" (D-129's "OPEN / PARTIALLY RULED" phrasing is caught; D-131's is not) even though sub-2..4 are OPEN. Candidate fix: widen the scan's status-line match or adopt a status-wording convention. No information is lost today (the ledger's hand-seeded block carries D-131).

2. netem-tc sudoers install VERIFIED (read-only)

  • The install command itself was operator-run interactively in the predecessor's window (fragment-header command, verbatim). This session re-verified without root: installed file 0440 root:root, byte count identical to the repo fragment (1888), and non-interactive sudo -n -l grant checks exit 0 for both verbs on virbr5 (the step-E target) plus spot checks on virbr7/virbr3 -- which also proves the installed file parses (a sudoers parse error disables the file). Capture: docs/audit/netem-sudo-install-20260721.txt.
  • CURRENT-STATE step-E paragraph corrected in this same commit (GA-R1 C1/C2): install PENDING -> INSTALLED+VERIFIED; remaining path stated (wire modules/netem-link into the outer root from its HELD comment, adds-only plan gate, then the gated placeholder run).
  • Revert: git revert this commit (records only); to remove the live grant: sudo rm /etc/sudoers.d/netem-tc && sudo visudo -c.

3. Predecessor bookend LANDED (this commit)

  • Bounded SESSION CLOSE entry for the close-and-delivery session appended to docs/session-ledger.md (GA-R4; second same-day disconnect-then-successor-lands cycle, same pattern as item 1 of the predecessor's own changelog).
  • Revert: git revert this commit (returns the predecessor to not-durable state; live cloud untouched).

4. modules/netem-link: LOCAL execution mode (D-128 amendment) + NEW harness

  • The module's always-SSH design assumed tofu runs from Office1 (D-103 era); D-128 fixed the outer root's execution ON vcloud, where the mesh bridges live. Measured: no self-ssh path exists on vcloud (no keypair in ~/.ssh, no vcloud/localhost known_hosts entry) -- an SSH self-hop would have required minting a NEW standing credential for nothing. Amendment: vcloud_host_ssh_target now OPTIONAL (default "" = run sudo tc directly on the invoking machine); non-empty keeps the original hop. Both provisioners (apply + destroy) conditional; destroy still references self.* only (DOCFIX-194 class guarded).
  • NEW tests/netem-link/run-tests.sh (12 cases: terraform_data not null_resource, triggers, input-stash/destroy-self discipline, both modes present, default-empty, replace/del||true idempotency, no virbrN literal in module, verbs covered by the sudoers fragment shapes, tofu-validate fixture instantiating BOTH modes). Gauntlet 75 -> 76 ALL GREEN.
  • Revert: git revert this commit's module+harness hunks; the wire block (item 5) must go with it (it relies on the default).

5. netem WIRED in the outer root (step E) -- target virbr5, placeholder profile

  • Replaced the HELD comment (opentofu/main.tf) with module "netem_vr1_dc0_vr1_dc1": bridge_name = virbr5, MEASURED at wire time (virsh net-info mesh-vr1-dc0-vr1-dc1 -> Bridge: virbr5; virbr7/virbr3 re-measured too, all matching the sudoers fragment). netem_args = "delay 3ms 1ms loss 0.01%" -- the RULED PLACEHOLDER (operator ruling 2026-07-16; buildout-design S6 same-metro lean; D-100 gap #11 final numbers remain UNRULED), recorded as PROVISIONAL.
  • FLAGGED runbook divergence (DOCFIX queued): phase2 runbook Step 11 (2026-07-15) targets the dc0<->office1 leg as "the only leg with substrate on both ends"; superseded in fact -- that leg now carries the live rack<->region transit (MAAS, node DNS, inner qemu+ssh), so netem there would perturb live operations. CURRENT-STATE's step-E re-measure + the operator-ruled netem-tc fragment both name virbr5 (dc0<->dc1, zero traffic) as the target; followed those.
  • Revert: tofu destroy -target=module.netem_vr1_dc0_vr1_dc1 (destroy provisioner clears the qdisc: tc qdisc del dev virbr5 root), then git revert the wire hunk.

6. STOP at 1/1/0, operator ruling, targeted apply EXECUTED (step E DONE, G10 CLOSED)

  • The wire plan (docs/audit/outer-plan-20260721-netem-wire.txt) came back 1/1/0 vs the expected adds-only: the extra in-place change is the office1 edge picking up D-129's channels = [] module-schema reconcile (traced to commit f5510c7; the office1 edge's state predates the attribute). Benign in config terms, but an in-place update against the LIVE unpinned-MAC office1 edge = the 07-20 MAC-regen class. STOP honored; presented as a gated decision.
  • RULING (GA-R5, one decision this exchange). Question as presented: "The netem-wire outer plan came back 1 add / 1 change / 0 destroy -- the add is the expected netem resource, but the change is an in-place update on the LIVE office1 edge (state-schema reconcile: channels absent -> [], from the D-129 module change). Section 5 says any non-expected diff is a STOP. How do we proceed?" Operator selection, exact: "Targeted netem apply (Recommended)".
  • Executed: saved -target plan, re-verified via tofu show (exact 1/0/0, only terraform_data.netem), applied. Provisioner ran the ruled command locally through the installed NOPASSWD grant. Step-12 verify: virbr5 carries netem delay 3ms 1ms loss 0.01%; virbr7/virbr3 noqueue (untouched). Convergence re-plan 0/1/0 = exactly the office1 residual. Captures: outer-plan-20260721-netem-targeted.txt, outer-apply-20260721-netem.txt, stepE-netem-20260721.txt, outer-plan-20260721-postE-residual.txt.
  • CURRENT-STATE updated in the same commit (GA-R1 C1): step E DONE, G10 CLOSED, section-5 expected plan re-recorded as 0/1/0, NEW gate G16 (office1 channels reconcile -- remainder split per E3, never a conditional close).
  • Revert: item 5's revert (destroy-target clears the qdisc and the state entry); CURRENT-STATE re-records section 5 back to ZERO DIFF.

7. Queued (logged, not executed)

  • DOCFIX: phase2 runbook Step 11 leg-selection text (item 5's flagged divergence). -> DELIVERED as item 9 below.
  • G16 mechanism ruling: state-only surgery (G6 precedent) vs ride the edge's next scheduled restart (with the D-129 office1 qga retrofit).
  • ledger-scan D-131 status-phrasing blind spot (item 1). -> DELIVERED as item 8 below.

8. ledger-scan: PARTIALLY-RULED decisions now surface (stage-close batch)

  • The last-Status-line match gained PARTIAL alongside PROPOSED|OPEN -- D-131 ("PARTIALLY RULED", the word OPEN on a continuation line) was invisible to the scan while its sub-2..4 are open. Blast radius checked before widening: exactly two PARTIAL status lines exist (D-129 already caught via OPEN; D-131 now caught). Harness gained a regression fixture (D-052, numbered below the fixture high-water so the next-free check stays true) -- 46 checks ALL PASS. Live scan now lists D-068/D-071/D-129/D-131.
  • Revert: git revert this hunk; D-131 disappears from the scan again (the ledger's hand-seeded block still carries it).

9. DOCFIX-197: phase2 runbook Step 11 leg selection corrected to executed reality

  • Two spots (the step-E summary line + the Step 11 body): target is the dc0<->dc1 leg (re-measure bridge at apply time), NOT dc0<->office1, which now carries live rack<->region transit; example block updated to the as-executed module shape (local mode, drifting-ID caveat, capture cites). The netem-parameters PLACEHOLDER note was already accurate and is untouched.
  • Revert: git revert (docs only).

10. D-103 + D-123 amendments and maas-vm-host header: pod-mechanism refutation RECORDED

  • Records the 2026-07-20 MEASURED refutation (snap confinement kills the local pod; qemu+ssh pod dies on volume-ref domblkinfo; pod unnecessary -- PXE delivered discovery) and the already-made 2026-07-20 ruling (per-machine power_type=virsh). This is recording, not a new ruling: both facts and the ruling predate this session and are cited to their captures/changelogs. modules/maas-vm-host header gains a DO-NOT-WIRE-FOR-DC note; retire-or-keep stays an OPEN stage-close ruling (SEC-013 tied).
  • Revert: git revert (docs + one module comment; no resource HCL touched).

11. GA-R7 memory review (stage-close item) DONE

  • Index (MEMORY.md) matches the two entries; both are compliant pointer/convention records (re-vetted in the 07-19 sweep). The one flagged possibly-memory-only fact (sandbox NetBox token location) verified REPO-CARRIED (vr1-office1-as-built rows 130-131) and confirmed live by name-only directory listing; memory updated to drop its caveat. No memory-only facts remain; nothing new needs memorizing (all stage lessons are repo-carried).

12. Stage-close read-only fact gathering (for the pending rulings)

  • Stale MAAS pod object CONFIRMED on the region: vm-hosts read -> id=4 vr1-dc0-inner type=virsh (beside the legitimate office1-lxd id=1). Its deletion is a gated mutation awaiting the retire-or-keep ruling cluster (with SEC-013's on-disk key + tfstate disposition).
  • Manifest gap noted (finding, not fixed): neither creds manifest tracks maas-api-key.txt or maas-virsh_ed25519 in ~/vr1-dc0-creds/ on voffice1 (SEC-012/013 rows are the only tracking). Candidate manifest addition at SEC-012's disposition.

13. RULING: vr1-dc0-maas RETIRED (stage-close ruling 1) + repo removal

  • RULING (GA-R5). Question as presented: "Stage-close ruling 1 of 3: what happens to opentofu/vr1-dc0-maas/ (the refuted pod root)?" with measured context (dead mechanism, stale pod id=4, SEC-013 surfaces). Operator selection, exact: "Retire fully (Recommended)".
  • Executed (repo part): git rm -r opentofu/vr1-dc0-maas/; opentofu-validate PASS with the root gone (validator iterates roots dynamically -- no reference edits needed).
  • Revert: git revert restores the root (history preserves it regardless; D-103/D-123 amendments remain the mechanism authority).

14. INCIDENT: stale-pod delete cascaded to the 9 DC0 machine records

  • Cleanup step 2 (maas admin vm-host delete 4, gated) ALSO deleted the nine DC0 machine records: the failed 2026-07-20 pod refresh had linked the existing machines to the pod (it died on domblkinfo DURING enumeration -- late enough to associate), and MAAS deletes pod-linked machine records on vm-host delete even without decompose.
  • AGENT ERROR, owned: the pod's machine-association list should have been read BEFORE the delete; it was checked after, on the wrong reasoning that a failed registration implies no associations. Lesson for appendix-A (queued): before ANY vm-host delete, read the pod's machines list and expect ZERO; a non-empty list is a STOP.
  • Blast radius measured (capture docs/audit/incident-20260721-pod-delete-cascade.txt): substrate fully intact (10/10 domains inside vvr1-dc0, edge running, nodes shut off; MAC pins config-carried; rack DNS/legs/dhcpd untouched). Lost: MAAS enlistment records, Ready state, per-machine power wiring. CURRENT-STATE corrected in the same commit (GA-R1 C2).
  • Recovery (operator-gated, PENDING): power-on -> PXE re-enlist -> maas-node-power.sh re-wire -> re-commission (the proven 07-21 fleet flow, ~minutes). SEC-013's remaining cleanup steps (tfstate + key file on voffice1) HELD until recovery is decided.
  • Revert: n/a (records); the MAAS-side loss is repaired by the recovery flow, not a git operation.

15. RECOVERY EXECUTED (operator-ruled) -- all 9 nodes Ready again; retire-fully COMPLETE

  • RULING (GA-R5). Question: recover now via the proven 07-21 flow? Operator selection, exact: "Recover now (Recommended)".
  • Executed, all gated: virsh start x9 on the rack -> PXE re-enlist (9/9 in ~2 min; pinned MACs = stable identity) -> voffice1 clone fast-forwarded -> maas-node-power.sh DRY (9/9 MAC-matched) then --commit (9/9 ok, each verified by query-power-state) -> commission x9 -> ALL 9 READY in ~3 min. Shapes verified exact to D-121 Option C (3x16/64GiB, 2x12/48GiB, 4x8/24GiB), power=virsh (docs/audit/incident-20260721-recovery-verify.txt). NOTE: new random MAAS hostnames (ace-robin, alert-cub, ... replace the pre-incident set); nothing durable quoted the old names as identity (the power script is MAC-matched by design).
  • Held cleanup then completed (the rest of ruling 1): voffice1's vr1-dc0-maas dir incl. terraform.tfstate REMOVED; SEC-013 key file ~/vr1-dc0-creds/maas-api-key.txt REMOVED; absence verified by listing (creds dir now: maas-virsh_ed25519 [SEC-012, in use] + the dc0 service keypair). SEC-013 row narrowed to CLI-profile-only; CURRENT-STATE incident paragraph resolved; NEW appendix-A entry (pod-delete cascade: check the pod's machine list BEFORE delete).
  • Revert: n/a for the live repairs; docs via git revert. The deleted key file is not restorable from the repo (by design) -- the operator re-mints from MAAS if a maas-provider root ever returns.

16. G16 CLOSED: office1 channels residual reconciled by state surgery (stage-close ruling 2)

  • RULING (GA-R5). Question as presented: "Stage-close ruling 2: G16 mechanism -- how does the office1 edge's channels = [] state-schema residual get reconciled?" Operator selection, exact: "State surgery (Recommended)".
  • Executed per the G6 precedent: tofu state pull -> verified the attribute is "channels": null on exactly ONE resource (module.office1_opnsense libvirt_domain.vm) -> injected [] -> serial 29 -> 30 -> tofu state push. Backup: opentofu/terraform.tfstate.pre-G16-20260721 (gitignored, verified). Convergence re-plan: no differences (docs/audit/outer-plan-20260721-postG16-converged.txt). Guests never touched (office1-opnsense Id 2 running throughout). CURRENT-STATE section 5 re-recorded to ZERO DIFF + G16 row CLOSED, same commit.
  • Revert: tofu state push opentofu/terraform.tfstate.pre-G16-20260721 (returns serial-29 state; the 0/1/0 residual and gate reopen).

17. STAGE 3 CLOSE (stage-close ruling 3) -- consolidation + skill sweep

  • RULING (GA-R5). Question as presented: "Stage-close ruling 3: close Stage 3 now?" (full close-out set enumerated in the question). Operator selection, exact: "Close and merge (Recommended)".
  • GA-R2 consolidation: the four closed execution-session changelogs (phase5-sweep, dc0-deploy-stepB, commissioning-diag, close-and-delivery -- 1,402 lines) moved to docs/archive/changelogs/; vr1-stage3-partial-record.md finalized as vr1-stage3-record.md with the close summary + closing-era session table. THIS changelog stays top-level until its session closes (record precedent).
  • Status surfaces updated same commit (GA-R1 C1): CURRENT-STATE stage line (CLOSED for vr1-dc0 scope; dc1 = designed HELD remainder G12), workflow doc Stage-3 State row (+ note that its Gate line predates the pod refutation + DOCFIX-197), runbooks/README row 3 (vm_host language corrected) + the stale NONE-EXECUTED banner demoted to a CURRENT-STATE pointer.
  • Skill sweep (GA-R6 close item): folded the D-131 sub-1 standing rack-forwarder invariant, the two MAAS standing traps (pods refuted -> per-machine virsh; pod delete cascades -- check machine list first), and the Stage-3 merge precedent. No stale facts found (the skill delegates facts to the repo by design).
  • Revert: git revert the consolidation commit (moves + record + status lines return); the merge itself is reverted only by operator decision (git revert -m 1 ).