Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260721-netem-install-verify.md
@JANeumatrix JANeumatrix 3 hours ago 9 KB Stage-close batch: ledger-scan PARTIAL fix; DOCFIX-197; D-103/D-123 pod-refutation amendments; GA-R7 memory review
Raw Blame History

Session changelog 2026-07-21 (third session) -- disconnect collapse + netem install verification

One changelog per session (GA-R2/D1). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts. Predecessor (close-and-delivery session, disconnected after the operator ran the netem-tc install): docs/changelog-20260721-close-and-delivery.md.

1. Lingering-session sweep (read-only; nothing to collapse but records)

  • Verified: working tree clean at pushed HEAD 8a9c04c; zero unpushed commits; single worktree; no stray run-logged windows or tofu processes; branches = stage branch + main only, both synced. The disconnected session's seven deliverable commits were ALL already durable -- the only outstanding items were its ledger bookend and the record of the operator-run netem-tc install.
  • One divergence FLAGGED (logged, not fixed -- queued finding): ledger-scan.sh omits D-131 from its PROPOSED/OPEN list because the Status line begins "PARTIALLY RULED" (D-129's "OPEN / PARTIALLY RULED" phrasing is caught; D-131's is not) even though sub-2..4 are OPEN. Candidate fix: widen the scan's status-line match or adopt a status-wording convention. No information is lost today (the ledger's hand-seeded block carries D-131).

2. netem-tc sudoers install VERIFIED (read-only)

  • The install command itself was operator-run interactively in the predecessor's window (fragment-header command, verbatim). This session re-verified without root: installed file 0440 root:root, byte count identical to the repo fragment (1888), and non-interactive sudo -n -l grant checks exit 0 for both verbs on virbr5 (the step-E target) plus spot checks on virbr7/virbr3 -- which also proves the installed file parses (a sudoers parse error disables the file). Capture: docs/audit/netem-sudo-install-20260721.txt.
  • CURRENT-STATE step-E paragraph corrected in this same commit (GA-R1 C1/C2): install PENDING -> INSTALLED+VERIFIED; remaining path stated (wire modules/netem-link into the outer root from its HELD comment, adds-only plan gate, then the gated placeholder run).
  • Revert: git revert this commit (records only); to remove the live grant: sudo rm /etc/sudoers.d/netem-tc && sudo visudo -c.

3. Predecessor bookend LANDED (this commit)

  • Bounded SESSION CLOSE entry for the close-and-delivery session appended to docs/session-ledger.md (GA-R4; second same-day disconnect-then-successor-lands cycle, same pattern as item 1 of the predecessor's own changelog).
  • Revert: git revert this commit (returns the predecessor to not-durable state; live cloud untouched).

4. modules/netem-link: LOCAL execution mode (D-128 amendment) + NEW harness

  • The module's always-SSH design assumed tofu runs from Office1 (D-103 era); D-128 fixed the outer root's execution ON vcloud, where the mesh bridges live. Measured: no self-ssh path exists on vcloud (no keypair in ~/.ssh, no vcloud/localhost known_hosts entry) -- an SSH self-hop would have required minting a NEW standing credential for nothing. Amendment: vcloud_host_ssh_target now OPTIONAL (default "" = run sudo tc directly on the invoking machine); non-empty keeps the original hop. Both provisioners (apply + destroy) conditional; destroy still references self.* only (DOCFIX-194 class guarded).
  • NEW tests/netem-link/run-tests.sh (12 cases: terraform_data not null_resource, triggers, input-stash/destroy-self discipline, both modes present, default-empty, replace/del||true idempotency, no virbrN literal in module, verbs covered by the sudoers fragment shapes, tofu-validate fixture instantiating BOTH modes). Gauntlet 75 -> 76 ALL GREEN.
  • Revert: git revert this commit's module+harness hunks; the wire block (item 5) must go with it (it relies on the default).

5. netem WIRED in the outer root (step E) -- target virbr5, placeholder profile

  • Replaced the HELD comment (opentofu/main.tf) with module "netem_vr1_dc0_vr1_dc1": bridge_name = virbr5, MEASURED at wire time (virsh net-info mesh-vr1-dc0-vr1-dc1 -> Bridge: virbr5; virbr7/virbr3 re-measured too, all matching the sudoers fragment). netem_args = "delay 3ms 1ms loss 0.01%" -- the RULED PLACEHOLDER (operator ruling 2026-07-16; buildout-design S6 same-metro lean; D-100 gap #11 final numbers remain UNRULED), recorded as PROVISIONAL.
  • FLAGGED runbook divergence (DOCFIX queued): phase2 runbook Step 11 (2026-07-15) targets the dc0<->office1 leg as "the only leg with substrate on both ends"; superseded in fact -- that leg now carries the live rack<->region transit (MAAS, node DNS, inner qemu+ssh), so netem there would perturb live operations. CURRENT-STATE's step-E re-measure + the operator-ruled netem-tc fragment both name virbr5 (dc0<->dc1, zero traffic) as the target; followed those.
  • Revert: tofu destroy -target=module.netem_vr1_dc0_vr1_dc1 (destroy provisioner clears the qdisc: tc qdisc del dev virbr5 root), then git revert the wire hunk.

6. STOP at 1/1/0, operator ruling, targeted apply EXECUTED (step E DONE, G10 CLOSED)

  • The wire plan (docs/audit/outer-plan-20260721-netem-wire.txt) came back 1/1/0 vs the expected adds-only: the extra in-place change is the office1 edge picking up D-129's channels = [] module-schema reconcile (traced to commit f5510c7; the office1 edge's state predates the attribute). Benign in config terms, but an in-place update against the LIVE unpinned-MAC office1 edge = the 07-20 MAC-regen class. STOP honored; presented as a gated decision.
  • RULING (GA-R5, one decision this exchange). Question as presented: "The netem-wire outer plan came back 1 add / 1 change / 0 destroy -- the add is the expected netem resource, but the change is an in-place update on the LIVE office1 edge (state-schema reconcile: channels absent -> [], from the D-129 module change). Section 5 says any non-expected diff is a STOP. How do we proceed?" Operator selection, exact: "Targeted netem apply (Recommended)".
  • Executed: saved -target plan, re-verified via tofu show (exact 1/0/0, only terraform_data.netem), applied. Provisioner ran the ruled command locally through the installed NOPASSWD grant. Step-12 verify: virbr5 carries netem delay 3ms 1ms loss 0.01%; virbr7/virbr3 noqueue (untouched). Convergence re-plan 0/1/0 = exactly the office1 residual. Captures: outer-plan-20260721-netem-targeted.txt, outer-apply-20260721-netem.txt, stepE-netem-20260721.txt, outer-plan-20260721-postE-residual.txt.
  • CURRENT-STATE updated in the same commit (GA-R1 C1): step E DONE, G10 CLOSED, section-5 expected plan re-recorded as 0/1/0, NEW gate G16 (office1 channels reconcile -- remainder split per E3, never a conditional close).
  • Revert: item 5's revert (destroy-target clears the qdisc and the state entry); CURRENT-STATE re-records section 5 back to ZERO DIFF.

7. Queued (logged, not executed)

  • DOCFIX: phase2 runbook Step 11 leg-selection text (item 5's flagged divergence). -> DELIVERED as item 9 below.
  • G16 mechanism ruling: state-only surgery (G6 precedent) vs ride the edge's next scheduled restart (with the D-129 office1 qga retrofit).
  • ledger-scan D-131 status-phrasing blind spot (item 1). -> DELIVERED as item 8 below.

8. ledger-scan: PARTIALLY-RULED decisions now surface (stage-close batch)

  • The last-Status-line match gained PARTIAL alongside PROPOSED|OPEN -- D-131 ("PARTIALLY RULED", the word OPEN on a continuation line) was invisible to the scan while its sub-2..4 are open. Blast radius checked before widening: exactly two PARTIAL status lines exist (D-129 already caught via OPEN; D-131 now caught). Harness gained a regression fixture (D-052, numbered below the fixture high-water so the next-free check stays true) -- 46 checks ALL PASS. Live scan now lists D-068/D-071/D-129/D-131.
  • Revert: git revert this hunk; D-131 disappears from the scan again (the ledger's hand-seeded block still carries it).

9. DOCFIX-197: phase2 runbook Step 11 leg selection corrected to executed reality

  • Two spots (the step-E summary line + the Step 11 body): target is the dc0<->dc1 leg (re-measure bridge at apply time), NOT dc0<->office1, which now carries live rack<->region transit; example block updated to the as-executed module shape (local mode, drifting-ID caveat, capture cites). The netem-parameters PLACEHOLDER note was already accurate and is untouched.
  • Revert: git revert (docs only).

10. D-103 + D-123 amendments and maas-vm-host header: pod-mechanism refutation RECORDED

  • Records the 2026-07-20 MEASURED refutation (snap confinement kills the local pod; qemu+ssh pod dies on volume-ref domblkinfo; pod unnecessary -- PXE delivered discovery) and the already-made 2026-07-20 ruling (per-machine power_type=virsh). This is recording, not a new ruling: both facts and the ruling predate this session and are cited to their captures/changelogs. modules/maas-vm-host header gains a DO-NOT-WIRE-FOR-DC note; retire-or-keep stays an OPEN stage-close ruling (SEC-013 tied).
  • Revert: git revert (docs + one module comment; no resource HCL touched).

11. GA-R7 memory review (stage-close item) DONE

  • Index (MEMORY.md) matches the two entries; both are compliant pointer/convention records (re-vetted in the 07-19 sweep). The one flagged possibly-memory-only fact (sandbox NetBox token location) verified REPO-CARRIED (vr1-office1-as-built rows 130-131) and confirmed live by name-only directory listing; memory updated to drop its caveat. No memory-only facts remain; nothing new needs memorizing (all stage lessons are repo-carried).

12. Stage-close read-only fact gathering (for the pending rulings)

  • Stale MAAS pod object CONFIRMED on the region: vm-hosts read -> id=4 vr1-dc0-inner type=virsh (beside the legitimate office1-lxd id=1). Its deletion is a gated mutation awaiting the retire-or-keep ruling cluster (with SEC-013's on-disk key + tfstate disposition).