# Session changelog 2026-07-21 (third session) -- disconnect collapse + netem install verification One changelog per session (GA-R2/D1). Status lives ONLY in docs/CURRENT-STATE.md; this file is narrative + reverts. Predecessor (close-and-delivery session, disconnected after the operator ran the netem-tc install): docs/changelog-20260721-close-and-delivery.md. ## 1. Lingering-session sweep (read-only; nothing to collapse but records) - Verified: working tree clean at pushed HEAD 8a9c04c; zero unpushed commits; single worktree; no stray run-logged windows or tofu processes; branches = stage branch + main only, both synced. The disconnected session's seven deliverable commits were ALL already durable -- the only outstanding items were its ledger bookend and the record of the operator-run netem-tc install. - One divergence FLAGGED (logged, not fixed -- queued finding): `ledger-scan.sh` omits D-131 from its PROPOSED/OPEN list because the Status line begins "PARTIALLY RULED" (D-129's "OPEN / PARTIALLY RULED" phrasing is caught; D-131's is not) even though sub-2..4 are OPEN. Candidate fix: widen the scan's status-line match or adopt a status-wording convention. No information is lost today (the ledger's hand-seeded block carries D-131). ## 2. netem-tc sudoers install VERIFIED (read-only) - The install command itself was operator-run interactively in the predecessor's window (fragment-header command, verbatim). This session re-verified without root: installed file 0440 root:root, byte count identical to the repo fragment (1888), and non-interactive `sudo -n -l` grant checks exit 0 for both verbs on virbr5 (the step-E target) plus spot checks on virbr7/virbr3 -- which also proves the installed file parses (a sudoers parse error disables the file). Capture: docs/audit/netem-sudo-install-20260721.txt. - CURRENT-STATE step-E paragraph corrected in this same commit (GA-R1 C1/C2): install PENDING -> INSTALLED+VERIFIED; remaining path stated (wire modules/netem-link into the outer root from its HELD comment, adds-only plan gate, then the gated placeholder run). - **Revert:** git revert this commit (records only); to remove the live grant: sudo rm /etc/sudoers.d/netem-tc && sudo visudo -c. ## 3. Predecessor bookend LANDED (this commit) - Bounded SESSION CLOSE entry for the close-and-delivery session appended to docs/session-ledger.md (GA-R4; second same-day disconnect-then-successor-lands cycle, same pattern as item 1 of the predecessor's own changelog). - **Revert:** git revert this commit (returns the predecessor to not-durable state; live cloud untouched). ## 4. modules/netem-link: LOCAL execution mode (D-128 amendment) + NEW harness - The module's always-SSH design assumed tofu runs from Office1 (D-103 era); D-128 fixed the outer root's execution ON vcloud, where the mesh bridges live. Measured: no self-ssh path exists on vcloud (no keypair in ~/.ssh, no vcloud/localhost known_hosts entry) -- an SSH self-hop would have required minting a NEW standing credential for nothing. Amendment: `vcloud_host_ssh_target` now OPTIONAL (default "" = run `sudo tc` directly on the invoking machine); non-empty keeps the original hop. Both provisioners (apply + destroy) conditional; destroy still references self.* only (DOCFIX-194 class guarded). - NEW tests/netem-link/run-tests.sh (12 cases: terraform_data not null_resource, triggers, input-stash/destroy-self discipline, both modes present, default-empty, replace/del||true idempotency, no virbrN literal in module, verbs covered by the sudoers fragment shapes, tofu-validate fixture instantiating BOTH modes). Gauntlet 75 -> 76 ALL GREEN. - **Revert:** git revert this commit's module+harness hunks; the wire block (item 5) must go with it (it relies on the default). ## 5. netem WIRED in the outer root (step E) -- target virbr5, placeholder profile - Replaced the HELD comment (opentofu/main.tf) with `module "netem_vr1_dc0_vr1_dc1"`: bridge_name = virbr5, MEASURED at wire time (`virsh net-info mesh-vr1-dc0-vr1-dc1` -> Bridge: virbr5; virbr7/virbr3 re-measured too, all matching the sudoers fragment). netem_args = "delay 3ms 1ms loss 0.01%" -- the RULED PLACEHOLDER (operator ruling 2026-07-16; buildout-design S6 same-metro lean; D-100 gap #11 final numbers remain UNRULED), recorded as PROVISIONAL. - FLAGGED runbook divergence (DOCFIX queued): phase2 runbook Step 11 (2026-07-15) targets the dc0<->office1 leg as "the only leg with substrate on both ends"; superseded in fact -- that leg now carries the live rack<->region transit (MAAS, node DNS, inner qemu+ssh), so netem there would perturb live operations. CURRENT-STATE's step-E re-measure + the operator-ruled netem-tc fragment both name virbr5 (dc0<->dc1, zero traffic) as the target; followed those. - **Revert:** `tofu destroy -target=module.netem_vr1_dc0_vr1_dc1` (destroy provisioner clears the qdisc: `tc qdisc del dev virbr5 root`), then git revert the wire hunk. ## 6. STOP at 1/1/0, operator ruling, targeted apply EXECUTED (step E DONE, G10 CLOSED) - The wire plan (docs/audit/outer-plan-20260721-netem-wire.txt) came back 1/1/0 vs the expected adds-only: the extra in-place change is the office1 edge picking up D-129's `channels = []` module-schema reconcile (traced to commit f5510c7; the office1 edge's state predates the attribute). Benign in config terms, but an in-place update against the LIVE unpinned-MAC office1 edge = the 07-20 MAC-regen class. STOP honored; presented as a gated decision. - RULING (GA-R5, one decision this exchange). Question as presented: "The netem-wire outer plan came back 1 add / 1 change / 0 destroy -- the add is the expected netem resource, but the change is an in-place update on the LIVE office1 edge (state-schema reconcile: `channels` absent -> `[]`, from the D-129 module change). Section 5 says any non-expected diff is a STOP. How do we proceed?" Operator selection, exact: "Targeted netem apply (Recommended)". - Executed: saved `-target` plan, re-verified via `tofu show` (exact 1/0/0, only terraform_data.netem), applied. Provisioner ran the ruled command locally through the installed NOPASSWD grant. Step-12 verify: virbr5 carries `netem delay 3ms 1ms loss 0.01%`; virbr7/virbr3 noqueue (untouched). Convergence re-plan 0/1/0 = exactly the office1 residual. Captures: outer-plan-20260721-netem-targeted.txt, outer-apply-20260721-netem.txt, stepE-netem-20260721.txt, outer-plan-20260721-postE-residual.txt. - CURRENT-STATE updated in the same commit (GA-R1 C1): step E DONE, G10 CLOSED, section-5 expected plan re-recorded as 0/1/0, NEW gate G16 (office1 channels reconcile -- remainder split per E3, never a conditional close). - **Revert:** item 5's revert (destroy-target clears the qdisc and the state entry); CURRENT-STATE re-records section 5 back to ZERO DIFF. ## 7. Queued (logged, not executed) - DOCFIX: phase2 runbook Step 11 leg-selection text (item 5's flagged divergence). -> DELIVERED as item 9 below. - G16 mechanism ruling: state-only surgery (G6 precedent) vs ride the edge's next scheduled restart (with the D-129 office1 qga retrofit). - ledger-scan D-131 status-phrasing blind spot (item 1). -> DELIVERED as item 8 below. ## 8. ledger-scan: PARTIALLY-RULED decisions now surface (stage-close batch) - The last-Status-line match gained PARTIAL alongside PROPOSED|OPEN -- D-131 ("PARTIALLY RULED", the word OPEN on a continuation line) was invisible to the scan while its sub-2..4 are open. Blast radius checked before widening: exactly two PARTIAL status lines exist (D-129 already caught via OPEN; D-131 now caught). Harness gained a regression fixture (D-052, numbered below the fixture high-water so the next-free check stays true) -- 46 checks ALL PASS. Live scan now lists D-068/D-071/D-129/D-131. - **Revert:** git revert this hunk; D-131 disappears from the scan again (the ledger's hand-seeded block still carries it). ## 9. DOCFIX-197: phase2 runbook Step 11 leg selection corrected to executed reality - Two spots (the step-E summary line + the Step 11 body): target is the dc0<->dc1 leg (re-measure bridge at apply time), NOT dc0<->office1, which now carries live rack<->region transit; example block updated to the as-executed module shape (local mode, drifting-ID caveat, capture cites). The netem-parameters PLACEHOLDER note was already accurate and is untouched. - **Revert:** git revert (docs only). ## 10. D-103 + D-123 amendments and maas-vm-host header: pod-mechanism refutation RECORDED - Records the 2026-07-20 MEASURED refutation (snap confinement kills the local pod; qemu+ssh pod dies on volume-ref domblkinfo; pod unnecessary -- PXE delivered discovery) and the already-made 2026-07-20 ruling (per-machine power_type=virsh). This is recording, not a new ruling: both facts and the ruling predate this session and are cited to their captures/changelogs. modules/maas-vm-host header gains a DO-NOT-WIRE-FOR-DC note; retire-or-keep stays an OPEN stage-close ruling (SEC-013 tied). - **Revert:** git revert (docs + one module comment; no resource HCL touched). ## 11. GA-R7 memory review (stage-close item) DONE - Index (MEMORY.md) matches the two entries; both are compliant pointer/convention records (re-vetted in the 07-19 sweep). The one flagged possibly-memory-only fact (sandbox NetBox token location) verified REPO-CARRIED (vr1-office1-as-built rows 130-131) and confirmed live by name-only directory listing; memory updated to drop its caveat. No memory-only facts remain; nothing new needs memorizing (all stage lessons are repo-carried). ## 12. Stage-close read-only fact gathering (for the pending rulings) - Stale MAAS pod object CONFIRMED on the region: `vm-hosts read` -> id=4 `vr1-dc0-inner` type=virsh (beside the legitimate office1-lxd id=1). Its deletion is a gated mutation awaiting the retire-or-keep ruling cluster (with SEC-013's on-disk key + tfstate disposition). - Manifest gap noted (finding, not fixed): neither creds manifest tracks `maas-api-key.txt` or `maas-virsh_ed25519` in `~/vr1-dc0-creds/` on voffice1 (SEC-012/013 rows are the only tracking). Candidate manifest addition at SEC-012's disposition. ## 13. RULING: vr1-dc0-maas RETIRED (stage-close ruling 1) + repo removal - RULING (GA-R5). Question as presented: "Stage-close ruling 1 of 3: what happens to `opentofu/vr1-dc0-maas/` (the refuted pod root)?" with measured context (dead mechanism, stale pod id=4, SEC-013 surfaces). Operator selection, exact: "Retire fully (Recommended)". - Executed (repo part): `git rm -r opentofu/vr1-dc0-maas/`; opentofu-validate PASS with the root gone (validator iterates roots dynamically -- no reference edits needed). - **Revert:** git revert restores the root (history preserves it regardless; D-103/D-123 amendments remain the mechanism authority). ## 14. INCIDENT: stale-pod delete cascaded to the 9 DC0 machine records - Cleanup step 2 (`maas admin vm-host delete 4`, gated) ALSO deleted the nine DC0 machine records: the failed 2026-07-20 pod refresh had linked the existing machines to the pod (it died on domblkinfo DURING enumeration -- late enough to associate), and MAAS deletes pod-linked machine records on vm-host delete even without decompose. - AGENT ERROR, owned: the pod's machine-association list should have been read BEFORE the delete; it was checked after, on the wrong reasoning that a failed registration implies no associations. Lesson for appendix-A (queued): before ANY `vm-host delete`, read the pod's machines list and expect ZERO; a non-empty list is a STOP. - Blast radius measured (capture docs/audit/incident-20260721-pod-delete-cascade.txt): substrate fully intact (10/10 domains inside vvr1-dc0, edge running, nodes shut off; MAC pins config-carried; rack DNS/legs/dhcpd untouched). Lost: MAAS enlistment records, Ready state, per-machine power wiring. CURRENT-STATE corrected in the same commit (GA-R1 C2). - Recovery (operator-gated, PENDING): power-on -> PXE re-enlist -> maas-node-power.sh re-wire -> re-commission (the proven 07-21 fleet flow, ~minutes). SEC-013's remaining cleanup steps (tfstate + key file on voffice1) HELD until recovery is decided. - **Revert:** n/a (records); the MAAS-side loss is repaired by the recovery flow, not a git operation. ## 15. RECOVERY EXECUTED (operator-ruled) -- all 9 nodes Ready again; retire-fully COMPLETE - RULING (GA-R5). Question: recover now via the proven 07-21 flow? Operator selection, exact: "Recover now (Recommended)". - Executed, all gated: virsh start x9 on the rack -> PXE re-enlist (9/9 in ~2 min; pinned MACs = stable identity) -> voffice1 clone fast-forwarded -> maas-node-power.sh DRY (9/9 MAC-matched) then --commit (9/9 ok, each verified by query-power-state) -> commission x9 -> ALL 9 READY in ~3 min. Shapes verified exact to D-121 Option C (3x16/64GiB, 2x12/48GiB, 4x8/24GiB), power=virsh (docs/audit/incident-20260721-recovery-verify.txt). NOTE: new random MAAS hostnames (ace-robin, alert-cub, ... replace the pre-incident set); nothing durable quoted the old names as identity (the power script is MAC-matched by design). - Held cleanup then completed (the rest of ruling 1): voffice1's vr1-dc0-maas dir incl. terraform.tfstate REMOVED; SEC-013 key file `~/vr1-dc0-creds/maas-api-key.txt` REMOVED; absence verified by listing (creds dir now: maas-virsh_ed25519 [SEC-012, in use] + the dc0 service keypair). SEC-013 row narrowed to CLI-profile-only; CURRENT-STATE incident paragraph resolved; NEW appendix-A entry (pod-delete cascade: check the pod's machine list BEFORE delete). - **Revert:** n/a for the live repairs; docs via git revert. The deleted key file is not restorable from the repo (by design) -- the operator re-mints from MAAS if a maas-provider root ever returns.