# Session changelog 2026-07-21 (third session) -- disconnect collapse + netem install verification

One changelog per session (GA-R2/D1). Status lives ONLY in
docs/CURRENT-STATE.md; this file is narrative + reverts. Predecessor
(close-and-delivery session, disconnected after the operator ran the
netem-tc install): docs/changelog-20260721-close-and-delivery.md.

## 1. Lingering-session sweep (read-only; nothing to collapse but records)

- Verified: working tree clean at pushed HEAD 8a9c04c; zero unpushed
  commits; single worktree; no stray run-logged windows or tofu
  processes; branches = stage branch + main only, both synced. The
  disconnected session's seven deliverable commits were ALL already
  durable -- the only outstanding items were its ledger bookend and the
  record of the operator-run netem-tc install.
- One divergence FLAGGED (logged, not fixed -- queued finding):
  `ledger-scan.sh` omits D-131 from its PROPOSED/OPEN list because the
  Status line begins "PARTIALLY RULED" (D-129's "OPEN / PARTIALLY
  RULED" phrasing is caught; D-131's is not) even though sub-2..4 are
  OPEN. Candidate fix: widen the scan's status-line match or adopt a
  status-wording convention. No information is lost today (the ledger's
  hand-seeded block carries D-131).

## 2. netem-tc sudoers install VERIFIED (read-only)

- The install command itself was operator-run interactively in the
  predecessor's window (fragment-header command, verbatim). This
  session re-verified without root: installed file 0440 root:root,
  byte count identical to the repo fragment (1888), and non-interactive
  `sudo -n -l` grant checks exit 0 for both verbs on virbr5 (the step-E
  target) plus spot checks on virbr7/virbr3 -- which also proves the
  installed file parses (a sudoers parse error disables the file).
  Capture: docs/audit/netem-sudo-install-20260721.txt.
- CURRENT-STATE step-E paragraph corrected in this same commit (GA-R1
  C1/C2): install PENDING -> INSTALLED+VERIFIED; remaining path stated
  (wire modules/netem-link into the outer root from its HELD comment,
  adds-only plan gate, then the gated placeholder run).
- **Revert:** git revert this commit (records only); to remove the live
  grant: sudo rm /etc/sudoers.d/netem-tc && sudo visudo -c.

## 3. Predecessor bookend LANDED (this commit)

- Bounded SESSION CLOSE entry for the close-and-delivery session
  appended to docs/session-ledger.md (GA-R4; second same-day
  disconnect-then-successor-lands cycle, same pattern as item 1 of the
  predecessor's own changelog).
- **Revert:** git revert this commit (returns the predecessor to
  not-durable state; live cloud untouched).

## 4. modules/netem-link: LOCAL execution mode (D-128 amendment) + NEW harness

- The module's always-SSH design assumed tofu runs from Office1 (D-103
  era); D-128 fixed the outer root's execution ON vcloud, where the
  mesh bridges live. Measured: no self-ssh path exists on vcloud (no
  keypair in ~/.ssh, no vcloud/localhost known_hosts entry) -- an SSH
  self-hop would have required minting a NEW standing credential for
  nothing. Amendment: `vcloud_host_ssh_target` now OPTIONAL (default
  "" = run `sudo tc` directly on the invoking machine); non-empty
  keeps the original hop. Both provisioners (apply + destroy)
  conditional; destroy still references self.* only (DOCFIX-194
  class guarded).
- NEW tests/netem-link/run-tests.sh (12 cases: terraform_data not
  null_resource, triggers, input-stash/destroy-self discipline, both
  modes present, default-empty, replace/del||true idempotency, no
  virbrN literal in module, verbs covered by the sudoers fragment
  shapes, tofu-validate fixture instantiating BOTH modes). Gauntlet
  75 -> 76 ALL GREEN.
- **Revert:** git revert this commit's module+harness hunks; the wire
  block (item 5) must go with it (it relies on the default).

## 5. netem WIRED in the outer root (step E) -- target virbr5, placeholder profile

- Replaced the HELD comment (opentofu/main.tf) with
  `module "netem_vr1_dc0_vr1_dc1"`: bridge_name = virbr5, MEASURED at
  wire time (`virsh net-info mesh-vr1-dc0-vr1-dc1` -> Bridge: virbr5;
  virbr7/virbr3 re-measured too, all matching the sudoers fragment).
  netem_args = "delay 3ms 1ms loss 0.01%" -- the RULED PLACEHOLDER
  (operator ruling 2026-07-16; buildout-design S6 same-metro lean;
  D-100 gap #11 final numbers remain UNRULED), recorded as PROVISIONAL.
- FLAGGED runbook divergence (DOCFIX queued): phase2 runbook Step 11
  (2026-07-15) targets the dc0<->office1 leg as "the only leg with
  substrate on both ends"; superseded in fact -- that leg now carries
  the live rack<->region transit (MAAS, node DNS, inner qemu+ssh), so
  netem there would perturb live operations. CURRENT-STATE's step-E
  re-measure + the operator-ruled netem-tc fragment both name virbr5
  (dc0<->dc1, zero traffic) as the target; followed those.
- **Revert:** `tofu destroy -target=module.netem_vr1_dc0_vr1_dc1`
  (destroy provisioner clears the qdisc: `tc qdisc del dev virbr5
  root`), then git revert the wire hunk.

## 6. STOP at 1/1/0, operator ruling, targeted apply EXECUTED (step E DONE, G10 CLOSED)

- The wire plan (docs/audit/outer-plan-20260721-netem-wire.txt) came
  back 1/1/0 vs the expected adds-only: the extra in-place change is
  the office1 edge picking up D-129's `channels = []` module-schema
  reconcile (traced to commit f5510c7; the office1 edge's state
  predates the attribute). Benign in config terms, but an in-place
  update against the LIVE unpinned-MAC office1 edge = the 07-20
  MAC-regen class. STOP honored; presented as a gated decision.
- RULING (GA-R5, one decision this exchange). Question as presented:
  "The netem-wire outer plan came back 1 add / 1 change / 0 destroy --
  the add is the expected netem resource, but the change is an
  in-place update on the LIVE office1 edge (state-schema reconcile:
  `channels` absent -> `[]`, from the D-129 module change). Section 5
  says any non-expected diff is a STOP. How do we proceed?" Operator
  selection, exact: "Targeted netem apply (Recommended)".
- Executed: saved `-target` plan, re-verified via `tofu show` (exact
  1/0/0, only terraform_data.netem), applied. Provisioner ran the
  ruled command locally through the installed NOPASSWD grant. Step-12
  verify: virbr5 carries `netem delay 3ms 1ms loss 0.01%`;
  virbr7/virbr3 noqueue (untouched). Convergence re-plan 0/1/0 =
  exactly the office1 residual. Captures:
  outer-plan-20260721-netem-targeted.txt, outer-apply-20260721-netem.txt,
  stepE-netem-20260721.txt, outer-plan-20260721-postE-residual.txt.
- CURRENT-STATE updated in the same commit (GA-R1 C1): step E DONE,
  G10 CLOSED, section-5 expected plan re-recorded as 0/1/0, NEW gate
  G16 (office1 channels reconcile -- remainder split per E3, never a
  conditional close).
- **Revert:** item 5's revert (destroy-target clears the qdisc and the
  state entry); CURRENT-STATE re-records section 5 back to ZERO DIFF.

## 7. Queued (logged, not executed)

- DOCFIX: phase2 runbook Step 11 leg-selection text (item 5's flagged
  divergence). -> DELIVERED as item 9 below.
- G16 mechanism ruling: state-only surgery (G6 precedent) vs ride the
  edge's next scheduled restart (with the D-129 office1 qga retrofit).
- ledger-scan D-131 status-phrasing blind spot (item 1). -> DELIVERED
  as item 8 below.

## 8. ledger-scan: PARTIALLY-RULED decisions now surface (stage-close batch)

- The last-Status-line match gained PARTIAL alongside PROPOSED|OPEN --
  D-131 ("PARTIALLY RULED", the word OPEN on a continuation line) was
  invisible to the scan while its sub-2..4 are open. Blast radius
  checked before widening: exactly two PARTIAL status lines exist
  (D-129 already caught via OPEN; D-131 now caught). Harness gained a
  regression fixture (D-052, numbered below the fixture high-water so
  the next-free check stays true) -- 46 checks ALL PASS. Live scan now
  lists D-068/D-071/D-129/D-131.
- **Revert:** git revert this hunk; D-131 disappears from the scan
  again (the ledger's hand-seeded block still carries it).

## 9. DOCFIX-197: phase2 runbook Step 11 leg selection corrected to executed reality

- Two spots (the step-E summary line + the Step 11 body): target is the
  dc0<->dc1 leg (re-measure bridge at apply time), NOT dc0<->office1,
  which now carries live rack<->region transit; example block updated
  to the as-executed module shape (local mode, drifting-ID caveat,
  capture cites). The netem-parameters PLACEHOLDER note was already
  accurate and is untouched.
- **Revert:** git revert (docs only).

## 10. D-103 + D-123 amendments and maas-vm-host header: pod-mechanism refutation RECORDED

- Records the 2026-07-20 MEASURED refutation (snap confinement kills
  the local pod; qemu+ssh pod dies on volume-ref domblkinfo; pod
  unnecessary -- PXE delivered discovery) and the already-made
  2026-07-20 ruling (per-machine power_type=virsh). This is recording,
  not a new ruling: both facts and the ruling predate this session and
  are cited to their captures/changelogs. modules/maas-vm-host header
  gains a DO-NOT-WIRE-FOR-DC note; retire-or-keep stays an OPEN
  stage-close ruling (SEC-013 tied).
- **Revert:** git revert (docs + one module comment; no resource HCL
  touched).

## 11. GA-R7 memory review (stage-close item) DONE

- Index (MEMORY.md) matches the two entries; both are compliant
  pointer/convention records (re-vetted in the 07-19 sweep). The one
  flagged possibly-memory-only fact (sandbox NetBox token location)
  verified REPO-CARRIED (vr1-office1-as-built rows 130-131) and
  confirmed live by name-only directory listing; memory updated to
  drop its caveat. No memory-only facts remain; nothing new needs
  memorizing (all stage lessons are repo-carried).

## 12. Stage-close read-only fact gathering (for the pending rulings)

- Stale MAAS pod object CONFIRMED on the region: `vm-hosts read` ->
  id=4 `vr1-dc0-inner` type=virsh (beside the legitimate office1-lxd
  id=1). Its deletion is a gated mutation awaiting the retire-or-keep
  ruling cluster (with SEC-013's on-disk key + tfstate disposition).
- Manifest gap noted (finding, not fixed): neither creds manifest
  tracks `maas-api-key.txt` or `maas-virsh_ed25519` in
  `~/vr1-dc0-creds/` on voffice1 (SEC-012/013 rows are the only
  tracking). Candidate manifest addition at SEC-012's disposition.

## 13. RULING: vr1-dc0-maas RETIRED (stage-close ruling 1) + repo removal

- RULING (GA-R5). Question as presented: "Stage-close ruling 1 of 3:
  what happens to `opentofu/vr1-dc0-maas/` (the refuted pod root)?"
  with measured context (dead mechanism, stale pod id=4, SEC-013
  surfaces). Operator selection, exact: "Retire fully (Recommended)".
- Executed (repo part): `git rm -r opentofu/vr1-dc0-maas/`;
  opentofu-validate PASS with the root gone (validator iterates roots
  dynamically -- no reference edits needed).
- **Revert:** git revert restores the root (history preserves it
  regardless; D-103/D-123 amendments remain the mechanism authority).

## 14. INCIDENT: stale-pod delete cascaded to the 9 DC0 machine records

- Cleanup step 2 (`maas admin vm-host delete 4`, gated) ALSO deleted
  the nine DC0 machine records: the failed 2026-07-20 pod refresh had
  linked the existing machines to the pod (it died on domblkinfo
  DURING enumeration -- late enough to associate), and MAAS deletes
  pod-linked machine records on vm-host delete even without decompose.
- AGENT ERROR, owned: the pod's machine-association list should have
  been read BEFORE the delete; it was checked after, on the wrong
  reasoning that a failed registration implies no associations. Lesson
  for appendix-A (queued): before ANY `vm-host delete`, read the pod's
  machines list and expect ZERO; a non-empty list is a STOP.
- Blast radius measured (capture
  docs/audit/incident-20260721-pod-delete-cascade.txt): substrate
  fully intact (10/10 domains inside vvr1-dc0, edge running, nodes
  shut off; MAC pins config-carried; rack DNS/legs/dhcpd untouched).
  Lost: MAAS enlistment records, Ready state, per-machine power
  wiring. CURRENT-STATE corrected in the same commit (GA-R1 C2).
- Recovery (operator-gated, PENDING): power-on -> PXE re-enlist ->
  maas-node-power.sh re-wire -> re-commission (the proven 07-21 fleet
  flow, ~minutes). SEC-013's remaining cleanup steps (tfstate + key
  file on voffice1) HELD until recovery is decided.
- **Revert:** n/a (records); the MAAS-side loss is repaired by the
  recovery flow, not a git operation.

## 15. RECOVERY EXECUTED (operator-ruled) -- all 9 nodes Ready again; retire-fully COMPLETE

- RULING (GA-R5). Question: recover now via the proven 07-21 flow?
  Operator selection, exact: "Recover now (Recommended)".
- Executed, all gated: virsh start x9 on the rack -> PXE re-enlist
  (9/9 in ~2 min; pinned MACs = stable identity) -> voffice1 clone
  fast-forwarded -> maas-node-power.sh DRY (9/9 MAC-matched) then
  --commit (9/9 ok, each verified by query-power-state) -> commission
  x9 -> ALL 9 READY in ~3 min. Shapes verified exact to D-121 Option C
  (3x16/64GiB, 2x12/48GiB, 4x8/24GiB), power=virsh
  (docs/audit/incident-20260721-recovery-verify.txt). NOTE: new random
  MAAS hostnames (ace-robin, alert-cub, ... replace the pre-incident
  set); nothing durable quoted the old names as identity (the power
  script is MAC-matched by design).
- Held cleanup then completed (the rest of ruling 1): voffice1's
  vr1-dc0-maas dir incl. terraform.tfstate REMOVED; SEC-013 key file
  `~/vr1-dc0-creds/maas-api-key.txt` REMOVED; absence verified by
  listing (creds dir now: maas-virsh_ed25519 [SEC-012, in use] + the
  dc0 service keypair). SEC-013 row narrowed to CLI-profile-only;
  CURRENT-STATE incident paragraph resolved; NEW appendix-A entry
  (pod-delete cascade: check the pod's machine list BEFORE delete).
- **Revert:** n/a for the live repairs; docs via git revert. The
  deleted key file is not restorable from the repo (by design) -- the
  operator re-mints from MAAS if a maas-provider root ever returns.

## 16. G16 CLOSED: office1 channels residual reconciled by state surgery (stage-close ruling 2)

- RULING (GA-R5). Question as presented: "Stage-close ruling 2: G16
  mechanism -- how does the office1 edge's `channels = []`
  state-schema residual get reconciled?" Operator selection, exact:
  "State surgery (Recommended)".
- Executed per the G6 precedent: `tofu state pull` -> verified the
  attribute is `"channels": null` on exactly ONE resource
  (module.office1_opnsense libvirt_domain.vm) -> injected `[]` ->
  serial 29 -> 30 -> `tofu state push`. Backup:
  `opentofu/terraform.tfstate.pre-G16-20260721` (gitignored,
  verified). Convergence re-plan: **no differences**
  (docs/audit/outer-plan-20260721-postG16-converged.txt). Guests never
  touched (office1-opnsense Id 2 running throughout). CURRENT-STATE
  section 5 re-recorded to ZERO DIFF + G16 row CLOSED, same commit.
- **Revert:** `tofu state push opentofu/terraform.tfstate.pre-G16-20260721`
  (returns serial-29 state; the 0/1/0 residual and gate reopen).

## 17. STAGE 3 CLOSE (stage-close ruling 3) -- consolidation + skill sweep

- RULING (GA-R5). Question as presented: "Stage-close ruling 3: close
  Stage 3 now?" (full close-out set enumerated in the question).
  Operator selection, exact: "Close and merge (Recommended)".
- GA-R2 consolidation: the four closed execution-session changelogs
  (phase5-sweep, dc0-deploy-stepB, commissioning-diag,
  close-and-delivery -- 1,402 lines) moved to docs/archive/changelogs/;
  vr1-stage3-partial-record.md finalized as vr1-stage3-record.md with
  the close summary + closing-era session table. THIS changelog stays
  top-level until its session closes (record precedent).
- Status surfaces updated same commit (GA-R1 C1): CURRENT-STATE stage
  line (CLOSED for vr1-dc0 scope; dc1 = designed HELD remainder G12),
  workflow doc Stage-3 State row (+ note that its Gate line predates
  the pod refutation + DOCFIX-197), runbooks/README row 3 (vm_host
  language corrected) + the stale NONE-EXECUTED banner demoted to a
  CURRENT-STATE pointer.
- Skill sweep (GA-R6 close item): folded the D-131 sub-1 standing
  rack-forwarder invariant, the two MAAS standing traps (pods refuted
  -> per-machine virsh; pod delete cascades -- check machine list
  first), and the Stage-3 merge precedent. No stale facts found (the
  skill delegates facts to the repo by design).
- **Revert:** git revert the consolidation commit (moves + record +
  status lines return); the merge itself is reverted only by operator
  decision (git revert -m 1 <merge>).
