diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index e89317f..a7e2546 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -222,21 +222,35 @@ re-runnable; nothing is lost. The committee record (`docs/audit/commissioning-committee-20260720.md`) is the durable authority for this diagnosis and its ranked live hypotheses. - REMAINING IN G10: netem only (step E, - NOT started -- target is the dc0<->dc1 mesh = **virbr5 on vcloud**, - re-measured 2026-07-21). The sudo blocker is RESOLVED end to end: - operator-ruled scoped NOPASSWD 2026-07-21, fragment SHIPPED - (`scripts/sudoers.d/netem-tc`, harness 9 cases, gauntlet 75 ALL - GREEN) and **INSTALLED on vcloud 2026-07-21 (operator-run)** -- - verified read-only by the successor session: installed file 0440 - root:root, byte-identical to the repo fragment, and `sudo -n -l` - grant checks exit 0 (both verbs on virbr5; spot checks virbr7/virbr3) - (`docs/audit/netem-sudo-install-20260721.txt`). Remaining: wire - `modules/netem-link` into the outer root (HELD comment, - `opentofu/main.tf:309`) with the ruled placeholder parameters -- - expected outer plan = ADDS ONLY, any other diff is a STOP (section - 5) -- then the gated step-E placeholder netem run. Session changelog - 2026-07-21 item 7; install verification: this session's changelog. + **STEP E (netem) DONE 2026-07-21 -- G10 CLOSED.** The sudo mechanism: + operator-ruled scoped NOPASSWD, fragment SHIPPED (gauntlet 75 ALL + GREEN) and INSTALLED on vcloud (operator-run; verified 0440 + root:root, byte-identical, `sudo -n -l` exit 0 -- + `docs/audit/netem-sudo-install-20260721.txt`). Wiring: `modules/ + netem-link` amended with a LOCAL execution mode (empty ssh target = + bare `sudo tc`; the module's Office1-era always-SSH assumption is + refuted by D-128 -- the outer root runs ON vcloud, and a self-hop + would have needed a new standing credential; NEW tests/netem-link + harness 12 cases, gauntlet 76 ALL GREEN). Target = the dc0<->dc1 + mesh leg **virbr5** (re-measured at wire time via `virsh net-info`; + the runbook Step-11 text targeting the office1 leg is a FLAGGED + divergence, DOCFIX queued -- that leg now carries the live + rack<->region transit, netem there would perturb operations). The + wire plan came back **1/1/0 = STOP** (section 5): the extra in-place + change is the office1 edge picking up D-129's `channels = []` + state-schema reconcile (commit `f5510c7`; benign in config terms but + an in-place update against the LIVE unpinned-MAC office1 edge -- the + 07-20 MAC-regen class). **RULED 2026-07-21: targeted netem apply** + (question + selection in session changelog). Applied via saved + `-target` plan, exact 1/0/0 + (`docs/audit/outer-{plan,apply}-20260721-netem*.txt`); placeholder + profile LIVE on virbr5: `netem delay 3ms 1ms loss 0.01%` + (PROVISIONAL -- S6 same-metro lean; D-100 gap #11 final numbers + remain unruled), virbr7/virbr3 untouched + (`docs/audit/stepE-netem-20260721.txt`). Convergence re-plan = + **0/1/0, exactly the office1 residual** -- split per E3 into NEW + gate G16 (office1 channels state reconcile). ACTIVE: G16 + the + stage-close set. - The grounding audit is COMPLETE and EXITED (2026-07-19): Phases 1-6 all closed (charter `148dcef`; rulings `docs/audit/ga-rulings.md`; the Phase-5 sweep ran as six operator-gated batches in one session; exit @@ -392,18 +406,19 @@ - pre-reboot gate was 5/0/6 (recorded at `docs/dc0-deploy-readiness.md:59`, `docs/session-ledger.md:278`). -The EXPECTED outer plan is **ZERO DIFF** ("no differences"), re-recorded -2026-07-20 with its evidencing capture -(`docs/audit/outer-plan-20260720-postfix-converged.txt`) after two gated -applies in the step-B window (both exact: voffice1 transit NIC 1/1/1, -vvr1-dc0 seed replace 2/0/2 -- session changelog items 2-4, captures -`outer-{plan,apply}-20260719-voffice1-transit.txt` and -`outer-{plan,apply}-20260719-vvr1dc0-netplan-fix.txt`). History: 7/2/7 -post-reboot symptom -> 6/2/6 post-D-130 -> 6/0/6 post-G6-reconcile -> -applied exact -> zero diff -> 1/1/1 (voffice1 transit, ruled+applied) -> -2/0/2 (rack netplan fix, applied) -> zero diff converged (this entry). -A future outer plan showing ANY diff is a STOP (investigate drift -before touching anything). +The EXPECTED outer plan is **0/1/0** -- exactly ONE in-place change on +`module.office1_opnsense.libvirt_domain.vm` (`channels` absent -> `[]`, +the D-129 module-schema reconcile, gate G16), re-recorded 2026-07-21 +with its evidencing capture +(`docs/audit/outer-plan-20260721-postE-residual.txt`) after the ruled +targeted netem apply. Any OTHER diff is a STOP (investigate drift +before touching anything); when G16 closes, the expected plan returns +to ZERO DIFF and this section gets re-recorded with the closing +capture. History: 7/2/7 post-reboot symptom -> 6/2/6 post-D-130 -> +6/0/6 post-G6-reconcile -> applied exact -> zero diff -> 1/1/1 +(voffice1 transit, ruled+applied) -> 2/0/2 (rack netplan fix, applied) +-> zero diff converged 2026-07-20 -> 1/1/0 netem-wire STOP -> targeted +netem apply 1/0/0 exact -> 0/1/0 office1 residual (this entry). ## 6. Open gates @@ -423,12 +438,13 @@ | G7 | New captured plan == the expected triple recorded in section 5 | [V] re-plan to a capture file after G5+G6 | session | CLOSED 2026-07-19: capture `docs/audit/outer-plan-20260719-postG6.txt` = 6/0/6, equals section 5 exactly | | G8 | Same-session pre-apply re-verify: 6 planes still empty | [V] run in the SAME session as the apply | session | CLOSED 2026-07-19: verified in the apply session itself (all six 0 leases; only office1 nets attached) immediately before step A | | G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via `site-headend-install.sh --host-nodes --check` on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched | -| G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). REMAINING: netem (step E) | +| G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (`docs/audit/stepE-netem-20260721.txt` + `outer-{plan,apply}-20260721-netem*.txt`). **G10 CLOSED 2026-07-21** | | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | | G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator | HELD (`docs/dc0-deploy-readiness.md:100-103`) | | G13 | D-129 residuals | [R] operator-gated live plugin install on the edge; qga channel retrofit at next scheduled edge restart; 4 sub-decisions (section 8) | operator | OPEN / PARTIALLY RULED (`docs/design-decisions.md:4017`) | | G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether `opentofu/vr1-dc0-maas` is retired | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-20 | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | PROPOSED/OPEN (status lines, section 8) | +| G16 | office1 edge `channels = []` state reconcile (the D-129 module-schema residual; expected outer plan 0/1/0 until closed, section 5) | [R] operator rules the mechanism: state-only surgery (G6 precedent) vs ride the in-place apply into the edge's next scheduled restart (where D-129 already defers the office1 qga retrofit) -- an in-place apply against the unpinned-MAC live edge is the 07-20 MAC-regen class, so it is NOT applied casually | operator | OPEN 2026-07-21: residual measured (`docs/audit/outer-plan-20260721-postE-residual.txt`, 0/1/0); ruled targeted-apply STOP handling recorded in session changelog | ## 7. Version pins (measured; the authority for every pin) diff --git a/docs/audit/outer-apply-20260721-netem.txt b/docs/audit/outer-apply-20260721-netem.txt new file mode 100644 index 0000000..d0f347d --- /dev/null +++ b/docs/audit/outer-apply-20260721-netem.txt @@ -0,0 +1,19 @@ +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Creating... +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Provisioning with 'local-exec'... +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem (local-exec): Executing: ["/bin/sh" "-c" "sudo tc qdisc replace dev virbr5 root netem delay 3ms 1ms loss 0.01%"] +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Creation complete after 0s [id=1ea36d3f-e7af-984c-8157-152724a0b85a] + +Warning: Applied changes may be incomplete + +The plan was created with the -target or the -exclude option in effect, so +some changes requested in the configuration may have been ignored and the +output values may not be fully updated. Run the following command to verify +that no other changes are pending: + tofu plan + +Note that the -target and -exclude options are not suitable for routine use, +and are provided only for exceptional situations such as recovering from +errors or mistakes, or when OpenTofu specifically suggests to use it as part +of an error message. + +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. diff --git a/docs/audit/outer-plan-20260721-netem-targeted.txt b/docs/audit/outer-plan-20260721-netem-targeted.txt new file mode 100644 index 0000000..d8a74d3 --- /dev/null +++ b/docs/audit/outer-plan-20260721-netem-targeted.txt @@ -0,0 +1,39 @@ + +OpenTofu used the selected providers to generate the following execution +plan. Resource actions are indicated with the following symbols: + + create + +OpenTofu will perform the following actions: + + # module.netem_vr1_dc0_vr1_dc1.terraform_data.netem will be created + + resource "terraform_data" "netem" { + + id = (known after apply) + + input = { + + bridge_name = "virbr5" + + ssh_target = "" + } + + output = (known after apply) + + triggers_replace = [ + + "virbr5", + + "delay 3ms 1ms loss 0.01%", + ] + } + +Plan: 1 to add, 0 to change, 0 to destroy. + +Warning: Resource targeting is in effect + +You are creating a plan with either the -target option or the -exclude +option, which means that the result of this plan may not represent all of the +changes requested by the current configuration. + +The -target and -exclude options are not for routine use, and are provided +only for exceptional situations such as recovering from errors or mistakes, +or when OpenTofu specifically suggests to use it as part of an error message. + +───────────────────────────────────────────────────────────────────────────── + +Saved the plan to: phase2-vr1-dc0-netem.tfplan + +To perform exactly these actions, run the following command to apply: + tofu apply "phase2-vr1-dc0-netem.tfplan" diff --git a/docs/audit/outer-plan-20260721-netem-wire.txt b/docs/audit/outer-plan-20260721-netem-wire.txt new file mode 100644 index 0000000..7e3ba3e --- /dev/null +++ b/docs/audit/outer-plan-20260721-netem-wire.txt @@ -0,0 +1,77 @@ +module.voffice1.libvirt_cloudinit_disk.seed: Refreshing state... [id=a4694210c663c9ce] +module.vvr1_dc0.libvirt_cloudinit_disk.seed: Refreshing state... [id=ff281478c6083cc3] +module.office1_network.libvirt_network.office1_local: Refreshing state... [id=8fdd2a97-417c-44d4-89e4-ae8d65594135] +module.mesh_vr1_dc1_office1.libvirt_network.link: Refreshing state... [id=38a20d2d-cd91-4604-a5f4-8e2a6609633c] +module.vr1_dc1_storage.libvirt_pool.dc: Refreshing state... [id=4a1df114-ee04-4c80-9233-cc0c140c8556] +module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link: Refreshing state... [id=9cbc8589-9f40-48e6-872e-ef3abfe29a93] +module.office1_storage.libvirt_pool.dc: Refreshing state... [id=5f94194c-69c1-4b04-a85f-c18d87303a03] +module.mesh_vr1_dc0_office1.libvirt_network.link: Refreshing state... [id=8318548f-c3d6-4e06-bef4-fe3f11d68125] +module.vr1_dc0_storage.libvirt_pool.dc: Refreshing state... [id=7ce1101c-a89e-40ca-9263-5f572bee40a9] +module.vr1_dc0_uplink.libvirt_network.site_wan: Refreshing state... [id=f3500153-e4de-45f1-8854-9c92974a6094] +module.vvr1_dc0.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-cloudinit.iso] +module.voffice1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso] +module.ubuntu_noble_base.libvirt_volume.base: Refreshing state... [id=/var/lib/libvirt/vr1/office1/ubuntu-24.04-base.qcow2] +module.office1_opnsense.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/office1-opnsense-disk.qcow2] +module.voffice1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-disk.qcow2] +module.vvr1_dc0.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-disk.qcow2] +module.office1_opnsense.libvirt_domain.vm: Refreshing state... [name=office1-opnsense] +module.voffice1.libvirt_domain.vm: Refreshing state... [name=voffice1] +module.vvr1_dc0.libvirt_domain.vm: Refreshing state... [name=vvr1-dc0] + +Note: Objects have changed outside of OpenTofu + +OpenTofu detected the following changes made outside of OpenTofu since the +last "tofu apply" which may have affected this plan: + + # module.vvr1_dc0.libvirt_domain.vm has changed + ~ resource "libvirt_domain" "vm" { + ~ id = 4 -> 7 + name = "vvr1-dc0" + # (11 unchanged attributes hidden) + } + + +Unless you have made equivalent changes to your configuration, or ignored the +relevant attributes using ignore_changes, the following plan may include +actions to undo or respond to these changes. + +───────────────────────────────────────────────────────────────────────────── + +OpenTofu used the selected providers to generate the following execution +plan. Resource actions are indicated with the following symbols: + + create + ~ update in-place (current -> planned) + +OpenTofu will perform the following actions: + + # module.netem_vr1_dc0_vr1_dc1.terraform_data.netem will be created + + resource "terraform_data" "netem" { + + id = (known after apply) + + input = { + + bridge_name = "virbr5" + + ssh_target = "" + } + + output = (known after apply) + + triggers_replace = [ + + "virbr5", + + "delay 3ms 1ms loss 0.01%", + ] + } + + # module.office1_opnsense.libvirt_domain.vm will be updated in-place + ~ resource "libvirt_domain" "vm" { + ~ devices = { + + channels = [] + # (3 unchanged attributes hidden) + } + id = 2 + name = "office1-opnsense" + # (10 unchanged attributes hidden) + } + +Plan: 1 to add, 1 to change, 0 to destroy. + +───────────────────────────────────────────────────────────────────────────── + +Note: You didn't use the -out option to save this plan, so OpenTofu can't +guarantee to take exactly these actions if you run "tofu apply" now. diff --git a/docs/audit/outer-plan-20260721-postE-residual.txt b/docs/audit/outer-plan-20260721-postE-residual.txt new file mode 100644 index 0000000..a4c3b2f --- /dev/null +++ b/docs/audit/outer-plan-20260721-postE-residual.txt @@ -0,0 +1,63 @@ +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Refreshing state... [id=1ea36d3f-e7af-984c-8157-152724a0b85a] +module.vvr1_dc0.libvirt_cloudinit_disk.seed: Refreshing state... [id=ff281478c6083cc3] +module.voffice1.libvirt_cloudinit_disk.seed: Refreshing state... [id=a4694210c663c9ce] +module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link: Refreshing state... [id=9cbc8589-9f40-48e6-872e-ef3abfe29a93] +module.office1_storage.libvirt_pool.dc: Refreshing state... [id=5f94194c-69c1-4b04-a85f-c18d87303a03] +module.office1_network.libvirt_network.office1_local: Refreshing state... [id=8fdd2a97-417c-44d4-89e4-ae8d65594135] +module.vr1_dc1_storage.libvirt_pool.dc: Refreshing state... [id=4a1df114-ee04-4c80-9233-cc0c140c8556] +module.mesh_vr1_dc1_office1.libvirt_network.link: Refreshing state... [id=38a20d2d-cd91-4604-a5f4-8e2a6609633c] +module.mesh_vr1_dc0_office1.libvirt_network.link: Refreshing state... [id=8318548f-c3d6-4e06-bef4-fe3f11d68125] +module.vr1_dc0_storage.libvirt_pool.dc: Refreshing state... [id=7ce1101c-a89e-40ca-9263-5f572bee40a9] +module.vr1_dc0_uplink.libvirt_network.site_wan: Refreshing state... [id=f3500153-e4de-45f1-8854-9c92974a6094] +module.ubuntu_noble_base.libvirt_volume.base: Refreshing state... [id=/var/lib/libvirt/vr1/office1/ubuntu-24.04-base.qcow2] +module.office1_opnsense.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/office1-opnsense-disk.qcow2] +module.voffice1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso] +module.vvr1_dc0.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-cloudinit.iso] +module.voffice1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-disk.qcow2] +module.vvr1_dc0.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-disk.qcow2] +module.office1_opnsense.libvirt_domain.vm: Refreshing state... [name=office1-opnsense] +module.vvr1_dc0.libvirt_domain.vm: Refreshing state... [name=vvr1-dc0] +module.voffice1.libvirt_domain.vm: Refreshing state... [name=voffice1] + +Note: Objects have changed outside of OpenTofu + +OpenTofu detected the following changes made outside of OpenTofu since the +last "tofu apply" which may have affected this plan: + + # module.vvr1_dc0.libvirt_domain.vm has changed + ~ resource "libvirt_domain" "vm" { + ~ id = 4 -> 7 + name = "vvr1-dc0" + # (11 unchanged attributes hidden) + } + + +Unless you have made equivalent changes to your configuration, or ignored the +relevant attributes using ignore_changes, the following plan may include +actions to undo or respond to these changes. + +───────────────────────────────────────────────────────────────────────────── + +OpenTofu used the selected providers to generate the following execution +plan. Resource actions are indicated with the following symbols: + ~ update in-place (current -> planned) + +OpenTofu will perform the following actions: + + # module.office1_opnsense.libvirt_domain.vm will be updated in-place + ~ resource "libvirt_domain" "vm" { + ~ devices = { + + channels = [] + # (3 unchanged attributes hidden) + } + id = 2 + name = "office1-opnsense" + # (10 unchanged attributes hidden) + } + +Plan: 0 to add, 1 to change, 0 to destroy. + +───────────────────────────────────────────────────────────────────────────── + +Note: You didn't use the -out option to save this plan, so OpenTofu can't +guarantee to take exactly these actions if you run "tofu apply" now. diff --git a/docs/audit/stepE-netem-20260721.txt b/docs/audit/stepE-netem-20260721.txt new file mode 100644 index 0000000..5f74a79 --- /dev/null +++ b/docs/audit/stepE-netem-20260721.txt @@ -0,0 +1,11 @@ +# Step E (netem placeholder) verification -- 2026-07-21 +# Gate check for G10 final item. Apply capture: outer-apply-20260721-netem.txt + +$ tc qdisc show dev virbr5 +qdisc netem 800c: root refcnt 2 limit 1000 delay 3ms 1ms loss 0.01% + +$ tc qdisc show dev virbr7 # untouched (live transit leg) +qdisc noqueue 0: root refcnt 2 + +$ tc qdisc show dev virbr3 # untouched +qdisc noqueue 0: root refcnt 2 diff --git a/docs/changelog-20260721-netem-install-verify.md b/docs/changelog-20260721-netem-install-verify.md index 0189177..c718814 100644 --- a/docs/changelog-20260721-netem-install-verify.md +++ b/docs/changelog-20260721-netem-install-verify.md @@ -46,3 +46,83 @@ predecessor's own changelog). - **Revert:** git revert this commit (returns the predecessor to not-durable state; live cloud untouched). + +## 4. modules/netem-link: LOCAL execution mode (D-128 amendment) + NEW harness + +- The module's always-SSH design assumed tofu runs from Office1 (D-103 + era); D-128 fixed the outer root's execution ON vcloud, where the + mesh bridges live. Measured: no self-ssh path exists on vcloud (no + keypair in ~/.ssh, no vcloud/localhost known_hosts entry) -- an SSH + self-hop would have required minting a NEW standing credential for + nothing. Amendment: `vcloud_host_ssh_target` now OPTIONAL (default + "" = run `sudo tc` directly on the invoking machine); non-empty + keeps the original hop. Both provisioners (apply + destroy) + conditional; destroy still references self.* only (DOCFIX-194 + class guarded). +- NEW tests/netem-link/run-tests.sh (12 cases: terraform_data not + null_resource, triggers, input-stash/destroy-self discipline, both + modes present, default-empty, replace/del||true idempotency, no + virbrN literal in module, verbs covered by the sudoers fragment + shapes, tofu-validate fixture instantiating BOTH modes). Gauntlet + 75 -> 76 ALL GREEN. +- **Revert:** git revert this commit's module+harness hunks; the wire + block (item 5) must go with it (it relies on the default). + +## 5. netem WIRED in the outer root (step E) -- target virbr5, placeholder profile + +- Replaced the HELD comment (opentofu/main.tf) with + `module "netem_vr1_dc0_vr1_dc1"`: bridge_name = virbr5, MEASURED at + wire time (`virsh net-info mesh-vr1-dc0-vr1-dc1` -> Bridge: virbr5; + virbr7/virbr3 re-measured too, all matching the sudoers fragment). + netem_args = "delay 3ms 1ms loss 0.01%" -- the RULED PLACEHOLDER + (operator ruling 2026-07-16; buildout-design S6 same-metro lean; + D-100 gap #11 final numbers remain UNRULED), recorded as PROVISIONAL. +- FLAGGED runbook divergence (DOCFIX queued): phase2 runbook Step 11 + (2026-07-15) targets the dc0<->office1 leg as "the only leg with + substrate on both ends"; superseded in fact -- that leg now carries + the live rack<->region transit (MAAS, node DNS, inner qemu+ssh), so + netem there would perturb live operations. CURRENT-STATE's step-E + re-measure + the operator-ruled netem-tc fragment both name virbr5 + (dc0<->dc1, zero traffic) as the target; followed those. +- **Revert:** `tofu destroy -target=module.netem_vr1_dc0_vr1_dc1` + (destroy provisioner clears the qdisc: `tc qdisc del dev virbr5 + root`), then git revert the wire hunk. + +## 6. STOP at 1/1/0, operator ruling, targeted apply EXECUTED (step E DONE, G10 CLOSED) + +- The wire plan (docs/audit/outer-plan-20260721-netem-wire.txt) came + back 1/1/0 vs the expected adds-only: the extra in-place change is + the office1 edge picking up D-129's `channels = []` module-schema + reconcile (traced to commit f5510c7; the office1 edge's state + predates the attribute). Benign in config terms, but an in-place + update against the LIVE unpinned-MAC office1 edge = the 07-20 + MAC-regen class. STOP honored; presented as a gated decision. +- RULING (GA-R5, one decision this exchange). Question as presented: + "The netem-wire outer plan came back 1 add / 1 change / 0 destroy -- + the add is the expected netem resource, but the change is an + in-place update on the LIVE office1 edge (state-schema reconcile: + `channels` absent -> `[]`, from the D-129 module change). Section 5 + says any non-expected diff is a STOP. How do we proceed?" Operator + selection, exact: "Targeted netem apply (Recommended)". +- Executed: saved `-target` plan, re-verified via `tofu show` (exact + 1/0/0, only terraform_data.netem), applied. Provisioner ran the + ruled command locally through the installed NOPASSWD grant. Step-12 + verify: virbr5 carries `netem delay 3ms 1ms loss 0.01%`; + virbr7/virbr3 noqueue (untouched). Convergence re-plan 0/1/0 = + exactly the office1 residual. Captures: + outer-plan-20260721-netem-targeted.txt, outer-apply-20260721-netem.txt, + stepE-netem-20260721.txt, outer-plan-20260721-postE-residual.txt. +- CURRENT-STATE updated in the same commit (GA-R1 C1): step E DONE, + G10 CLOSED, section-5 expected plan re-recorded as 0/1/0, NEW gate + G16 (office1 channels reconcile -- remainder split per E3, never a + conditional close). +- **Revert:** item 5's revert (destroy-target clears the qdisc and the + state entry); CURRENT-STATE re-records section 5 back to ZERO DIFF. + +## 7. Queued (logged, not executed) + +- DOCFIX: phase2 runbook Step 11 leg-selection text (item 5's flagged + divergence). +- G16 mechanism ruling: state-only surgery (G6 precedent) vs ride the + edge's next scheduled restart (with the D-129 office1 qga retrofit). +- ledger-scan D-131 status-phrasing blind spot (item 1). diff --git a/opentofu/main.tf b/opentofu/main.tf index 5c668f6..ec828c0 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -323,18 +323,32 @@ # kvm nested=1 + stage the opnsense base image) -> INNER root (planes/wan/edge/nodes). # ===================================================================== -# netem (runbook Step 11) -- HELD as a comment. It is a POST-apply step that runs -# SSH-wrapped provisioners from Office1 to the vcloud host, so it needs a real -# `vcloud_host_ssh_target` (a runtime value) and the netem params are an unruled -# D-100 sub-item (gap #11, placeholder only). Measured input ready: the -# vr1-dc0<->office1 mesh leg is bridge `virbr7`. Wire at Step 11, not before. -# module "netem_vr1_dc0_office1" { -# source = "./modules/netem-link" -# link_name = "vr1-dc0-office1" -# bridge_name = "virbr7" # measured 2026-07-15 -# vcloud_host_ssh_target = "" -# netem_args = "" -# } +# netem (runbook Step 11 / deploy step E) -- WIRED 2026-07-21, the last G10 item. +# TARGET: the dc0<->dc1 mesh leg, bridge virbr5 -- MEASURED this session +# (`virsh net-info mesh-vr1-dc0-vr1-dc1` -> Bridge: virbr5; virbrN is a +# drifting ID, re-measure before every apply) -- per the CURRENT-STATE step-E +# re-measure (2026-07-21) and the operator-ruled netem-tc sudoers fragment, +# which names virbr5 as the step-E target. FLAGGED runbook divergence +# (DOCFIX queued): Step 11's 2026-07-15 text targeted the dc0<->office1 leg +# (virbr7) as "the only leg with substrate on both ends", but that leg now +# CARRIES the live rack<->region transit (MAAS, node DNS, the inner root's +# qemu+ssh path), so netem there would perturb live operations; the +# zero-traffic dc0<->dc1 leg exercises the mechanism -- step E's whole point +# -- with no operational impact. vcloud_host_ssh_target stays at its empty +# default = LOCAL mode (D-128: this root executes ON vcloud, where the +# bridges live; module amendment 2026-07-21 -- an SSH self-hop would have +# required minting a new standing credential). netem_args is the RULED +# PLACEHOLDER (operator ruling 2026-07-16; D-100 gap #11's final numbers +# remain UNRULED): buildout-design Section 6 same-metro dark-fiber lean -- +# low single-digit-ms delay, modest jitter, negligible loss. PROVISIONAL, +# not a measured or ruled link profile; re-tune when the Roosevelt inter-DC +# target exists. +module "netem_vr1_dc0_vr1_dc1" { + source = "./modules/netem-link" + link_name = "vr1-dc0-vr1-dc1" + bridge_name = "virbr5" # MEASURED 2026-07-21: virsh net-info mesh-vr1-dc0-vr1-dc1 + netem_args = "delay 3ms 1ms loss 0.01%" # PLACEHOLDER -- S6 lean; D-100 gap #11 unruled +} # ===================================================================== # D-125 (Model B bridge-in, closes OBS-3): the vr1-dc0 simulated-ISP uplink, at diff --git a/opentofu/modules/netem-link/main.tf b/opentofu/modules/netem-link/main.tf index 3ce5608..a465007 100644 --- a/opentofu/modules/netem-link/main.tf +++ b/opentofu/modules/netem-link/main.tf @@ -7,11 +7,17 @@ # over the older `null_resource` for exactly this "provisioner with no # logical managed resource to attach to" case -- used here, not null_resource. # -# Runs over SSH, not a bare local-exec command: OpenTofu itself runs from -# the Office1 operator VM (D-103), not on the vcloud host where the bridge -# interfaces exist, so the command has to hop there explicitly. Assumes -# passwordless SSH + passwordless sudo to var.vcloud_host_ssh_target as this -# repo's usual jumphost-session conventions provide -- not configured here. +# Execution locus (amended 2026-07-21): the original design assumed OpenTofu +# runs from the Office1 operator VM (D-103) and therefore ALWAYS hopped via +# SSH. D-128 fixed the outer root's execution ON the vcloud host itself +# (Plane 1), where the mesh bridges live -- so an SSH self-hop would demand a +# new standing self-ssh credential for nothing. `vcloud_host_ssh_target` is +# now OPTIONAL: empty (the default) runs `sudo tc` directly on the machine +# invoking `tofu apply` (correct for the D-128 outer root); non-empty keeps +# the original SSH hop for any future root that runs remotely. Passwordless +# sudo for the tc verbs is assumed either way -- provided on vcloud by the +# operator-installed scoped fragment /etc/sudoers.d/netem-tc (repo copy +# scripts/sudoers.d/netem-tc; operator-ruled 2026-07-21), not configured here. # # UNVERIFIED, flagged plainly: this whole resource has not been run for # real this session (no `tofu` binary, no live vcloud host, no bridge to @@ -42,11 +48,11 @@ } provisioner "local-exec" { - command = "ssh ${var.vcloud_host_ssh_target} 'sudo tc qdisc replace dev ${var.bridge_name} root netem ${var.netem_args}'" + command = var.vcloud_host_ssh_target != "" ? "ssh ${var.vcloud_host_ssh_target} 'sudo tc qdisc replace dev ${var.bridge_name} root netem ${var.netem_args}'" : "sudo tc qdisc replace dev ${var.bridge_name} root netem ${var.netem_args}" } provisioner "local-exec" { when = destroy - command = "ssh ${self.input.ssh_target} 'sudo tc qdisc del dev ${self.input.bridge_name} root' || true" + command = self.input.ssh_target != "" ? "ssh ${self.input.ssh_target} 'sudo tc qdisc del dev ${self.input.bridge_name} root' || true" : "sudo tc qdisc del dev ${self.input.bridge_name} root || true" } } diff --git a/opentofu/modules/netem-link/variables.tf b/opentofu/modules/netem-link/variables.tf index 59616cf..fff8e97 100644 --- a/opentofu/modules/netem-link/variables.tf +++ b/opentofu/modules/netem-link/variables.tf @@ -5,16 +5,17 @@ variable "vcloud_host_ssh_target" { description = <<-EOT - SSH target for the vcloud host, e.g. "user@vcloud-host" -- REQUIRED - because OpenTofu itself runs from the Office1 operator VM (D-103), not on - the vcloud host where the bridge interfaces actually live; a plain - local-exec provisioner runs commands on the machine invoking `tofu - apply` (Office1), so applying tc netem to a vcloud-host bridge needs an - explicit SSH hop, not a bare local command. Passwordless (key-based) SSH - + passwordless sudo on the vcloud host for the invoking user are assumed - prerequisites, not configured by this module. No default -- measured. + SSH target for the host carrying the mesh bridges, e.g. + "user@vcloud-host" -- or EMPTY (the default) to run `sudo tc` directly + on the machine invoking `tofu apply`. Empty is correct for the D-128 + outer root, which executes ON the vcloud host itself (Plane 1) where + the bridges live; the SSH hop remains for any root that runs remotely + (the module's original D-103 Office1 assumption, amended 2026-07-21). + Passwordless sudo for the tc verbs (and, in SSH mode, key-based SSH) + are assumed prerequisites, not configured by this module. EOT type = string + default = "" } variable "bridge_name" { diff --git a/tests/netem-link/run-tests.sh b/tests/netem-link/run-tests.sh new file mode 100755 index 0000000..b2a3ed2 --- /dev/null +++ b/tests/netem-link/run-tests.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash +# tests/netem-link/run-tests.sh -- guard for opentofu/modules/netem-link +# (D-100 netem mechanism; local-mode amendment 2026-07-21 per D-128: the +# outer root runs ON vcloud, so empty vcloud_host_ssh_target = bare local +# `sudo tc`, non-empty = the original Office1-era SSH hop). Static shape +# assertions + a tofu-validate fixture when tofu is available. +# Exit: 0 all pass | 1 any failed. ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MOD="$(cd "$HERE/../.." && pwd)/opentofu/modules/netem-link" +FRAG="$(cd "$HERE/../.." && pwd)/scripts/sudoers.d/netem-tc" +PASS=0; FAIL=0 +ok(){ echo " PASS $1"; PASS=$((PASS+1)); } +no(){ echo " FAIL $1"; FAIL=$((FAIL+1)); } + +# T1: module files present +for f in main.tf variables.tf outputs.tf; do + [ -f "$MOD/$f" ] || { no "T1 module files present ($f missing)"; echo; echo "netem-link: $PASS passed, $((FAIL)) failed"; exit 1; } +done +ok "T1 module files present" + +# T2: terraform_data, not null_resource (OpenTofu's own recommendation) +grep -q 'resource "terraform_data" "netem"' "$MOD/main.tf" \ + && ! grep -v '^ *#' "$MOD/main.tf" | grep -q 'null_resource' \ + && ok "T2 terraform_data used (no null_resource)" \ + || no "T2 terraform_data used (no null_resource)" + +# T3: re-apply triggers on bridge + args +grep -q 'triggers_replace = \[var.bridge_name, var.netem_args\]' "$MOD/main.tf" \ + && ok "T3 triggers_replace covers bridge_name + netem_args" \ + || no "T3 triggers_replace covers bridge_name + netem_args" + +# T4: destroy-time values stashed in input (DOCFIX-194 pattern) +grep -q 'ssh_target *= *var.vcloud_host_ssh_target' "$MOD/main.tf" \ + && grep -q 'bridge_name *= *var.bridge_name' "$MOD/main.tf" \ + && ok "T4 input stash present for destroy provisioner" \ + || no "T4 input stash present for destroy provisioner" + +# T5: destroy provisioner references ONLY self.* (var.* there is an INIT-time +# error -- the DOCFIX-194 class); assert no var. between 'when = destroy' and +# the end of that provisioner block +DESTROY_BLOCK="$(awk '/when *= *destroy/,/^ }/' "$MOD/main.tf")" +[ -n "$DESTROY_BLOCK" ] && ! grep -q 'var\.' <<<"$DESTROY_BLOCK" \ + && ok "T5 destroy provisioner references self.* only" \ + || no "T5 destroy provisioner references self.* only" + +# T6: local mode -- empty target runs bare sudo tc in BOTH provisioners +grep -q 'var.vcloud_host_ssh_target != "" ?' "$MOD/main.tf" \ + && grep -q 'self.input.ssh_target != "" ?' "$MOD/main.tf" \ + && ok "T6 empty-target conditional present in apply + destroy" \ + || no "T6 empty-target conditional present in apply + destroy" + +# T7: ssh mode retained (non-empty target still hops) +grep -q "ssh \${var.vcloud_host_ssh_target} 'sudo tc qdisc replace" "$MOD/main.tf" \ + && grep -q "ssh \${self.input.ssh_target} 'sudo tc qdisc del" "$MOD/main.tf" \ + && ok "T7 ssh hop retained for non-empty target" \ + || no "T7 ssh hop retained for non-empty target" + +# T8: vcloud_host_ssh_target defaults to "" (local mode is the default) +awk '/variable "vcloud_host_ssh_target"/,/^}/' "$MOD/variables.tf" | grep -q 'default *= *""' \ + && ok "T8 vcloud_host_ssh_target default is empty (local)" \ + || no "T8 vcloud_host_ssh_target default is empty (local)" + +# T9: idempotent verbs -- replace on apply, del || true on destroy +grep -q 'tc qdisc replace dev' "$MOD/main.tf" && grep -q "tc qdisc del dev" "$MOD/main.tf" \ + && grep -q '|| true' "$MOD/main.tf" \ + && ok "T9 replace on apply, del||true on destroy" \ + || no "T9 replace on apply, del||true on destroy" + +# T10: no hardcoded virbrN in the module (bridge is a measured input) +grep -q 'virbr[0-9]' "$MOD"/*.tf \ + && no "T10 no virbrN literal in module sources" \ + || ok "T10 no virbrN literal in module sources" + +# T11: module verbs stay inside the sudoers fragment's granted shapes +if [ -f "$FRAG" ]; then + grep -q 'qdisc replace dev virbr[0-9]* root netem \*' "$FRAG" \ + && grep -q 'qdisc del dev virbr[0-9]* root$' "$FRAG" \ + && ok "T11 module verbs covered by netem-tc fragment shapes" \ + || no "T11 module verbs covered by netem-tc fragment shapes" +else + echo " SKIP T11 sudoers fragment not present" +fi + +# T12: tofu validate on a throwaway fixture instantiating BOTH modes +if command -v tofu >/dev/null 2>&1; then + TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT + cat >"$TMP/main.tf" </dev/null 2>&1 && tofu validate >/dev/null 2>&1); then + ok "T12 tofu validate passes (both modes instantiated)" + else + no "T12 tofu validate passes (both modes instantiated)" + fi +else + echo " SKIP T12 tofu not available" +fi + +echo; echo "netem-link: $PASS passed, $FAIL failed" +[ "$FAIL" -eq 0 ]