diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f8b7a09..e578927 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -82,10 +82,20 @@ D-134 utility .4 + edge default route -- egress path MEASURED working, edge DNS answers; debmirror GPG-verified jammy triple + UCA caracal; harness 18/18; gauntlet **ALL GREEN (77)**, - `docs/audit/gauntlet-20260723-stage4-dcmirror.txt`). NEXT: gated - installs on both racks + initial sync -> Step 6/7/8 verify closure - -> lib-hosts.sh per-DC population + lib-net VID comment (delivery - items) -> mirror reachability gate -> stage close-out set. + `docs/audit/gauntlet-20260723-stage4-dcmirror.txt`). **INSTALLED + BOTH RACKS 2026-07-23** (check PASS 15/15 each; mirrors answer 200 + on 10.12.8.4 / 10.12.68.4; initial syncs RUNNING -- HOME-under- + systemd fix shipped same-hour, harness 19/19). **INCIDENT resolved + same-hour (changelog item 9):** the fresh dc1 edge passed NO LAN + traffic -- pf ruleset never regenerated after v4 addressing (the + set-interface script reloads the interface, not the filter; dc0 was + masked by its 07-20 plugin work). Fix: `configctl filter reload`; + dc1 rack egress now 0% loss / HTTP 301. Appendix-A entry + standup + DoD addition + set-interface amendment QUEUED to stage close. NEXT: + syncs complete (last-sync.status OK both) -> mirror node-side + reachability gate -> Step 6/7/8 verify closure -> lib-hosts.sh + per-DC population + lib-net VID comment (delivery items) -> stage + close-out set. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/changelog-20260723-stage4-open.md b/docs/changelog-20260723-stage4-open.md index 13ea3b5..8639a39 100644 --- a/docs/changelog-20260723-stage4-open.md +++ b/docs/changelog-20260723-stage4-open.md @@ -173,6 +173,47 @@ `maas admin tag update-nodes remove=...` x9 + tags delete; boot-source-selection delete id=2 (images age out). +## Item 9 -- INCIDENT (resolved): dc1 edge passed NO LAN traffic -- pf ruleset never regenerated + +- Found at the dc1 mirror install: apt through the edge timed out. Measured + chain: rack default route present, edge LAN gw ping 0%, internet 100% loss; + `pfctl -s nat` on the dc1 edge showed ZERO outbound NAT rules while dc0's + edge (identical config -- both carry the single WAN_GW gateway_item) showed + the full automatic set. Root cause: the fresh dc1 edge's pf ruleset was + NEVER REGENERATED after v4 addressing -- `opnsense-set-interface-v4 + --commit` runs `configctl interface reconfigure` but NOT a filter reload. + dc0 was masked: its 07-20 plugin/qga work triggered filter reloads as a + side effect. My dc0-only egress probe earlier this session was wrongly + taken as covering dc1 (symmetry inference -- owned). +- Fix (minimal, logged): `configctl filter reload` on the dc1 edge -> NAT + rules present, rack egress 0% loss / HTTP 301 -- identical to dc0. +- QUEUED: appendix-A entry (symptom: LAN clients of a freshly-addressed DC + edge get 100% loss with reachable gateway; check `pfctl -s nat`); DC-standup + DoD addition (filter reload + LAN-client egress probe after edge + addressing); candidate opnsense-set-interface-v4 amendment (add the filter + reload) -- a built-surface change, presented at stage close, not slipped in. +- **Revert:** n/a (the reload materializes the edge's own committed config). + +## Item 10 -- dc-mirror installed BOTH racks; syncs RUNNING; HOME fix (harness 19) + +- dc0: install pass-1 net layer up -> apt debmirror/nginx/ubuntu-cloud-keyring + -> install PASS (15 checks; mirror answers http://10.12.8.4/ 200). +- dc1: same, after the item-9 incident fix (install PASS; http://10.12.68.4/ + 200). +- First sync attempt FAILED both debmirror runs: "HOME not defined in + environment!" under systemd's empty env (debmirror needs a GPG home). + Fixed in gen_sync_unit (`Environment=HOME=/root`), harness case T19 added + (19/19), REINSTALLED both racks (config-diff regeneration path exercised), + syncs restarted: both RUNNING and downloading (dc0 400M / dc1 343M within + the first minute). Daily timers enabled. +- Mirror gate remains OPEN until: initial syncs complete (last-sync.status + OK) + mirror serves a real node (node-side probe at Stage 5 first boot or + a gated rescue-boot check). Stage 4 close blocks on the mirror gate per + the operator's build-now ruling. +- **Revert:** `systemctl stop/disable -mirror-sync.timer + -net.service`, + rm the generated files, `rm -rf /var/lib/dc-mirror`, apt remove the three + packages; the edge filter reload stands (item 9). + ## Item 8 -- OWNED: pipe-masked L10 lint FAIL on the D-135 commit (abd6b92) - The D-135 commit changed a design-decisions Status line WITHOUT diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 6cb278a..3a03c34 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -278,6 +278,26 @@ refresh fragility); D-068 item1, D-131 sub-4, D-132 pinned. Next stage branches off main. - Full body: docs/archive/changelogs/changelog-20260723-g12-dc1-edge.md. Status ONLY in CURRENT-STATE.md. +## SESSION 2026-07-23 -- Stage 4 OPEN + carve + mirrors (IN FLIGHT; bounded, GA-R4) + +- Stage 4 OPENED (branch dc-dc-stage4-phase3-maas-deploy); discovery captured; D-133 + (flat per-NIC carve; hw-faithful NICs next deployment) + D-134 (+amendment: contiguous + bands, Option B swap) ADOPTED, utterances quoted. +- CARVE EXECUTED both DCs (logged stage4-carve): ranges .201-.254, 10 fabrics, 90 NIC + re-homes, subnets+spaces, 18 nodes statics + br-ex; all Ready; capture committed. +- Deploy model RULED: Ready handoff (DOCFIX-200 -- runbook Step 4 would have broken + Stage 5); jammy images synced; per-DC tags applied 9+9. +- Mirror: exception REJECTED -> D-135 ADOPTED; dc-mirror.sh shipped (19/19, gauntlet 77); + INSTALLED both racks; initial syncs RUNNING at last check (hours; verify + last-sync.status before citing). INCIDENT: dc1 edge pf never regenerated post- + addressing -- filter reload fixed; appendix-A + DoD + script amendment QUEUED. +- OWNED: pipe-masked L10 lint FAIL (repaired); dc0-only egress probe wrongly assumed + symmetric (item 9). +- OPEN in-stage: sync completion + node-side mirror gate; Step 6/7/8 closure notes; + lib-hosts per-DC population + lib-net VID comment; queued appendix-A/DOCFIX set; + permission-rules prune flag (`Bash(ssh *)`); stage close-out set. +- Details: docs/changelog-20260723-stage4-open.md. Status ONLY in CURRENT-STATE.md. + ## SESSION CLOSE 2026-07-23 -- queue pass + D-068 rulings (bounded, GA-R4) - Queue fully processed. Repo: 5 script/harness fixes (prep-image bash -s; headend hints + diff --git a/scripts/dc-mirror.sh b/scripts/dc-mirror.sh index be942b1..8e83779 100755 --- a/scripts/dc-mirror.sh +++ b/scripts/dc-mirror.sh @@ -178,6 +178,9 @@ [Service] Type=oneshot +# debmirror dies with "HOME not defined in environment!" under systemd's +# empty environment (it needs a GPG home) -- measured on the first dc0 run. +Environment=HOME=/root ExecStart=${SYNC_HELPER} Nice=10 IOSchedulingClass=idle diff --git a/tests/dc-mirror/run-tests.sh b/tests/dc-mirror/run-tests.sh index c0f42aa..ffbe242 100755 --- a/tests/dc-mirror/run-tests.sh +++ b/tests/dc-mirror/run-tests.sh @@ -102,5 +102,11 @@ && ok "T18 sync unit is niced + idle IO" \ || no "T18 sync unit is niced + idle IO" +# T19: sync unit sets HOME -- debmirror dies without it under systemd's empty +# environment (measured on the first dc0 run, 2026-07-23) +grep -q 'Environment=HOME=/root' "$SCRIPT" \ + && ok "T19 sync unit sets HOME (debmirror GPG home)" \ + || no "T19 sync unit sets HOME (debmirror GPG home)" + echo; echo "dc-mirror: $PASS passed, $FAIL failed" [ "$FAIL" -eq 0 ]