diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index a56c105..bf8c0b5 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -475,7 +475,7 @@ | G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via `site-headend-install.sh --host-nodes --check` on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched | | G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (`docs/audit/stepE-netem-20260721.txt` + `outer-{plan,apply}-20260721-netem*.txt`). **G10 CLOSED 2026-07-21** | | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | -| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). NEXT (gated): D-125 egress gate on br-vr1-dc1-wan, edge bootstrap (D-112(c)/D-113(a2)) + v4 addressing (WAN 172.30.3.2, LAN 10.12.64.1/22), rack standup DoD (dc-rack-net dc1 + forwarder 10.12.68.3, region-side DHCP primary_rack nmpcq4 + D-120 range, maas-node-power) | +| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). NEXT (gated): edge bootstrap (D-112(c)/D-113(a2)) + v4 addressing (WAN 172.30.3.2, LAN 10.12.64.1/22), rack standup DoD (dc-rack-net dc1 + forwarder 10.12.68.3, region-side DHCP primary_rack nmpcq4 + D-120 range, maas-node-power) | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | OPEN (execution only; decision content complete) | | G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether `opentofu/vr1-dc0-maas` is retired | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-20 | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 remains PROPOSED/OPEN (items 2-3 + the item-1 re-scoped migration plan) | diff --git a/docs/audit/d125-egress-gate-20260722-dc1.txt b/docs/audit/d125-egress-gate-20260722-dc1.txt new file mode 100644 index 0000000..1acf6da --- /dev/null +++ b/docs/audit/d125-egress-gate-20260722-dc1.txt @@ -0,0 +1,30 @@ +# D-125 egress gate, vr1-dc1 -- 2026-07-22T22:02:58Z -- run 1 of 2 +WARNING Requested memory 1024 MiB is less than the recommended 3072 MiB for OS ubuntu24.04 +== guest booted; waiting for the probe to report (<=180s) == +=== D-125 PROBE RESULT === +[ 14.871107] cloud-init[674]: ci-info: | enp1s0 | True | 172.30.3.50 | 255.255.255.0 | global | 52:54:00:22:37:09 | +enp1s0 UP 172.30.3.50/24 fe80::5054:ff:fe22:3709/64 +172.30.3.0/24 dev enp1s0 proto kernel scope link src 172.30.3.50 +GW-PING-RC=0 +d125-throwaway login: NET-PING-RC=0 +LAN-TCP-http=000 +NET-TCP-http=301 +NET-TCP-ubuntu=200 +D125-TEST-COMPLETE +=== teardown === +leftover-domains=0 +# run 2 of 2 +WARNING Requested memory 1024 MiB is less than the recommended 3072 MiB for OS ubuntu24.04 +== guest booted; waiting for the probe to report (<=180s) == +=== D-125 PROBE RESULT === +[ 14.685552] cloud-init[672]: ci-info: | enp1s0 | True | 172.30.3.50 | 255.255.255.0 | global | 52:54:00:a7:58:67 | +enp1s0 UP 172.30.3.50/24 fe80::5054:ff:fea7:5867/64 +172.30.3.0/24 dev enp1s0 proto kernel scope link src 172.30.3.50 +GW-PING-RC=0 +d125-throwaway login: NET-PING-RC=0 +LAN-TCP-http=000 +NET-TCP-http=301 +NET-TCP-ubuntu=200 +D125-TEST-COMPLETE +=== teardown === +leftover-domains=0 diff --git a/docs/changelog-20260722-g12-dc1-build.md b/docs/changelog-20260722-g12-dc1-build.md index da50a3f..9e8ef69 100644 --- a/docs/changelog-20260722-g12-dc1-build.md +++ b/docs/changelog-20260722-g12-dc1-build.md @@ -179,13 +179,27 @@ Revert: `dc-dc-teardown-rollback.md` tree; inner root destroy from voffice1 tears down the 28 (containment VM unaffected). +## Item 10 -- D-125 egress isolation gate: PASS (vr1-dc1) + +Procedure of record = dc0's `d125-egress-test.sh` (recovered from the dc0 +rack's home), adapted per-DC (pool `vr1-dc1-inner`, bridge `br-vr1-dc1-wan`, +probe 172.30.3.50/24 via .1 -- D-124 amendment /24) and staged to the dc1 +rack. Prereqs installed via the region proxy (virtinst, cloud-image-utils); +`/tmp/noble.img` streamed dc0->dc1. **Two identical consecutive runs**: +GW-PING-RC=0, NET-PING-RC=0, NET-TCP-http=301, NET-TCP-ubuntu=200, +LAN-TCP-http=000 (the region-isolation check), leftover-domains=0 -- +`docs/audit/d125-egress-gate-20260722-dc1.txt`. Bridge-in egress PROVEN +end-to-end for dc1; the double-NAT fallback is NOT needed. dc0's +one-time unexplained first-run ICMP failure did NOT recur. + +Revert: none needed (throwaway fully torn down, verified); delete the +capture + the staged script + /tmp/noble.img on the rack if desired. + ## Next (gated, not run here) -D-125 egress isolation gate on `br-vr1-dc1-wan` (throwaway guest, two -identical runs); edge bootstrap (D-112(c) console -> key-only SSH -> -D-113(a2) API key on 26.7); edge addressing via `opnsense-set-interface-v4` -(WAN 172.30.3.2/24 gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); -rack standup DoD (dc-rack-net.sh install dc1 + forwarder 10.12.68.3, -region-side DHCP on metal-admin naming nmpcq4 primary_rack, dynamic range -10.12.68.100-.200 per D-120, maas-node-power dc1 arm). Runbook + -CURRENT-STATE govern. +Edge bootstrap (D-112(c) console -> key-only SSH -> D-113(a2) API key on +26.7); edge addressing via `opnsense-set-interface-v4` (WAN 172.30.3.2/24 +gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); rack standup DoD +(dc-rack-net.sh install dc1 + forwarder 10.12.68.3, region-side DHCP on +metal-admin naming nmpcq4 primary_rack, dynamic range 10.12.68.100-.200 +per D-120, maas-node-power dc1 arm). Runbook + CURRENT-STATE govern.