diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 02fd26a..00a219e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -475,7 +475,7 @@ | G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via `site-headend-install.sh --host-nodes --check` on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched | | G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (`docs/audit/stepE-netem-20260721.txt` + `outer-{plan,apply}-20260721-netem*.txt`). **G10 CLOSED 2026-07-21** | | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | -| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). NEXT (gated): rack standup DoD (dc-rack-net dc1 arm + harness, then install dc1 + forwarder 10.12.68.3, region-side DHCP primary_rack nmpcq4 + D-120 range 10.12.68.100-.200, maas-node-power dc1 arm), commission 9/9 to Ready, then G12 close-out (gauntlet/repo-lint/consolidation/memory review/skill sweep/operator-gated merge to main) | +| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation, not SEC-012 reuse). NEXT (gated): wire the SEC-016 dedicated power key -> maas-node-power.sh dc1 (dry then --commit) -> commission 9/9 to Ready (D-121 Option C), then G12 close-out (gauntlet/repo-lint/consolidation/memory review/skill sweep/operator-gated merge to main). NOTE open SEC rows now include SEC-014/-015/-016 (G14 count stale -- reconcile at close). | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | OPEN (execution only; decision content complete) | | G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether `opentofu/vr1-dc0-maas` is retired | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-20 | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 remains PROPOSED/OPEN (items 2-3 + the item-1 re-scoped migration plan) | diff --git a/docs/audit/dc1-rack-net-install-20260723.txt b/docs/audit/dc1-rack-net-install-20260723.txt new file mode 100644 index 0000000..2c946cf --- /dev/null +++ b/docs/audit/dc1-rack-net-install-20260723.txt @@ -0,0 +1,15 @@ +=== dc-rack-net check dc1 (2026-07-23T00:20:23Z) === + OK /usr/local/sbin/dc1-rack-net-apply matches + OK /etc/systemd/system/dc1-rack-legs.service matches + OK /etc/dnsmasq-dc1-node.conf matches + OK /etc/systemd/system/dc1-node-dns.service matches + OK dc1-rack-legs enabled + OK dc1-node-dns enabled + OK dc1-node-dns active + OK 10.12.68.2/22 on virbr6 (vr1-dc1-metal-admin) + OK 10.12.68.3/22 on virbr6 (vr1-dc1-metal-admin) + OK 10.12.64.2/22 on virbr4 (vr1-dc1-provider-public) +dc-rack-net check (dc1): PASS +=== forwarder probe: SOA maas-internal via 10.12.68.3 === +maas-internal. nobody.example.com. 1877 600 1800 604800 15 +rc=0 diff --git a/docs/changelog-20260723-g12-dc1-edge.md b/docs/changelog-20260723-g12-dc1-edge.md index 237f127..e0f5dfc 100644 --- a/docs/changelog-20260723-g12-dc1-edge.md +++ b/docs/changelog-20260723-g12-dc1-edge.md @@ -131,9 +131,57 @@ D-131 fix that pre-empts dc0's commissioning SERVFAIL on the isolated rack. Revert: `git checkout` scripts/dc-rack-net.sh tests/dc-rack-net/run-tests.sh. +## Item 8 -- dc-rack-net install on the dc1 rack (persistent legs + forwarder) + +`install dc1` run on the rack (operator-gated). Check PASS 10/10 +(`docs/audit/dc1-rack-net-install-20260723.txt`): persistent metal-admin legs +`10.12.68.2/22` + `10.12.68.3/22` on virbr6, provider-public `10.12.64.2/22` on +virbr4, `dc1-rack-legs` + `dc1-node-dns` enabled+active. Behavioral proof: the +forwarder answers authoritative `maas-internal SOA` via region BIND -- the +D-131 fix that pre-empts dc0's commissioning SERVFAIL. Revert: +`dc-rack-net.sh` install is idempotent; to undo, disable the two units + remove +the generated files + the interim legs. + +## Item 9 -- region-side MAAS config for dc1 metal-admin (DHCP + DNS + range) + +MAAS auto-discovered dc1 planes from the rack interfaces: metal-admin +`10.12.68.0/22` = subnet id 11 (VLAN fabric 142/vid 0), provider-public +`10.12.64.0/22` = subnet id 10. On subnet 11 (all operator-gated): +- D-120 dynamic range `10.12.68.100-10.12.68.200` created (iprange id 3) -- + the ruled band applied to dc1's CIDR. +- D-131: `dns_servers=10.12.68.3 allow_dns=false` (nodes resolve via the rack + forwarder, not MAAS -- the SERVFAIL fix). +- DHCP: VLAN fabric 142/vid 0 `dhcp_on=true primary_rack=nmpcq4`. +Verified BEHAVIORALLY (dc0 lesson -- not the self-report): dhcpd RUNNING on the +rack (`dhcpd -4 ... virbr6`, dhcpd.conf freshly generated); no Temporal +incident (the dc0 region restart fixed it fleet-wide). Revert: +`maas admin vlan update 142 0 dhcp_on=false`; `subnet update 11 dns_servers= +allow_dns=true`; `ipranges delete 3`. + +## Item 10 -- dc1 enlistment PROVEN (canary) + +`virsh reset vr1-dc1-control-01` -> the node PXE-booted, got DHCP from nmpcq4, +and ENLISTED in MAAS ~2 min later (machine count 11 -> 12). The +DHCP->PXE->enlist chain works end to end for dc1; with the forwarder +pre-installed, dc0's two stacked commissioning faults are pre-mitigated. +Revert: n/a (enlistment; the machine is deleted/re-commissioned as needed). + +## Item 11 -- SEC-016 ruling: dedicated dc1 MAAS->libvirt power key (GA-R5) + +Commissioning needs the region MAAS snap to SSH the dc1 rack libvirt +(`power_type=virsh`). The dc0 rack authorizes the SEC-012 MAAS key; the dc1 +rack does not. Operator RULED (AskUserQuestion, exact utterance): **"Mint a +dedicated dc1 power key"** -- per-DC isolation, NOT cross-DC reuse of SEC-012. +Recorded as **SEC-016** (security-ledger). Wiring (dependent work, next): +mint `vr1-dc1-maas-power_ed25519`; authorize its pubkey on the dc1 rack; +install privkey in the region MAAS snap + snap `ssh config` Host 172.31.0.6; +give the maas-node-power script's virsh reach from voffice1 via the dc1 SERVICE +key (dc0 split). Then `maas-node-power.sh` dc1 (dry -> --commit), commission +9/9. Revert: deauthorize the pubkey on the rack, remove the snap key + config. + ## Next (gated) -Rack standup DoD: `install dc1` on the rack (this arm) +Wire SEC-016 dedicated power key -> `maas-node-power.sh` dc1 (dry then --commit) (provider-public `10.12.64.2/22`, metal-admin `10.12.68.2/22`, forwarder `10.12.68.3` -> region BIND `10.10.0.20`) + harness update, then `install dc1`; region-side MAAS (metal-admin `dns_servers=10.12.68.3 diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 1f396e7..ffa9473 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -25,6 +25,8 @@ | SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk). | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` | operator | **CLOSED 2026-07-16 (operator ruling -- keep uniform 6-NIC).** Review R3-F10: A2's cross-examination refuted the attack-surface concern -- in the isolated-L2 sim the unbound vNICs have no reachability out, and pruning would INCREASE Roosevelt-delta (baremetal trunks all VLANs to every node on bonded NICs, so all planes are present regardless of L3 binding). Uniform 6-NIC is the more Roosevelt-faithful model. Accepted non-issue; no code change. | | SEC-015 | 2026-07-23 | **dc1 edge SSH private key staged transiently on the DC1 rack.** The `vr1-dc1-edge_ed25519` private key (root-granting on `vr1-dc1-opnsense`) was scp'd to `vvr1-dc1:~/vr1-dc1-edge_ed25519` (0600) so `opnsense-bootstrap-apikey.sh` -- which SSHes the edge directly, no ProxyJump -- could mint the API key there (dc0 method). It existed on the rack only for the mint + v4-addressing phase and was `shred -u`'d immediately after (absence verified by `ls`). The permanent copy stays on vcloud `~/vr1-dc1-creds/` (SEC-007 pattern: edge SSH is the only management path -- a rotation obligation, not a delete-me). | 2026-07-23 session changelog item 6; docs/changelog-20260723-g12-dc1-edge.md | operator | **OPEN -- rotation obligation (transient exposure already remediated).** The on-disk rack copy is gone; the standing surface is the vcloud key + the minted API key/secret (`~/vr1-dc1-creds/opnsense-api.txt`). Rotate both at v1 close, or immediately if vcloud is rebuilt/shared. Roosevelt note: metal edges reached differently may avoid the rack-staging step entirely (a ProxyCommand-from-vcloud mint keeps the key off the rack -- deferred, not adopted this build to stay on the dc0-proven path). | +| SEC-016 | 2026-07-23 | **Dedicated dc1 MAAS->libvirt power key (per-DC isolation -- operator-ruled).** dc1 commissioning needs `power_type=virsh`, which has the REGION's MAAS snap SSH to the dc1 rack's libvirt to power-cycle nodes. RULING (GA-R5) -- question presented: "How should the region drive the dc1 rack's libvirt for node power control (power_type=virsh)?" options (A) reuse the SEC-012 key on dc1, (B) mint a dedicated dc1 power key. Operator answer, exact utterance (option selected): **"Mint a dedicated dc1 power key"** -- a SEPARATE per-DC MAAS->libvirt keypair, NOT cross-DC reuse of SEC-012's key. Mechanism: private half in the region MAAS snap (`/var/snap/maas/current/root/.ssh/` + per-host `ssh config` Host 172.31.0.6 -> that IdentityFile, so 172.31.0.2/dc0 keeps SEC-012 and 172.31.0.6/dc1 uses this one); public half authorizes jessea123 (libvirt group) on the dc1 rack. The maas-node-power SCRIPT's own mapping virsh (jessea123 on voffice1) uses the dc1 SERVICE key via voffice1 `ssh config` -- the dc0 split exactly (script=svc key, MAAS=dedicated key). | 2026-07-23 session changelog; docs/changelog-20260723-g12-dc1-edge.md; dc0 precedent SEC-012 | operator | **OPEN -- rotation obligation.** (1) ROTATE at v1 close or if the region/dc1 rack is rebuilt/shared. (2) Same libvirt-group SCOPE hardening candidate as SEC-012 (power-only polkit grant). (3) FRAGILITY: the snap-side key + ssh config live under per-revision `/var/snap/maas/current/` and may not survive a snap refresh -- re-assert after any refresh (queued to DC standup DoD). Roosevelt-relevance: establishes PER-DC power-credential isolation (bare metal: per-DC IPMI/BMC creds, never a shared cross-DC power credential) -- whether that becomes a standing D-principle is a follow-up, not blocking. | + **STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive files AND environment/config files for a site live in a single `~/-creds/` folder on vcloud, mode 0700, files 0600 (public keys 0644). No loose env files in `~`. Sites: `~/vr1-office1-creds/`