diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index 7b4422a..711453d 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -143,6 +143,15 @@ `scripts/maas-node-power.sh` is the ruled shape -- D-103/D-123 amendments 2026-07-20), and a pod delete CASCADES to its linked machine records -- read the pod's machine list BEFORE any `vm-host delete`; non-empty = STOP (appendix-A, 2026-07-21 incident). +**Per-DC power credential (SEC-012 dc0, SEC-016 dc1, RULED 2026-07-23 -- standing +DC-standup invariant): each DC gets its OWN dedicated MAAS->libvirt power key, NEVER +a cross-DC reuse.** The REGION's MAAS snap dials the power address, so the split is: +the maas-node-power SCRIPT's mapping virsh uses the DC SERVICE key (via the region's +`~/.ssh/config` Host ), while MAAS's own power ops use the dedicated +key installed in the snap (`/var/snap/maas/current/root/.ssh/` + a per-host `ssh +config` block so each rack uses its own key). FRAGILITY: the snap-side key lives under +per-revision `/var/snap/maas/current/` and may not survive a snap refresh -- re-assert +after any refresh. Roosevelt analog: per-DC IPMI/BMC credentials. **Change-delivery loop:** grep for prior art (zeroth decision) -> grep design-decisions for the governing D-NNN -> edit -> `bash scripts/repo-lint.sh` diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index e53daad..50278e5 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -26,8 +26,10 @@ - Stage: Stage 3 / Phase 2 -- "OpenTofu builds each DC substrate" (`docs/dc-dc-deployment-workflow.md:148`; runbook `runbooks/dc-dc-phase2-tofu-dc-substrate.md`) -- **CLOSED 2026-07-21 - for its vr1-dc0 scope** (operator-ruled "Close and merge"; dc1 is - the stage's designed HELD remainder, gate G12). Close-out set: + for its vr1-dc0 scope** (operator-ruled "Close and merge"; dc1 was + the stage's designed HELD remainder, gate G12 -- **now also CLOSED + 2026-07-23: dc1 substrate built + commissioned 9/9, merged to `main`, + branch retired; see the G12 gate row**). Close-out set: gauntlet ALL GREEN + repo-lint 0-fail + this consolidation commit + GA-R7 memory review + merge of `dc-dc-stage3-phase2-dc-substrate` to `main` (merge commit) + branch retirement; stage record @@ -475,7 +477,7 @@ | G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via `site-headend-install.sh --host-nodes --check` on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched | | G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (`docs/audit/stepE-netem-20260721.txt` + `outer-{plan,apply}-20260721-netem*.txt`). **G10 CLOSED 2026-07-21** | | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | -| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED + WIRED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). **COMMISSIONING 9/9 READY 2026-07-23** (`docs/audit/dc1-commissioning-verify-20260723.txt`): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, `power_type=virsh` set + verified by real query-power-state (SEC-016 path proven), commissioned to **ALL 9 READY in ~3.5 min** (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. **G12 [V] leg (the dc1 build) is COMPLETE.** NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, **operator-gated merge of `dc-dc-g12-dc1-substrate` -> `main`** (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close). | +| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | CLOSED 2026-07-23 (operator-ruled "Merge to main + full close"; commissioning 9/9 READY, merge commit on `main`, branch retired) -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED + WIRED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). **COMMISSIONING 9/9 READY 2026-07-23** (`docs/audit/dc1-commissioning-verify-20260723.txt`): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, `power_type=virsh` set + verified by real query-power-state (SEC-016 path proven), commissioned to **ALL 9 READY in ~3.5 min** (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. **G12 [V] leg (the dc1 build) is COMPLETE.** NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, **operator-gated merge of `dc-dc-g12-dc1-substrate` -> `main`** (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close). | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | OPEN (execution only; decision content complete) | | G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether `opentofu/vr1-dc0-maas` is retired | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-20 | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 remains PROPOSED/OPEN (items 2-3 + the item-1 re-scoped migration plan) | diff --git a/docs/archive/changelogs/changelog-20260721-g12-dc1-authoring.md b/docs/archive/changelogs/changelog-20260721-g12-dc1-authoring.md new file mode 100644 index 0000000..652a4ab --- /dev/null +++ b/docs/archive/changelogs/changelog-20260721-g12-dc1-authoring.md @@ -0,0 +1,149 @@ +# 2026-07-21 -- G12 [V] leg: vr1-dc1 substrate authoring (session changelog) + +Session scope: the G12 remaining [V] leg per the D-124 AMENDMENT (2026-07-21): +apex confirm-free -> vr1_dc1 vars -> dc1 substrate authoring -> gated build. +Branch: `dc-dc-g12-dc1-substrate` (off post-Stage-3-merge `main`, per the +stage-close rule). One changelog for the session (GA-R2/D1); every item +carries its revert. + +## Item 1 -- dc1 apex confirm-free capture (read-only; the [V] leg's first step) + +Read-only GETs on office1-netbox (token used on-host, never printed), captured +to `docs/audit/dc1-apex-confirm-20260721.txt`. Findings: + +- The six dc1 plane /22s ALREADY EXIST in the apex, scoped `vr1-dc1`, exactly + matching the ratified D-124-amendment scheme (provider-public 10.12.64.0/22 + ... replication 10.12.84.0/22, roles per D-101). Confirm-CONSISTENT: the + assignment landed at the original D-101/D-115 import; the ruled derivation + matches it. +- Transit 172.31.0.4/30 FREE (container 172.31.0.0/24 + dc0's /30 are its only + occupants; zero ip-addresses in the /30). +- Uplink 172.30.3.0/24 already registered to vr1-dc1 (D-115 edge import, + 2026-07-17). Rack IP 10.12.68.2 + forwarder .3 free (zero ip-addresses in + 10.12.68.0/22). Site `vr1-dc1` and role `transit` exist. + +Consequence: the ONLY apex write still owed for dc1 is the transit /30 + rack +metal-admin IP -- exactly the dc-rack-mgmt-import.py surface (dc0 precedent: +those are the only two objects dc0 registered either; transit endpoint IPs and +the D-131 forwarder alias are NOT apex objects for dc0 and dc1 mirrors that). + +Revert: delete the capture file (no state touched). + +## Item 2 -- dc-rack-mgmt-import.py: per-site support (vr1-dc0 + vr1-dc1) + +What: the importer was dc0-hardcoded (SITE_SLUG/METAL_ADMIN/RACK_DNS/desc +constants). Replaced with a `SITES` map (dc0: 10.12.8.0/22 / vvr1-dc0, ratified +2026-07-16; dc1: 10.12.68.0/22 / vvr1-dc1, D-124 amendment 2026-07-21) selected +by a new REQUIRED `--site {vr1-dc0,vr1-dc1}` flag (env: RACK_SITE). Explicit by +design: one DC's values can never land scoped to another site. An env-supplied +site is validated against SITES in code (argparse `choices` does not validate +env-default values). ROLE_SLUG / CONTAINER / D-120 band offsets stay global +(convention-wide). Also generalized dc0-specific error/usage text and fixed two +stale "(Cloud)" labels on the container messages (the container has been +172.31.0.0/24, not Cloud, since 2026-07-16). + +Why: unblocks the one owed dc1 apex write; the D-124 scheme "generalizes +per-leg" and the tool now encodes that instead of forking a dc1 copy. + +Harness: `tests/dc-rack-mgmt-import/` extended in the same change -- all CLI +cases pass `--site`, new cases: missing `--site` dies; bogus RACK_SITE env +dies; dc1 happy path (site id binding, /22 mask, dns vvr1-dc1); cross-site +guard (dc0 rack IP under --site vr1-dc1 rejected, zero writes); dc1 band edges +.2/.49 accepted, .1 gateway rejected; missing vr1-dc1 site precondition dies. +Structural pins moved from the retired globals to the SITES map. +**117/117 PASS** (was 96). + +Live dry-run vs the real apex (read-only, on office1-netbox): +`docs/audit/dc1-rack-import-dryrun-20260721.txt` -- whole preflight PASS, +plan = would-create exactly 172.31.0.4/30 (role transit, scope vr1-dc1) + +10.12.68.2/22 (dns vvr1-dc1). The `--commit` is a GATED operator step (apex +mutation), not run here. + +Revert: `git checkout main -- netbox/dc-rack-mgmt-import.py tests/dc-rack-mgmt-import/` +(restores the dc0-only tool); delete the two capture files. No apex state +touched (dry-run only). + +## Item 3 -- IN FLIGHT AT DISCONNECT (drop-state record, landed 2026-07-22 by the successor session; NOT a delivery) + +The session DISCONNECTED mid-item, before its bookend. This item records what +exists so the next session resumes without re-derivation; nothing here claims +delivery. Authored and sitting UNCOMMITTED in the tree at the drop: + +- `opentofu/variables.tf`: `vr1_dc1_planes` (apex-verified D-124-amendment + /22s), `vvr1_dc1_{vcpu,memory_mib,disk_bytes}` sizing, `vr1_dc1_ssh_pubkey_path`, + and the deliberately NO-DEFAULT rack/transit vars (tfvars-only after the + gated apex `--commit` -- hard rule 2). +- `opentofu/main.tf`: `module "vr1_dc1_uplink"` (172.30.3.0/24) + + `module "vvr1_dc1"` containment VM (dc0 pattern of record, dedicated dc1 key). +- `opentofu/vr1-dc1-substrate/` (NEW): full inner root mirroring dc0's + (same file set + variable shape); all 54 node MACs PRE-PINNED from first + apply, scheme `52:54:01:d1:NN:PP` (locally-administered, outside libvirt's + 52:54:00 auto space). +- `scripts/lib-net.sh`: vr1-dc1 arm (literals land; unruled OpenStack-layer + values UNSET) + a one-selection-per-shell guard. +- `tests/dc-selector/run-tests.sh`: reconciled, 40/40 PASS. + +DROP POINT: after the dc-selector harness went green, BEFORE the full +gauntlet. Successor-run gauntlet = **6/76 FAILED** (capture +`docs/audit/gauntlet-20260722-g12-dropstate.txt`): carve-host-interfaces, +dc-dc-dr-drill, dc-dc-radosgw-multisite, dc-dc-rbd-mirror, +phase-00-maas-standup, reenroll-hosts -- all still assert the retired +"vr1-dc1 fails loud (NetBox gap)" selector behavior that the new dc1 arm +retires. Also sighted for the reconcile sweep: `tests/dc-dc-prefixes-import/ +test_logic.py` greps the old message (currently passing); dc-dc-rbd-mirror's +reminder text prints dc0's replication CIDR under `dc=vr1-dc1` (cross-DC +value surfacing). repo-lint 0-fail; `tofu fmt -check` clean on all touched tf. + +OWED to finish the item: reconcile the six harnesses (+ the two sighted +surfaces) -> gauntlet ALL GREEN -> delivery entry here -> commit + push. +THEN G12's next gated step is unchanged: operator-gated apex `--commit` +(transit /30 + rack IP), tfvars, build. + +Revert (drops the in-flight work, NOT items 1-2): `git checkout -- +opentofu/main.tf opentofu/variables.tf scripts/lib-net.sh +tests/dc-selector/run-tests.sh && rm -rf opentofu/vr1-dc1-substrate/`; +delete the gauntlet capture file. + +## Item 4 -- harness reconcile: item 3 delivered (successor session, 2026-07-22) + +What: the six red harnesses reconciled to the ratified dc1 selector arm, plus +the two sighted surfaces dispositioned. Per script: + +- `phase-00-maas-standup.sh`: NEW PLANES/lib-net parity guard (exit 2, + precondition) -- the PLANES table is a DC0-hardcoded D-052/D-053 literal, and + with dc1 now selectable the script would otherwise have planned DC0 topology + under `DC=vr1-dc1` (the drop-state gauntlet showed exactly that: exit 0 + + WOULD: leak). Any selected DC whose lib-net plane set diverges from the table + is refused. Harness asserts the refusal (exit 2, no WOULD:/DO: leak). +- `dc-dc-rbd-mirror.sh` + `dc-dc-radosgw-multisite.sh`: the sighted cross-DC + value surfacing FIXED IN BOTH (radosgw carried the same defect as the sighted + rbd-mirror): the $DC gate ran only in a subshell, so the replication-plane + reminder always read the file's flat DC0 defaults -- under `--dc vr1-dc1` it + printed 10.12.36.0/22 (dc0's). On gate success the script now selects the DC + in its own shell; new harness cases pin the dc1 reminder to 10.12.84.0/22. +- `dc-dc-dr-drill` / `radosgw-multisite` / `rbd-mirror` harnesses: the dc1 + gate-refusal tests (want rc 3) became gate-OK dry-run + juju-guarded --apply + (rc 2) tests. The enforce path (exit 3) is retained in the scripts as + defense-in-depth but is no longer CLI-reachable (arg validation admits only + the two ratified tokens) -- noted in the test comments. +- `carve-host-interfaces` / `reenroll-hosts` harnesses: vr1-dc1 now fails at + the HOSTS layer like vr1-dc0 ("no enrolled hosts yet"); tests assert that + message + the no-mutation guard. Stale dc1/dc2-era header comments in both + scripts (and phase-00) updated. +- Sighting re-check: `tests/dc-dc-prefixes-import/` carries NO grep of the + retired selector message at HEAD -- the drop-state sighting does not + reproduce; harness passes unmodified (91 checks). No edit made there. + +Verification: the six harnesses individually green; full gauntlet **ALL GREEN +(76 harnesses)** -- `docs/audit/gauntlet-20260722-g12-reconcile.txt`; +repo-lint 0 fail. This item + the item-3 tree state land in one commit +(this delivery); G12's next gated step is unchanged: operator-gated apex +`--commit`, then tfvars + gated build. + +Revert: `git checkout main -- scripts/phase-00-maas-standup.sh +scripts/dc-dc-rbd-mirror.sh scripts/dc-dc-radosgw-multisite.sh +scripts/carve-host-interfaces.sh scripts/reenroll-hosts.sh +tests/phase-00-maas-standup/ tests/dc-dc-rbd-mirror/ +tests/dc-dc-radosgw-multisite/ tests/dc-dc-dr-drill/ +tests/carve-host-interfaces/ tests/reenroll-hosts/`; delete the reconcile +gauntlet capture (item-3 revert covers the authored substrate). diff --git a/docs/archive/changelogs/changelog-20260722-g12-dc1-build.md b/docs/archive/changelogs/changelog-20260722-g12-dc1-build.md new file mode 100644 index 0000000..9e8ef69 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260722-g12-dc1-build.md @@ -0,0 +1,205 @@ +# 2026-07-22 -- G12 [V] leg: dc1 apex write + substrate build step A (session changelog) + +Session scope: continue the G12 [V] leg from the reconciled tree (predecessor +delivery landed as changelog-20260721-g12-dc1-authoring.md items 3-4, commit +`d2bf743`): the one owed apex write, tfvars, and the gated outer build. +Branch: `dc-dc-g12-dc1-substrate`. One changelog for the session (GA-R2/D1); +every item carries its revert. + +## Item 1 -- apex `--commit` EXECUTED (the one owed dc1 apex write; operator-gated) + +Same-session read-only preflight FIRST (verify-before-mutate): dry-run re-run +against the live apex = would-create exactly 2, already-present 0 -- +`docs/audit/dc1-rack-import-dryrun-20260722.txt`, identical to the 07-21 +preflight. Then the gated write (piped to office1-netbox over ssh; token +sourced on-host from `/root/netbox-secrets/api.token` per +`creds-manifests/vr1-office1.manifest`, value never printed or brought into +context): + +- CREATED `172.31.0.4/30` (prefix id=139) role=transit scope=dcim.site:vr1-dc1 +- CREATED `10.12.68.2/22` (ip id=4) dns=vvr1-dc1 + +Post-commit idempotency read-back: would-create 0 / already-present 2. Both +runs captured in `docs/audit/dc1-rack-import-commit-20260722.txt`. Values +verbatim from the D-124 amendment (ruled 2026-07-21); exactly the two objects +the dc0 precedent registered. + +Revert: delete the two apex objects by id (prefix 139, ip-address 4) via the +NetBox UI/API on office1-netbox; delete the capture files. + +## Item 2 -- vr1-dc1 service keypair MINTED (manifest-prescribed at-deploy step) + +`~/vr1-dc1-creds/vr1-dc1_svc_ed25519{,.pub}` generated on vcloud (ed25519, +modes 600/644) per `creds-manifests/vr1-dc1.manifest` ("MINTED AT DC DEPLOY"). +`bash scripts/creds-audit.sh vr1-dc1` = CLEAN. D-126 option (a) per-env key; +pubkey feeds vvr1-dc1 cloud-init (item 3), private half jumphost-local. + +Revert: remove both key files (couples to item 4 -- vvr1-dc1's cloud-init +authorizes this pubkey; re-mint requires a seed rebuild). + +## Item 3 -- dc1 tfvars authored (LOCAL, gitignored -- recorded here, not committed) + +`opentofu/d124-rack.auto.tfvars` gained the five vr1_dc1_* values, verbatim +from the D-124 amendment's own tfvars line (rack_metal_admin_ip=10.12.68.2, +rack_transit_ip=172.31.0.6, rack_transit_prefix=30, +rack_transit_peer_ip=172.31.0.5) + `vr1_dc1_ssh_pubkey_path` to the item-2 +pubkey. Stale "dc1 DEFERRED" header comment replaced. `tofu fmt` applied; +`scripts/opentofu-validate.sh` PASS (all roots). + +Revert: remove the vr1_dc1_* block from the local file. + +## Item 4 -- outer apply: dc1 substrate step A (operator-gated, logged, saved-plan exact) + +Preconditions measured in-session: host RAM 1007 GiB with 450 GiB committed +(voffice1 32 + edge 2 + vvr1-dc0 416) -> +416 GiB fits with ~140 GiB headroom; +vCPU 234/256 post-apply; dc-dc-whole-host-budget 13/13 PASS. + +Saved plan `tfplan-dc1-20260722` = **5/0/0 exact** (vr1-dc1-uplink network + +vvr1-dc1 domain/disk/seed/cloudinit; ZERO touches to live resources) -- +capture `docs/audit/outer-plan-20260722-dc1-substrate.txt`. Applied via the +saved plan under a per-command as-executed wrap +(`~/as-executed/2026-07-22-dc1-deploy.log`; index row added): **5 added, 0 +changed, 0 destroyed**. Convergence re-plan = **zero diff** +(`docs/audit/outer-plan-20260722-postdc1-converged.txt`). Live verify: +vvr1-dc1 RUNNING (Id 8), voffice1/office1-opnsense/vvr1-dc0 untouched and +running; vr1-dc1-uplink + both dc1 mesh legs active. + +Revert: `runbooks/dc-dc-teardown-rollback.md` decision tree; the targeted +destroy set is the five applied resources (module.vvr1_dc1.* + +module.vr1_dc1_uplink.*); delete the plan/capture files and the tfplan. + +CORRECTION (same session): commit `0bd6342` accidentally included the spent +binary saved plan (repo-lint L1 x2 -- the lint's exit code was masked by a +pipeline; owned). The file deviated from the `.gitignore` naming convention +(`opentofu/**/*.tfplan`) -- saved plans must be named `*.tfplan`. Removed in +the follow-up commit; lint back to 0-fail. The applied-plan record remains +the two dated captures. + +## Item 5 -- cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (gated apply) + +What: `modules/cloudinit-vm` gained `interface_macs` (ported verbatim-in-intent +from `modules/node-vm`, same validations; harness +4 cases, cloudinit-vm 10/10) +-- the 2026-07-20 voffice1 Kea/MAC-regen incident class, closed at the module +layer. Root wiring: `module.voffice1` NIC3 -> `mesh-vr1-dc1-office1` with all +three MACs pinned (NIC1/NIC2 = measured live values; NIC3 = pre-pinned +`52:54:01:d1:fe:01`, dc1 scheme, fe = region-side); `module.vvr1_dc1` pin-adopts +its two measured step-A MACs (standup DoD invariant). Saved plan +`voffice1-dc1nic-20260722.tfplan` = **0/2/0, zero replaces**, all five MAC +values verified in the diff (`docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`); +applied logged (trap 1e: BOTH domains bounced -- presented as such). Post-bounce +battery ALL PASS: 4/4 domains running, voffice1 MACs exactly as pinned, Kea +lease 10.10.0.20 intact, region dhcpd up, netbox 302 + tailscale up (nested LXD +VMs self-recovered, ~3 min), dc0 rack: dhcpd + dc0-node-dns + dc0-rack-legs +active, forwarder SOA answers. Convergence re-plan ZERO DIFF. + +Revert: remove NIC3 + interface_macs from main.tf, revert the module + +harness (`git checkout` the four files), re-plan/apply (bounces again). + +## Item 6 -- voffice1 dc1 transit leg (in-guest netplan, gated) + +`/etc/netplan/61-transit-dc1.yaml` (0600) on voffice1: enp3s0 static +`172.31.0.5/30` (per-DC drop-in; dc0's `60-transit.yaml` untouched). Verified: +enp3s0 UP with .5/30, ping 172.31.0.6 = 0% loss, first ssh into vvr1-dc1 over +the transit with the dc1 key OK (hostname vvr1-dc1, nested-KVM module +present, region route + 10.10.0.20 reach pre-staged by step-A cloud-init). +Revert: remove the drop-in + `netplan apply`. + +## Item 7 -- rack bootstrap EXECUTED (site-headend-install, dc1-parameterized) + +Enrollment secret staged region->rack as a host-to-host pipe (0600 +`/root/region-enroll.secret`, 32 bytes verified by count, value never in +context). Snap system proxy set to `http://10.10.0.20:8000` (dc0-measured +mechanism; env proxy for apt). Dry-run FIRST exposed that the script's +host-nodes DEFAULTS are dc0-flavored -- all overridable by existing flags; +run with `--wan-bridge br-vr1-dc1-wan --inner-pool-path +/var/lib/libvirt/vr1-dc1-inner --opnsense-base ...26.7...`. First real run +FAILED exit 4: stale base-image apt index -> 404s on superseded debs via the +proxy; fixed with `apt-get update`, idempotent re-run **exit 0**: rack +ENROLLED, nested KVM on, inner pool + AppArmor, SEC-010 rack end, WAN bridge +verified with enp2s0 enslaved. Region-side verify: `maas admin +rack-controllers read` lists **vvr1-dc1 (nmpcq4)**. SEC-010 REGION end: +voffice1's `/etc/nftables-sec010.nft` extended with the enp3s0 drop pair +(dc0 idiom), table reloaded clean, ruleset = both legs dropped. +`--check` capture: `docs/audit/dc1-stepB-check-20260722.txt` (sole [--] = +opnsense base, item 8). + +QUEUED findings (logged, not fixed mid-step): (a) site-headend-install +NOTE/hint prose still says "26.1" and "vr1-dc0-substrate" even when +parameterized for dc1 (cosmetic, misleads operators); (b) +`nftables-sec010.nft` reload is NOT idempotent -- `nft -f` on a live table +appends, so a service restart duplicates rules (observed; cleaned via +`nft delete table` + restart; the file should flush first); (c) +`opnsense-prep-image.sh` dies on `BASH_SOURCE[0]: unbound variable` when +piped via `bash -s` under `set -u` (ran into it before the mirror guard). + +Revert: rack side -- `snap remove maas`, remove nftables-sec010 + +sec010-fw + kvm-nested modprobe + inner pool dir on vvr1-dc1; region side -- +remove the enp3s0 pair from voffice1's nft file + restart unit; delete +the staged secret file. + +## Item 8 -- OPNsense 26.7 base staged on dc1 (proven-artifact copy) + +`opnsense-prep-image.sh` requires `OPNSENSE_MIRROR_BASE` (deliberately not +repo-recorded; mirrors change). Instead of choosing a mirror, the dc0 rack's +OPERATOR-RULED and boot-PROVEN 26.7 base was streamed rack->rack through the +jump path (direct rack-to-rack is SEC-010-dropped, correctly) and +sha256-verified on arrival against the dc0 source hash +(`3981821e3a3c...476627d`). Same bits that passed the D-112 boot path on dc0. +Revert: delete `/var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2`. + +## Item 9 -- inner apply: dc1 substrate step C EXECUTED (from voffice1, D-128 Plane 2) + +Staging on voffice1 (dc0-precedent set): repo clone fetched + switched from the +retired stage-3 branch to `dc-dc-g12-dc1-substrate` @ 61c416e (dc0 inner +tfstate untouched, verified); dc1 private key piped in (0600, 432 bytes by +count); 172.31.0.6 host keys scanned into known_hosts (all 3 types -- dc0 +trap); the 26.7 base copied rack->voffice1 to `~/vr1-dc1-images/` and +sha256-verified (provider streams volume content itself -- the dc0 measured +fix; rack-side copy satisfies only the bootstrap check). Inner tfvars +`d124-inner.auto.tfvars` written on voffice1, every value sourced (measured +transit .6, outer-committed planes map verbatim, Stage-1 mtu 9000, D-106 +suffix, keyfile + local base paths). + +Plan `dc1-inner-20260722.tfplan` = **28/0/0** (dc0's exact step-C count), +verified in-capture: 54 pinned `52:54:01:d1:*` MACs, 9 Option-C nodes + edge, +qga channel present (`docs/audit/inner-plan-20260722-dc1-stepC.txt`). First +apply FAILED at 18/28: the edge domain's serial-log dir +`/var/lib/libvirt/vr1/staging/` did not exist inside vvr1-dc1 (hand-created +on dc0 during the 07-20 serial work; QUEUED finding -- belongs in the +bootstrap/standup DoD so dc2+ does not repeat it). Fix-forward per the +rollback tree: dir created (root:root 0755, dc0 mirror), resume plan 10/0/0, +**apply exit 0**. Final: 28/28 in state, convergence ZERO DIFF +(`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING +inside vvr1-dc1**, edge serial log at the FreeBSD login prompt -- OPNsense +26.7, factory LAN 192.168.1.1/24, WAN unaddressed (second D-112 boot-path +datapoint on 26.7; exactly dc0's post-step-C state). The dc1 INNER tfstate +lives ON voffice1 (`opentofu/vr1-dc1-substrate/terraform.tfstate` -- add to +the site backup set, same as dc0's). + +Revert: `dc-dc-teardown-rollback.md` tree; inner root destroy from voffice1 +tears down the 28 (containment VM unaffected). + +## Item 10 -- D-125 egress isolation gate: PASS (vr1-dc1) + +Procedure of record = dc0's `d125-egress-test.sh` (recovered from the dc0 +rack's home), adapted per-DC (pool `vr1-dc1-inner`, bridge `br-vr1-dc1-wan`, +probe 172.30.3.50/24 via .1 -- D-124 amendment /24) and staged to the dc1 +rack. Prereqs installed via the region proxy (virtinst, cloud-image-utils); +`/tmp/noble.img` streamed dc0->dc1. **Two identical consecutive runs**: +GW-PING-RC=0, NET-PING-RC=0, NET-TCP-http=301, NET-TCP-ubuntu=200, +LAN-TCP-http=000 (the region-isolation check), leftover-domains=0 -- +`docs/audit/d125-egress-gate-20260722-dc1.txt`. Bridge-in egress PROVEN +end-to-end for dc1; the double-NAT fallback is NOT needed. dc0's +one-time unexplained first-run ICMP failure did NOT recur. + +Revert: none needed (throwaway fully torn down, verified); delete the +capture + the staged script + /tmp/noble.img on the rack if desired. + +## Next (gated, not run here) + +Edge bootstrap (D-112(c) console -> key-only SSH -> D-113(a2) API key on +26.7); edge addressing via `opnsense-set-interface-v4` (WAN 172.30.3.2/24 +gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); rack standup DoD +(dc-rack-net.sh install dc1 + forwarder 10.12.68.3, region-side DHCP on +metal-admin naming nmpcq4 primary_rack, dynamic range 10.12.68.100-.200 +per D-120, maas-node-power dc1 arm). Runbook + CURRENT-STATE govern. diff --git a/docs/archive/changelogs/changelog-20260723-g12-dc1-edge.md b/docs/archive/changelogs/changelog-20260723-g12-dc1-edge.md new file mode 100644 index 0000000..2c17800 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260723-g12-dc1-edge.md @@ -0,0 +1,229 @@ +# 2026-07-23 -- G12 [V] leg: dc1 edge bootstrap + v4 addressing (session changelog) + +Session scope: resume the G12 [V] leg from the 2026-07-22 close handoff +(edge bootstrap onward). Branch `dc-dc-g12-dc1-substrate`. One changelog per +session (GA-R2/D1); every item carries its revert. Live mutations gated +(operator ran in `manual` permission mode -- see Item 0). Edge-phase mutations +mostly run UNWRAPPED (approval prompt = the gate); the console bootstrap went +through the `~/as-executed/2026-07-22-dc1-deploy.log` wrap. + +## Item 0 -- permission-mode friction (recorded; no repo change) + +The session opened in `auto` mode, whose classifier BLOCKS remote +`sudo`/mutation shapes (matched by the project `ssh * sudo *` ask rule) +instead of surfacing them for approval -- so it cannot deliver "operator +approves each mutation." Resolved by switching to `manual` mode (alias for +`default`; no classifier): allow-rules flow read-only rack probes, project ask +rules prompt every gated mutation, deny rules + `guard-destructive.py` stay +active. Session allow-rules for the dc1/dc0 rack ssh shapes were added to +`.claude/settings.local.json` (the operator later broadened to `ssh *`). +Revert: none (session-scoped settings; remove the added allow lines at close +if desired). + +## Item 1 -- dc1 edge service keypair minted + manifest + +`~/vr1-dc1-creds/vr1-dc1-edge_ed25519` (0600/0644) minted on vcloud, +`creds-manifests/vr1-dc1.manifest` extended (edge key + `opnsense-api.txt`), +`creds-audit vr1-dc1` CLEAN (5 entries). Revert: `git checkout` the manifest; +delete the keypair. + +## Item 2 -- D-112(c) console bootstrap COMPLETE (edge SSH + service key) + +Driver `d112c-console-dc1.py` (dc1 adaptation of the dc0-PROVEN v6, retrieved +verbatim from `vvr1-dc0:~/d112c-console.py`): serial console -> factory +root/opnsense -> shell -> ship+run bootstrap PHP (enable ssh, permitrootlogin, +install dc1 edge pubkey, `write_config`) -> materialize PHP +(`local_user_set()`, dc0 lesson v). Transcript `~/d112c-console-dc1.log` on +the rack: `CONFIG-WRITTEN`, `AK-IN-CONFIG=152`, `USER-MATERIALIZED`; +`/root/.ssh/authorized_keys` (115 B) created (was absent). Edge banner +confirms **vtnet0 = LAN 192.168.1.1/24, vtnet1 = WAN** (dc0 mapping). +**FINDING (logged, not fixed mid-step):** the dc0 `d112c-bootstrap.php` on the +rack required only `config.inc`, so my first bootstrap PHP threw +`Call to undefined function shell_safe()` at `config.inc:311` (in +`write_config()->make_config_revision_entry()`) and aborted BEFORE writing -- +this is **dc0 lesson (iv)** (shell_safe lives in `util.inc`, measured on dc0's +identical 26.7 image). Added `require_once("util.inc")`; two diagnostic lines +made csh-safe (edge root shell is tcsh; `2>&1` = "Ambiguous output redirect"). +Re-ran clean. The proven dc0 `.b64` bootstrap artifact is missing this include +-- a real gap if replayed; queue: fold `util.inc` into the canonical console +bootstrap payload. Revert: config-only; re-run factory reset on the edge to +undo (or leave -- it is the intended state). + +## Item 3 -- key-only SSH to the edge VERIFIED (D-112(c) proof) + +Reached the edge via a ProxyCommand chain from vcloud (rack key for the jump, +edge key for the final hop -- per-hop keys; a plain `-J` applies one key to all +hops and the rack rejected it). `root@192.168.1.1` -> `uname -r` = +`15.1-RELEASE-p1`, `ifconfig -l` = vtnet0/vtnet1/lo0/enc0/pfsync0/pflog0. +Second D-112(c) datapoint after dc0. + +## Item 4 -- D-113(a2) API key MINTED + smoke test PASS + +Interim reach leg `192.168.1.2/24` added on the rack provider-public bridge +`virbr4` (measured this session) to reach the factory edge LAN. Edge-config +scripts + edge key staged on the rack (dc0 method; the mint script SSHes +edge-direct, no ProxyJump). Key minted via the vendor `opnsense-mint-apikey.php` +(`apikeys->add()` -- the GUI's own path); key/secret 80 chars, 173 B -> +`~/opnsense-api.txt` (0600, secret never printed). Smoke test from the rack: +`GET core/firmware/status` -> exit 0, `product_abi 26.7`. First proof the +D-113(a2) API path works on 26.7 for dc1. +**FINDING (logged):** `opnsense-bootstrap-apikey.sh`'s first `scp` to the edge +failed once with `scp: Connection closed` -- TRANSIENT: the edge sshd was not +ready in the moment right after the console bootstrap's `configctl openssh +restart` (both default `scp` and `scp -O` succeeded minutes later; the edge is +reachable and scp works). Unblocked via an ssh-pipe mint (vendor PHP shipped +by `ssh 'cat >'`, retrieved by `ssh -n 'cat' > file`) -- during which a +missing `-n` on the edge ssh calls let them EAT the rack `bash -s` heredoc +stdin (mint ran, retrieve/cleanup lines were swallowed); re-run with `-n` +retrieved the already-minted key cleanly. Queue: a post-restart sshd +readiness wait / one scp retry in `opnsense-bootstrap-apikey.sh`. Revert: delete +the API key via the edge API + remove `~/vr1-dc1-creds/opnsense-api.txt`. + +## Item 5 -- edge v4 addressing (WAN then LAN; reach-handoff) + +`opnsense-set-interface-v4.sh` run VERBATIM from the rack (plain scp works +after warm-up; no repo edit). Values confirmed from committed `lib-net.sh` +(provider-public `10.12.64.0/22` gw `10.12.64.1`, D-124 amendment) + the D-124 +uplink `172.30.3.0/24`. +- **WAN first** (script ordering rule): dhcp -> `172.30.3.2/24` gw + `172.30.3.1`, applied + read back on the kernel (vtnet1). Edge egress + VERIFIED: ping 1.1.1.1 = 0% loss. +- **Permanent rack leg** `10.12.64.2/22` added on `virbr4` (so the rack keeps + reach after the edge LAN moves). +- **LAN last**: `192.168.1.1/24` -> `10.12.64.1/22` (vtnet0). The apply drops + the `192.168.1.1` session mid-command (expected; the foreground ssh hung on + the dead interface and was TaskStop-ped -- config was already saved+applied). + Verified independently at the NEW LAN via the rack's `10.12.64.2` leg: + `ifconfig vtnet0` = `inet 10.12.64.1 netmask 0xfffffc00` (/22); API + `GET core/firmware/status` at `10.12.64.1` -> exit 0, abi 26.7. +- Interim `192.168.1.2/24` removed; `virbr4` now carries only `10.12.64.2/22` + (dc0 end-state exactly). +Revert: re-address LAN/WAN back via the same script; the leg adds/dels are +`ip addr add/del` (non-persistent). + +## Item 6 -- creds consolidated + rack edge-key WIPED (SEC-015 closed transient) + +API creds pulled rack -> `~/vr1-dc1-creds/opnsense-api.txt` (0600, 1 key + 1 +secret). Rack edge-key copy `shred -u`'d; staged edge-config scripts + +console driver + rack creds copy removed (`ls` confirms none remain). The +non-secret console transcript `~/d112c-console-dc1.log` retained as an audit +artifact. **SEC-015** opened for the transient edge-key-on-rack exposure +(minted->wiped same phase; disclosed). Revert: n/a (cleanup). + +## Milestone + +dc1 edge fully bootstrapped + addressed -- D-112(c) console bootstrap and +D-113(a2) REST API both PROVEN on 26.7 (second datapoint after dc0). Edge: +WAN `172.30.3.2/24` gw `172.30.3.1` (egress 0% loss), LAN `10.12.64.1/22` +(the ruled provider-public gateway), API answering at the new LAN. + +## Item 7 -- dc-rack-net.sh dc1 site-table arm + harness (D-131 sub-1 delivery) + +Added the `dc1)` arm to `scripts/dc-rack-net.sh` (the STANDING per-DC forwarder ++ rack-legs pattern), MEASURED/apex-committed values, network-name keyed (no +virbrN literal -- T6 holds): + vr1-dc1-metal-admin 10.12.68.2/22 rack MAAS/DHCP leg (apex-committed rack IP) + vr1-dc1-metal-admin 10.12.68.3/22 node-DNS forwarder listen alias (D-120 static) + vr1-dc1-provider-public 10.12.64.2/22 edge-LAN leg (measured live on the bridge 2026-07-23) + DNS_LISTEN=10.12.68.3 DNS_UPSTREAM=10.10.0.20 (region BIND over transit). +Harness `tests/dc-rack-net/` extended T15-T18 (dc1 identity + MEASURED-tag); +**18/18**, gauntlet **ALL GREEN (76)**, repo-lint 0-fail. This forwarder is the +D-131 fix that pre-empts dc0's commissioning SERVFAIL on the isolated rack. +Revert: `git checkout` scripts/dc-rack-net.sh tests/dc-rack-net/run-tests.sh. + +## Item 8 -- dc-rack-net install on the dc1 rack (persistent legs + forwarder) + +`install dc1` run on the rack (operator-gated). Check PASS 10/10 +(`docs/audit/dc1-rack-net-install-20260723.txt`): persistent metal-admin legs +`10.12.68.2/22` + `10.12.68.3/22` on virbr6, provider-public `10.12.64.2/22` on +virbr4, `dc1-rack-legs` + `dc1-node-dns` enabled+active. Behavioral proof: the +forwarder answers authoritative `maas-internal SOA` via region BIND -- the +D-131 fix that pre-empts dc0's commissioning SERVFAIL. Revert: +`dc-rack-net.sh` install is idempotent; to undo, disable the two units + remove +the generated files + the interim legs. + +## Item 9 -- region-side MAAS config for dc1 metal-admin (DHCP + DNS + range) + +MAAS auto-discovered dc1 planes from the rack interfaces: metal-admin +`10.12.68.0/22` = subnet id 11 (VLAN fabric 142/vid 0), provider-public +`10.12.64.0/22` = subnet id 10. On subnet 11 (all operator-gated): +- D-120 dynamic range `10.12.68.100-10.12.68.200` created (iprange id 3) -- + the ruled band applied to dc1's CIDR. +- D-131: `dns_servers=10.12.68.3 allow_dns=false` (nodes resolve via the rack + forwarder, not MAAS -- the SERVFAIL fix). +- DHCP: VLAN fabric 142/vid 0 `dhcp_on=true primary_rack=nmpcq4`. +Verified BEHAVIORALLY (dc0 lesson -- not the self-report): dhcpd RUNNING on the +rack (`dhcpd -4 ... virbr6`, dhcpd.conf freshly generated); no Temporal +incident (the dc0 region restart fixed it fleet-wide). Revert: +`maas admin vlan update 142 0 dhcp_on=false`; `subnet update 11 dns_servers= +allow_dns=true`; `ipranges delete 3`. + +## Item 10 -- dc1 enlistment PROVEN (canary) + +`virsh reset vr1-dc1-control-01` -> the node PXE-booted, got DHCP from nmpcq4, +and ENLISTED in MAAS ~2 min later (machine count 11 -> 12). The +DHCP->PXE->enlist chain works end to end for dc1; with the forwarder +pre-installed, dc0's two stacked commissioning faults are pre-mitigated. +Revert: n/a (enlistment; the machine is deleted/re-commissioned as needed). + +## Item 11 -- SEC-016 ruling: dedicated dc1 MAAS->libvirt power key (GA-R5) + +Commissioning needs the region MAAS snap to SSH the dc1 rack libvirt +(`power_type=virsh`). The dc0 rack authorizes the SEC-012 MAAS key; the dc1 +rack does not. Operator RULED (AskUserQuestion, exact utterance): **"Mint a +dedicated dc1 power key"** -- per-DC isolation, NOT cross-DC reuse of SEC-012. +Recorded as **SEC-016** (security-ledger). Wiring (dependent work, next): +mint `vr1-dc1-maas-power_ed25519`; authorize its pubkey on the dc1 rack; +install privkey in the region MAAS snap + snap `ssh config` Host 172.31.0.6; +give the maas-node-power script's virsh reach from voffice1 via the dc1 SERVICE +key (dc0 split). Then `maas-node-power.sh` dc1 (dry -> --commit), commission +9/9. Revert: deauthorize the pubkey on the rack, remove the snap key + config. + +## Item 12 -- SEC-016 power key WIRED (dedicated dc1 MAAS->libvirt) + +Per the SEC-016 ruling, replicating the dc0 split (script=svc key, MAAS=dedicated +key): +- Minted `~/vr1-dc1-creds/vr1-dc1-maas-power_ed25519` (manifest updated, + creds-audit CLEAN 7 entries). +- Public half authorized on the dc1 rack jessea123 (2 keys now, matching dc0). +- SCRIPT virsh path: dc1 SERVICE key copied to voffice1 `~/vr1-dc1-creds/` + + voffice1 `~/.ssh/config` Host 172.31.0.6 -> svc key (voffice1 reaches the rack + directly on its 172.31.0.5/30 transit leg). Verified: voffice1 virsh lists 9 + dc1 domains. +- MAAS power path: dedicated power PRIVKEY installed in the region MAAS snap + (`/var/snap/maas/current/root/.ssh/id_dc1_power`, 0600 root) + snap + `ssh config` Host 172.31.0.6 -> that key (dc0's 172.31.0.2 keeps SEC-012's + default id_ed25519 -- true per-DC isolation). Revert: deauthorize the pubkey + on the rack, remove the snap key + Host block, remove the voffice1 svc key + + Host block. + +## Item 13 -- power control set + commissioning 9/9 READY + +All 9 dc1 nodes reset -> PXE enlisted (by pinned 52:54:01:d1 MACs; machines +11->20). `maas-node-power.sh --commit` set `power_type=virsh` on all 9, each +verified by a real `query-power-state` -- which PROVES the SEC-016 dedicated +key drives the dc1 rack from the region (2 transient mid-shutdown power=error +cleared on re-query). Region MAAS config: D-120 range + D-131 forwarder DNS + +DHCP primary_rack=nmpcq4. Commissioned all 9 -> **ALL 9 READY in ~3.5 min** +(Commissioning -> Testing -> Ready, NO timeout, NO SERVFAIL), shapes EXACT to +D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB), power=virsh +(`docs/audit/dc1-commissioning-verify-20260723.txt`). dc0's two stacked faults +(MAC regen, rack-resolver SERVFAIL) were pre-empted by the pinned MACs (step C) ++ the dc-rack-net forwarder (item 8). Revert: `maas admin machine delete ` +per node; power-off domains. + +## Milestone -- G12 [V] leg COMPLETE + +dc1 substrate fully built and commissioned: outer vvr1-dc1 + inner 28/28 (prior +sessions) -> edge bootstrapped + addressed (D-112c/D-113a2 on 26.7) -> rack-net ++ region MAAS -> 9/9 nodes READY, shapes exact. G12's remaining leg was the +build; that is done. NEXT: G12 close-out (this changelog consolidated, gauntlet ++ repo-lint, GA-R7 memory review, skill sweep, operator-gated merge to `main`, +branch retirement). +(provider-public `10.12.64.2/22`, metal-admin `10.12.68.2/22`, forwarder +`10.12.68.3` -> region BIND `10.10.0.20`) + harness update, then +`install dc1`; region-side MAAS (metal-admin `dns_servers=10.12.68.3 +allow_dns=false`, DHCP `primary_rack=nmpcq4`, D-120 range +`10.12.68.100-.200`); `maas-node-power.sh` dc1 arm; commission 9/9 to Ready +(D-121 Option C); then G12 close-out (gauntlet, repo-lint, consolidation, +memory review, skill sweep, operator-gated merge to `main`). diff --git a/docs/archive/stage-records/vr1-stage3-record.md b/docs/archive/stage-records/vr1-stage3-record.md index 12e77a0..4f89354 100644 --- a/docs/archive/stage-records/vr1-stage3-record.md +++ b/docs/archive/stage-records/vr1-stage3-record.md @@ -1,4 +1,22 @@ -# VR1 Stage 3 record -- CLOSED 2026-07-21 (dc0 substrate; dc1 HELD under G12) +# VR1 Stage 3 record -- CLOSED 2026-07-21 (dc0 substrate); G12 dc1 remainder CLOSED 2026-07-23 + +**G12 CLOSED 2026-07-23 (dc1 substrate; operator-ruled "Merge to main + full close").** +The HELD dc1 remainder was built to a commissioned fleet on branch +`dc-dc-g12-dc1-substrate` and merged to `main`. Sequence (all operator-gated, +`docs/CURRENT-STATE.md` G12 row is the status authority): apex `--commit` + outer +step-A (5/0/0) + step-B transit/rack/SEC-010 + step-C inner 28/28 + D-125 egress +gate (2026-07-22); then edge bootstrap D-112(c)/D-113(a2) on 26.7 + v4 addressing, +`dc-rack-net` dc1 arm (harness 18/18) + install (forwarder answers `maas-internal`), +region MAAS (D-120 range + D-131 forwarder DNS + DHCP primary_rack=nmpcq4), SEC-016 +dedicated per-DC MAAS->libvirt power key (operator-ruled), and **commissioning 9/9 +READY in ~3.5 min, shapes exact to D-121 Option C** (2026-07-23). dc0's two stacked +commissioning faults (MAC regen, rack-resolver SERVFAIL) were pre-empted by pinned +MACs (step C) + the pre-installed forwarder. New SEC rows: SEC-015 (transient edge +key on rack, remediated), SEC-016 (dedicated dc1 power key). Three findings logged +(edge-bootstrap `util.inc`/`shell_safe()`, transient post-restart-sshd scp, +MAAS-snap-key refresh fragility). dc1 session changelogs added to the table below. + +# VR1 Stage 3 record -- dc0 substrate CLOSED 2026-07-21 (original title preserved) **Stage close (operator-ruled 2026-07-21, "Close and merge"):** all dc0 gates CLOSED with captures -- G9 (outer apply), G10 (deploy steps B-E: bootstrap, depth-4 boot, D-125 egress, MAAS + all 9 @@ -24,6 +42,9 @@ | changelog-20260719-dc0-deploy-stepB.md | deploy steps A-D: transit, bootstrap, inner apply, edge, enlistment, power | 838 | | changelog-20260721-commissioning-diag.md | commissioning double-fault diagnosis + repair (9/9 Ready) | 132 | | changelog-20260721-close-and-delivery.md | MAC pinning + dc-rack-net + incident docs + netem sudo mechanism | 140 | +| changelog-20260721-g12-dc1-authoring.md | G12 dc1: apex confirm-free + importer dc1 support + vars/substrate authoring (drop-state) | -- | +| changelog-20260722-g12-dc1-build.md | G12 dc1: apex write + outer step A + step B/C + D-125 egress gate | 257 | +| changelog-20260723-g12-dc1-edge.md | G12 dc1: edge bootstrap/addressing + rack-net + region MAAS + SEC-016 + commissioning 9/9 | 240 | | file (now docs/archive/changelogs/) | title line | lines | |---|---|---| diff --git a/docs/changelog-20260721-g12-dc1-authoring.md b/docs/changelog-20260721-g12-dc1-authoring.md deleted file mode 100644 index 652a4ab..0000000 --- a/docs/changelog-20260721-g12-dc1-authoring.md +++ /dev/null @@ -1,149 +0,0 @@ -# 2026-07-21 -- G12 [V] leg: vr1-dc1 substrate authoring (session changelog) - -Session scope: the G12 remaining [V] leg per the D-124 AMENDMENT (2026-07-21): -apex confirm-free -> vr1_dc1 vars -> dc1 substrate authoring -> gated build. -Branch: `dc-dc-g12-dc1-substrate` (off post-Stage-3-merge `main`, per the -stage-close rule). One changelog for the session (GA-R2/D1); every item -carries its revert. - -## Item 1 -- dc1 apex confirm-free capture (read-only; the [V] leg's first step) - -Read-only GETs on office1-netbox (token used on-host, never printed), captured -to `docs/audit/dc1-apex-confirm-20260721.txt`. Findings: - -- The six dc1 plane /22s ALREADY EXIST in the apex, scoped `vr1-dc1`, exactly - matching the ratified D-124-amendment scheme (provider-public 10.12.64.0/22 - ... replication 10.12.84.0/22, roles per D-101). Confirm-CONSISTENT: the - assignment landed at the original D-101/D-115 import; the ruled derivation - matches it. -- Transit 172.31.0.4/30 FREE (container 172.31.0.0/24 + dc0's /30 are its only - occupants; zero ip-addresses in the /30). -- Uplink 172.30.3.0/24 already registered to vr1-dc1 (D-115 edge import, - 2026-07-17). Rack IP 10.12.68.2 + forwarder .3 free (zero ip-addresses in - 10.12.68.0/22). Site `vr1-dc1` and role `transit` exist. - -Consequence: the ONLY apex write still owed for dc1 is the transit /30 + rack -metal-admin IP -- exactly the dc-rack-mgmt-import.py surface (dc0 precedent: -those are the only two objects dc0 registered either; transit endpoint IPs and -the D-131 forwarder alias are NOT apex objects for dc0 and dc1 mirrors that). - -Revert: delete the capture file (no state touched). - -## Item 2 -- dc-rack-mgmt-import.py: per-site support (vr1-dc0 + vr1-dc1) - -What: the importer was dc0-hardcoded (SITE_SLUG/METAL_ADMIN/RACK_DNS/desc -constants). Replaced with a `SITES` map (dc0: 10.12.8.0/22 / vvr1-dc0, ratified -2026-07-16; dc1: 10.12.68.0/22 / vvr1-dc1, D-124 amendment 2026-07-21) selected -by a new REQUIRED `--site {vr1-dc0,vr1-dc1}` flag (env: RACK_SITE). Explicit by -design: one DC's values can never land scoped to another site. An env-supplied -site is validated against SITES in code (argparse `choices` does not validate -env-default values). ROLE_SLUG / CONTAINER / D-120 band offsets stay global -(convention-wide). Also generalized dc0-specific error/usage text and fixed two -stale "(Cloud)" labels on the container messages (the container has been -172.31.0.0/24, not Cloud, since 2026-07-16). - -Why: unblocks the one owed dc1 apex write; the D-124 scheme "generalizes -per-leg" and the tool now encodes that instead of forking a dc1 copy. - -Harness: `tests/dc-rack-mgmt-import/` extended in the same change -- all CLI -cases pass `--site`, new cases: missing `--site` dies; bogus RACK_SITE env -dies; dc1 happy path (site id binding, /22 mask, dns vvr1-dc1); cross-site -guard (dc0 rack IP under --site vr1-dc1 rejected, zero writes); dc1 band edges -.2/.49 accepted, .1 gateway rejected; missing vr1-dc1 site precondition dies. -Structural pins moved from the retired globals to the SITES map. -**117/117 PASS** (was 96). - -Live dry-run vs the real apex (read-only, on office1-netbox): -`docs/audit/dc1-rack-import-dryrun-20260721.txt` -- whole preflight PASS, -plan = would-create exactly 172.31.0.4/30 (role transit, scope vr1-dc1) + -10.12.68.2/22 (dns vvr1-dc1). The `--commit` is a GATED operator step (apex -mutation), not run here. - -Revert: `git checkout main -- netbox/dc-rack-mgmt-import.py tests/dc-rack-mgmt-import/` -(restores the dc0-only tool); delete the two capture files. No apex state -touched (dry-run only). - -## Item 3 -- IN FLIGHT AT DISCONNECT (drop-state record, landed 2026-07-22 by the successor session; NOT a delivery) - -The session DISCONNECTED mid-item, before its bookend. This item records what -exists so the next session resumes without re-derivation; nothing here claims -delivery. Authored and sitting UNCOMMITTED in the tree at the drop: - -- `opentofu/variables.tf`: `vr1_dc1_planes` (apex-verified D-124-amendment - /22s), `vvr1_dc1_{vcpu,memory_mib,disk_bytes}` sizing, `vr1_dc1_ssh_pubkey_path`, - and the deliberately NO-DEFAULT rack/transit vars (tfvars-only after the - gated apex `--commit` -- hard rule 2). -- `opentofu/main.tf`: `module "vr1_dc1_uplink"` (172.30.3.0/24) + - `module "vvr1_dc1"` containment VM (dc0 pattern of record, dedicated dc1 key). -- `opentofu/vr1-dc1-substrate/` (NEW): full inner root mirroring dc0's - (same file set + variable shape); all 54 node MACs PRE-PINNED from first - apply, scheme `52:54:01:d1:NN:PP` (locally-administered, outside libvirt's - 52:54:00 auto space). -- `scripts/lib-net.sh`: vr1-dc1 arm (literals land; unruled OpenStack-layer - values UNSET) + a one-selection-per-shell guard. -- `tests/dc-selector/run-tests.sh`: reconciled, 40/40 PASS. - -DROP POINT: after the dc-selector harness went green, BEFORE the full -gauntlet. Successor-run gauntlet = **6/76 FAILED** (capture -`docs/audit/gauntlet-20260722-g12-dropstate.txt`): carve-host-interfaces, -dc-dc-dr-drill, dc-dc-radosgw-multisite, dc-dc-rbd-mirror, -phase-00-maas-standup, reenroll-hosts -- all still assert the retired -"vr1-dc1 fails loud (NetBox gap)" selector behavior that the new dc1 arm -retires. Also sighted for the reconcile sweep: `tests/dc-dc-prefixes-import/ -test_logic.py` greps the old message (currently passing); dc-dc-rbd-mirror's -reminder text prints dc0's replication CIDR under `dc=vr1-dc1` (cross-DC -value surfacing). repo-lint 0-fail; `tofu fmt -check` clean on all touched tf. - -OWED to finish the item: reconcile the six harnesses (+ the two sighted -surfaces) -> gauntlet ALL GREEN -> delivery entry here -> commit + push. -THEN G12's next gated step is unchanged: operator-gated apex `--commit` -(transit /30 + rack IP), tfvars, build. - -Revert (drops the in-flight work, NOT items 1-2): `git checkout -- -opentofu/main.tf opentofu/variables.tf scripts/lib-net.sh -tests/dc-selector/run-tests.sh && rm -rf opentofu/vr1-dc1-substrate/`; -delete the gauntlet capture file. - -## Item 4 -- harness reconcile: item 3 delivered (successor session, 2026-07-22) - -What: the six red harnesses reconciled to the ratified dc1 selector arm, plus -the two sighted surfaces dispositioned. Per script: - -- `phase-00-maas-standup.sh`: NEW PLANES/lib-net parity guard (exit 2, - precondition) -- the PLANES table is a DC0-hardcoded D-052/D-053 literal, and - with dc1 now selectable the script would otherwise have planned DC0 topology - under `DC=vr1-dc1` (the drop-state gauntlet showed exactly that: exit 0 + - WOULD: leak). Any selected DC whose lib-net plane set diverges from the table - is refused. Harness asserts the refusal (exit 2, no WOULD:/DO: leak). -- `dc-dc-rbd-mirror.sh` + `dc-dc-radosgw-multisite.sh`: the sighted cross-DC - value surfacing FIXED IN BOTH (radosgw carried the same defect as the sighted - rbd-mirror): the $DC gate ran only in a subshell, so the replication-plane - reminder always read the file's flat DC0 defaults -- under `--dc vr1-dc1` it - printed 10.12.36.0/22 (dc0's). On gate success the script now selects the DC - in its own shell; new harness cases pin the dc1 reminder to 10.12.84.0/22. -- `dc-dc-dr-drill` / `radosgw-multisite` / `rbd-mirror` harnesses: the dc1 - gate-refusal tests (want rc 3) became gate-OK dry-run + juju-guarded --apply - (rc 2) tests. The enforce path (exit 3) is retained in the scripts as - defense-in-depth but is no longer CLI-reachable (arg validation admits only - the two ratified tokens) -- noted in the test comments. -- `carve-host-interfaces` / `reenroll-hosts` harnesses: vr1-dc1 now fails at - the HOSTS layer like vr1-dc0 ("no enrolled hosts yet"); tests assert that - message + the no-mutation guard. Stale dc1/dc2-era header comments in both - scripts (and phase-00) updated. -- Sighting re-check: `tests/dc-dc-prefixes-import/` carries NO grep of the - retired selector message at HEAD -- the drop-state sighting does not - reproduce; harness passes unmodified (91 checks). No edit made there. - -Verification: the six harnesses individually green; full gauntlet **ALL GREEN -(76 harnesses)** -- `docs/audit/gauntlet-20260722-g12-reconcile.txt`; -repo-lint 0 fail. This item + the item-3 tree state land in one commit -(this delivery); G12's next gated step is unchanged: operator-gated apex -`--commit`, then tfvars + gated build. - -Revert: `git checkout main -- scripts/phase-00-maas-standup.sh -scripts/dc-dc-rbd-mirror.sh scripts/dc-dc-radosgw-multisite.sh -scripts/carve-host-interfaces.sh scripts/reenroll-hosts.sh -tests/phase-00-maas-standup/ tests/dc-dc-rbd-mirror/ -tests/dc-dc-radosgw-multisite/ tests/dc-dc-dr-drill/ -tests/carve-host-interfaces/ tests/reenroll-hosts/`; delete the reconcile -gauntlet capture (item-3 revert covers the authored substrate). diff --git a/docs/changelog-20260722-g12-dc1-build.md b/docs/changelog-20260722-g12-dc1-build.md deleted file mode 100644 index 9e8ef69..0000000 --- a/docs/changelog-20260722-g12-dc1-build.md +++ /dev/null @@ -1,205 +0,0 @@ -# 2026-07-22 -- G12 [V] leg: dc1 apex write + substrate build step A (session changelog) - -Session scope: continue the G12 [V] leg from the reconciled tree (predecessor -delivery landed as changelog-20260721-g12-dc1-authoring.md items 3-4, commit -`d2bf743`): the one owed apex write, tfvars, and the gated outer build. -Branch: `dc-dc-g12-dc1-substrate`. One changelog for the session (GA-R2/D1); -every item carries its revert. - -## Item 1 -- apex `--commit` EXECUTED (the one owed dc1 apex write; operator-gated) - -Same-session read-only preflight FIRST (verify-before-mutate): dry-run re-run -against the live apex = would-create exactly 2, already-present 0 -- -`docs/audit/dc1-rack-import-dryrun-20260722.txt`, identical to the 07-21 -preflight. Then the gated write (piped to office1-netbox over ssh; token -sourced on-host from `/root/netbox-secrets/api.token` per -`creds-manifests/vr1-office1.manifest`, value never printed or brought into -context): - -- CREATED `172.31.0.4/30` (prefix id=139) role=transit scope=dcim.site:vr1-dc1 -- CREATED `10.12.68.2/22` (ip id=4) dns=vvr1-dc1 - -Post-commit idempotency read-back: would-create 0 / already-present 2. Both -runs captured in `docs/audit/dc1-rack-import-commit-20260722.txt`. Values -verbatim from the D-124 amendment (ruled 2026-07-21); exactly the two objects -the dc0 precedent registered. - -Revert: delete the two apex objects by id (prefix 139, ip-address 4) via the -NetBox UI/API on office1-netbox; delete the capture files. - -## Item 2 -- vr1-dc1 service keypair MINTED (manifest-prescribed at-deploy step) - -`~/vr1-dc1-creds/vr1-dc1_svc_ed25519{,.pub}` generated on vcloud (ed25519, -modes 600/644) per `creds-manifests/vr1-dc1.manifest` ("MINTED AT DC DEPLOY"). -`bash scripts/creds-audit.sh vr1-dc1` = CLEAN. D-126 option (a) per-env key; -pubkey feeds vvr1-dc1 cloud-init (item 3), private half jumphost-local. - -Revert: remove both key files (couples to item 4 -- vvr1-dc1's cloud-init -authorizes this pubkey; re-mint requires a seed rebuild). - -## Item 3 -- dc1 tfvars authored (LOCAL, gitignored -- recorded here, not committed) - -`opentofu/d124-rack.auto.tfvars` gained the five vr1_dc1_* values, verbatim -from the D-124 amendment's own tfvars line (rack_metal_admin_ip=10.12.68.2, -rack_transit_ip=172.31.0.6, rack_transit_prefix=30, -rack_transit_peer_ip=172.31.0.5) + `vr1_dc1_ssh_pubkey_path` to the item-2 -pubkey. Stale "dc1 DEFERRED" header comment replaced. `tofu fmt` applied; -`scripts/opentofu-validate.sh` PASS (all roots). - -Revert: remove the vr1_dc1_* block from the local file. - -## Item 4 -- outer apply: dc1 substrate step A (operator-gated, logged, saved-plan exact) - -Preconditions measured in-session: host RAM 1007 GiB with 450 GiB committed -(voffice1 32 + edge 2 + vvr1-dc0 416) -> +416 GiB fits with ~140 GiB headroom; -vCPU 234/256 post-apply; dc-dc-whole-host-budget 13/13 PASS. - -Saved plan `tfplan-dc1-20260722` = **5/0/0 exact** (vr1-dc1-uplink network + -vvr1-dc1 domain/disk/seed/cloudinit; ZERO touches to live resources) -- -capture `docs/audit/outer-plan-20260722-dc1-substrate.txt`. Applied via the -saved plan under a per-command as-executed wrap -(`~/as-executed/2026-07-22-dc1-deploy.log`; index row added): **5 added, 0 -changed, 0 destroyed**. Convergence re-plan = **zero diff** -(`docs/audit/outer-plan-20260722-postdc1-converged.txt`). Live verify: -vvr1-dc1 RUNNING (Id 8), voffice1/office1-opnsense/vvr1-dc0 untouched and -running; vr1-dc1-uplink + both dc1 mesh legs active. - -Revert: `runbooks/dc-dc-teardown-rollback.md` decision tree; the targeted -destroy set is the five applied resources (module.vvr1_dc1.* + -module.vr1_dc1_uplink.*); delete the plan/capture files and the tfplan. - -CORRECTION (same session): commit `0bd6342` accidentally included the spent -binary saved plan (repo-lint L1 x2 -- the lint's exit code was masked by a -pipeline; owned). The file deviated from the `.gitignore` naming convention -(`opentofu/**/*.tfplan`) -- saved plans must be named `*.tfplan`. Removed in -the follow-up commit; lint back to 0-fail. The applied-plan record remains -the two dated captures. - -## Item 5 -- cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (gated apply) - -What: `modules/cloudinit-vm` gained `interface_macs` (ported verbatim-in-intent -from `modules/node-vm`, same validations; harness +4 cases, cloudinit-vm 10/10) --- the 2026-07-20 voffice1 Kea/MAC-regen incident class, closed at the module -layer. Root wiring: `module.voffice1` NIC3 -> `mesh-vr1-dc1-office1` with all -three MACs pinned (NIC1/NIC2 = measured live values; NIC3 = pre-pinned -`52:54:01:d1:fe:01`, dc1 scheme, fe = region-side); `module.vvr1_dc1` pin-adopts -its two measured step-A MACs (standup DoD invariant). Saved plan -`voffice1-dc1nic-20260722.tfplan` = **0/2/0, zero replaces**, all five MAC -values verified in the diff (`docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`); -applied logged (trap 1e: BOTH domains bounced -- presented as such). Post-bounce -battery ALL PASS: 4/4 domains running, voffice1 MACs exactly as pinned, Kea -lease 10.10.0.20 intact, region dhcpd up, netbox 302 + tailscale up (nested LXD -VMs self-recovered, ~3 min), dc0 rack: dhcpd + dc0-node-dns + dc0-rack-legs -active, forwarder SOA answers. Convergence re-plan ZERO DIFF. - -Revert: remove NIC3 + interface_macs from main.tf, revert the module + -harness (`git checkout` the four files), re-plan/apply (bounces again). - -## Item 6 -- voffice1 dc1 transit leg (in-guest netplan, gated) - -`/etc/netplan/61-transit-dc1.yaml` (0600) on voffice1: enp3s0 static -`172.31.0.5/30` (per-DC drop-in; dc0's `60-transit.yaml` untouched). Verified: -enp3s0 UP with .5/30, ping 172.31.0.6 = 0% loss, first ssh into vvr1-dc1 over -the transit with the dc1 key OK (hostname vvr1-dc1, nested-KVM module -present, region route + 10.10.0.20 reach pre-staged by step-A cloud-init). -Revert: remove the drop-in + `netplan apply`. - -## Item 7 -- rack bootstrap EXECUTED (site-headend-install, dc1-parameterized) - -Enrollment secret staged region->rack as a host-to-host pipe (0600 -`/root/region-enroll.secret`, 32 bytes verified by count, value never in -context). Snap system proxy set to `http://10.10.0.20:8000` (dc0-measured -mechanism; env proxy for apt). Dry-run FIRST exposed that the script's -host-nodes DEFAULTS are dc0-flavored -- all overridable by existing flags; -run with `--wan-bridge br-vr1-dc1-wan --inner-pool-path -/var/lib/libvirt/vr1-dc1-inner --opnsense-base ...26.7...`. First real run -FAILED exit 4: stale base-image apt index -> 404s on superseded debs via the -proxy; fixed with `apt-get update`, idempotent re-run **exit 0**: rack -ENROLLED, nested KVM on, inner pool + AppArmor, SEC-010 rack end, WAN bridge -verified with enp2s0 enslaved. Region-side verify: `maas admin -rack-controllers read` lists **vvr1-dc1 (nmpcq4)**. SEC-010 REGION end: -voffice1's `/etc/nftables-sec010.nft` extended with the enp3s0 drop pair -(dc0 idiom), table reloaded clean, ruleset = both legs dropped. -`--check` capture: `docs/audit/dc1-stepB-check-20260722.txt` (sole [--] = -opnsense base, item 8). - -QUEUED findings (logged, not fixed mid-step): (a) site-headend-install -NOTE/hint prose still says "26.1" and "vr1-dc0-substrate" even when -parameterized for dc1 (cosmetic, misleads operators); (b) -`nftables-sec010.nft` reload is NOT idempotent -- `nft -f` on a live table -appends, so a service restart duplicates rules (observed; cleaned via -`nft delete table` + restart; the file should flush first); (c) -`opnsense-prep-image.sh` dies on `BASH_SOURCE[0]: unbound variable` when -piped via `bash -s` under `set -u` (ran into it before the mirror guard). - -Revert: rack side -- `snap remove maas`, remove nftables-sec010 + -sec010-fw + kvm-nested modprobe + inner pool dir on vvr1-dc1; region side -- -remove the enp3s0 pair from voffice1's nft file + restart unit; delete -the staged secret file. - -## Item 8 -- OPNsense 26.7 base staged on dc1 (proven-artifact copy) - -`opnsense-prep-image.sh` requires `OPNSENSE_MIRROR_BASE` (deliberately not -repo-recorded; mirrors change). Instead of choosing a mirror, the dc0 rack's -OPERATOR-RULED and boot-PROVEN 26.7 base was streamed rack->rack through the -jump path (direct rack-to-rack is SEC-010-dropped, correctly) and -sha256-verified on arrival against the dc0 source hash -(`3981821e3a3c...476627d`). Same bits that passed the D-112 boot path on dc0. -Revert: delete `/var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2`. - -## Item 9 -- inner apply: dc1 substrate step C EXECUTED (from voffice1, D-128 Plane 2) - -Staging on voffice1 (dc0-precedent set): repo clone fetched + switched from the -retired stage-3 branch to `dc-dc-g12-dc1-substrate` @ 61c416e (dc0 inner -tfstate untouched, verified); dc1 private key piped in (0600, 432 bytes by -count); 172.31.0.6 host keys scanned into known_hosts (all 3 types -- dc0 -trap); the 26.7 base copied rack->voffice1 to `~/vr1-dc1-images/` and -sha256-verified (provider streams volume content itself -- the dc0 measured -fix; rack-side copy satisfies only the bootstrap check). Inner tfvars -`d124-inner.auto.tfvars` written on voffice1, every value sourced (measured -transit .6, outer-committed planes map verbatim, Stage-1 mtu 9000, D-106 -suffix, keyfile + local base paths). - -Plan `dc1-inner-20260722.tfplan` = **28/0/0** (dc0's exact step-C count), -verified in-capture: 54 pinned `52:54:01:d1:*` MACs, 9 Option-C nodes + edge, -qga channel present (`docs/audit/inner-plan-20260722-dc1-stepC.txt`). First -apply FAILED at 18/28: the edge domain's serial-log dir -`/var/lib/libvirt/vr1/staging/` did not exist inside vvr1-dc1 (hand-created -on dc0 during the 07-20 serial work; QUEUED finding -- belongs in the -bootstrap/standup DoD so dc2+ does not repeat it). Fix-forward per the -rollback tree: dir created (root:root 0755, dc0 mirror), resume plan 10/0/0, -**apply exit 0**. Final: 28/28 in state, convergence ZERO DIFF -(`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING -inside vvr1-dc1**, edge serial log at the FreeBSD login prompt -- OPNsense -26.7, factory LAN 192.168.1.1/24, WAN unaddressed (second D-112 boot-path -datapoint on 26.7; exactly dc0's post-step-C state). The dc1 INNER tfstate -lives ON voffice1 (`opentofu/vr1-dc1-substrate/terraform.tfstate` -- add to -the site backup set, same as dc0's). - -Revert: `dc-dc-teardown-rollback.md` tree; inner root destroy from voffice1 -tears down the 28 (containment VM unaffected). - -## Item 10 -- D-125 egress isolation gate: PASS (vr1-dc1) - -Procedure of record = dc0's `d125-egress-test.sh` (recovered from the dc0 -rack's home), adapted per-DC (pool `vr1-dc1-inner`, bridge `br-vr1-dc1-wan`, -probe 172.30.3.50/24 via .1 -- D-124 amendment /24) and staged to the dc1 -rack. Prereqs installed via the region proxy (virtinst, cloud-image-utils); -`/tmp/noble.img` streamed dc0->dc1. **Two identical consecutive runs**: -GW-PING-RC=0, NET-PING-RC=0, NET-TCP-http=301, NET-TCP-ubuntu=200, -LAN-TCP-http=000 (the region-isolation check), leftover-domains=0 -- -`docs/audit/d125-egress-gate-20260722-dc1.txt`. Bridge-in egress PROVEN -end-to-end for dc1; the double-NAT fallback is NOT needed. dc0's -one-time unexplained first-run ICMP failure did NOT recur. - -Revert: none needed (throwaway fully torn down, verified); delete the -capture + the staged script + /tmp/noble.img on the rack if desired. - -## Next (gated, not run here) - -Edge bootstrap (D-112(c) console -> key-only SSH -> D-113(a2) API key on -26.7); edge addressing via `opnsense-set-interface-v4` (WAN 172.30.3.2/24 -gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); rack standup DoD -(dc-rack-net.sh install dc1 + forwarder 10.12.68.3, region-side DHCP on -metal-admin naming nmpcq4 primary_rack, dynamic range 10.12.68.100-.200 -per D-120, maas-node-power dc1 arm). Runbook + CURRENT-STATE govern. diff --git a/docs/changelog-20260723-g12-dc1-edge.md b/docs/changelog-20260723-g12-dc1-edge.md deleted file mode 100644 index 2c17800..0000000 --- a/docs/changelog-20260723-g12-dc1-edge.md +++ /dev/null @@ -1,229 +0,0 @@ -# 2026-07-23 -- G12 [V] leg: dc1 edge bootstrap + v4 addressing (session changelog) - -Session scope: resume the G12 [V] leg from the 2026-07-22 close handoff -(edge bootstrap onward). Branch `dc-dc-g12-dc1-substrate`. One changelog per -session (GA-R2/D1); every item carries its revert. Live mutations gated -(operator ran in `manual` permission mode -- see Item 0). Edge-phase mutations -mostly run UNWRAPPED (approval prompt = the gate); the console bootstrap went -through the `~/as-executed/2026-07-22-dc1-deploy.log` wrap. - -## Item 0 -- permission-mode friction (recorded; no repo change) - -The session opened in `auto` mode, whose classifier BLOCKS remote -`sudo`/mutation shapes (matched by the project `ssh * sudo *` ask rule) -instead of surfacing them for approval -- so it cannot deliver "operator -approves each mutation." Resolved by switching to `manual` mode (alias for -`default`; no classifier): allow-rules flow read-only rack probes, project ask -rules prompt every gated mutation, deny rules + `guard-destructive.py` stay -active. Session allow-rules for the dc1/dc0 rack ssh shapes were added to -`.claude/settings.local.json` (the operator later broadened to `ssh *`). -Revert: none (session-scoped settings; remove the added allow lines at close -if desired). - -## Item 1 -- dc1 edge service keypair minted + manifest - -`~/vr1-dc1-creds/vr1-dc1-edge_ed25519` (0600/0644) minted on vcloud, -`creds-manifests/vr1-dc1.manifest` extended (edge key + `opnsense-api.txt`), -`creds-audit vr1-dc1` CLEAN (5 entries). Revert: `git checkout` the manifest; -delete the keypair. - -## Item 2 -- D-112(c) console bootstrap COMPLETE (edge SSH + service key) - -Driver `d112c-console-dc1.py` (dc1 adaptation of the dc0-PROVEN v6, retrieved -verbatim from `vvr1-dc0:~/d112c-console.py`): serial console -> factory -root/opnsense -> shell -> ship+run bootstrap PHP (enable ssh, permitrootlogin, -install dc1 edge pubkey, `write_config`) -> materialize PHP -(`local_user_set()`, dc0 lesson v). Transcript `~/d112c-console-dc1.log` on -the rack: `CONFIG-WRITTEN`, `AK-IN-CONFIG=152`, `USER-MATERIALIZED`; -`/root/.ssh/authorized_keys` (115 B) created (was absent). Edge banner -confirms **vtnet0 = LAN 192.168.1.1/24, vtnet1 = WAN** (dc0 mapping). -**FINDING (logged, not fixed mid-step):** the dc0 `d112c-bootstrap.php` on the -rack required only `config.inc`, so my first bootstrap PHP threw -`Call to undefined function shell_safe()` at `config.inc:311` (in -`write_config()->make_config_revision_entry()`) and aborted BEFORE writing -- -this is **dc0 lesson (iv)** (shell_safe lives in `util.inc`, measured on dc0's -identical 26.7 image). Added `require_once("util.inc")`; two diagnostic lines -made csh-safe (edge root shell is tcsh; `2>&1` = "Ambiguous output redirect"). -Re-ran clean. The proven dc0 `.b64` bootstrap artifact is missing this include --- a real gap if replayed; queue: fold `util.inc` into the canonical console -bootstrap payload. Revert: config-only; re-run factory reset on the edge to -undo (or leave -- it is the intended state). - -## Item 3 -- key-only SSH to the edge VERIFIED (D-112(c) proof) - -Reached the edge via a ProxyCommand chain from vcloud (rack key for the jump, -edge key for the final hop -- per-hop keys; a plain `-J` applies one key to all -hops and the rack rejected it). `root@192.168.1.1` -> `uname -r` = -`15.1-RELEASE-p1`, `ifconfig -l` = vtnet0/vtnet1/lo0/enc0/pfsync0/pflog0. -Second D-112(c) datapoint after dc0. - -## Item 4 -- D-113(a2) API key MINTED + smoke test PASS - -Interim reach leg `192.168.1.2/24` added on the rack provider-public bridge -`virbr4` (measured this session) to reach the factory edge LAN. Edge-config -scripts + edge key staged on the rack (dc0 method; the mint script SSHes -edge-direct, no ProxyJump). Key minted via the vendor `opnsense-mint-apikey.php` -(`apikeys->add()` -- the GUI's own path); key/secret 80 chars, 173 B -> -`~/opnsense-api.txt` (0600, secret never printed). Smoke test from the rack: -`GET core/firmware/status` -> exit 0, `product_abi 26.7`. First proof the -D-113(a2) API path works on 26.7 for dc1. -**FINDING (logged):** `opnsense-bootstrap-apikey.sh`'s first `scp` to the edge -failed once with `scp: Connection closed` -- TRANSIENT: the edge sshd was not -ready in the moment right after the console bootstrap's `configctl openssh -restart` (both default `scp` and `scp -O` succeeded minutes later; the edge is -reachable and scp works). Unblocked via an ssh-pipe mint (vendor PHP shipped -by `ssh 'cat >'`, retrieved by `ssh -n 'cat' > file`) -- during which a -missing `-n` on the edge ssh calls let them EAT the rack `bash -s` heredoc -stdin (mint ran, retrieve/cleanup lines were swallowed); re-run with `-n` -retrieved the already-minted key cleanly. Queue: a post-restart sshd -readiness wait / one scp retry in `opnsense-bootstrap-apikey.sh`. Revert: delete -the API key via the edge API + remove `~/vr1-dc1-creds/opnsense-api.txt`. - -## Item 5 -- edge v4 addressing (WAN then LAN; reach-handoff) - -`opnsense-set-interface-v4.sh` run VERBATIM from the rack (plain scp works -after warm-up; no repo edit). Values confirmed from committed `lib-net.sh` -(provider-public `10.12.64.0/22` gw `10.12.64.1`, D-124 amendment) + the D-124 -uplink `172.30.3.0/24`. -- **WAN first** (script ordering rule): dhcp -> `172.30.3.2/24` gw - `172.30.3.1`, applied + read back on the kernel (vtnet1). Edge egress - VERIFIED: ping 1.1.1.1 = 0% loss. -- **Permanent rack leg** `10.12.64.2/22` added on `virbr4` (so the rack keeps - reach after the edge LAN moves). -- **LAN last**: `192.168.1.1/24` -> `10.12.64.1/22` (vtnet0). The apply drops - the `192.168.1.1` session mid-command (expected; the foreground ssh hung on - the dead interface and was TaskStop-ped -- config was already saved+applied). - Verified independently at the NEW LAN via the rack's `10.12.64.2` leg: - `ifconfig vtnet0` = `inet 10.12.64.1 netmask 0xfffffc00` (/22); API - `GET core/firmware/status` at `10.12.64.1` -> exit 0, abi 26.7. -- Interim `192.168.1.2/24` removed; `virbr4` now carries only `10.12.64.2/22` - (dc0 end-state exactly). -Revert: re-address LAN/WAN back via the same script; the leg adds/dels are -`ip addr add/del` (non-persistent). - -## Item 6 -- creds consolidated + rack edge-key WIPED (SEC-015 closed transient) - -API creds pulled rack -> `~/vr1-dc1-creds/opnsense-api.txt` (0600, 1 key + 1 -secret). Rack edge-key copy `shred -u`'d; staged edge-config scripts + -console driver + rack creds copy removed (`ls` confirms none remain). The -non-secret console transcript `~/d112c-console-dc1.log` retained as an audit -artifact. **SEC-015** opened for the transient edge-key-on-rack exposure -(minted->wiped same phase; disclosed). Revert: n/a (cleanup). - -## Milestone - -dc1 edge fully bootstrapped + addressed -- D-112(c) console bootstrap and -D-113(a2) REST API both PROVEN on 26.7 (second datapoint after dc0). Edge: -WAN `172.30.3.2/24` gw `172.30.3.1` (egress 0% loss), LAN `10.12.64.1/22` -(the ruled provider-public gateway), API answering at the new LAN. - -## Item 7 -- dc-rack-net.sh dc1 site-table arm + harness (D-131 sub-1 delivery) - -Added the `dc1)` arm to `scripts/dc-rack-net.sh` (the STANDING per-DC forwarder -+ rack-legs pattern), MEASURED/apex-committed values, network-name keyed (no -virbrN literal -- T6 holds): - vr1-dc1-metal-admin 10.12.68.2/22 rack MAAS/DHCP leg (apex-committed rack IP) - vr1-dc1-metal-admin 10.12.68.3/22 node-DNS forwarder listen alias (D-120 static) - vr1-dc1-provider-public 10.12.64.2/22 edge-LAN leg (measured live on the bridge 2026-07-23) - DNS_LISTEN=10.12.68.3 DNS_UPSTREAM=10.10.0.20 (region BIND over transit). -Harness `tests/dc-rack-net/` extended T15-T18 (dc1 identity + MEASURED-tag); -**18/18**, gauntlet **ALL GREEN (76)**, repo-lint 0-fail. This forwarder is the -D-131 fix that pre-empts dc0's commissioning SERVFAIL on the isolated rack. -Revert: `git checkout` scripts/dc-rack-net.sh tests/dc-rack-net/run-tests.sh. - -## Item 8 -- dc-rack-net install on the dc1 rack (persistent legs + forwarder) - -`install dc1` run on the rack (operator-gated). Check PASS 10/10 -(`docs/audit/dc1-rack-net-install-20260723.txt`): persistent metal-admin legs -`10.12.68.2/22` + `10.12.68.3/22` on virbr6, provider-public `10.12.64.2/22` on -virbr4, `dc1-rack-legs` + `dc1-node-dns` enabled+active. Behavioral proof: the -forwarder answers authoritative `maas-internal SOA` via region BIND -- the -D-131 fix that pre-empts dc0's commissioning SERVFAIL. Revert: -`dc-rack-net.sh` install is idempotent; to undo, disable the two units + remove -the generated files + the interim legs. - -## Item 9 -- region-side MAAS config for dc1 metal-admin (DHCP + DNS + range) - -MAAS auto-discovered dc1 planes from the rack interfaces: metal-admin -`10.12.68.0/22` = subnet id 11 (VLAN fabric 142/vid 0), provider-public -`10.12.64.0/22` = subnet id 10. On subnet 11 (all operator-gated): -- D-120 dynamic range `10.12.68.100-10.12.68.200` created (iprange id 3) -- - the ruled band applied to dc1's CIDR. -- D-131: `dns_servers=10.12.68.3 allow_dns=false` (nodes resolve via the rack - forwarder, not MAAS -- the SERVFAIL fix). -- DHCP: VLAN fabric 142/vid 0 `dhcp_on=true primary_rack=nmpcq4`. -Verified BEHAVIORALLY (dc0 lesson -- not the self-report): dhcpd RUNNING on the -rack (`dhcpd -4 ... virbr6`, dhcpd.conf freshly generated); no Temporal -incident (the dc0 region restart fixed it fleet-wide). Revert: -`maas admin vlan update 142 0 dhcp_on=false`; `subnet update 11 dns_servers= -allow_dns=true`; `ipranges delete 3`. - -## Item 10 -- dc1 enlistment PROVEN (canary) - -`virsh reset vr1-dc1-control-01` -> the node PXE-booted, got DHCP from nmpcq4, -and ENLISTED in MAAS ~2 min later (machine count 11 -> 12). The -DHCP->PXE->enlist chain works end to end for dc1; with the forwarder -pre-installed, dc0's two stacked commissioning faults are pre-mitigated. -Revert: n/a (enlistment; the machine is deleted/re-commissioned as needed). - -## Item 11 -- SEC-016 ruling: dedicated dc1 MAAS->libvirt power key (GA-R5) - -Commissioning needs the region MAAS snap to SSH the dc1 rack libvirt -(`power_type=virsh`). The dc0 rack authorizes the SEC-012 MAAS key; the dc1 -rack does not. Operator RULED (AskUserQuestion, exact utterance): **"Mint a -dedicated dc1 power key"** -- per-DC isolation, NOT cross-DC reuse of SEC-012. -Recorded as **SEC-016** (security-ledger). Wiring (dependent work, next): -mint `vr1-dc1-maas-power_ed25519`; authorize its pubkey on the dc1 rack; -install privkey in the region MAAS snap + snap `ssh config` Host 172.31.0.6; -give the maas-node-power script's virsh reach from voffice1 via the dc1 SERVICE -key (dc0 split). Then `maas-node-power.sh` dc1 (dry -> --commit), commission -9/9. Revert: deauthorize the pubkey on the rack, remove the snap key + config. - -## Item 12 -- SEC-016 power key WIRED (dedicated dc1 MAAS->libvirt) - -Per the SEC-016 ruling, replicating the dc0 split (script=svc key, MAAS=dedicated -key): -- Minted `~/vr1-dc1-creds/vr1-dc1-maas-power_ed25519` (manifest updated, - creds-audit CLEAN 7 entries). -- Public half authorized on the dc1 rack jessea123 (2 keys now, matching dc0). -- SCRIPT virsh path: dc1 SERVICE key copied to voffice1 `~/vr1-dc1-creds/` + - voffice1 `~/.ssh/config` Host 172.31.0.6 -> svc key (voffice1 reaches the rack - directly on its 172.31.0.5/30 transit leg). Verified: voffice1 virsh lists 9 - dc1 domains. -- MAAS power path: dedicated power PRIVKEY installed in the region MAAS snap - (`/var/snap/maas/current/root/.ssh/id_dc1_power`, 0600 root) + snap - `ssh config` Host 172.31.0.6 -> that key (dc0's 172.31.0.2 keeps SEC-012's - default id_ed25519 -- true per-DC isolation). Revert: deauthorize the pubkey - on the rack, remove the snap key + Host block, remove the voffice1 svc key + - Host block. - -## Item 13 -- power control set + commissioning 9/9 READY - -All 9 dc1 nodes reset -> PXE enlisted (by pinned 52:54:01:d1 MACs; machines -11->20). `maas-node-power.sh --commit` set `power_type=virsh` on all 9, each -verified by a real `query-power-state` -- which PROVES the SEC-016 dedicated -key drives the dc1 rack from the region (2 transient mid-shutdown power=error -cleared on re-query). Region MAAS config: D-120 range + D-131 forwarder DNS + -DHCP primary_rack=nmpcq4. Commissioned all 9 -> **ALL 9 READY in ~3.5 min** -(Commissioning -> Testing -> Ready, NO timeout, NO SERVFAIL), shapes EXACT to -D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB), power=virsh -(`docs/audit/dc1-commissioning-verify-20260723.txt`). dc0's two stacked faults -(MAC regen, rack-resolver SERVFAIL) were pre-empted by the pinned MACs (step C) -+ the dc-rack-net forwarder (item 8). Revert: `maas admin machine delete ` -per node; power-off domains. - -## Milestone -- G12 [V] leg COMPLETE - -dc1 substrate fully built and commissioned: outer vvr1-dc1 + inner 28/28 (prior -sessions) -> edge bootstrapped + addressed (D-112c/D-113a2 on 26.7) -> rack-net -+ region MAAS -> 9/9 nodes READY, shapes exact. G12's remaining leg was the -build; that is done. NEXT: G12 close-out (this changelog consolidated, gauntlet -+ repo-lint, GA-R7 memory review, skill sweep, operator-gated merge to `main`, -branch retirement). -(provider-public `10.12.64.2/22`, metal-admin `10.12.68.2/22`, forwarder -`10.12.68.3` -> region BIND `10.10.0.20`) + harness update, then -`install dc1`; region-side MAAS (metal-admin `dns_servers=10.12.68.3 -allow_dns=false`, DHCP `primary_rack=nmpcq4`, D-120 range -`10.12.68.100-.200`); `maas-node-power.sh` dc1 arm; commission 9/9 to Ready -(D-121 Option C); then G12 close-out (gauntlet, repo-lint, consolidation, -memory review, skill sweep, operator-gated merge to `main`).