diff --git a/.claude/skills/openstack-cloud-ops/SKILL.md b/.claude/skills/openstack-cloud-ops/SKILL.md index 7b4422a..711453d 100644 --- a/.claude/skills/openstack-cloud-ops/SKILL.md +++ b/.claude/skills/openstack-cloud-ops/SKILL.md @@ -143,6 +143,15 @@ `scripts/maas-node-power.sh` is the ruled shape -- D-103/D-123 amendments 2026-07-20), and a pod delete CASCADES to its linked machine records -- read the pod's machine list BEFORE any `vm-host delete`; non-empty = STOP (appendix-A, 2026-07-21 incident). +**Per-DC power credential (SEC-012 dc0, SEC-016 dc1, RULED 2026-07-23 -- standing +DC-standup invariant): each DC gets its OWN dedicated MAAS->libvirt power key, NEVER +a cross-DC reuse.** The REGION's MAAS snap dials the power address, so the split is: +the maas-node-power SCRIPT's mapping virsh uses the DC SERVICE key (via the region's +`~/.ssh/config` Host ), while MAAS's own power ops use the dedicated +key installed in the snap (`/var/snap/maas/current/root/.ssh/` + a per-host `ssh +config` block so each rack uses its own key). FRAGILITY: the snap-side key lives under +per-revision `/var/snap/maas/current/` and may not survive a snap refresh -- re-assert +after any refresh. Roosevelt analog: per-DC IPMI/BMC credentials. **Change-delivery loop:** grep for prior art (zeroth decision) -> grep design-decisions for the governing D-NNN -> edit -> `bash scripts/repo-lint.sh` diff --git a/creds-manifests/vr1-dc1.manifest b/creds-manifests/vr1-dc1.manifest index a0ec24f..d53bd99 100644 --- a/creds-manifests/vr1-dc1.manifest +++ b/creds-manifests/vr1-dc1.manifest @@ -15,3 +15,16 @@ vr1-dc1_svc_ed25519 600 local vr1-dc1_svc_ed25519.pub 644 local +# Edge service keypair (D-112(c) console bootstrap; minted 2026-07-22 on vcloud +# at the G12 dc1 edge-bootstrap step -- dc0 precedent, whose manifest rows are +# a queued backfill finding). +vr1-dc1-edge_ed25519 600 local +vr1-dc1-edge_ed25519.pub 644 local +# Edge REST API key/secret, minted on the edge via the vendor model +# (D-113(a2)); consolidated to vcloud 2026-07-22 at the G12 edge-address step. +opnsense-api.txt 600 local +# Dedicated MAAS->libvirt power key (SEC-016, per-DC isolation; minted +# 2026-07-23 at the G12 commissioning step). Private half also placed on +# voffice1's region MAAS snap (off-manifest -- the snap is not this folder). +vr1-dc1-maas-power_ed25519 600 local +vr1-dc1-maas-power_ed25519.pub 644 local diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 287122f..50278e5 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -26,8 +26,10 @@ - Stage: Stage 3 / Phase 2 -- "OpenTofu builds each DC substrate" (`docs/dc-dc-deployment-workflow.md:148`; runbook `runbooks/dc-dc-phase2-tofu-dc-substrate.md`) -- **CLOSED 2026-07-21 - for its vr1-dc0 scope** (operator-ruled "Close and merge"; dc1 is - the stage's designed HELD remainder, gate G12). Close-out set: + for its vr1-dc0 scope** (operator-ruled "Close and merge"; dc1 was + the stage's designed HELD remainder, gate G12 -- **now also CLOSED + 2026-07-23: dc1 substrate built + commissioned 9/9, merged to `main`, + branch retired; see the G12 gate row**). Close-out set: gauntlet ALL GREEN + repo-lint 0-fail + this consolidation commit + GA-R7 memory review + merge of `dc-dc-stage3-phase2-dc-substrate` to `main` (merge commit) + branch retirement; stage record @@ -441,19 +443,19 @@ `docs/session-ledger.md:278`). The EXPECTED outer plan is **ZERO DIFF** ("no differences"), -re-recorded 2026-07-21 with its evidencing capture -(`docs/audit/outer-plan-20260721-postG16-converged.txt`) after the -operator-ruled G16 state surgery (channels null -> [] injected on the -office1 edge's state entry, serial 29 -> 30, backup -`terraform.tfstate.pre-G16-20260721`; guests never touched -- G6 -precedent). A future outer plan showing ANY diff is a STOP -(investigate drift before touching anything). History: 7/2/7 +re-recorded 2026-07-22 with its evidencing capture +(`docs/audit/outer-plan-20260722-postdc1-converged.txt`) after the +G12 dc1 substrate step-A apply (saved plan 5/0/0 exact -- vvr1-dc1 + +vr1-dc1-uplink adds only, zero touches to live resources). A future +outer plan showing ANY diff is a STOP (investigate drift before +touching anything). History: 7/2/7 post-reboot symptom -> 6/2/6 post-D-130 -> 6/0/6 post-G6-reconcile -> applied exact -> zero diff -> 1/1/1 (voffice1 transit, ruled+applied) -> 2/0/2 (rack netplan fix, applied) -> zero diff converged 2026-07-20 -> 1/1/0 netem-wire STOP -> targeted netem apply 1/0/0 exact -> 0/1/0 -office1 residual -> G16 state surgery -> zero diff converged (this -entry). +office1 residual -> G16 state surgery -> zero diff converged 2026-07-21 +-> 5/0/0 dc1 substrate adds (G12 step A, saved-plan applied exact +2026-07-22) -> zero diff converged (this entry). ## 6. Open gates @@ -475,7 +477,7 @@ | G9 | DC0 outer apply (deploy step A) | [V] operator-gated, logged (`run-logged.sh`), after G1-G8; audit exit criteria met (charter Phase 6). SEC pre-apply dependency (S2): SEC-010's transit FORWARD-drop is applied+verified at deploy step B via `site-headend-install.sh --host-nodes --check` on vvr1-dc0 (gate G10) -- the ONLY SEC row gated on this apply (register of record: security-ledger). CANONICAL ENTRY DOC (probe hole H1): `runbooks/dc-dc-phase2-tofu-dc-substrate.md`, with `docs/dc0-deploy-readiness.md` section E as the step table | operator | CLOSED 2026-07-19: G8 same-session planes check passed (6x 0 leases, 0 attachments); saved plan == 6/0/6 applied in the logged dc0-deploy window; convergence re-plan = no differences; vvr1-dc0 running, prior guests untouched | | G10 | Deploy steps B-E in-sequence gates: SEC-010 `--host-nodes --check` on vvr1-dc0; depth-4 nested boot; D-125 foreign-MAC egress test; MAAS reachability + `TF_VAR_maas_api_key` before step D; netem placeholder step E | [V] exercised during the gated deploy | session (each mutation operator-approved) | Step B DONE 2026-07-20 (`--check` EXIT 0 incl. SEC-010, `docs/audit/stepB-check-20260720-final.txt`; interfaces enp1s0/enp2s0). Depth-4 nested boot DONE (10 domains running inside vvr1-dc0). D-125 egress isolation test PASS 2026-07-20 (`docs/audit/d125-egress-gate-20260720-matrix.txt`), and the edge itself now egresses 0% loss after the v4 addressing. Step D COMPLETE incl. commissioning: ALL 9 NODES READY 2026-07-21 (two stacked faults diagnosed + fixed -- `docs/audit/commissioning-diag-20260721.txt`; section 1). Step E (netem) DONE 2026-07-21: sudo fragment installed+verified, module local-mode amendment, targeted apply 1/0/0 exact (operator-ruled at the 1/1/0 STOP), placeholder profile live on virbr5, virbr7/virbr3 untouched (`docs/audit/stepE-netem-20260721.txt` + `outer-{plan,apply}-20260721-netem*.txt`). **G10 CLOSED 2026-07-21** | | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | -| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | OPEN -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). Remaining [V] leg: apex confirm-free, vr1_dc1_rack_* vars, dc1 substrate root, build | +| G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | CLOSED 2026-07-23 (operator-ruled "Merge to main + full close"; commissioning 9/9 READY, merge commit on `main`, branch retired) -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED + WIRED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). **COMMISSIONING 9/9 READY 2026-07-23** (`docs/audit/dc1-commissioning-verify-20260723.txt`): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, `power_type=virsh` set + verified by real query-power-state (SEC-016 path proven), commissioned to **ALL 9 READY in ~3.5 min** (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. **G12 [V] leg (the dc1 build) is COMPLETE.** NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, **operator-gated merge of `dc-dc-g12-dc1-substrate` -> `main`** (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close). | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | OPEN (execution only; decision content complete) | | G14 | 9 OPEN SEC rows (SEC-001, -003..-008, plus SEC-012 + SEC-013 opened 2026-07-20 for credentials this deploy created; SEC-010 CLOSED 2026-07-20, operator-ruled, applied+verified both transit ends) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012 also carries a SCOPE question (libvirt-group grant is broader than the power verbs MAAS needs), SEC-013 is tied to whether `opentofu/vr1-dc0-maas` is retired | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-20 | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 remains PROPOSED/OPEN (items 2-3 + the item-1 re-scoped migration plan) | diff --git a/docs/archive/changelogs/changelog-20260721-g12-dc1-authoring.md b/docs/archive/changelogs/changelog-20260721-g12-dc1-authoring.md new file mode 100644 index 0000000..652a4ab --- /dev/null +++ b/docs/archive/changelogs/changelog-20260721-g12-dc1-authoring.md @@ -0,0 +1,149 @@ +# 2026-07-21 -- G12 [V] leg: vr1-dc1 substrate authoring (session changelog) + +Session scope: the G12 remaining [V] leg per the D-124 AMENDMENT (2026-07-21): +apex confirm-free -> vr1_dc1 vars -> dc1 substrate authoring -> gated build. +Branch: `dc-dc-g12-dc1-substrate` (off post-Stage-3-merge `main`, per the +stage-close rule). One changelog for the session (GA-R2/D1); every item +carries its revert. + +## Item 1 -- dc1 apex confirm-free capture (read-only; the [V] leg's first step) + +Read-only GETs on office1-netbox (token used on-host, never printed), captured +to `docs/audit/dc1-apex-confirm-20260721.txt`. Findings: + +- The six dc1 plane /22s ALREADY EXIST in the apex, scoped `vr1-dc1`, exactly + matching the ratified D-124-amendment scheme (provider-public 10.12.64.0/22 + ... replication 10.12.84.0/22, roles per D-101). Confirm-CONSISTENT: the + assignment landed at the original D-101/D-115 import; the ruled derivation + matches it. +- Transit 172.31.0.4/30 FREE (container 172.31.0.0/24 + dc0's /30 are its only + occupants; zero ip-addresses in the /30). +- Uplink 172.30.3.0/24 already registered to vr1-dc1 (D-115 edge import, + 2026-07-17). Rack IP 10.12.68.2 + forwarder .3 free (zero ip-addresses in + 10.12.68.0/22). Site `vr1-dc1` and role `transit` exist. + +Consequence: the ONLY apex write still owed for dc1 is the transit /30 + rack +metal-admin IP -- exactly the dc-rack-mgmt-import.py surface (dc0 precedent: +those are the only two objects dc0 registered either; transit endpoint IPs and +the D-131 forwarder alias are NOT apex objects for dc0 and dc1 mirrors that). + +Revert: delete the capture file (no state touched). + +## Item 2 -- dc-rack-mgmt-import.py: per-site support (vr1-dc0 + vr1-dc1) + +What: the importer was dc0-hardcoded (SITE_SLUG/METAL_ADMIN/RACK_DNS/desc +constants). Replaced with a `SITES` map (dc0: 10.12.8.0/22 / vvr1-dc0, ratified +2026-07-16; dc1: 10.12.68.0/22 / vvr1-dc1, D-124 amendment 2026-07-21) selected +by a new REQUIRED `--site {vr1-dc0,vr1-dc1}` flag (env: RACK_SITE). Explicit by +design: one DC's values can never land scoped to another site. An env-supplied +site is validated against SITES in code (argparse `choices` does not validate +env-default values). ROLE_SLUG / CONTAINER / D-120 band offsets stay global +(convention-wide). Also generalized dc0-specific error/usage text and fixed two +stale "(Cloud)" labels on the container messages (the container has been +172.31.0.0/24, not Cloud, since 2026-07-16). + +Why: unblocks the one owed dc1 apex write; the D-124 scheme "generalizes +per-leg" and the tool now encodes that instead of forking a dc1 copy. + +Harness: `tests/dc-rack-mgmt-import/` extended in the same change -- all CLI +cases pass `--site`, new cases: missing `--site` dies; bogus RACK_SITE env +dies; dc1 happy path (site id binding, /22 mask, dns vvr1-dc1); cross-site +guard (dc0 rack IP under --site vr1-dc1 rejected, zero writes); dc1 band edges +.2/.49 accepted, .1 gateway rejected; missing vr1-dc1 site precondition dies. +Structural pins moved from the retired globals to the SITES map. +**117/117 PASS** (was 96). + +Live dry-run vs the real apex (read-only, on office1-netbox): +`docs/audit/dc1-rack-import-dryrun-20260721.txt` -- whole preflight PASS, +plan = would-create exactly 172.31.0.4/30 (role transit, scope vr1-dc1) + +10.12.68.2/22 (dns vvr1-dc1). The `--commit` is a GATED operator step (apex +mutation), not run here. + +Revert: `git checkout main -- netbox/dc-rack-mgmt-import.py tests/dc-rack-mgmt-import/` +(restores the dc0-only tool); delete the two capture files. No apex state +touched (dry-run only). + +## Item 3 -- IN FLIGHT AT DISCONNECT (drop-state record, landed 2026-07-22 by the successor session; NOT a delivery) + +The session DISCONNECTED mid-item, before its bookend. This item records what +exists so the next session resumes without re-derivation; nothing here claims +delivery. Authored and sitting UNCOMMITTED in the tree at the drop: + +- `opentofu/variables.tf`: `vr1_dc1_planes` (apex-verified D-124-amendment + /22s), `vvr1_dc1_{vcpu,memory_mib,disk_bytes}` sizing, `vr1_dc1_ssh_pubkey_path`, + and the deliberately NO-DEFAULT rack/transit vars (tfvars-only after the + gated apex `--commit` -- hard rule 2). +- `opentofu/main.tf`: `module "vr1_dc1_uplink"` (172.30.3.0/24) + + `module "vvr1_dc1"` containment VM (dc0 pattern of record, dedicated dc1 key). +- `opentofu/vr1-dc1-substrate/` (NEW): full inner root mirroring dc0's + (same file set + variable shape); all 54 node MACs PRE-PINNED from first + apply, scheme `52:54:01:d1:NN:PP` (locally-administered, outside libvirt's + 52:54:00 auto space). +- `scripts/lib-net.sh`: vr1-dc1 arm (literals land; unruled OpenStack-layer + values UNSET) + a one-selection-per-shell guard. +- `tests/dc-selector/run-tests.sh`: reconciled, 40/40 PASS. + +DROP POINT: after the dc-selector harness went green, BEFORE the full +gauntlet. Successor-run gauntlet = **6/76 FAILED** (capture +`docs/audit/gauntlet-20260722-g12-dropstate.txt`): carve-host-interfaces, +dc-dc-dr-drill, dc-dc-radosgw-multisite, dc-dc-rbd-mirror, +phase-00-maas-standup, reenroll-hosts -- all still assert the retired +"vr1-dc1 fails loud (NetBox gap)" selector behavior that the new dc1 arm +retires. Also sighted for the reconcile sweep: `tests/dc-dc-prefixes-import/ +test_logic.py` greps the old message (currently passing); dc-dc-rbd-mirror's +reminder text prints dc0's replication CIDR under `dc=vr1-dc1` (cross-DC +value surfacing). repo-lint 0-fail; `tofu fmt -check` clean on all touched tf. + +OWED to finish the item: reconcile the six harnesses (+ the two sighted +surfaces) -> gauntlet ALL GREEN -> delivery entry here -> commit + push. +THEN G12's next gated step is unchanged: operator-gated apex `--commit` +(transit /30 + rack IP), tfvars, build. + +Revert (drops the in-flight work, NOT items 1-2): `git checkout -- +opentofu/main.tf opentofu/variables.tf scripts/lib-net.sh +tests/dc-selector/run-tests.sh && rm -rf opentofu/vr1-dc1-substrate/`; +delete the gauntlet capture file. + +## Item 4 -- harness reconcile: item 3 delivered (successor session, 2026-07-22) + +What: the six red harnesses reconciled to the ratified dc1 selector arm, plus +the two sighted surfaces dispositioned. Per script: + +- `phase-00-maas-standup.sh`: NEW PLANES/lib-net parity guard (exit 2, + precondition) -- the PLANES table is a DC0-hardcoded D-052/D-053 literal, and + with dc1 now selectable the script would otherwise have planned DC0 topology + under `DC=vr1-dc1` (the drop-state gauntlet showed exactly that: exit 0 + + WOULD: leak). Any selected DC whose lib-net plane set diverges from the table + is refused. Harness asserts the refusal (exit 2, no WOULD:/DO: leak). +- `dc-dc-rbd-mirror.sh` + `dc-dc-radosgw-multisite.sh`: the sighted cross-DC + value surfacing FIXED IN BOTH (radosgw carried the same defect as the sighted + rbd-mirror): the $DC gate ran only in a subshell, so the replication-plane + reminder always read the file's flat DC0 defaults -- under `--dc vr1-dc1` it + printed 10.12.36.0/22 (dc0's). On gate success the script now selects the DC + in its own shell; new harness cases pin the dc1 reminder to 10.12.84.0/22. +- `dc-dc-dr-drill` / `radosgw-multisite` / `rbd-mirror` harnesses: the dc1 + gate-refusal tests (want rc 3) became gate-OK dry-run + juju-guarded --apply + (rc 2) tests. The enforce path (exit 3) is retained in the scripts as + defense-in-depth but is no longer CLI-reachable (arg validation admits only + the two ratified tokens) -- noted in the test comments. +- `carve-host-interfaces` / `reenroll-hosts` harnesses: vr1-dc1 now fails at + the HOSTS layer like vr1-dc0 ("no enrolled hosts yet"); tests assert that + message + the no-mutation guard. Stale dc1/dc2-era header comments in both + scripts (and phase-00) updated. +- Sighting re-check: `tests/dc-dc-prefixes-import/` carries NO grep of the + retired selector message at HEAD -- the drop-state sighting does not + reproduce; harness passes unmodified (91 checks). No edit made there. + +Verification: the six harnesses individually green; full gauntlet **ALL GREEN +(76 harnesses)** -- `docs/audit/gauntlet-20260722-g12-reconcile.txt`; +repo-lint 0 fail. This item + the item-3 tree state land in one commit +(this delivery); G12's next gated step is unchanged: operator-gated apex +`--commit`, then tfvars + gated build. + +Revert: `git checkout main -- scripts/phase-00-maas-standup.sh +scripts/dc-dc-rbd-mirror.sh scripts/dc-dc-radosgw-multisite.sh +scripts/carve-host-interfaces.sh scripts/reenroll-hosts.sh +tests/phase-00-maas-standup/ tests/dc-dc-rbd-mirror/ +tests/dc-dc-radosgw-multisite/ tests/dc-dc-dr-drill/ +tests/carve-host-interfaces/ tests/reenroll-hosts/`; delete the reconcile +gauntlet capture (item-3 revert covers the authored substrate). diff --git a/docs/archive/changelogs/changelog-20260722-g12-dc1-build.md b/docs/archive/changelogs/changelog-20260722-g12-dc1-build.md new file mode 100644 index 0000000..9e8ef69 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260722-g12-dc1-build.md @@ -0,0 +1,205 @@ +# 2026-07-22 -- G12 [V] leg: dc1 apex write + substrate build step A (session changelog) + +Session scope: continue the G12 [V] leg from the reconciled tree (predecessor +delivery landed as changelog-20260721-g12-dc1-authoring.md items 3-4, commit +`d2bf743`): the one owed apex write, tfvars, and the gated outer build. +Branch: `dc-dc-g12-dc1-substrate`. One changelog for the session (GA-R2/D1); +every item carries its revert. + +## Item 1 -- apex `--commit` EXECUTED (the one owed dc1 apex write; operator-gated) + +Same-session read-only preflight FIRST (verify-before-mutate): dry-run re-run +against the live apex = would-create exactly 2, already-present 0 -- +`docs/audit/dc1-rack-import-dryrun-20260722.txt`, identical to the 07-21 +preflight. Then the gated write (piped to office1-netbox over ssh; token +sourced on-host from `/root/netbox-secrets/api.token` per +`creds-manifests/vr1-office1.manifest`, value never printed or brought into +context): + +- CREATED `172.31.0.4/30` (prefix id=139) role=transit scope=dcim.site:vr1-dc1 +- CREATED `10.12.68.2/22` (ip id=4) dns=vvr1-dc1 + +Post-commit idempotency read-back: would-create 0 / already-present 2. Both +runs captured in `docs/audit/dc1-rack-import-commit-20260722.txt`. Values +verbatim from the D-124 amendment (ruled 2026-07-21); exactly the two objects +the dc0 precedent registered. + +Revert: delete the two apex objects by id (prefix 139, ip-address 4) via the +NetBox UI/API on office1-netbox; delete the capture files. + +## Item 2 -- vr1-dc1 service keypair MINTED (manifest-prescribed at-deploy step) + +`~/vr1-dc1-creds/vr1-dc1_svc_ed25519{,.pub}` generated on vcloud (ed25519, +modes 600/644) per `creds-manifests/vr1-dc1.manifest` ("MINTED AT DC DEPLOY"). +`bash scripts/creds-audit.sh vr1-dc1` = CLEAN. D-126 option (a) per-env key; +pubkey feeds vvr1-dc1 cloud-init (item 3), private half jumphost-local. + +Revert: remove both key files (couples to item 4 -- vvr1-dc1's cloud-init +authorizes this pubkey; re-mint requires a seed rebuild). + +## Item 3 -- dc1 tfvars authored (LOCAL, gitignored -- recorded here, not committed) + +`opentofu/d124-rack.auto.tfvars` gained the five vr1_dc1_* values, verbatim +from the D-124 amendment's own tfvars line (rack_metal_admin_ip=10.12.68.2, +rack_transit_ip=172.31.0.6, rack_transit_prefix=30, +rack_transit_peer_ip=172.31.0.5) + `vr1_dc1_ssh_pubkey_path` to the item-2 +pubkey. Stale "dc1 DEFERRED" header comment replaced. `tofu fmt` applied; +`scripts/opentofu-validate.sh` PASS (all roots). + +Revert: remove the vr1_dc1_* block from the local file. + +## Item 4 -- outer apply: dc1 substrate step A (operator-gated, logged, saved-plan exact) + +Preconditions measured in-session: host RAM 1007 GiB with 450 GiB committed +(voffice1 32 + edge 2 + vvr1-dc0 416) -> +416 GiB fits with ~140 GiB headroom; +vCPU 234/256 post-apply; dc-dc-whole-host-budget 13/13 PASS. + +Saved plan `tfplan-dc1-20260722` = **5/0/0 exact** (vr1-dc1-uplink network + +vvr1-dc1 domain/disk/seed/cloudinit; ZERO touches to live resources) -- +capture `docs/audit/outer-plan-20260722-dc1-substrate.txt`. Applied via the +saved plan under a per-command as-executed wrap +(`~/as-executed/2026-07-22-dc1-deploy.log`; index row added): **5 added, 0 +changed, 0 destroyed**. Convergence re-plan = **zero diff** +(`docs/audit/outer-plan-20260722-postdc1-converged.txt`). Live verify: +vvr1-dc1 RUNNING (Id 8), voffice1/office1-opnsense/vvr1-dc0 untouched and +running; vr1-dc1-uplink + both dc1 mesh legs active. + +Revert: `runbooks/dc-dc-teardown-rollback.md` decision tree; the targeted +destroy set is the five applied resources (module.vvr1_dc1.* + +module.vr1_dc1_uplink.*); delete the plan/capture files and the tfplan. + +CORRECTION (same session): commit `0bd6342` accidentally included the spent +binary saved plan (repo-lint L1 x2 -- the lint's exit code was masked by a +pipeline; owned). The file deviated from the `.gitignore` naming convention +(`opentofu/**/*.tfplan`) -- saved plans must be named `*.tfplan`. Removed in +the follow-up commit; lint back to 0-fail. The applied-plan record remains +the two dated captures. + +## Item 5 -- cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (gated apply) + +What: `modules/cloudinit-vm` gained `interface_macs` (ported verbatim-in-intent +from `modules/node-vm`, same validations; harness +4 cases, cloudinit-vm 10/10) +-- the 2026-07-20 voffice1 Kea/MAC-regen incident class, closed at the module +layer. Root wiring: `module.voffice1` NIC3 -> `mesh-vr1-dc1-office1` with all +three MACs pinned (NIC1/NIC2 = measured live values; NIC3 = pre-pinned +`52:54:01:d1:fe:01`, dc1 scheme, fe = region-side); `module.vvr1_dc1` pin-adopts +its two measured step-A MACs (standup DoD invariant). Saved plan +`voffice1-dc1nic-20260722.tfplan` = **0/2/0, zero replaces**, all five MAC +values verified in the diff (`docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`); +applied logged (trap 1e: BOTH domains bounced -- presented as such). Post-bounce +battery ALL PASS: 4/4 domains running, voffice1 MACs exactly as pinned, Kea +lease 10.10.0.20 intact, region dhcpd up, netbox 302 + tailscale up (nested LXD +VMs self-recovered, ~3 min), dc0 rack: dhcpd + dc0-node-dns + dc0-rack-legs +active, forwarder SOA answers. Convergence re-plan ZERO DIFF. + +Revert: remove NIC3 + interface_macs from main.tf, revert the module + +harness (`git checkout` the four files), re-plan/apply (bounces again). + +## Item 6 -- voffice1 dc1 transit leg (in-guest netplan, gated) + +`/etc/netplan/61-transit-dc1.yaml` (0600) on voffice1: enp3s0 static +`172.31.0.5/30` (per-DC drop-in; dc0's `60-transit.yaml` untouched). Verified: +enp3s0 UP with .5/30, ping 172.31.0.6 = 0% loss, first ssh into vvr1-dc1 over +the transit with the dc1 key OK (hostname vvr1-dc1, nested-KVM module +present, region route + 10.10.0.20 reach pre-staged by step-A cloud-init). +Revert: remove the drop-in + `netplan apply`. + +## Item 7 -- rack bootstrap EXECUTED (site-headend-install, dc1-parameterized) + +Enrollment secret staged region->rack as a host-to-host pipe (0600 +`/root/region-enroll.secret`, 32 bytes verified by count, value never in +context). Snap system proxy set to `http://10.10.0.20:8000` (dc0-measured +mechanism; env proxy for apt). Dry-run FIRST exposed that the script's +host-nodes DEFAULTS are dc0-flavored -- all overridable by existing flags; +run with `--wan-bridge br-vr1-dc1-wan --inner-pool-path +/var/lib/libvirt/vr1-dc1-inner --opnsense-base ...26.7...`. First real run +FAILED exit 4: stale base-image apt index -> 404s on superseded debs via the +proxy; fixed with `apt-get update`, idempotent re-run **exit 0**: rack +ENROLLED, nested KVM on, inner pool + AppArmor, SEC-010 rack end, WAN bridge +verified with enp2s0 enslaved. Region-side verify: `maas admin +rack-controllers read` lists **vvr1-dc1 (nmpcq4)**. SEC-010 REGION end: +voffice1's `/etc/nftables-sec010.nft` extended with the enp3s0 drop pair +(dc0 idiom), table reloaded clean, ruleset = both legs dropped. +`--check` capture: `docs/audit/dc1-stepB-check-20260722.txt` (sole [--] = +opnsense base, item 8). + +QUEUED findings (logged, not fixed mid-step): (a) site-headend-install +NOTE/hint prose still says "26.1" and "vr1-dc0-substrate" even when +parameterized for dc1 (cosmetic, misleads operators); (b) +`nftables-sec010.nft` reload is NOT idempotent -- `nft -f` on a live table +appends, so a service restart duplicates rules (observed; cleaned via +`nft delete table` + restart; the file should flush first); (c) +`opnsense-prep-image.sh` dies on `BASH_SOURCE[0]: unbound variable` when +piped via `bash -s` under `set -u` (ran into it before the mirror guard). + +Revert: rack side -- `snap remove maas`, remove nftables-sec010 + +sec010-fw + kvm-nested modprobe + inner pool dir on vvr1-dc1; region side -- +remove the enp3s0 pair from voffice1's nft file + restart unit; delete +the staged secret file. + +## Item 8 -- OPNsense 26.7 base staged on dc1 (proven-artifact copy) + +`opnsense-prep-image.sh` requires `OPNSENSE_MIRROR_BASE` (deliberately not +repo-recorded; mirrors change). Instead of choosing a mirror, the dc0 rack's +OPERATOR-RULED and boot-PROVEN 26.7 base was streamed rack->rack through the +jump path (direct rack-to-rack is SEC-010-dropped, correctly) and +sha256-verified on arrival against the dc0 source hash +(`3981821e3a3c...476627d`). Same bits that passed the D-112 boot path on dc0. +Revert: delete `/var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2`. + +## Item 9 -- inner apply: dc1 substrate step C EXECUTED (from voffice1, D-128 Plane 2) + +Staging on voffice1 (dc0-precedent set): repo clone fetched + switched from the +retired stage-3 branch to `dc-dc-g12-dc1-substrate` @ 61c416e (dc0 inner +tfstate untouched, verified); dc1 private key piped in (0600, 432 bytes by +count); 172.31.0.6 host keys scanned into known_hosts (all 3 types -- dc0 +trap); the 26.7 base copied rack->voffice1 to `~/vr1-dc1-images/` and +sha256-verified (provider streams volume content itself -- the dc0 measured +fix; rack-side copy satisfies only the bootstrap check). Inner tfvars +`d124-inner.auto.tfvars` written on voffice1, every value sourced (measured +transit .6, outer-committed planes map verbatim, Stage-1 mtu 9000, D-106 +suffix, keyfile + local base paths). + +Plan `dc1-inner-20260722.tfplan` = **28/0/0** (dc0's exact step-C count), +verified in-capture: 54 pinned `52:54:01:d1:*` MACs, 9 Option-C nodes + edge, +qga channel present (`docs/audit/inner-plan-20260722-dc1-stepC.txt`). First +apply FAILED at 18/28: the edge domain's serial-log dir +`/var/lib/libvirt/vr1/staging/` did not exist inside vvr1-dc1 (hand-created +on dc0 during the 07-20 serial work; QUEUED finding -- belongs in the +bootstrap/standup DoD so dc2+ does not repeat it). Fix-forward per the +rollback tree: dir created (root:root 0755, dc0 mirror), resume plan 10/0/0, +**apply exit 0**. Final: 28/28 in state, convergence ZERO DIFF +(`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING +inside vvr1-dc1**, edge serial log at the FreeBSD login prompt -- OPNsense +26.7, factory LAN 192.168.1.1/24, WAN unaddressed (second D-112 boot-path +datapoint on 26.7; exactly dc0's post-step-C state). The dc1 INNER tfstate +lives ON voffice1 (`opentofu/vr1-dc1-substrate/terraform.tfstate` -- add to +the site backup set, same as dc0's). + +Revert: `dc-dc-teardown-rollback.md` tree; inner root destroy from voffice1 +tears down the 28 (containment VM unaffected). + +## Item 10 -- D-125 egress isolation gate: PASS (vr1-dc1) + +Procedure of record = dc0's `d125-egress-test.sh` (recovered from the dc0 +rack's home), adapted per-DC (pool `vr1-dc1-inner`, bridge `br-vr1-dc1-wan`, +probe 172.30.3.50/24 via .1 -- D-124 amendment /24) and staged to the dc1 +rack. Prereqs installed via the region proxy (virtinst, cloud-image-utils); +`/tmp/noble.img` streamed dc0->dc1. **Two identical consecutive runs**: +GW-PING-RC=0, NET-PING-RC=0, NET-TCP-http=301, NET-TCP-ubuntu=200, +LAN-TCP-http=000 (the region-isolation check), leftover-domains=0 -- +`docs/audit/d125-egress-gate-20260722-dc1.txt`. Bridge-in egress PROVEN +end-to-end for dc1; the double-NAT fallback is NOT needed. dc0's +one-time unexplained first-run ICMP failure did NOT recur. + +Revert: none needed (throwaway fully torn down, verified); delete the +capture + the staged script + /tmp/noble.img on the rack if desired. + +## Next (gated, not run here) + +Edge bootstrap (D-112(c) console -> key-only SSH -> D-113(a2) API key on +26.7); edge addressing via `opnsense-set-interface-v4` (WAN 172.30.3.2/24 +gw .1, LAN -> 10.12.64.1/22 per the D-124 amendment); rack standup DoD +(dc-rack-net.sh install dc1 + forwarder 10.12.68.3, region-side DHCP on +metal-admin naming nmpcq4 primary_rack, dynamic range 10.12.68.100-.200 +per D-120, maas-node-power dc1 arm). Runbook + CURRENT-STATE govern. diff --git a/docs/archive/changelogs/changelog-20260723-g12-dc1-edge.md b/docs/archive/changelogs/changelog-20260723-g12-dc1-edge.md new file mode 100644 index 0000000..2c17800 --- /dev/null +++ b/docs/archive/changelogs/changelog-20260723-g12-dc1-edge.md @@ -0,0 +1,229 @@ +# 2026-07-23 -- G12 [V] leg: dc1 edge bootstrap + v4 addressing (session changelog) + +Session scope: resume the G12 [V] leg from the 2026-07-22 close handoff +(edge bootstrap onward). Branch `dc-dc-g12-dc1-substrate`. One changelog per +session (GA-R2/D1); every item carries its revert. Live mutations gated +(operator ran in `manual` permission mode -- see Item 0). Edge-phase mutations +mostly run UNWRAPPED (approval prompt = the gate); the console bootstrap went +through the `~/as-executed/2026-07-22-dc1-deploy.log` wrap. + +## Item 0 -- permission-mode friction (recorded; no repo change) + +The session opened in `auto` mode, whose classifier BLOCKS remote +`sudo`/mutation shapes (matched by the project `ssh * sudo *` ask rule) +instead of surfacing them for approval -- so it cannot deliver "operator +approves each mutation." Resolved by switching to `manual` mode (alias for +`default`; no classifier): allow-rules flow read-only rack probes, project ask +rules prompt every gated mutation, deny rules + `guard-destructive.py` stay +active. Session allow-rules for the dc1/dc0 rack ssh shapes were added to +`.claude/settings.local.json` (the operator later broadened to `ssh *`). +Revert: none (session-scoped settings; remove the added allow lines at close +if desired). + +## Item 1 -- dc1 edge service keypair minted + manifest + +`~/vr1-dc1-creds/vr1-dc1-edge_ed25519` (0600/0644) minted on vcloud, +`creds-manifests/vr1-dc1.manifest` extended (edge key + `opnsense-api.txt`), +`creds-audit vr1-dc1` CLEAN (5 entries). Revert: `git checkout` the manifest; +delete the keypair. + +## Item 2 -- D-112(c) console bootstrap COMPLETE (edge SSH + service key) + +Driver `d112c-console-dc1.py` (dc1 adaptation of the dc0-PROVEN v6, retrieved +verbatim from `vvr1-dc0:~/d112c-console.py`): serial console -> factory +root/opnsense -> shell -> ship+run bootstrap PHP (enable ssh, permitrootlogin, +install dc1 edge pubkey, `write_config`) -> materialize PHP +(`local_user_set()`, dc0 lesson v). Transcript `~/d112c-console-dc1.log` on +the rack: `CONFIG-WRITTEN`, `AK-IN-CONFIG=152`, `USER-MATERIALIZED`; +`/root/.ssh/authorized_keys` (115 B) created (was absent). Edge banner +confirms **vtnet0 = LAN 192.168.1.1/24, vtnet1 = WAN** (dc0 mapping). +**FINDING (logged, not fixed mid-step):** the dc0 `d112c-bootstrap.php` on the +rack required only `config.inc`, so my first bootstrap PHP threw +`Call to undefined function shell_safe()` at `config.inc:311` (in +`write_config()->make_config_revision_entry()`) and aborted BEFORE writing -- +this is **dc0 lesson (iv)** (shell_safe lives in `util.inc`, measured on dc0's +identical 26.7 image). Added `require_once("util.inc")`; two diagnostic lines +made csh-safe (edge root shell is tcsh; `2>&1` = "Ambiguous output redirect"). +Re-ran clean. The proven dc0 `.b64` bootstrap artifact is missing this include +-- a real gap if replayed; queue: fold `util.inc` into the canonical console +bootstrap payload. Revert: config-only; re-run factory reset on the edge to +undo (or leave -- it is the intended state). + +## Item 3 -- key-only SSH to the edge VERIFIED (D-112(c) proof) + +Reached the edge via a ProxyCommand chain from vcloud (rack key for the jump, +edge key for the final hop -- per-hop keys; a plain `-J` applies one key to all +hops and the rack rejected it). `root@192.168.1.1` -> `uname -r` = +`15.1-RELEASE-p1`, `ifconfig -l` = vtnet0/vtnet1/lo0/enc0/pfsync0/pflog0. +Second D-112(c) datapoint after dc0. + +## Item 4 -- D-113(a2) API key MINTED + smoke test PASS + +Interim reach leg `192.168.1.2/24` added on the rack provider-public bridge +`virbr4` (measured this session) to reach the factory edge LAN. Edge-config +scripts + edge key staged on the rack (dc0 method; the mint script SSHes +edge-direct, no ProxyJump). Key minted via the vendor `opnsense-mint-apikey.php` +(`apikeys->add()` -- the GUI's own path); key/secret 80 chars, 173 B -> +`~/opnsense-api.txt` (0600, secret never printed). Smoke test from the rack: +`GET core/firmware/status` -> exit 0, `product_abi 26.7`. First proof the +D-113(a2) API path works on 26.7 for dc1. +**FINDING (logged):** `opnsense-bootstrap-apikey.sh`'s first `scp` to the edge +failed once with `scp: Connection closed` -- TRANSIENT: the edge sshd was not +ready in the moment right after the console bootstrap's `configctl openssh +restart` (both default `scp` and `scp -O` succeeded minutes later; the edge is +reachable and scp works). Unblocked via an ssh-pipe mint (vendor PHP shipped +by `ssh 'cat >'`, retrieved by `ssh -n 'cat' > file`) -- during which a +missing `-n` on the edge ssh calls let them EAT the rack `bash -s` heredoc +stdin (mint ran, retrieve/cleanup lines were swallowed); re-run with `-n` +retrieved the already-minted key cleanly. Queue: a post-restart sshd +readiness wait / one scp retry in `opnsense-bootstrap-apikey.sh`. Revert: delete +the API key via the edge API + remove `~/vr1-dc1-creds/opnsense-api.txt`. + +## Item 5 -- edge v4 addressing (WAN then LAN; reach-handoff) + +`opnsense-set-interface-v4.sh` run VERBATIM from the rack (plain scp works +after warm-up; no repo edit). Values confirmed from committed `lib-net.sh` +(provider-public `10.12.64.0/22` gw `10.12.64.1`, D-124 amendment) + the D-124 +uplink `172.30.3.0/24`. +- **WAN first** (script ordering rule): dhcp -> `172.30.3.2/24` gw + `172.30.3.1`, applied + read back on the kernel (vtnet1). Edge egress + VERIFIED: ping 1.1.1.1 = 0% loss. +- **Permanent rack leg** `10.12.64.2/22` added on `virbr4` (so the rack keeps + reach after the edge LAN moves). +- **LAN last**: `192.168.1.1/24` -> `10.12.64.1/22` (vtnet0). The apply drops + the `192.168.1.1` session mid-command (expected; the foreground ssh hung on + the dead interface and was TaskStop-ped -- config was already saved+applied). + Verified independently at the NEW LAN via the rack's `10.12.64.2` leg: + `ifconfig vtnet0` = `inet 10.12.64.1 netmask 0xfffffc00` (/22); API + `GET core/firmware/status` at `10.12.64.1` -> exit 0, abi 26.7. +- Interim `192.168.1.2/24` removed; `virbr4` now carries only `10.12.64.2/22` + (dc0 end-state exactly). +Revert: re-address LAN/WAN back via the same script; the leg adds/dels are +`ip addr add/del` (non-persistent). + +## Item 6 -- creds consolidated + rack edge-key WIPED (SEC-015 closed transient) + +API creds pulled rack -> `~/vr1-dc1-creds/opnsense-api.txt` (0600, 1 key + 1 +secret). Rack edge-key copy `shred -u`'d; staged edge-config scripts + +console driver + rack creds copy removed (`ls` confirms none remain). The +non-secret console transcript `~/d112c-console-dc1.log` retained as an audit +artifact. **SEC-015** opened for the transient edge-key-on-rack exposure +(minted->wiped same phase; disclosed). Revert: n/a (cleanup). + +## Milestone + +dc1 edge fully bootstrapped + addressed -- D-112(c) console bootstrap and +D-113(a2) REST API both PROVEN on 26.7 (second datapoint after dc0). Edge: +WAN `172.30.3.2/24` gw `172.30.3.1` (egress 0% loss), LAN `10.12.64.1/22` +(the ruled provider-public gateway), API answering at the new LAN. + +## Item 7 -- dc-rack-net.sh dc1 site-table arm + harness (D-131 sub-1 delivery) + +Added the `dc1)` arm to `scripts/dc-rack-net.sh` (the STANDING per-DC forwarder ++ rack-legs pattern), MEASURED/apex-committed values, network-name keyed (no +virbrN literal -- T6 holds): + vr1-dc1-metal-admin 10.12.68.2/22 rack MAAS/DHCP leg (apex-committed rack IP) + vr1-dc1-metal-admin 10.12.68.3/22 node-DNS forwarder listen alias (D-120 static) + vr1-dc1-provider-public 10.12.64.2/22 edge-LAN leg (measured live on the bridge 2026-07-23) + DNS_LISTEN=10.12.68.3 DNS_UPSTREAM=10.10.0.20 (region BIND over transit). +Harness `tests/dc-rack-net/` extended T15-T18 (dc1 identity + MEASURED-tag); +**18/18**, gauntlet **ALL GREEN (76)**, repo-lint 0-fail. This forwarder is the +D-131 fix that pre-empts dc0's commissioning SERVFAIL on the isolated rack. +Revert: `git checkout` scripts/dc-rack-net.sh tests/dc-rack-net/run-tests.sh. + +## Item 8 -- dc-rack-net install on the dc1 rack (persistent legs + forwarder) + +`install dc1` run on the rack (operator-gated). Check PASS 10/10 +(`docs/audit/dc1-rack-net-install-20260723.txt`): persistent metal-admin legs +`10.12.68.2/22` + `10.12.68.3/22` on virbr6, provider-public `10.12.64.2/22` on +virbr4, `dc1-rack-legs` + `dc1-node-dns` enabled+active. Behavioral proof: the +forwarder answers authoritative `maas-internal SOA` via region BIND -- the +D-131 fix that pre-empts dc0's commissioning SERVFAIL. Revert: +`dc-rack-net.sh` install is idempotent; to undo, disable the two units + remove +the generated files + the interim legs. + +## Item 9 -- region-side MAAS config for dc1 metal-admin (DHCP + DNS + range) + +MAAS auto-discovered dc1 planes from the rack interfaces: metal-admin +`10.12.68.0/22` = subnet id 11 (VLAN fabric 142/vid 0), provider-public +`10.12.64.0/22` = subnet id 10. On subnet 11 (all operator-gated): +- D-120 dynamic range `10.12.68.100-10.12.68.200` created (iprange id 3) -- + the ruled band applied to dc1's CIDR. +- D-131: `dns_servers=10.12.68.3 allow_dns=false` (nodes resolve via the rack + forwarder, not MAAS -- the SERVFAIL fix). +- DHCP: VLAN fabric 142/vid 0 `dhcp_on=true primary_rack=nmpcq4`. +Verified BEHAVIORALLY (dc0 lesson -- not the self-report): dhcpd RUNNING on the +rack (`dhcpd -4 ... virbr6`, dhcpd.conf freshly generated); no Temporal +incident (the dc0 region restart fixed it fleet-wide). Revert: +`maas admin vlan update 142 0 dhcp_on=false`; `subnet update 11 dns_servers= +allow_dns=true`; `ipranges delete 3`. + +## Item 10 -- dc1 enlistment PROVEN (canary) + +`virsh reset vr1-dc1-control-01` -> the node PXE-booted, got DHCP from nmpcq4, +and ENLISTED in MAAS ~2 min later (machine count 11 -> 12). The +DHCP->PXE->enlist chain works end to end for dc1; with the forwarder +pre-installed, dc0's two stacked commissioning faults are pre-mitigated. +Revert: n/a (enlistment; the machine is deleted/re-commissioned as needed). + +## Item 11 -- SEC-016 ruling: dedicated dc1 MAAS->libvirt power key (GA-R5) + +Commissioning needs the region MAAS snap to SSH the dc1 rack libvirt +(`power_type=virsh`). The dc0 rack authorizes the SEC-012 MAAS key; the dc1 +rack does not. Operator RULED (AskUserQuestion, exact utterance): **"Mint a +dedicated dc1 power key"** -- per-DC isolation, NOT cross-DC reuse of SEC-012. +Recorded as **SEC-016** (security-ledger). Wiring (dependent work, next): +mint `vr1-dc1-maas-power_ed25519`; authorize its pubkey on the dc1 rack; +install privkey in the region MAAS snap + snap `ssh config` Host 172.31.0.6; +give the maas-node-power script's virsh reach from voffice1 via the dc1 SERVICE +key (dc0 split). Then `maas-node-power.sh` dc1 (dry -> --commit), commission +9/9. Revert: deauthorize the pubkey on the rack, remove the snap key + config. + +## Item 12 -- SEC-016 power key WIRED (dedicated dc1 MAAS->libvirt) + +Per the SEC-016 ruling, replicating the dc0 split (script=svc key, MAAS=dedicated +key): +- Minted `~/vr1-dc1-creds/vr1-dc1-maas-power_ed25519` (manifest updated, + creds-audit CLEAN 7 entries). +- Public half authorized on the dc1 rack jessea123 (2 keys now, matching dc0). +- SCRIPT virsh path: dc1 SERVICE key copied to voffice1 `~/vr1-dc1-creds/` + + voffice1 `~/.ssh/config` Host 172.31.0.6 -> svc key (voffice1 reaches the rack + directly on its 172.31.0.5/30 transit leg). Verified: voffice1 virsh lists 9 + dc1 domains. +- MAAS power path: dedicated power PRIVKEY installed in the region MAAS snap + (`/var/snap/maas/current/root/.ssh/id_dc1_power`, 0600 root) + snap + `ssh config` Host 172.31.0.6 -> that key (dc0's 172.31.0.2 keeps SEC-012's + default id_ed25519 -- true per-DC isolation). Revert: deauthorize the pubkey + on the rack, remove the snap key + Host block, remove the voffice1 svc key + + Host block. + +## Item 13 -- power control set + commissioning 9/9 READY + +All 9 dc1 nodes reset -> PXE enlisted (by pinned 52:54:01:d1 MACs; machines +11->20). `maas-node-power.sh --commit` set `power_type=virsh` on all 9, each +verified by a real `query-power-state` -- which PROVES the SEC-016 dedicated +key drives the dc1 rack from the region (2 transient mid-shutdown power=error +cleared on re-query). Region MAAS config: D-120 range + D-131 forwarder DNS + +DHCP primary_rack=nmpcq4. Commissioned all 9 -> **ALL 9 READY in ~3.5 min** +(Commissioning -> Testing -> Ready, NO timeout, NO SERVFAIL), shapes EXACT to +D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB), power=virsh +(`docs/audit/dc1-commissioning-verify-20260723.txt`). dc0's two stacked faults +(MAC regen, rack-resolver SERVFAIL) were pre-empted by the pinned MACs (step C) ++ the dc-rack-net forwarder (item 8). Revert: `maas admin machine delete ` +per node; power-off domains. + +## Milestone -- G12 [V] leg COMPLETE + +dc1 substrate fully built and commissioned: outer vvr1-dc1 + inner 28/28 (prior +sessions) -> edge bootstrapped + addressed (D-112c/D-113a2 on 26.7) -> rack-net ++ region MAAS -> 9/9 nodes READY, shapes exact. G12's remaining leg was the +build; that is done. NEXT: G12 close-out (this changelog consolidated, gauntlet ++ repo-lint, GA-R7 memory review, skill sweep, operator-gated merge to `main`, +branch retirement). +(provider-public `10.12.64.2/22`, metal-admin `10.12.68.2/22`, forwarder +`10.12.68.3` -> region BIND `10.10.0.20`) + harness update, then +`install dc1`; region-side MAAS (metal-admin `dns_servers=10.12.68.3 +allow_dns=false`, DHCP `primary_rack=nmpcq4`, D-120 range +`10.12.68.100-.200`); `maas-node-power.sh` dc1 arm; commission 9/9 to Ready +(D-121 Option C); then G12 close-out (gauntlet, repo-lint, consolidation, +memory review, skill sweep, operator-gated merge to `main`). diff --git a/docs/archive/stage-records/vr1-stage3-record.md b/docs/archive/stage-records/vr1-stage3-record.md index 12e77a0..4f89354 100644 --- a/docs/archive/stage-records/vr1-stage3-record.md +++ b/docs/archive/stage-records/vr1-stage3-record.md @@ -1,4 +1,22 @@ -# VR1 Stage 3 record -- CLOSED 2026-07-21 (dc0 substrate; dc1 HELD under G12) +# VR1 Stage 3 record -- CLOSED 2026-07-21 (dc0 substrate); G12 dc1 remainder CLOSED 2026-07-23 + +**G12 CLOSED 2026-07-23 (dc1 substrate; operator-ruled "Merge to main + full close").** +The HELD dc1 remainder was built to a commissioned fleet on branch +`dc-dc-g12-dc1-substrate` and merged to `main`. Sequence (all operator-gated, +`docs/CURRENT-STATE.md` G12 row is the status authority): apex `--commit` + outer +step-A (5/0/0) + step-B transit/rack/SEC-010 + step-C inner 28/28 + D-125 egress +gate (2026-07-22); then edge bootstrap D-112(c)/D-113(a2) on 26.7 + v4 addressing, +`dc-rack-net` dc1 arm (harness 18/18) + install (forwarder answers `maas-internal`), +region MAAS (D-120 range + D-131 forwarder DNS + DHCP primary_rack=nmpcq4), SEC-016 +dedicated per-DC MAAS->libvirt power key (operator-ruled), and **commissioning 9/9 +READY in ~3.5 min, shapes exact to D-121 Option C** (2026-07-23). dc0's two stacked +commissioning faults (MAC regen, rack-resolver SERVFAIL) were pre-empted by pinned +MACs (step C) + the pre-installed forwarder. New SEC rows: SEC-015 (transient edge +key on rack, remediated), SEC-016 (dedicated dc1 power key). Three findings logged +(edge-bootstrap `util.inc`/`shell_safe()`, transient post-restart-sshd scp, +MAAS-snap-key refresh fragility). dc1 session changelogs added to the table below. + +# VR1 Stage 3 record -- dc0 substrate CLOSED 2026-07-21 (original title preserved) **Stage close (operator-ruled 2026-07-21, "Close and merge"):** all dc0 gates CLOSED with captures -- G9 (outer apply), G10 (deploy steps B-E: bootstrap, depth-4 boot, D-125 egress, MAAS + all 9 @@ -24,6 +42,9 @@ | changelog-20260719-dc0-deploy-stepB.md | deploy steps A-D: transit, bootstrap, inner apply, edge, enlistment, power | 838 | | changelog-20260721-commissioning-diag.md | commissioning double-fault diagnosis + repair (9/9 Ready) | 132 | | changelog-20260721-close-and-delivery.md | MAC pinning + dc-rack-net + incident docs + netem sudo mechanism | 140 | +| changelog-20260721-g12-dc1-authoring.md | G12 dc1: apex confirm-free + importer dc1 support + vars/substrate authoring (drop-state) | -- | +| changelog-20260722-g12-dc1-build.md | G12 dc1: apex write + outer step A + step B/C + D-125 egress gate | 257 | +| changelog-20260723-g12-dc1-edge.md | G12 dc1: edge bootstrap/addressing + rack-net + region MAAS + SEC-016 + commissioning 9/9 | 240 | | file (now docs/archive/changelogs/) | title line | lines | |---|---|---| diff --git a/docs/audit/d125-egress-gate-20260722-dc1.txt b/docs/audit/d125-egress-gate-20260722-dc1.txt new file mode 100644 index 0000000..1acf6da --- /dev/null +++ b/docs/audit/d125-egress-gate-20260722-dc1.txt @@ -0,0 +1,30 @@ +# D-125 egress gate, vr1-dc1 -- 2026-07-22T22:02:58Z -- run 1 of 2 +WARNING Requested memory 1024 MiB is less than the recommended 3072 MiB for OS ubuntu24.04 +== guest booted; waiting for the probe to report (<=180s) == +=== D-125 PROBE RESULT === +[ 14.871107] cloud-init[674]: ci-info: | enp1s0 | True | 172.30.3.50 | 255.255.255.0 | global | 52:54:00:22:37:09 | +enp1s0 UP 172.30.3.50/24 fe80::5054:ff:fe22:3709/64 +172.30.3.0/24 dev enp1s0 proto kernel scope link src 172.30.3.50 +GW-PING-RC=0 +d125-throwaway login: NET-PING-RC=0 +LAN-TCP-http=000 +NET-TCP-http=301 +NET-TCP-ubuntu=200 +D125-TEST-COMPLETE +=== teardown === +leftover-domains=0 +# run 2 of 2 +WARNING Requested memory 1024 MiB is less than the recommended 3072 MiB for OS ubuntu24.04 +== guest booted; waiting for the probe to report (<=180s) == +=== D-125 PROBE RESULT === +[ 14.685552] cloud-init[672]: ci-info: | enp1s0 | True | 172.30.3.50 | 255.255.255.0 | global | 52:54:00:a7:58:67 | +enp1s0 UP 172.30.3.50/24 fe80::5054:ff:fea7:5867/64 +172.30.3.0/24 dev enp1s0 proto kernel scope link src 172.30.3.50 +GW-PING-RC=0 +d125-throwaway login: NET-PING-RC=0 +LAN-TCP-http=000 +NET-TCP-http=301 +NET-TCP-ubuntu=200 +D125-TEST-COMPLETE +=== teardown === +leftover-domains=0 diff --git a/docs/audit/dc1-apex-confirm-20260721.txt b/docs/audit/dc1-apex-confirm-20260721.txt new file mode 100644 index 0000000..4bf2512 --- /dev/null +++ b/docs/audit/dc1-apex-confirm-20260721.txt @@ -0,0 +1,32 @@ +# dc1 apex confirm-free query -- 2026-07-21T23:45:49Z -- run on office1-netbox (read-only GETs) +== prefixes within_include 10.12.64.0/19 (dc1 supernet) == +count: 6 + 10.12.64.0/22 role=provider-public status=active scope=vr1-dc1 desc=VR1 DC1 provider-public (v4; D-101) + 10.12.68.0/22 role=metal-admin status=active scope=vr1-dc1 desc=VR1 DC1 metal-admin (v4; D-101) + 10.12.72.0/22 role=metal-internal status=active scope=vr1-dc1 desc=VR1 DC1 metal-internal (v4; D-101) + 10.12.76.0/22 role=data-tenant status=active scope=vr1-dc1 desc=VR1 DC1 data-tenant (v4; D-101) + 10.12.80.0/22 role=storage status=active scope=vr1-dc1 desc=VR1 DC1 storage (v4; D-101) + 10.12.84.0/22 role=replication status=active scope=vr1-dc1 desc=VR1 DC1 replication (v4; D-101) +== prefixes within_include 172.31.0.0/24 (transit container) == +count: 2 + 172.31.0.0/24 role=transit status=container desc=Transit (v4) -- dedicated per-DC point-to-point transit supernet (D-12 + 172.31.0.0/30 role=transit status=active scope=vr1-dc0 desc=office1<->dc0 management transit -- region<->rack MAAS control path (D +== prefix exact 172.30.3.0/24 (dc1 uplink) == +count: 1 + 172.30.3.0/24 role=edge status=active scope=vr1-dc1 desc=vr1-dc1 simulated-ISP edge WAN (D-115) +== prefixes within_include 172.30.0.0/16 (edge supernet) == +count: 4 + 172.30.0.0/16 role=edge status=container desc=Edge (v4) -- simulated ISP/WAN segments; mirrors v6 2602:f3e2:fe::/48 + 172.30.1.0/24 role=edge status=active scope=vr1-off1 desc=VR1 Off1 office1-wan -- simulated ISP uplink (D-115) + 172.30.2.0/24 role=edge status=active scope=vr1-dc0 desc=vr1-dc0 simulated-ISP edge WAN (D-115) + 172.30.3.0/24 role=edge status=active scope=vr1-dc1 desc=vr1-dc1 simulated-ISP edge WAN (D-115) +== ip-addresses parent 10.12.68.0/22 (dc1 metal-admin band) == +count: 0 +== ip-addresses parent 172.31.0.4/30 (dc1 transit) == +count: 0 +== site vr1-dc1 == +count: 1 + vr1-dc1 status=active desc=Virtual Site for testing +== role transit == +count: 1 + transit diff --git a/docs/audit/dc1-commissioning-verify-20260723.txt b/docs/audit/dc1-commissioning-verify-20260723.txt new file mode 100644 index 0000000..b6c4390 --- /dev/null +++ b/docs/audit/dc1-commissioning-verify-20260723.txt @@ -0,0 +1,13 @@ +=== dc1 commissioning verify (2026-07-23T00:58:02Z) === +dc1 nodes: 9 statuses: {'Ready': 9} +shapes (cpu,GiB): {(16, 64): 3, (8, 24): 4, (12, 48): 2} + apt-ocelot Ready power=virsh/off 16cpu/64GiB + big-trout Ready power=virsh/off 16cpu/64GiB + real-wasp Ready power=virsh/off 16cpu/64GiB + ample-buck Ready power=virsh/off 12cpu/48GiB + divine-heron Ready power=virsh/off 12cpu/48GiB + big-eel Ready power=virsh/off 8cpu/24GiB + cute-satyr Ready power=virsh/off 8cpu/24GiB + right-spider Ready power=virsh/off 8cpu/24GiB + tender-elk Ready power=virsh/off 8cpu/24GiB +D-121 Option C expected: 3x(16,64) + 2x(12,48) + 4x(8,24) diff --git a/docs/audit/dc1-rack-import-commit-20260722.txt b/docs/audit/dc1-rack-import-commit-20260722.txt new file mode 100644 index 0000000..0cbe340 --- /dev/null +++ b/docs/audit/dc1-rack-import-commit-20260722.txt @@ -0,0 +1,29 @@ +# dc1 rack/transit importer COMMIT -- 2026-07-22T20:12:52Z -- piped to office1-netbox, token on-host (never printed) +Target : http://localhost:8000 (SANDBOX) + +*** COMMITTING. *** + +Transit prefix (office1<->vr1-dc1 mesh leg): + CREATED 172.31.0.4/30 (id=139) role=transit scope=vr1-dc1 + +Rack metal-admin static IP: + CREATED 10.12.68.2/22 (id=4) dns=vvr1-dc1 + +================================================================== +created: 2 already present: 0 +exit=0 +# dc1 rack/transit importer POST-COMMIT idempotency read-back -- 2026-07-22T20:13:54Z +Target : http://localhost:8000 (SANDBOX) + +*** DRY RUN -- nothing will be written. Re-run with --commit. *** + +Transit prefix (office1<->vr1-dc1 mesh leg): + EXISTS 172.31.0.4/30 + +Rack metal-admin static IP: + EXISTS 10.12.68.2/22 + +================================================================== +would create: 0 already present: 2 +DRY RUN -- nothing was written. Re-run with --commit. +exit=0 diff --git a/docs/audit/dc1-rack-import-dryrun-20260721.txt b/docs/audit/dc1-rack-import-dryrun-20260721.txt new file mode 100644 index 0000000..1307c8c --- /dev/null +++ b/docs/audit/dc1-rack-import-dryrun-20260721.txt @@ -0,0 +1,14 @@ +# dc1 rack/transit importer DRY-RUN vs live apex -- 2026-07-21T23:53:24Z -- on office1-netbox +Target : http://localhost:8000 (SANDBOX) + +*** DRY RUN -- nothing will be written. Re-run with --commit. *** + +Transit prefix (office1<->vr1-dc1 mesh leg): + [dry-run] would CREATE 172.31.0.4/30 role=transit scope=dcim.site:vr1-dc1 + +Rack metal-admin static IP: + [dry-run] would CREATE 10.12.68.2/22 dns=vvr1-dc1 (metal-admin static band) + +================================================================== +would create: 2 already present: 0 +DRY RUN -- nothing was written. Re-run with --commit. diff --git a/docs/audit/dc1-rack-import-dryrun-20260722.txt b/docs/audit/dc1-rack-import-dryrun-20260722.txt new file mode 100644 index 0000000..66f4a7d --- /dev/null +++ b/docs/audit/dc1-rack-import-dryrun-20260722.txt @@ -0,0 +1,15 @@ +# dc1 rack/transit importer DRY-RUN re-verify -- 2026-07-22T19:33:47Z -- piped to office1-netbox, token on-host (/root/netbox-secrets/api.token, never printed) +Target : http://localhost:8000 (SANDBOX) + +*** DRY RUN -- nothing will be written. Re-run with --commit. *** + +Transit prefix (office1<->vr1-dc1 mesh leg): + [dry-run] would CREATE 172.31.0.4/30 role=transit scope=dcim.site:vr1-dc1 + +Rack metal-admin static IP: + [dry-run] would CREATE 10.12.68.2/22 dns=vvr1-dc1 (metal-admin static band) + +================================================================== +would create: 2 already present: 0 +DRY RUN -- nothing was written. Re-run with --commit. +exit=0 diff --git a/docs/audit/dc1-rack-net-install-20260723.txt b/docs/audit/dc1-rack-net-install-20260723.txt new file mode 100644 index 0000000..2c946cf --- /dev/null +++ b/docs/audit/dc1-rack-net-install-20260723.txt @@ -0,0 +1,15 @@ +=== dc-rack-net check dc1 (2026-07-23T00:20:23Z) === + OK /usr/local/sbin/dc1-rack-net-apply matches + OK /etc/systemd/system/dc1-rack-legs.service matches + OK /etc/dnsmasq-dc1-node.conf matches + OK /etc/systemd/system/dc1-node-dns.service matches + OK dc1-rack-legs enabled + OK dc1-node-dns enabled + OK dc1-node-dns active + OK 10.12.68.2/22 on virbr6 (vr1-dc1-metal-admin) + OK 10.12.68.3/22 on virbr6 (vr1-dc1-metal-admin) + OK 10.12.64.2/22 on virbr4 (vr1-dc1-provider-public) +dc-rack-net check (dc1): PASS +=== forwarder probe: SOA maas-internal via 10.12.68.3 === +maas-internal. nobody.example.com. 1877 600 1800 604800 15 +rc=0 diff --git a/docs/audit/dc1-stepB-check-20260722-final.txt b/docs/audit/dc1-stepB-check-20260722-final.txt new file mode 100644 index 0000000..9af8e62 --- /dev/null +++ b/docs/audit/dc1-stepB-check-20260722-final.txt @@ -0,0 +1,17 @@ +# dc1 step-B gate FINAL: site-headend-install --check on vvr1-dc1 -- 2026-07-22T21:37:37Z +OK: maas snap 3.7.2-17972-g.35e297c4d (3.7/stable) +OK: MAAS rackd running +OK: enrolled to region http://10.10.0.20:5240/MAAS +== node-host (Model B) readiness == + [ok] libvirt/qemu installed + [ok] nested KVM ON (kvm_amd) + [ok] jessea123 in libvirt group + [ok] inner pool dir /var/lib/libvirt/vr1-dc1-inner + [ok] opnsense base /var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2 + [ok] SEC-010 transit FORWARD-drop present + transit interface 'enp1s0' exists + [ok] D-125 WAN bridge 'br-vr1-dc1-wan' present + uplink 'enp2s0' enslaved + [gate] DEPLOY-TIME (unprovable here, like the depth-4 boot gate): before the OPNsense chain + rides it, prove egress in ISOLATION -- attach a throwaway guest to 'br-vr1-dc1-wan', + confirm it gets a vcloud-ISP address and pings out. FAIL => revert to the D-125 + double-NAT fallback, NOT a redesign. +check-exit=0 diff --git a/docs/audit/dc1-stepB-check-20260722.txt b/docs/audit/dc1-stepB-check-20260722.txt new file mode 100644 index 0000000..e93e407 --- /dev/null +++ b/docs/audit/dc1-stepB-check-20260722.txt @@ -0,0 +1,17 @@ +# dc1 step-B gate: site-headend-install --check on vvr1-dc1 -- 2026-07-22T21:32:50Z +OK: maas snap 3.7.2-17972-g.35e297c4d (3.7/stable) +OK: MAAS rackd running +OK: enrolled to region http://10.10.0.20:5240/MAAS +== node-host (Model B) readiness == + [ok] libvirt/qemu installed + [ok] nested KVM ON (kvm_amd) + [ok] jessea123 in libvirt group + [ok] inner pool dir /var/lib/libvirt/vr1-dc1-inner + [--] opnsense base missing -- run: opnsense-prep-image.sh 26.1 /var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2 + [ok] SEC-010 transit FORWARD-drop present + transit interface 'enp1s0' exists + [ok] D-125 WAN bridge 'br-vr1-dc1-wan' present + uplink 'enp2s0' enslaved + [gate] DEPLOY-TIME (unprovable here, like the depth-4 boot gate): before the OPNsense chain + rides it, prove egress in ISOLATION -- attach a throwaway guest to 'br-vr1-dc1-wan', + confirm it gets a vcloud-ISP address and pings out. FAIL => revert to the D-125 + double-NAT fallback, NOT a redesign. +check-exit=1 diff --git a/docs/audit/gauntlet-20260722-g12-dropstate.txt b/docs/audit/gauntlet-20260722-g12-dropstate.txt new file mode 100644 index 0000000..333e399 --- /dev/null +++ b/docs/audit/gauntlet-20260722-g12-dropstate.txt @@ -0,0 +1,92 @@ + carve-host-interfaces SOME FAILED + MISS expected /no assigned network literals yet/ + checks ALL PASS (62 checks) + claude-guard ALL PASS + clientdocs-scripts ALL PASS + clientdocs-skill ALL PASS + cloud-assert ALL PASS + cloudinit-vm cloudinit-vm: 6 passed, 0 failed + cloud-snapshot ALL PASS + creds-audit creds-audit: 7/7 PASS + d063-apply ALL PASS (7/7) + d115-office-carve d115-office-carve: 20/20 PASS + d120-compose-bands d120-compose-bands: 16/16 PASS + d124-transit-seed d124-transit-seed: 21/21 PASS + dc-dc-ceph-disk-budget ALL PASS + dc-dc-dr-drill RESULT: PASS=21 FAIL=2 + FAIL T21 + FAIL T22 failback --apply is BLOCKED by the dc2 gate (rc 3) (rc=2 want=3) + FAIL: juju required on PATH for --apply + dc-dc-mtu-geneve-budget ALL PASS + dc-dc-prefixes-import ALL PASS (91 checks) + dc-dc-radosgw-multisite RESULT: PASS=17 FAIL=2 + FAIL T16 + FAIL T17 dc2 --apply is BLOCKED by the gate (rc 3) (rc=2 want=3) + FAIL: juju required on PATH for --apply + dc-dc-rbd-mirror RESULT: PASS=17 FAIL=2 + FAIL T17 + FAIL T18 dc2 --apply is BLOCKED by the gate (rc 3) (rc=2 want=3) + FAIL: juju required on PATH for --apply + dc-dc-whole-host-budget dc-dc-whole-host-budget: 13/13 PASS + dc-edge-wan-import dc-edge-wan-import: 58/58 PASS + dc-rack-mgmt-import dc-rack-mgmt-import: 117/117 PASS + dc-rack-net dc-rack-net: 14 passed, 0 failed + dc-selector ALL PASS (40 checks) + juju-spaces-check ALL PASS (5/5) + keystone-policy-drift ALL PASS + ledger-scan ALL PASS (46 checks) + lib-validate ALL PASS (35 checks) + maas-node-power maas-node-power: 24/24 PASS + netem-link netem-link: 12 passed, 0 failed + netem-sudoers netem-sudoers: 9 passed, 0 failed + node-vm node-vm: 15 passed, 0 failed + opentofu-validate ALL PASS + opnsense-api ALL PASS + opnsense-bootstrap-apikey ALL PASS + opnsense-plugins opnsense-plugins: 20 passed, 0 failed + opnsense-prep-image ALL PASS + opnsense-set-interface-v4 opnsense-set-interface-v4: 53/53 PASS + opnsense-set-interface-v6 opnsense-set-interface-v6: 40/40 PASS + phase-00-maas-standup SOME FAILED + [XX] DC=vr1-dc1: net selector fails loud (NetBox gap) exit 0 (want 1) + MISS expected /no assigned network literals yet/ + LEAK forbidden /DO:|WOULD:/ + phase-00-teardown-d061 ALL PASS + phase-02 ALL PASS + phase-03-adminrc ALL PASS + phase-03 ALL PASS + phase-04-create ALL PASS + phase-04-internal-cert-san ALL PASS + phase-04 ALL PASS + phase-05-amphora ALL PASS + phase-05 ALL PASS + phase-06-bootstrap ALL PASS + phase-06-capi-stack ALL PASS + phase-06-k8s-bootstrap ALL PASS + phase-06-kubeconfig-gate ALL PASS + phase-06-mgmt-vm ALL PASS + phase-06-net-setup ALL PASS + phase-07-conductor-graft ALL PASS + preflight ALL PASS + prereqs prereqs: 32/32 PASS + provider-bundle-check ALL PASS + record-audit record-audit: 35/35 PASS + reenroll-hosts SOME FAILED + MISS expected /no assigned network literals yet/ + repo-lint ALL PASS + roles-aggregates-import roles-aggregates-import: 24/24 PASS + sandbox-fidelity-check sandbox-fidelity-check: 14/14 PASS + sandbox-seed sandbox-seed: 22/22 PASS + site-baseleg site-baseleg: 24 passed, 0 failed + site-forward site-forward: 38/38 PASS + site-headend-install site-headend-install: 54/54 PASS + site-ssh-config site-ssh-config: 30/30 PASS + tenant-acceptance ALL PASS + tenant-assert ALL PASS + tenant-offboard ALL PASS (22/22) + tenant-onboard ALL PASS + trust_filter ALL PASS + validate ALL PASS (29 checks) + vault-kv-health ALL PASS (10/10) + +GAUNTLET: 6/76 FAILED -- carve-host-interfaces dc-dc-dr-drill dc-dc-radosgw-multisite dc-dc-rbd-mirror phase-00-maas-standup reenroll-hosts diff --git a/docs/audit/gauntlet-20260722-g12-reconcile.txt b/docs/audit/gauntlet-20260722-g12-reconcile.txt new file mode 100644 index 0000000..c289ec6 --- /dev/null +++ b/docs/audit/gauntlet-20260722-g12-reconcile.txt @@ -0,0 +1,78 @@ + carve-host-interfaces ALL PASS + checks ALL PASS (62 checks) + claude-guard ALL PASS + clientdocs-scripts ALL PASS + clientdocs-skill ALL PASS + cloud-assert ALL PASS + cloudinit-vm cloudinit-vm: 6 passed, 0 failed + cloud-snapshot ALL PASS + creds-audit creds-audit: 7/7 PASS + d063-apply ALL PASS (7/7) + d115-office-carve d115-office-carve: 20/20 PASS + d120-compose-bands d120-compose-bands: 16/16 PASS + d124-transit-seed d124-transit-seed: 21/21 PASS + dc-dc-ceph-disk-budget ALL PASS + dc-dc-dr-drill ALL PASS + dc-dc-mtu-geneve-budget ALL PASS + dc-dc-prefixes-import ALL PASS (91 checks) + dc-dc-radosgw-multisite ALL PASS + dc-dc-rbd-mirror ALL PASS + dc-dc-whole-host-budget dc-dc-whole-host-budget: 13/13 PASS + dc-edge-wan-import dc-edge-wan-import: 58/58 PASS + dc-rack-mgmt-import dc-rack-mgmt-import: 117/117 PASS + dc-rack-net dc-rack-net: 14 passed, 0 failed + dc-selector ALL PASS (40 checks) + juju-spaces-check ALL PASS (5/5) + keystone-policy-drift ALL PASS + ledger-scan ALL PASS (46 checks) + lib-validate ALL PASS (35 checks) + maas-node-power maas-node-power: 24/24 PASS + netem-link netem-link: 12 passed, 0 failed + netem-sudoers netem-sudoers: 9 passed, 0 failed + node-vm node-vm: 15 passed, 0 failed + opentofu-validate ALL PASS + opnsense-api ALL PASS + opnsense-bootstrap-apikey ALL PASS + opnsense-plugins opnsense-plugins: 20 passed, 0 failed + opnsense-prep-image ALL PASS + opnsense-set-interface-v4 opnsense-set-interface-v4: 53/53 PASS + opnsense-set-interface-v6 opnsense-set-interface-v6: 40/40 PASS + phase-00-maas-standup ALL PASS + phase-00-teardown-d061 ALL PASS + phase-02 ALL PASS + phase-03-adminrc ALL PASS + phase-03 ALL PASS + phase-04-create ALL PASS + phase-04-internal-cert-san ALL PASS + phase-04 ALL PASS + phase-05-amphora ALL PASS + phase-05 ALL PASS + phase-06-bootstrap ALL PASS + phase-06-capi-stack ALL PASS + phase-06-k8s-bootstrap ALL PASS + phase-06-kubeconfig-gate ALL PASS + phase-06-mgmt-vm ALL PASS + phase-06-net-setup ALL PASS + phase-07-conductor-graft ALL PASS + preflight ALL PASS + prereqs prereqs: 32/32 PASS + provider-bundle-check ALL PASS + record-audit record-audit: 35/35 PASS + reenroll-hosts ALL PASS + repo-lint ALL PASS + roles-aggregates-import roles-aggregates-import: 24/24 PASS + sandbox-fidelity-check sandbox-fidelity-check: 14/14 PASS + sandbox-seed sandbox-seed: 22/22 PASS + site-baseleg site-baseleg: 24 passed, 0 failed + site-forward site-forward: 38/38 PASS + site-headend-install site-headend-install: 54/54 PASS + site-ssh-config site-ssh-config: 30/30 PASS + tenant-acceptance ALL PASS + tenant-assert ALL PASS + tenant-offboard ALL PASS (22/22) + tenant-onboard ALL PASS + trust_filter ALL PASS + validate ALL PASS (29 checks) + vault-kv-health ALL PASS (10/10) + +GAUNTLET: ALL GREEN (76 harnesses) diff --git a/docs/audit/inner-converge-20260722-dc1-stepC.txt b/docs/audit/inner-converge-20260722-dc1-stepC.txt new file mode 100644 index 0000000..020c88e --- /dev/null +++ b/docs/audit/inner-converge-20260722-dc1-stepC.txt @@ -0,0 +1,2 @@ +28 +No changes. Your infrastructure matches the configuration. diff --git a/docs/audit/inner-plan-20260722-dc1-stepC.txt b/docs/audit/inner-plan-20260722-dc1-stepC.txt new file mode 100644 index 0000000..328e574 --- /dev/null +++ b/docs/audit/inner-plan-20260722-dc1-stepC.txt @@ -0,0 +1,1678 @@ + +OpenTofu used the selected providers to generate the following execution +plan. Resource actions are indicated with the following symbols: + + create + +OpenTofu will perform the following actions: + + # module.inner_storage.libvirt_pool.dc will be created + + resource "libvirt_pool" "dc" { + + allocation = (known after apply) + + available = (known after apply) + + capacity = (known after apply) + + id = (known after apply) + + name = "vr1-dc1-inner-pool" + + target = { + + path = "/var/lib/libvirt/vr1-dc1-inner" + } + + type = "dir" + + uuid = (known after apply) + } + + # module.vr1_dc1_node["vr1-dc1-compute-01"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-compute-01-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:04:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:04:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-compute-01-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 49152 + + memory_unit = "MiB" + + name = "vr1-dc1-compute-01" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 12 + } + + # module.vr1_dc1_node["vr1-dc1-compute-01"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 107374182400 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-compute-01-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-compute-02"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-compute-02-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:05:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:05:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-compute-02-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 49152 + + memory_unit = "MiB" + + name = "vr1-dc1-compute-02" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 12 + } + + # module.vr1_dc1_node["vr1-dc1-compute-02"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 107374182400 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-compute-02-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-control-01"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-control-01-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:01:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:01:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-control-01-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 65536 + + memory_unit = "MiB" + + name = "vr1-dc1-control-01" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 16 + } + + # module.vr1_dc1_node["vr1-dc1-control-01"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 161061273600 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-control-01-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-control-02"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-control-02-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:02:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:02:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-control-02-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 65536 + + memory_unit = "MiB" + + name = "vr1-dc1-control-02" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 16 + } + + # module.vr1_dc1_node["vr1-dc1-control-02"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 161061273600 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-control-02-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-control-03"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-control-03-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:03:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:03:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-control-03-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 65536 + + memory_unit = "MiB" + + name = "vr1-dc1-control-03" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 16 + } + + # module.vr1_dc1_node["vr1-dc1-control-03"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 161061273600 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-control-03-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-01"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-01-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:06:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:06:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-01-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-01" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-01"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-01-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-02"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-02-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:07:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:07:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-02-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-02" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-02"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-02-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-03"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-03-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:08:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:08:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-03-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-03" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-03"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-03-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_node["vr1-dc1-storage-04"].libvirt_domain.node will be created + + resource "libvirt_domain" "node" { + + autostart = false + + cpu = { + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + boot = { + + order = 2 + } + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-storage-04-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + boot = { + + order = 1 + } + + mac = { + + address = "52:54:01:d1:09:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-admin" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:02" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:03" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-metal-internal" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:04" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-data-tenant" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:05" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-storage" + } + } + }, + + { + + mac = { + + address = "52:54:01:d1:09:06" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-replication" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-storage-04-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 24576 + + memory_unit = "MiB" + + name = "vr1-dc1-storage-04" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 8 + } + + # module.vr1_dc1_node["vr1-dc1-storage-04"].libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = 590558003200 + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-storage-04-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_opnsense.libvirt_domain.vm will be created + + resource "libvirt_domain" "vm" { + + autostart = true + + cpu = { + + features = [ + + { + + name = "svm" + + policy = "disable" + }, + ] + + mode = "host-passthrough" + } + + devices = { + + channels = [ + + { + + source = { + + unix = { + + mode = "bind" + } + } + + target = { + + virt_io = { + + name = "org.qemu.guest_agent.0" + } + } + }, + ] + + disks = [ + + { + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-inner-pool" + + volume = "vr1-dc1-opnsense-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + ] + + interfaces = [ + + { + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-provider-public" + } + } + }, + + { + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-wan" + } + } + }, + ] + + serials = [ + + { + + log = { + + append = "on" + + file = "/var/lib/libvirt/vr1/staging/vr1-dc1-opnsense-serial.log" + } + + target = { + + port = 0 + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 2048 + + memory_unit = "MiB" + + name = "vr1-dc1-opnsense" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "pc" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 2 + } + + # module.vr1_dc1_opnsense.libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + capacity = (known after apply) + + create = { + + content = { + + url = "/home/jessea123/vr1-dc1-images/opnsense-26.7-nano.qcow2" + } + } + + id = (known after apply) + + key = (known after apply) + + name = "vr1-dc1-opnsense-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-inner-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vr1_dc1_planes.libvirt_network.plane["data-tenant"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "data-tenant.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-data-tenant" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["metal-admin"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "metal-admin.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-metal-admin" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["metal-internal"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "metal-internal.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-metal-internal" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["provider-public"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "provider-public.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-provider-public" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["replication"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "replication.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-replication" + + uuid = (known after apply) + } + + # module.vr1_dc1_planes.libvirt_network.plane["storage"] will be created + + resource "libvirt_network" "plane" { + + autostart = true + + domain = { + + name = "storage.vr1-dc1.cloud.neumatrix.local" + } + + id = (known after apply) + + mtu = { + + size = 9000 + } + + name = "vr1-dc1-storage" + + uuid = (known after apply) + } + + # module.vr1_dc1_wan.libvirt_network.wan_bridge will be created + + resource "libvirt_network" "wan_bridge" { + + autostart = true + + bridge = { + + name = "br-vr1-dc1-wan" + } + + forward = { + + mode = "bridge" + } + + id = (known after apply) + + name = "vr1-dc1-wan" + + uuid = (known after apply) + } + +Plan: 28 to add, 0 to change, 0 to destroy. + +───────────────────────────────────────────────────────────────────────────── + +Saved the plan to: dc1-inner-20260722.tfplan + +To perform exactly these actions, run the following command to apply: + tofu apply "dc1-inner-20260722.tfplan" diff --git a/docs/audit/outer-plan-20260722-dc1-substrate.txt b/docs/audit/outer-plan-20260722-dc1-substrate.txt new file mode 100644 index 0000000..631243d --- /dev/null +++ b/docs/audit/outer-plan-20260722-dc1-substrate.txt @@ -0,0 +1,252 @@ +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Refreshing state... [id=1ea36d3f-e7af-984c-8157-152724a0b85a] +module.voffice1.libvirt_cloudinit_disk.seed: Refreshing state... [id=a4694210c663c9ce] +module.vvr1_dc0.libvirt_cloudinit_disk.seed: Refreshing state... [id=ff281478c6083cc3] +module.office1_storage.libvirt_pool.dc: Refreshing state... [id=5f94194c-69c1-4b04-a85f-c18d87303a03] +module.vr1_dc1_storage.libvirt_pool.dc: Refreshing state... [id=4a1df114-ee04-4c80-9233-cc0c140c8556] +module.mesh_vr1_dc0_office1.libvirt_network.link: Refreshing state... [id=8318548f-c3d6-4e06-bef4-fe3f11d68125] +module.office1_network.libvirt_network.office1_local: Refreshing state... [id=8fdd2a97-417c-44d4-89e4-ae8d65594135] +module.mesh_vr1_dc1_office1.libvirt_network.link: Refreshing state... [id=38a20d2d-cd91-4604-a5f4-8e2a6609633c] +module.vr1_dc0_uplink.libvirt_network.site_wan: Refreshing state... [id=f3500153-e4de-45f1-8854-9c92974a6094] +module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link: Refreshing state... [id=9cbc8589-9f40-48e6-872e-ef3abfe29a93] +module.vr1_dc0_storage.libvirt_pool.dc: Refreshing state... [id=7ce1101c-a89e-40ca-9263-5f572bee40a9] +module.ubuntu_noble_base.libvirt_volume.base: Refreshing state... [id=/var/lib/libvirt/vr1/office1/ubuntu-24.04-base.qcow2] +module.voffice1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso] +module.office1_opnsense.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/office1-opnsense-disk.qcow2] +module.vvr1_dc0.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-cloudinit.iso] +module.voffice1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-disk.qcow2] +module.vvr1_dc0.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-disk.qcow2] +module.office1_opnsense.libvirt_domain.vm: Refreshing state... [name=office1-opnsense] +module.voffice1.libvirt_domain.vm: Refreshing state... [name=voffice1] +module.vvr1_dc0.libvirt_domain.vm: Refreshing state... [name=vvr1-dc0] + +Note: Objects have changed outside of OpenTofu + +OpenTofu detected the following changes made outside of OpenTofu since the +last "tofu apply" which may have affected this plan: + + # module.vvr1_dc0.libvirt_domain.vm has changed + ~ resource "libvirt_domain" "vm" { + ~ id = 4 -> 7 + name = "vvr1-dc0" + # (11 unchanged attributes hidden) + } + + +Unless you have made equivalent changes to your configuration, or ignored the +relevant attributes using ignore_changes, the following plan may include +actions to undo or respond to these changes. + +───────────────────────────────────────────────────────────────────────────── + +OpenTofu used the selected providers to generate the following execution +plan. Resource actions are indicated with the following symbols: + + create + +OpenTofu will perform the following actions: + + # module.vr1_dc1_uplink.libvirt_network.site_wan will be created + + resource "libvirt_network" "site_wan" { + + autostart = true + + domain = { + + name = "vr1-dc1-uplink" + } + + forward = { + + mode = "nat" + } + + id = (known after apply) + + ips = [ + + { + + address = "172.30.3.1" + + prefix = 24 + }, + ] + + mtu = { + + size = 1500 + } + + name = "vr1-dc1-uplink" + + uuid = (known after apply) + } + + # module.vvr1_dc1.libvirt_cloudinit_disk.seed will be created + + resource "libvirt_cloudinit_disk" "seed" { + + id = (known after apply) + + meta_data = <<-EOT + instance-id: vvr1-dc1-d124a + local-hostname: vvr1-dc1 + EOT + + name = "vvr1-dc1-cloudinit" + + network_config = <<-EOT + version: 2 + ethernets: + transit: + match: + name: "enp1s0" + addresses: ["172.31.0.6/30"] + routes: + - to: "10.10.0.0/22" + via: "172.31.0.5" + uplink: + match: + name: "enp2s0" + dhcp4: false + dhcp6: false + bridges: + br-vr1-dc1-wan: + interfaces: [uplink] + dhcp4: false + dhcp6: false + parameters: + stp: false + forward-delay: 0 + EOT + + path = (known after apply) + + size = (known after apply) + + user_data = <<-EOT + #cloud-config + hostname: vvr1-dc1 + fqdn: vvr1-dc1.cloud.neumatrix.local + manage_etc_hosts: true + users: + - name: jessea123 + groups: [adm, sudo] + shell: /bin/bash + sudo: "ALL=(ALL) NOPASSWD:ALL" + ssh_authorized_keys: + # D-126 per-env-key: vvr1-dc1 authorizes the DEDICATED dc1 key ONLY + # (~/vr1-dc1-creds/, D-124 amendment 2026-07-21). Inner root's qemu+ssh matches. + - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWB2qmc9Qrb/iaWYy6aNioqARVM/H+VHaHPqBi+4g2t vr1-dc1_svc (D-126 per-env key) + package_update: true + packages: + - qemu-guest-agent + runcmd: + - [systemctl, enable, --now, qemu-guest-agent] + EOT + } + + # module.vvr1_dc1.libvirt_domain.vm will be created + + resource "libvirt_domain" "vm" { + + autostart = false + + cpu = { + + features = [] + + mode = "host-passthrough" + } + + devices = { + + disks = [ + + { + + driver = { + + type = "qcow2" + } + + source = { + + volume = { + + pool = "vr1-dc1-pool" + + volume = "vvr1-dc1-disk.qcow2" + } + } + + target = { + + bus = "virtio" + + dev = "vda" + } + }, + + { + + device = "cdrom" + + source = { + + volume = { + + pool = "vr1-dc1-pool" + + volume = "vvr1-dc1-cloudinit.iso" + } + } + + target = { + + bus = "sata" + + dev = "sda" + } + }, + ] + + interfaces = [ + + { + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "mesh-vr1-dc1-office1" + } + } + }, + + { + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "vr1-dc1-uplink" + } + } + }, + ] + } + + features = { + + acpi = true + + apic = {} + } + + id = (known after apply) + + memory = 425984 + + memory_unit = "MiB" + + name = "vvr1-dc1" + + os = { + + type = "hvm" + + type_arch = "x86_64" + + type_machine = "q35" + } + + running = true + + type = "kvm" + + uuid = (known after apply) + + vcpu = 108 + } + + # module.vvr1_dc1.libvirt_volume.disk will be created + + resource "libvirt_volume" "disk" { + + allocation = (known after apply) + + backing_store = { + + format = { + + type = "qcow2" + } + + path = "/var/lib/libvirt/vr1/office1/ubuntu-24.04-base.qcow2" + } + + capacity = 3221225472000 + + id = (known after apply) + + key = (known after apply) + + name = "vvr1-dc1-disk.qcow2" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-pool" + + target = { + + format = { + + type = "qcow2" + } + + path = (known after apply) + } + } + + # module.vvr1_dc1.libvirt_volume.seed will be created + + resource "libvirt_volume" "seed" { + + allocation = (known after apply) + + capacity = (known after apply) + + create = { + + content = { + + url = (known after apply) + } + } + + id = (known after apply) + + key = (known after apply) + + name = "vvr1-dc1-cloudinit.iso" + + path = (known after apply) + + physical = (known after apply) + + pool = "vr1-dc1-pool" + } + +Plan: 5 to add, 0 to change, 0 to destroy. + +───────────────────────────────────────────────────────────────────────────── + +Note: You didn't use the -out option to save this plan, so OpenTofu can't +guarantee to take exactly these actions if you run "tofu apply" now. diff --git a/docs/audit/outer-plan-20260722-postdc1-converged.txt b/docs/audit/outer-plan-20260722-postdc1-converged.txt new file mode 100644 index 0000000..d71168b --- /dev/null +++ b/docs/audit/outer-plan-20260722-postdc1-converged.txt @@ -0,0 +1,30 @@ +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Refreshing state... [id=1ea36d3f-e7af-984c-8157-152724a0b85a] +module.voffice1.libvirt_cloudinit_disk.seed: Refreshing state... [id=a4694210c663c9ce] +module.mesh_vr1_dc0_office1.libvirt_network.link: Refreshing state... [id=8318548f-c3d6-4e06-bef4-fe3f11d68125] +module.vvr1_dc1.libvirt_cloudinit_disk.seed: Refreshing state... [id=41e9ec4b712038fc] +module.vvr1_dc0.libvirt_cloudinit_disk.seed: Refreshing state... [id=ff281478c6083cc3] +module.office1_storage.libvirt_pool.dc: Refreshing state... [id=5f94194c-69c1-4b04-a85f-c18d87303a03] +module.vr1_dc0_storage.libvirt_pool.dc: Refreshing state... [id=7ce1101c-a89e-40ca-9263-5f572bee40a9] +module.mesh_vr1_dc1_office1.libvirt_network.link: Refreshing state... [id=38a20d2d-cd91-4604-a5f4-8e2a6609633c] +module.office1_network.libvirt_network.office1_local: Refreshing state... [id=8fdd2a97-417c-44d4-89e4-ae8d65594135] +module.vr1_dc1_storage.libvirt_pool.dc: Refreshing state... [id=4a1df114-ee04-4c80-9233-cc0c140c8556] +module.vr1_dc0_uplink.libvirt_network.site_wan: Refreshing state... [id=f3500153-e4de-45f1-8854-9c92974a6094] +module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link: Refreshing state... [id=9cbc8589-9f40-48e6-872e-ef3abfe29a93] +module.vr1_dc1_uplink.libvirt_network.site_wan: Refreshing state... [id=4aad75c2-924f-410c-96bb-fa9217f8b4ea] +module.voffice1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso] +module.office1_opnsense.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/office1-opnsense-disk.qcow2] +module.vvr1_dc0.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-cloudinit.iso] +module.ubuntu_noble_base.libvirt_volume.base: Refreshing state... [id=/var/lib/libvirt/vr1/office1/ubuntu-24.04-base.qcow2] +module.vvr1_dc1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc1/vvr1-dc1-cloudinit.iso] +module.voffice1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-disk.qcow2] +module.vvr1_dc0.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-disk.qcow2] +module.vvr1_dc1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc1/vvr1-dc1-disk.qcow2] +module.office1_opnsense.libvirt_domain.vm: Refreshing state... [name=office1-opnsense] +module.vvr1_dc0.libvirt_domain.vm: Refreshing state... [name=vvr1-dc0] +module.vvr1_dc1.libvirt_domain.vm: Refreshing state... [name=vvr1-dc1] +module.voffice1.libvirt_domain.vm: Refreshing state... [name=voffice1] + +No changes. Your infrastructure matches the configuration. + +OpenTofu has compared your real infrastructure against your configuration and +found no differences, so no changes are needed. diff --git a/docs/audit/outer-plan-20260722-voffice1-dc1nic.txt b/docs/audit/outer-plan-20260722-voffice1-dc1nic.txt new file mode 100644 index 0000000..c978ca8 --- /dev/null +++ b/docs/audit/outer-plan-20260722-voffice1-dc1nic.txt @@ -0,0 +1,99 @@ +module.netem_vr1_dc0_vr1_dc1.terraform_data.netem: Refreshing state... [id=1ea36d3f-e7af-984c-8157-152724a0b85a] +module.vr1_dc1_storage.libvirt_pool.dc: Refreshing state... [id=4a1df114-ee04-4c80-9233-cc0c140c8556] +module.office1_storage.libvirt_pool.dc: Refreshing state... [id=5f94194c-69c1-4b04-a85f-c18d87303a03] +module.mesh_vr1_dc0_office1.libvirt_network.link: Refreshing state... [id=8318548f-c3d6-4e06-bef4-fe3f11d68125] +module.voffice1.libvirt_cloudinit_disk.seed: Refreshing state... [id=a4694210c663c9ce] +module.vvr1_dc0.libvirt_cloudinit_disk.seed: Refreshing state... [id=ff281478c6083cc3] +module.office1_network.libvirt_network.office1_local: Refreshing state... [id=8fdd2a97-417c-44d4-89e4-ae8d65594135] +module.mesh_vr1_dc0_vr1_dc1.libvirt_network.link: Refreshing state... [id=9cbc8589-9f40-48e6-872e-ef3abfe29a93] +module.mesh_vr1_dc1_office1.libvirt_network.link: Refreshing state... [id=38a20d2d-cd91-4604-a5f4-8e2a6609633c] +module.vr1_dc1_uplink.libvirt_network.site_wan: Refreshing state... [id=4aad75c2-924f-410c-96bb-fa9217f8b4ea] +module.vr1_dc0_uplink.libvirt_network.site_wan: Refreshing state... [id=f3500153-e4de-45f1-8854-9c92974a6094] +module.vvr1_dc1.libvirt_cloudinit_disk.seed: Refreshing state... [id=41e9ec4b712038fc] +module.vr1_dc0_storage.libvirt_pool.dc: Refreshing state... [id=7ce1101c-a89e-40ca-9263-5f572bee40a9] +module.voffice1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-cloudinit.iso] +module.office1_opnsense.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/office1-opnsense-disk.qcow2] +module.ubuntu_noble_base.libvirt_volume.base: Refreshing state... [id=/var/lib/libvirt/vr1/office1/ubuntu-24.04-base.qcow2] +module.vvr1_dc1.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc1/vvr1-dc1-cloudinit.iso] +module.vvr1_dc0.libvirt_volume.seed: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-cloudinit.iso] +module.vvr1_dc1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc1/vvr1-dc1-disk.qcow2] +module.voffice1.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/office1/voffice1-disk.qcow2] +module.vvr1_dc0.libvirt_volume.disk: Refreshing state... [id=/var/lib/libvirt/vr1/vr1-dc0/vvr1-dc0-disk.qcow2] +module.office1_opnsense.libvirt_domain.vm: Refreshing state... [name=office1-opnsense] +module.voffice1.libvirt_domain.vm: Refreshing state... [name=voffice1] +module.vvr1_dc0.libvirt_domain.vm: Refreshing state... [name=vvr1-dc0] +module.vvr1_dc1.libvirt_domain.vm: Refreshing state... [name=vvr1-dc1] + +OpenTofu used the selected providers to generate the following execution +plan. Resource actions are indicated with the following symbols: + ~ update in-place (current -> planned) + +OpenTofu will perform the following actions: + + # module.voffice1.libvirt_domain.vm will be updated in-place + ~ resource "libvirt_domain" "vm" { + ~ devices = { + ~ interfaces = [ + ~ { + + mac = { + + address = "52:54:00:89:e1:19" + } + # (2 unchanged attributes hidden) + }, + ~ { + + mac = { + + address = "52:54:00:99:04:01" + } + # (2 unchanged attributes hidden) + }, + + { + + mac = { + + address = "52:54:01:d1:fe:01" + } + + model = { + + type = "virtio" + } + + source = { + + network = { + + network = "mesh-vr1-dc1-office1" + } + } + }, + ] + # (1 unchanged attribute hidden) + } + id = 5 + name = "voffice1" + # (10 unchanged attributes hidden) + } + + # module.vvr1_dc1.libvirt_domain.vm will be updated in-place + ~ resource "libvirt_domain" "vm" { + ~ devices = { + ~ interfaces = [ + ~ { + + mac = { + + address = "52:54:00:11:39:ce" + } + # (2 unchanged attributes hidden) + }, + ~ { + + mac = { + + address = "52:54:00:3c:a6:a8" + } + # (2 unchanged attributes hidden) + }, + ] + # (1 unchanged attribute hidden) + } + id = 8 + name = "vvr1-dc1" + # (10 unchanged attributes hidden) + } + +Plan: 0 to add, 2 to change, 0 to destroy. + +───────────────────────────────────────────────────────────────────────────── + +Note: You didn't use the -out option to save this plan, so OpenTofu can't +guarantee to take exactly these actions if you run "tofu apply" now. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 8e93cdd..ffa9473 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -23,6 +23,9 @@ | SEC-013 | 2026-07-20 | **MAAS API key materialized to disk on the region.** An admin-scoped MAAS API key (`consumer:token:secret`) was placed by the operator in a 0600 file on `voffice1` so the `opentofu/vr1-dc0-maas` root could consume it via `TF_VAR_maas_api_key`. It grants FULL MAAS admin API access (machines, power, deploy, users). Two exposure surfaces beyond the file itself: (a) the OpenTofu **state file** of any root that uses the maas provider records it -- unavoidable with this provider, so that state inherits credential handling (0600, never committed); (b) a MAAS CLI profile was also created for the operator user from the same key. It was verified by FORMAT ONLY (71 bytes, 3 colon-separated parts) and never printed, echoed, or passed in argv. | 2026-07-20 step-D part 2; `opentofu/vr1-dc0-maas/main.tf` header; session changelog item 15 | operator | **OPEN -- rotation obligation (surface NARROWED 2026-07-21).** The retire-fully ruling was executed 2026-07-21: `opentofu/vr1-dc0-maas` removed from the repo, the on-disk key file (`~/vr1-dc0-creds/maas-api-key.txt` on voffice1) and the root's `terraform.tfstate` DELETED, absence verified by listing (session changelog 2026-07-21 items 13-15; NOTE the pod-object deletion cascaded to the 9 machine records -- incident capture `docs/audit/incident-20260721-pod-delete-cascade.txt`, recovered same-day). REMAINING surface: the operator's MAAS CLI profile only. Rotate the underlying API key at v1 close, or immediately if `voffice1` is rebuilt/shared. Custody detail off-repo per D-069. | | SEC-014 | 2026-07-21 | **Rack cluster secret exposed to session context.** During the commissioning diagnosis, a read of `/var/snap/maas//agent.yaml` on the DC0 rack (hunting the agent resolver's config surface) returned the rack's MAAS cluster `secret` into the Claude session context, the operator terminal scrollback, and the as-executed log (`~/as-executed/2026-07-21-ops-commissioning-diag.log`, 0600, jumphost-only). The secret authenticates rack<->region enrollment. The read was not anticipated to contain a credential (config file, not a key file); disclosed same-session. | 2026-07-21 session changelog item 6; docs/audit/commissioning-diag-20260721.txt | operator | **OPEN -- rotation obligation.** Rotate the MAAS shared secret (= rack re-enrollment for `7chphy`) at a convenient maintenance point, or immediately if session artifacts leave the jumphost. Process fix queued: add `agent.yaml` to the guard hook's never-read list alongside key/cred globs. | | SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk). | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` | operator | **CLOSED 2026-07-16 (operator ruling -- keep uniform 6-NIC).** Review R3-F10: A2's cross-examination refuted the attack-surface concern -- in the isolated-L2 sim the unbound vNICs have no reachability out, and pruning would INCREASE Roosevelt-delta (baremetal trunks all VLANs to every node on bonded NICs, so all planes are present regardless of L3 binding). Uniform 6-NIC is the more Roosevelt-faithful model. Accepted non-issue; no code change. | +| SEC-015 | 2026-07-23 | **dc1 edge SSH private key staged transiently on the DC1 rack.** The `vr1-dc1-edge_ed25519` private key (root-granting on `vr1-dc1-opnsense`) was scp'd to `vvr1-dc1:~/vr1-dc1-edge_ed25519` (0600) so `opnsense-bootstrap-apikey.sh` -- which SSHes the edge directly, no ProxyJump -- could mint the API key there (dc0 method). It existed on the rack only for the mint + v4-addressing phase and was `shred -u`'d immediately after (absence verified by `ls`). The permanent copy stays on vcloud `~/vr1-dc1-creds/` (SEC-007 pattern: edge SSH is the only management path -- a rotation obligation, not a delete-me). | 2026-07-23 session changelog item 6; docs/changelog-20260723-g12-dc1-edge.md | operator | **OPEN -- rotation obligation (transient exposure already remediated).** The on-disk rack copy is gone; the standing surface is the vcloud key + the minted API key/secret (`~/vr1-dc1-creds/opnsense-api.txt`). Rotate both at v1 close, or immediately if vcloud is rebuilt/shared. Roosevelt note: metal edges reached differently may avoid the rack-staging step entirely (a ProxyCommand-from-vcloud mint keeps the key off the rack -- deferred, not adopted this build to stay on the dc0-proven path). | + +| SEC-016 | 2026-07-23 | **Dedicated dc1 MAAS->libvirt power key (per-DC isolation -- operator-ruled).** dc1 commissioning needs `power_type=virsh`, which has the REGION's MAAS snap SSH to the dc1 rack's libvirt to power-cycle nodes. RULING (GA-R5) -- question presented: "How should the region drive the dc1 rack's libvirt for node power control (power_type=virsh)?" options (A) reuse the SEC-012 key on dc1, (B) mint a dedicated dc1 power key. Operator answer, exact utterance (option selected): **"Mint a dedicated dc1 power key"** -- a SEPARATE per-DC MAAS->libvirt keypair, NOT cross-DC reuse of SEC-012's key. Mechanism: private half in the region MAAS snap (`/var/snap/maas/current/root/.ssh/` + per-host `ssh config` Host 172.31.0.6 -> that IdentityFile, so 172.31.0.2/dc0 keeps SEC-012 and 172.31.0.6/dc1 uses this one); public half authorizes jessea123 (libvirt group) on the dc1 rack. The maas-node-power SCRIPT's own mapping virsh (jessea123 on voffice1) uses the dc1 SERVICE key via voffice1 `ssh config` -- the dc0 split exactly (script=svc key, MAAS=dedicated key). | 2026-07-23 session changelog; docs/changelog-20260723-g12-dc1-edge.md; dc0 precedent SEC-012 | operator | **OPEN -- rotation obligation.** (1) ROTATE at v1 close or if the region/dc1 rack is rebuilt/shared. (2) Same libvirt-group SCOPE hardening candidate as SEC-012 (power-only polkit grant). (3) FRAGILITY: the snap-side key + ssh config live under per-revision `/var/snap/maas/current/` and may not survive a snap refresh -- re-assert after any refresh (queued to DC standup DoD). Roosevelt-relevance: establishes PER-DC power-credential isolation (bare metal: per-DC IPMI/BMC creds, never a shared cross-DC power credential) -- whether that becomes a standing D-principle is a follow-up, not blocking. | **STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive files AND environment/config files for a site live in a single `~/-creds/` folder on vcloud, diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 6aad762..885062d 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -220,3 +220,37 @@ (pinned), G12 [V], G13 execution, 10 SEC rows, LP filing (operator). - Details: docs/archive/changelogs/changelog-20260721-tenant-review-pin.md. Status lives ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-07-21 -- G12 dc1-authoring session (fifth same-day; bounded, GA-R4) + +- Session DISCONNECTED before its bookend; entry landed by the successor session 2026-07-22 + after read-only re-verification (tree, branch, harnesses). No cloud state was touched. +- Delivered pre-drop (commit cbc1104 on branch dc-dc-g12-dc1-substrate -- branch UNPUSHED at + the drop): dc1 apex confirm-free capture (transit /30 + rack 10.12.68.2 FREE, planes already + consistent) + dc-rack-mgmt-import per-site --site support (117/117; live dry-run PASS). +- INTERRUPTED mid-item-3, authored UNCOMMITTED in tree: vr1_dc1 vars + vvr1-dc1 module + + vr1-dc1-uplink, inner root opentofu/vr1-dc1-substrate/ (54 MACs pre-pinned 52:54:01:d1), + lib-net.sh dc1 arm + selection guard, dc-selector 40/40. Drop point: BEFORE the gauntlet -- + successor run = 6/76 FAILED, six harnesses still assert the retired vr1-dc1 fail-loud arm + (docs/audit/gauntlet-20260722-g12-dropstate.txt). Owed: reconcile -> ALL GREEN -> delivery + entry -> commit+push; THEN the operator-gated apex --commit stays G12's next gated step. +- Details: docs/changelog-20260721-g12-dc1-authoring.md (item 3 = drop-state record). Status + lives ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-07-22 -- G12 dc1 build session (bounded, GA-R4) + +- Opened from the drop-state; predecessor's owed reconcile DELIVERED (six harnesses + two sighted + surfaces; gauntlet 76 ALL GREEN; landed as authoring-changelog item 4). Duplicate idle Claude + session checked at operator request -- already exited, nothing culled. +- G12 [V] leg executed through the D-125 gate, every mutation operator-gated, all pushed: + apex --commit (transit /30 + rack IP, read-back idempotent); dc1 svc key minted (audit CLEAN); + outer step A 5/0/0 exact; cloudinit-vm interface_macs shipped + voffice1 dc1-transit NIC 0/2/0 + (full MAC pins, post-bounce battery ALL PASS); transit .5<->.6 live; rack ENROLLED (nmpcq4) with + SEC-010 both ends + 26.7 base (hash-verified dc0 copy); inner step C 28/28 from voffice1 (one + fix-forward: serial-staging dir), 10/10 domains RUNNING, edge at the 26.7 login prompt; D-125 + egress gate PASS (two identical runs). Queued findings: changelog items 7/9 (four tool gaps). +- NEXT session: edge bootstrap (D-112(c) driver rebuild -- v6 + five measured lessons at + changelog-20260719-dc0-deploy-stepB.md:241-260; edge keypair mint first) -> API key -> v4 + addressing -> rack standup DoD -> commissioning 9/9 -> G12 close-out; operator-gated merge to + main is the hard new-session gate after that. +- Details: docs/changelog-20260722-g12-dc1-build.md (items 1-10). Status lives ONLY in CURRENT-STATE.md. diff --git a/logs/as-executed-index.md b/logs/as-executed-index.md index ef011f3..7be6d69 100644 --- a/logs/as-executed-index.md +++ b/logs/as-executed-index.md @@ -14,3 +14,4 @@ | 2026-07-08 | ops-decommission | jesse.austin (Claude Code lane, per-command wrap) | decommission window: lbtest LB reclaim + beta-cluster teardown + foil1 offboard (first offboard v2 --apply; E0 always-403 defect logged) + magnum orphan-sweep audit (clean); open/close cloud-assert PASS | | 2026-07-08 | ops-devteam-template | jesse.austin (Claude Code lane, per-command wrap) | deliver devteam-k8s cluster template (stage5, calico, hidden=False) after stuck flannel TestCluster2 confirmed gone + RAM reclaimed; closes the addendum-39 no-delivered-template gap | | 2026-07-21 | ops-commissioning-diag | jesse.austin (Claude Code lane, per-command wrap) | G10 commissioning diagnosis: instrumented run per committee; MAC-drift fault (all 9) repaired in place (ruled); MAAS 3.7 rack-only agent resolver defect found, dc0-node-dns forwarder workaround (ruled); ALL 9 NODES READY; SEC-014 opened; D-131 proposed | +| 2026-07-22 | dc1-deploy | jesse.austin (Claude Code lane, per-command wrap) | G12 dc1 build: apex --commit (transit /30 + rack IP, 2 objects), dc1 svc key minted, outer apply 5/0/0 exact (vvr1-dc1 + uplink), converged zero diff | diff --git a/netbox/dc-rack-mgmt-import.py b/netbox/dc-rack-mgmt-import.py index 9011ba3..b6e7630 100644 --- a/netbox/dc-rack-mgmt-import.py +++ b/netbox/dc-rack-mgmt-import.py @@ -1,17 +1,21 @@ #!/usr/bin/env python3 """ Register the D-124 Office1-region <-> DC-rack MANAGEMENT TRANSIT addressing in -office1-netbox (the VR1 IPAM apex): the point-to-point transit on the office1<->dc0 +office1-netbox (the VR1 IPAM apex): the point-to-point transit on the office1<->DC mesh leg, plus the rack's metal-admin static IP. Two objects, nothing else. D-124 (ADOPTED 2026-07-16, Scheme A -- transit-numbered mesh) rules that the region<->rack MAAS control path rides a small point-to-point transit on the -office1<->dc0 mesh leg (NOT metal-admin, which D-100 keeps DC-local and node-facing). -The rack (vvr1-dc0) STRADDLES both legs: the transit (region-facing) and metal-admin -(node-facing, 10.12.8.0/22). So this tool registers exactly: +office1<->DC mesh leg (NOT metal-admin, which D-100 keeps DC-local and node-facing). +The rack VM STRADDLES both legs: the transit (region-facing) and metal-admin +(node-facing, the DC's own /22). The DC is selected with --site (REQUIRED -- +explicit so dc1 values can never land scoped to dc0 or vice versa; the scheme +"generalizes per-leg", D-124). Per-site facts (metal-admin /22, rack VM name) +live in SITES below -- dc0 per the 2026-07-16 ratification, dc1 per the D-124 +AMENDMENT 2026-07-21 (vr1-dc1 addressing RATIFIED). So this tool registers exactly: - 1. the transit prefix role=transit, scope=dcim.site:vr1-dc0 (the /30 or /31) - 2. the rack ip-address /22 in metal-admin, D-120 static band .2-.49 + 1. the transit prefix role=transit, scope=dcim.site:<--site> (the /30 or /31) + 2. the rack ip-address /22 in that DC's metal-admin, D-120 static band .2-.49 Both LITERALS ARE OPERATOR INPUTS (--transit-cidr / --rack-ip): D-124, like D-115/ D-117, rules the SCHEME and leaves the actual CIDR + IP to office1-netbox (the apex; @@ -47,18 +51,21 @@ BEFORE this runs; a missing/collision container makes the tool DIE (fail-loud is the backstop for the candidate block being picked against a stale draft). This gates ACCEPTANCE only -- NetBox auto-nests by CIDR, we set no parent. - * transit prefix SCOPE = dcim.site:vr1-dc0 (RULED 2026-07-16: "mirror the edge" -- + * transit prefix SCOPE = dcim.site:<--site> (RULED 2026-07-16: "mirror the edge" -- site-scoped like the D-115 DC-edge /24s, 172.30.2.0/24 -> vr1-dc0). - * The rack IP band (metal-admin 10.12.8.0/22, static .2-.49, gw .1) is enforced + * The rack IP band (the DC's metal-admin /22, static .2-.49, gw .1) is enforced ARITHMETICALLY (D-120/D-124), NOT via a prefix-object lookup: metal-admin is not registered as a /22 prefix in the apex draft, and the band is a convention, not a container. This asymmetry with the transit check is intentional. -Usage (on office1-netbox / through a tunnel, with the sandbox token). D-124 values -operator-ratified 2026-07-16 (transit supernet 172.31.0.0/24, transit /30 -172.31.0.0/30, rack IP 10.12.8.2) -- confirm free against the LIVE apex first: +Usage (on office1-netbox / through a tunnel, with the sandbox token). Ratified values -- +confirm free against the LIVE apex first: + dc0 (ratified 2026-07-16: transit /30 172.31.0.0/30, rack IP 10.12.8.2; APPLIED): NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN= \ - python3 netbox/dc-rack-mgmt-import.py --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.2 + python3 netbox/dc-rack-mgmt-import.py --site vr1-dc0 --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.2 + dc1 (D-124 AMENDMENT ratified 2026-07-21: transit /30 172.31.0.4/30, rack IP 10.12.68.2): + NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN= \ + python3 netbox/dc-rack-mgmt-import.py --site vr1-dc1 --transit-cidr 172.31.0.4/30 --rack-ip 10.12.68.2 ... same, add --commit, to write. """ import argparse @@ -78,18 +85,32 @@ ROLE_SLUG = "transit" # the transit's OWN role (D-124); a precondition here CONTAINER = "172.31.0.0/24" # D-124 dedicated transit supernet (operator-pinned 2026-07-16); transit /30 must be subnet_of this (precondition) -SITE_SLUG = "vr1-dc0" # the transit prefix is site-scoped here (mirrors the D-115 edge /24s) STATUS = "active" # a real segment on the wire -# metal-admin band arithmetic (D-052/D-124). The rack's metal-admin IP lands in the -# D-120 static band .2-.49 of this /22, and NOT the .1 gateway. -METAL_ADMIN = "10.12.8.0/22" +# D-120 static band arithmetic: the rack's metal-admin IP lands in .2-.49 of the DC's +# metal-admin /22, and NOT the .1 gateway. Band offsets are convention-global; the /22 +# itself is per-site (SITES below). STATIC_BAND_LOW = 2 # .2 (first static site-services address, D-120) STATIC_BAND_HIGH = 49 # .49 (last static site-services address, D-120) -RACK_DNS = "vvr1-dc0" # the rack-controller VM (D-124 cloudinit-vm on the two legs) -TRANSIT_DESC = "office1<->dc0 management transit -- region<->rack MAAS control path (D-124 Scheme A)" -RACK_DESC = "vr1-dc0 MAAS rack controller (vvr1-dc0) metal-admin static IP (D-124; D-120 static band)" +# Per-site facts. The transit prefix is scoped dcim.site: (mirrors the D-115 edge +# /24s). dc0: ratified 2026-07-16 (applied to the apex 2026-07-16/17). dc1: D-124 +# AMENDMENT ratified 2026-07-21 -- metal-admin 10.12.68.0/22 within the vr1-dc1 +# supernet 10.12.64.0/19 (D-115), rack VM vvr1-dc1. +SITES = { + "vr1-dc0": { + "metal_admin": "10.12.8.0/22", + "rack_dns": "vvr1-dc0", + "transit_desc": "office1<->dc0 management transit -- region<->rack MAAS control path (D-124 Scheme A)", + "rack_desc": "vr1-dc0 MAAS rack controller (vvr1-dc0) metal-admin static IP (D-124; D-120 static band)", + }, + "vr1-dc1": { + "metal_admin": "10.12.68.0/22", + "rack_dns": "vvr1-dc1", + "transit_desc": "office1<->dc1 management transit -- region<->rack MAAS control path (D-124 Scheme A)", + "rack_desc": "vr1-dc1 MAAS rack controller (vvr1-dc1) metal-admin static IP (D-124 amendment 2026-07-21; D-120 static band)", + }, +} def die(msg: str): @@ -146,21 +167,21 @@ except ValueError as exc: die(f"--transit-cidr {raw!r} is not a valid network (host bits set?): {exc}") if net.version != 4: - die(f"--transit-cidr {raw} must be IPv4 (the office1<->dc0 transit is v4).") + die(f"--transit-cidr {raw} must be IPv4 (the office1<->DC transits are v4).") if net.prefixlen not in (30, 31): die(f"--transit-cidr {raw} must be a /30 or /31 point-to-point (got /{net.prefixlen}).") return net def parse_rack_ip(raw: str) -> ipaddress.IPv4Address: - """Accept a bare host (10.12.8.X) or a masked form; if masked, the mask MUST be the + """Accept a bare host or a masked form; if masked, the mask MUST be the metal-admin /22. Returns the host address; band placement is checked in preflight.""" try: if "/" in raw: iface = ipaddress.ip_interface(raw) if iface.network.prefixlen != 22: die(f"--rack-ip {raw} carries /{iface.network.prefixlen}; metal-admin is a /22. " - f"Pass a bare host (10.12.8.X) or the /22 form.") + f"Pass a bare host or the /22 form.") host = iface.ip else: host = ipaddress.ip_address(raw) @@ -173,12 +194,16 @@ def main() -> int: ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--site", choices=sorted(SITES), default=os.environ.get("RACK_SITE"), + help="REQUIRED. Which DC's transit + rack IP to register (env: RACK_SITE). " + "Explicit -- never inferred -- so one DC's values cannot land scoped " + "to another site.") ap.add_argument("--transit-cidr", default=os.environ.get("TRANSIT_CIDR"), - help="REQUIRED. The /30 or /31 point-to-point for the office1<->dc0 " + help="REQUIRED. The /30 or /31 point-to-point for the office1<->DC " "transit leg (operator-supplied, NetBox-assigned; env: TRANSIT_CIDR).") ap.add_argument("--rack-ip", default=os.environ.get("RACK_IP"), - help="REQUIRED. The rack's metal-admin static IP within 10.12.8.0/22, " - "static band .2-.49 (operator-supplied; env: RACK_IP).") + help="REQUIRED. The rack's metal-admin static IP within the site's " + "metal-admin /22, static band .2-.49 (operator-supplied; env: RACK_IP).") ap.add_argument("--commit", action="store_true", help="WRITE. Default is a DRY RUN that writes nothing.") ap.add_argument("--yes-write-upstream", action="store_true", @@ -192,6 +217,18 @@ # Inputs are args-or-env, so they cannot be argparse required=True (that would break # the env fallback). Hand-roll the missing checks, each with its own die. + if not args.site: + die("--site (or RACK_SITE) is REQUIRED -- the target DC is never inferred " + "(one DC's transit/rack values must not land scoped to another site).") + if args.site not in SITES: + # argparse `choices` does not validate a value arriving via the env-var default. + die(f"--site {args.site!r} is not a known DC (expected one of: {', '.join(sorted(SITES))}).") + site_cfg = SITES[args.site] + site_slug = args.site + metal_admin = site_cfg["metal_admin"] + rack_dns = site_cfg["rack_dns"] + transit_desc = site_cfg["transit_desc"] + rack_desc = site_cfg["rack_desc"] if not args.transit_cidr: die("--transit-cidr (or TRANSIT_CIDR) is REQUIRED -- the NetBox-assigned transit " "/30 or /31. No literal is invented in-repo (D-124).") @@ -216,24 +253,27 @@ transit = parse_transit_cidr(args.transit_cidr) container = ipaddress.ip_network(CONTAINER) if not transit.subnet_of(container): - die(f"transit {transit} is outside the container {CONTAINER} (Cloud) -- refusing to " - f"place it. If the transit is carved elsewhere, that is an operator/D-124 call.") + die(f"transit {transit} is outside the container {CONTAINER} (transit supernet) -- " + f"refusing to place it. If the transit is carved elsewhere, that is an " + f"operator/D-124 call.") - # (b) Rack IP band: within metal-admin /22, in the D-120 static band .2-.49, NOT the - # .1 gateway (a distinct die so the ".1 rejected" property is unambiguous). + # (b) Rack IP band: within the site's metal-admin /22, in the D-120 static band + # .2-.49, NOT the .1 gateway (a distinct die so the ".1 rejected" property is + # unambiguous). rack = parse_rack_ip(args.rack_ip) - metal = ipaddress.ip_network(METAL_ADMIN) + metal = ipaddress.ip_network(metal_admin) if rack not in metal: - die(f"rack IP {rack} is outside metal-admin {METAL_ADMIN} -- refusing to place it.") + die(f"rack IP {rack} is outside metal-admin {metal_admin} ({site_slug}) -- refusing " + f"to place it.") gateway = metal.network_address + 1 band_low = metal.network_address + STATIC_BAND_LOW band_high = metal.network_address + STATIC_BAND_HIGH if rack == gateway: - die(f"rack IP {rack} is the .1 GATEWAY of {METAL_ADMIN} -- refusing (D-120: .1 is the " + die(f"rack IP {rack} is the .1 GATEWAY of {metal_admin} -- refusing (D-120: .1 is the " f"site gateway, not a static-service address).") if not (band_low <= rack <= band_high): die(f"rack IP {rack} is outside the D-120 static band {band_low}-{band_high} of " - f"{METAL_ADMIN} -- the rack's metal-admin IP must land in .2-.49 (not the dynamic/" + f"{metal_admin} -- the rack's metal-admin IP must land in .2-.49 (not the dynamic/" f"node bands).") rack_addr = f"{rack}/{metal.prefixlen}" # NetBox stores the host WITH the plane mask @@ -247,12 +287,12 @@ f"place the transit prefix with no role.") if nb.one("ipam/prefixes", prefix=CONTAINER) is None: - die(f"container {CONTAINER} (Cloud) absent in the apex -- refusing to place the transit " - f"in an unallocated supernet.") + die(f"container {CONTAINER} (transit supernet) absent in the apex -- refusing to place " + f"the transit in an unallocated supernet.") - site = nb.one("dcim/sites", slug=SITE_SLUG) + site = nb.one("dcim/sites", slug=site_slug) if site is None: - die(f"site '{SITE_SLUG}' absent -- cannot scope the transit prefix. The DC site is a " + die(f"site '{site_slug}' absent -- cannot scope the transit prefix. The DC site is a " f"precondition (it already exists in the apex); this tool never creates it.") # (d) Present-state (idempotency) -- resolved up front, part of the whole-plan gate. @@ -264,19 +304,19 @@ # ---- CREATE (transit prefix, then rack ip-address) -- only reached once the WHOLE # plan above is viable, so this loop cannot half-write the apex. ---- - print("\nTransit prefix (office1<->dc0 mesh leg):") + print(f"\nTransit prefix (office1<->{site_slug} mesh leg):") if transit_present: print(f" EXISTS {transit_str}") existing += 1 elif not args.commit: print(f" [dry-run] would CREATE {transit_str} role={ROLE_SLUG} " - f"scope=dcim.site:{SITE_SLUG}") + f"scope=dcim.site:{site_slug}") created += 1 else: payload = {"prefix": transit_str, "role": role["id"], "status": STATUS, - "description": TRANSIT_DESC, "scope_type": "dcim.site", "scope_id": site["id"]} + "description": transit_desc, "scope_type": "dcim.site", "scope_id": site["id"]} o = nb.create("ipam/prefixes", payload) - print(f" CREATED {transit_str} (id={o['id']}) role={ROLE_SLUG} scope={SITE_SLUG}") + print(f" CREATED {transit_str} (id={o['id']}) role={ROLE_SLUG} scope={site_slug}") created += 1 print("\nRack metal-admin static IP:") @@ -284,13 +324,13 @@ print(f" EXISTS {rack_addr}") existing += 1 elif not args.commit: - print(f" [dry-run] would CREATE {rack_addr} dns={RACK_DNS} (metal-admin static band)") + print(f" [dry-run] would CREATE {rack_addr} dns={rack_dns} (metal-admin static band)") created += 1 else: payload = {"address": rack_addr, "status": STATUS, - "dns_name": RACK_DNS, "description": RACK_DESC} + "dns_name": rack_dns, "description": rack_desc} o = nb.create("ipam/ip-addresses", payload) - print(f" CREATED {rack_addr} (id={o['id']}) dns={RACK_DNS}") + print(f" CREATED {rack_addr} (id={o['id']}) dns={rack_dns}") created += 1 verb = "would create" if not args.commit else "created" diff --git a/opentofu/main.tf b/opentofu/main.tf index ec828c0..80efbee 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -38,22 +38,13 @@ target_path = var.vr1_dc0_pool_path } -# ---- vr1-dc1 (VR1's SECOND DC): PLANES deferred by SEQUENCING, not a missing -# literal. Its supernet IS assigned -- D-115 moved it to 10.12.64.0/19 (ADOPTED, -# imported to office1-netbox), superseding the old "wait for NetBox" premise. It -# stays out of scope until vr1-dc0 completes (operator ruling 2026-07-10, Option B: -# vr1-dc0-first). When vr1-dc1 is started, add a `vr1_dc1_planes` variable (same -# shape as `vr1_dc0_planes`, CIDRs derived per D-115) and uncomment this block. -# Do NOT uncomment it now. The storage pool + mesh legs are NOT address-dependent, -# so they ARE wired now. -# -# module "vr1_dc1_planes" { -# source = "./modules/dc-planes" -# dc_name = "vr1-dc1" -# domain_suffix = var.domain_suffix -# mtu = var.underlay_mtu -# planes = var.vr1_dc1_planes -# } +# ---- vr1-dc1 (VR1's SECOND DC): STARTED 2026-07-21 (gate G12; the D-124 +# AMENDMENT ratified its addressing, apex confirm-free captured). Under D-123 +# Model B its six planes are NOT created here: like dc0's, they are created by +# the INNER root (opentofu/vr1-dc1-substrate/) on vvr1-dc1's own libvirt. The +# committed values-of-record live in var.vr1_dc1_planes (variables.tf); the old +# pre-Model-B `module "vr1_dc1_planes"` skeleton that stood here is retired -- +# do not re-add an outer planes module for any DC. ---- module "vr1_dc1_storage" { source = "./modules/dc-storage-pool" @@ -195,6 +186,18 @@ network_names = [ module.office1_network.network_name, # NIC1 enp1s0 -> lan (office1-local, DHCP/Kea) module.mesh_vr1_dc0_office1.network_name, # NIC2 -> transit (region end, static .1/30) + module.mesh_vr1_dc1_office1.network_name, # NIC3 -> dc1 transit (region end, static .5/30, D-124 amendment) + ] + # MAC pins (interface_macs, ported to cloudinit-vm 2026-07-22): NIC1/NIC2 = + # MEASURED live values (virsh domiflist voffice1) -- NIC1's MAC keys the Kea + # 10.10.0.20 reservation, the 2026-07-20 regen incident class. NIC3 = NEW, + # pre-pinned from first apply (dc1 standup DoD invariant) in the dc1 + # locally-administered scheme (52:54:01:d1:NN:PP; fe = region-side host, + # outside the node NN range). + interface_macs = [ + "52:54:00:89:e1:19", # NIC1 office1-local (Kea reservation key) + "52:54:00:99:04:01", # NIC2 dc0 mesh transit + "52:54:01:d1:fe:01", # NIC3 dc1 mesh transit (pre-pinned) ] expose_nested_virt = true @@ -376,6 +379,18 @@ # mtu defaults to 1500 (the ISP-uplink domain; never jumbo). } +# vr1-dc1's uplink -- same D-125 bridge-in shape as dc0's above, same reasoning. +# The /24 is D-115's vr1-dc1 sim-ISP edge WAN, registered in the apex scoped +# vr1-dc1 (verified 2026-07-21, docs/audit/dc1-apex-confirm-20260721.txt) -- a +# RULED literal, not a HELD value. The D-125 egress isolation gate re-runs at +# dc1 standup (per-DC deploy-time gate), though bridge-in itself is PROVEN on dc0. +module "vr1_dc1_uplink" { + source = "./modules/site-wan" + network_name = "vr1-dc1-uplink" + cidr = "172.30.3.0/24" # D-115 vr1-dc1 sim-ISP edge WAN (D-124 amendment 2026-07-21) + # mtu defaults to 1500 (the ISP-uplink domain; never jumbo). +} + # ===================================================================== # D-123 MODEL B / D-124 AMENDMENT: vvr1-dc0 -- the vr1-dc0 SITE CONTAINMENT VM. Under # Model B it is BOTH the MAAS rack controller (enrolled to the Office1 region) AND the @@ -498,3 +513,107 @@ forward-delay: 0 EOT } + +# ===================================================================== +# G12 (2026-07-21): vvr1-dc1 -- the vr1-dc1 SITE CONTAINMENT VM. The dc0 block +# above is the PATTERN OF RECORD; this mirrors it exactly with the D-124-AMENDMENT +# addressing (transit 172.31.0.4/30 via tfvars after the apex --commit; uplink +# 172.30.3.0/24). Same Model B role: MAAS rack controller (enrolled to the Office1 +# region) AND nested libvirt host for dc1's 9-node fleet, created by the inner +# root opentofu/vr1-dc1-substrate/. Sizing identical (D-121 Option C is per-DC; +# the 2-DC whole-host budget was validated when dc0 was sized). D-126 per-env key: +# the DEDICATED dc1 key, not office1's or dc0's. NOT YET APPLIED -- the apply, the +# bootstrap gate (site-headend-install.sh node-host mode) and the inner-root apply +# are gated dc1-standup steps. Standup definition-of-done additionally carries +# (D-124 amendment): scripts/dc-rack-net.sh site row with legs MEASURED at standup +# + install; the D-131 forwarder (10.12.68.3); MAC pinning from the FIRST apply +# (the inner root pre-assigns; see vr1-dc1-substrate/main.tf); per-machine virsh +# power via maas-node-power.sh (pods refuted, D-103/D-123 amendments). +# ===================================================================== +module "vvr1_dc1" { + source = "./modules/cloudinit-vm" + vm_name = "vvr1-dc1" + autostart = false # D-127: DC containment VM -- MANUAL (gated bring-up), never on host boot + vcpu = var.vvr1_dc1_vcpu # same Option-C fleet derivation as dc0 + memory_mib = var.vvr1_dc1_memory_mib # derived 416 GiB (384 node + 32 overhead) + disk_size_bytes = var.vvr1_dc1_disk_bytes # derived ~3000 GiB (holds the inner libvirt pool) + pool_name = module.vr1_dc1_storage.pool_name + base_volume_path = module.ubuntu_noble_base.path + expose_nested_virt = true # D-123 Model B: LOAD-BEARING -- inner node VMs need svm passthrough + + # TWO outer legs, dc0 pattern: + # NIC1 (enp1s0 -> mgmt) = office1<->dc1 transit (D-124 Scheme A; its OWN mesh leg, + # NOT dc0's). SEC-010/--transit-if key on THIS interface. + # NIC2 (enp2s0 -> uplink) = D-125 bridge-in: IP-less port of br-vr1-dc1-wan on the + # vcloud ISP NAT (vr1-dc1-uplink). + network_names = [ + module.mesh_vr1_dc1_office1.network_name, # NIC1 enp1s0 -> mgmt (transit; SEC-010 keys here) + module.vr1_dc1_uplink.network_name, # NIC2 enp2s0 -> uplink (IP-less port of br-vr1-dc1-wan) + ] + # MAC pin-adoption (2026-07-22, step-A first-apply values MEASURED via + # virsh domiflist -- the dc1 standup DoD invariant: pinned before any + # in-place update can regenerate them). + interface_macs = [ + "52:54:00:11:39:ce", # NIC1 dc1 mesh transit + "52:54:00:3c:a6:a8", # NIC2 dc1 uplink (br-vr1-dc1-wan port) + ] + + user_data = <<-EOT + #cloud-config + hostname: vvr1-dc1 + fqdn: vvr1-dc1.${var.domain_suffix} + manage_etc_hosts: true + users: + - name: jessea123 + groups: [adm, sudo] + shell: /bin/bash + sudo: "ALL=(ALL) NOPASSWD:ALL" + ssh_authorized_keys: + # D-126 per-env-key: vvr1-dc1 authorizes the DEDICATED dc1 key ONLY + # (~/vr1-dc1-creds/, D-124 amendment 2026-07-21). Inner root's qemu+ssh matches. + - ${trimspace(file(var.vr1_dc1_ssh_pubkey_path))} + package_update: true + packages: + - qemu-guest-agent + runcmd: + - [systemctl, enable, --now, qemu-guest-agent] + EOT + + meta_data = <<-EOT + instance-id: vvr1-dc1-d124a + local-hostname: vvr1-dc1 + EOT + + # STATIC on the transit leg; uplink is an IP-less bridge port (dc0 pattern, + # comments there are the authority). INTERFACE NAMING: enp1s0/enp2s0 are the + # names dc0 MEASURED on this exact q35 2-NIC cloudinit-vm shape (kernel rename + # log, 2026-07-19) -- expected to hold here, CONFIRM on vvr1-dc1's first boot + # before relying on it. set-name deliberately absent (dc0's measured netplan + # pitfall). The region route rides the transit toward Office1 (10.10.0.0/22). + # br_netfilter CONSTRAINT (D-125): SEC-010's FORWARD-drop stays scoped to + # enp1s0 -- never a global drop (would kill bridged WAN frames). + network_config = <<-EOT + version: 2 + ethernets: + transit: + match: + name: "enp1s0" + addresses: ["${var.vr1_dc1_rack_transit_ip}/${var.vr1_dc1_rack_transit_prefix}"] + routes: + - to: "10.10.0.0/22" + via: "${var.vr1_dc1_rack_transit_peer_ip}" + uplink: + match: + name: "enp2s0" + dhcp4: false + dhcp6: false + bridges: + br-vr1-dc1-wan: + interfaces: [uplink] + dhcp4: false + dhcp6: false + parameters: + stp: false + forward-delay: 0 + EOT +} diff --git a/opentofu/modules/cloudinit-vm/main.tf b/opentofu/modules/cloudinit-vm/main.tf index 6f6d907..6aa8e49 100644 --- a/opentofu/modules/cloudinit-vm/main.tf +++ b/opentofu/modules/cloudinit-vm/main.tf @@ -154,10 +154,15 @@ ] interfaces = [ - for net_name in var.network_names : { + for i, net_name in var.network_names : { model = { type = "virtio" } + # MAC pinning (see variables.tf interface_macs): an unpinned MAC is + # provider-owned and an "in-place" apply can regenerate it (measured + # 2026-07-20, voffice1 Kea incident). Same wiring as modules/node-vm, + # verified against dmacvicar/libvirt 0.9.8 schema 2026-07-21. + mac = length(var.interface_macs) > 0 ? { address = var.interface_macs[i] } : null source = { network = { network = net_name diff --git a/opentofu/modules/cloudinit-vm/variables.tf b/opentofu/modules/cloudinit-vm/variables.tf index dfcb498..fb4933d 100644 --- a/opentofu/modules/cloudinit-vm/variables.tf +++ b/opentofu/modules/cloudinit-vm/variables.tf @@ -39,6 +39,30 @@ type = list(string) } +variable "interface_macs" { + description = <<-EOT + Optional ordered list of MAC addresses, one per network_names entry (same + order), PINNING each NIC's MAC in config. Empty (the default) lets libvirt + generate MACs. Ported from modules/node-vm (same incident class): an + unpinned MAC is provider-owned and an "in-place" domain update can + silently REGENERATE it -- measured 2026-07-20 on voffice1, where the + transit-NIC add regenerated the office1-local NIC's MAC and broke its Kea + reservation. Pin any VM whose address is leased/reserved by MAC. + EOT + type = list(string) + default = [] + + validation { + condition = length(var.interface_macs) == 0 || length(var.interface_macs) == length(var.network_names) + error_message = "interface_macs must be empty or exactly one MAC per network_names entry (partial pinning would silently leave some NICs drift-prone)." + } + + validation { + condition = alltrue([for m in var.interface_macs : can(regex("^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$", m))]) + error_message = "Each interface_macs entry must be a colon-separated 6-byte hex MAC (e.g. 52:54:00:ab:cd:ef)." + } +} + variable "user_data" { description = <<-EOT Full #cloud-config YAML content (a real, chosen configuration -- hostname, diff --git a/opentofu/variables.tf b/opentofu/variables.tf index de04094..6c92524 100644 --- a/opentofu/variables.tf +++ b/opentofu/variables.tf @@ -58,13 +58,33 @@ } } -# vr1_dc1_planes has NO variable/default here yet, deliberately: VR1 DC1's v4 -# supernet is a D-101 OPEN SUB-ITEM ("exact prefixes assigned in NetBox, -# non-colliding with..."), not yet wired here. Do not add a guessed default -- -# once NetBox assigns it, add a vr1_dc1_planes variable shaped exactly like -# vr1_dc0_planes above and a matching module "vr1_dc1_planes" block in main.tf -# (see the commented skeleton there). -# +variable "vr1_dc1_planes" { + description = <<-EOT + vr1-dc1's six planes (D-124 AMENDMENT 2026-07-21, addressing RATIFIED; + supernet 10.12.64.0/19 per D-115). NOT a copy of dc0's layout: the six + /22s are CONTIGUOUS keeping dc0's role ORDER (dc0's 4/8/12/16/32/36 + offsets cannot fit inside a /19 -- deliberate, documented divergence; + 10.12.88+92.0/22 spare). These values are apex-VERIFIED, not invented: + all six exist in office1-netbox scoped dcim.site:vr1-dc1 with matching + roles (docs/audit/dc1-apex-confirm-20260721.txt). Under D-123 Model B + the planes are CREATED by the INNER root (opentofu/vr1-dc1-substrate/); + this default is the committed values-of-record the inner tfvars copies + (same relationship as vr1_dc0_planes above / lib-net.sh's vr1-dc1 arm -- + change one, change all in the SAME commit). + EOT + type = map(object({ + cidr = string + })) + default = { + provider-public = { cidr = "10.12.64.0/22" } + metal-admin = { cidr = "10.12.68.0/22" } + metal-internal = { cidr = "10.12.72.0/22" } + data-tenant = { cidr = "10.12.76.0/22" } + storage = { cidr = "10.12.80.0/22" } + replication = { cidr = "10.12.84.0/22" } + } +} + # D-119 NAMING: vr1-dc0 is VR1's FIRST DC, vr1-dc1 its SECOND -- matching the # NetBox apex slugs exactly. Never write a bare "dcN" for a VR1 DC here: "dc0" # means VR0's LIVE testcloud in scripts/lib-net.sh, and that collision is what @@ -132,6 +152,66 @@ default = 3221225472000 } +variable "vr1_dc1_ssh_pubkey_path" { + description = <<-EOT + Path to the PUBLIC half of the DEDICATED vr1-dc1 service SSH key (D-126 per-env-key + ruling, option a; dc1 row ratified in the D-124 amendment 2026-07-21: + ~/vr1-dc1-creds/vr1-dc1_svc_ed25519). Injected into vvr1-dc1's cloud-init; the inner + root's qemu+ssh authenticates with the matching private half (jumphost-local, never + read). Read with file() at plan time -- the .pub must exist. Per-env blast-radius + isolation from Office1 AND from dc0. + EOT + type = string +} + +# ---- vvr1-dc1 sizing: SAME derivation as vvr1-dc0 (D-121 Option C is per-DC -- +# identical 3 control + 2 compute + 4 storage fleet), via +# scripts/dc-dc-whole-host-budget.py: 104 node vCPU / 384 GiB / ~2850 GiB thin + +# containment overhead. The whole-host 2-DC total (870/1024 GiB, 85%) was budgeted +# FIT when dc0's sizing was derived -- dc1 IS the second half of that budget. ---- +variable "vvr1_dc1_vcpu" { + description = "vvr1-dc1 vCPU (Model B: hosts one DC's 9 node VMs). Derived: 104 node + 4 overhead." + type = number + default = 108 +} + +variable "vvr1_dc1_memory_mib" { + description = "vvr1-dc1 RAM in MiB (Model B). Derived: 384 GiB node fleet + 32 GiB overhead = 416 GiB." + type = number + default = 425984 +} + +variable "vvr1_dc1_disk_bytes" { + description = "vvr1-dc1 boot disk in bytes (Model B: holds the inner libvirt pool). Derived: ~2850 GiB node disks (thin) + ~150 GiB OS = ~3000 GiB." + type = number + default = 3221225472000 +} + +# ---- D-124 AMENDMENT (2026-07-21): vr1-dc1 MAAS rack controller (vvr1-dc1) +# addressing. NO defaults -- ratified values enter via tfvars ONLY AFTER the +# operator-gated apex --commit registers them in office1-netbox (dc0 precedent: +# rack_transit_ip=172.31.0.6, prefix=30, peer=172.31.0.5, metal_admin=10.12.68.2). +# Do NOT invent or pre-fill them here (hard rule 2). ---- +variable "vr1_dc1_rack_metal_admin_ip" { + description = "vvr1-dc1 rack static IP in metal-admin (10.12.68.0/22), in the .2-.49 static band, not the .1 gateway. NetBox-assigned (D-124 amendment)." + type = string +} + +variable "vr1_dc1_rack_transit_ip" { + description = "vvr1-dc1 rack IP on the office1<->dc1 mesh transit (D-124 Scheme A, next /30 in 172.31.0.0/24). NetBox-assigned." + type = string +} + +variable "vr1_dc1_rack_transit_prefix" { + description = "office1<->dc1 transit link prefix length (30 or 31, D-124 Scheme A)." + type = number +} + +variable "vr1_dc1_rack_transit_peer_ip" { + description = "Office1 region's IP on the office1<->dc1 transit link -- the rack's next-hop toward the MAAS region (10.10.0.0/22). NetBox-assigned (D-124 amendment)." + type = string +} + # ---- D-124: vr1-dc0 MAAS rack controller (vvr1-dc0) addressing. NO defaults -- # these come from office1-netbox (the IPAM apex) via the D-124 importer, then # into a tfvars entry. Do NOT invent them here (hard rule 2). The rack straddles diff --git a/opentofu/vr1-dc1-substrate/main.tf b/opentofu/vr1-dc1-substrate/main.tf new file mode 100644 index 0000000..f3a8c68 --- /dev/null +++ b/opentofu/vr1-dc1-substrate/main.tf @@ -0,0 +1,151 @@ +# ===================================================================== +# vr1-dc1 INNER SUBSTRATE (G12, authored 2026-07-21) -- D-123 MODEL B, created +# inside vvr1-dc1. MIRROR of opentofu/vr1-dc0-substrate/main.tf (the pattern of +# record: its comments carry the measured traps -- provider dial, known_hosts, +# upload-from-executing-host -- and apply here unchanged). Run this root AFTER: +# (1) the outer opentofu/ root has booted + sized vvr1-dc1, and (2) the bootstrap +# gate (site-headend-install.sh node-host mode) has installed libvirtd + the +# inner pool + kvm nested=1 + the OPNsense base image on vvr1-dc1. +# ===================================================================== + +provider "libvirt" { + # qemu+ssh to vvr1-dc1 over the office1<->dc1 transit. Key auth + known_hosts + # verified out of band (NO no_verify). The IP is MEASURED after the outer apply + # (hard rule 2). keyfile + sshauth are REQUIRED URI params (dc0, measured + # 2026-07-20: the provider's Go ssh transport ignores ~/.ssh/config and default + # identities). known_hosts: ALL key types must be present (dc0's measured + # "knownhosts: key mismatch" trap; seed via ssh-keyscan, cross-checked). + uri = "qemu+ssh://${var.vvr1_dc1_ssh_user}@${var.vvr1_dc1_transit_ip}/system?keyfile=${var.vvr1_dc1_ssh_keyfile}&sshauth=privkey" +} + +# Inner storage pool (backs the node + edge disks) -- created inside vvr1-dc1 at +# the path the bootstrap step provisioned. +module "inner_storage" { + source = "../modules/dc-storage-pool" + dc_name = "vr1-dc1-inner" + target_path = var.inner_pool_path +} + +# The six vr1-dc1 planes (D-124 amendment 2026-07-21 layout -- contiguous /22s in +# 10.12.64.0/19, dc0's role ORDER kept) -- isolated-L2 networks INSIDE vvr1-dc1. +# metal-admin is where the rack serves PXE/DHCP to the inner nodes. +module "vr1_dc1_planes" { + source = "../modules/dc-planes" + dc_name = "vr1-dc1" + domain_suffix = var.domain_suffix + mtu = var.underlay_mtu + planes = var.vr1_dc1_planes +} + +# D-125 bridge-in (dc0 pattern, PROVEN there end to end): the DC WAN segment is a +# BRIDGE onto vvr1-dc1's br-vr1-dc1-wan (the IP-less uplink NIC on the vcloud ISP +# NAT `vr1-dc1-uplink` = 172.30.3.0/24, declared in the outer root's vvr1-dc1 +# netplan). ONE NAT, at vcloud. host_bridge MUST match the netplan bridge name in +# opentofu/main.tf. The D-125 egress isolation gate re-runs at dc1 standup. +module "vr1_dc1_wan" { + source = "../modules/wan-bridge" + network_name = "vr1-dc1-wan" + host_bridge = "br-vr1-dc1-wan" + # mtu defaults to 1500 (ISP-uplink domain; NOT the jumbo planes/mesh). +} + +# D-122: the DC edge -- 2-NIC (WAN + LAN), Office1 pattern (2048/2/nano). LAN = +# provider-public (external gateway, D-100; ruled gw 10.12.64.1 per the D-124 +# amendment); WAN = the dedicated uplink above (static .2 on 172.30.3.0/24, set +# via the D-113 API path at the standup addressing step, not here). +module "vr1_dc1_opnsense" { + source = "../modules/opnsense-edge" + vm_name = "vr1-dc1-opnsense" + autostart = true # D-127: the DC edge comes up with its containment VM (routing for the site) + memory_mib = 2048 + vcpu = 2 + pool_name = module.inner_storage.pool_name + base_volume_path = var.opnsense_base_path + lan_network_name = module.vr1_dc1_planes.network_names["provider-public"] + wan_network_name = module.vr1_dc1_wan.network_name + # D-129: NEW DC edges boot WITH the qga channel so they never need the retrofit + # (operator-ruled 2026-07-20). + expose_qga_channel = true +} + +# D-121 Option C layout (per-DC): 3 control + 2 compute + 4 storage = 9 nodes. +# Six NICs each (one per plane); metal-admin FIRST = PXE/boot plane (D-052 default +# binding). +locals { + vr1_dc1_node_nics = [ + module.vr1_dc1_planes.network_names["metal-admin"], # PXE / boot fabric first + module.vr1_dc1_planes.network_names["provider-public"], + module.vr1_dc1_planes.network_names["metal-internal"], + module.vr1_dc1_planes.network_names["data-tenant"], + module.vr1_dc1_planes.network_names["storage"], + module.vr1_dc1_planes.network_names["replication"], + ] + + # macs: PRE-ASSIGNED at authoring (2026-07-21) so the fleet is pinned from the + # FIRST apply -- the D-124-amendment standup invariant, and the direct lesson of + # dc0's 2026-07-20 incident (unpinned MACs regenerated by an in-place apply + # stranded the whole fleet in MAAS; dc0 had to measure-and-pin after the fact). + # Scheme, deterministic and collision-free by construction: + # 52:54:01:d1:NN:PP + # ^^^^^^^^ locally-administered unicast (bit 0x02 of octet 1), one octet away + # from libvirt's auto-generation space 52:54:00:* -- an auto MAC can + # never collide with these. + # ^^ d1 tags vr1-DC1 (dc0's pins are measured 52:54:00 values, not this + # scheme; future DCs take their own tag octet). + # NN = node ordinal 01..09 in the map order below; + # PP = NIC ordinal 01..06 in vr1_dc1_node_nics order (metal-admin + # first). These are IDENTITY values, not tunables -- change one and + # MAAS no longer knows the node. + vr1_dc1_nodes = { + "vr1-dc1-control-01" = { vcpu = 16, mem = 65536, disk_gib = 150, macs = [ + "52:54:01:d1:01:01", "52:54:01:d1:01:02", "52:54:01:d1:01:03", + "52:54:01:d1:01:04", "52:54:01:d1:01:05", "52:54:01:d1:01:06", + ] } + "vr1-dc1-control-02" = { vcpu = 16, mem = 65536, disk_gib = 150, macs = [ + "52:54:01:d1:02:01", "52:54:01:d1:02:02", "52:54:01:d1:02:03", + "52:54:01:d1:02:04", "52:54:01:d1:02:05", "52:54:01:d1:02:06", + ] } + "vr1-dc1-control-03" = { vcpu = 16, mem = 65536, disk_gib = 150, macs = [ + "52:54:01:d1:03:01", "52:54:01:d1:03:02", "52:54:01:d1:03:03", + "52:54:01:d1:03:04", "52:54:01:d1:03:05", "52:54:01:d1:03:06", + ] } + "vr1-dc1-compute-01" = { vcpu = 12, mem = 49152, disk_gib = 100, macs = [ + "52:54:01:d1:04:01", "52:54:01:d1:04:02", "52:54:01:d1:04:03", + "52:54:01:d1:04:04", "52:54:01:d1:04:05", "52:54:01:d1:04:06", + ] } + "vr1-dc1-compute-02" = { vcpu = 12, mem = 49152, disk_gib = 100, macs = [ + "52:54:01:d1:05:01", "52:54:01:d1:05:02", "52:54:01:d1:05:03", + "52:54:01:d1:05:04", "52:54:01:d1:05:05", "52:54:01:d1:05:06", + ] } + "vr1-dc1-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "52:54:01:d1:06:01", "52:54:01:d1:06:02", "52:54:01:d1:06:03", + "52:54:01:d1:06:04", "52:54:01:d1:06:05", "52:54:01:d1:06:06", + ] } + "vr1-dc1-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "52:54:01:d1:07:01", "52:54:01:d1:07:02", "52:54:01:d1:07:03", + "52:54:01:d1:07:04", "52:54:01:d1:07:05", "52:54:01:d1:07:06", + ] } + "vr1-dc1-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ + "52:54:01:d1:08:01", "52:54:01:d1:08:02", "52:54:01:d1:08:03", + "52:54:01:d1:08:04", "52:54:01:d1:08:05", "52:54:01:d1:08:06", + ] } + "vr1-dc1-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550, macs = [ # Option C: 4th OSD host + "52:54:01:d1:09:01", "52:54:01:d1:09:02", "52:54:01:d1:09:03", + "52:54:01:d1:09:04", "52:54:01:d1:09:05", "52:54:01:d1:09:06", + ] } + } +} + +module "vr1_dc1_node" { + for_each = local.vr1_dc1_nodes + + source = "../modules/node-vm" + vm_name = each.key + autostart = false # D-127: node VMs are MAAS-power-controlled -- MANUAL, never on host boot + vcpu = each.value.vcpu + memory_mib = each.value.mem + disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 + pool_name = module.inner_storage.pool_name + network_names = local.vr1_dc1_node_nics + interface_macs = each.value.macs # pinned FROM THE FIRST APPLY -- see the scheme comment +} diff --git a/opentofu/vr1-dc1-substrate/variables.tf b/opentofu/vr1-dc1-substrate/variables.tf new file mode 100644 index 0000000..4f695d2 --- /dev/null +++ b/opentofu/vr1-dc1-substrate/variables.tf @@ -0,0 +1,48 @@ +# Inner-root inputs, mirroring vr1-dc0-substrate/variables.tf (the pattern of +# record -- its per-variable comments carry the measured traps and apply here +# unchanged). The two connection values (transit IP + ssh user) are MEASURED +# after vvr1-dc1 boots (hard rule 2 -- not invented); the rest mirror the outer +# root. + +variable "vvr1_dc1_transit_ip" { + description = "vvr1-dc1's static transit IP (D-124 Scheme A, NetBox-assigned via the amendment's apex commit) -- where this root's qemu+ssh inner provider connects. MEASURED after the outer apply boots vvr1-dc1." + type = string +} + +variable "vvr1_dc1_ssh_user" { + description = "SSH user on vvr1-dc1 for the qemu+ssh inner provider (the cloud-init user, e.g. jessea123). Key auth + known_hosts verified out of band (NO no_verify; ALL key types in known_hosts -- dc0's measured trap)." + type = string + default = "jessea123" +} + +variable "vvr1_dc1_ssh_keyfile" { + description = "PATH (on the EXECUTING host, i.e. voffice1 per D-128) to the DEDICATED dc1 private key for the qemu+ssh provider (D-126 option a; ~/vr1-dc1-creds/vr1-dc1_svc_ed25519). A path, never key material. REQUIRED: the provider's ssh dial reads NEITHER ~/.ssh/config nor default identities (dc0, measured 2026-07-20)." + type = string +} + +variable "inner_pool_path" { + description = "Directory path INSIDE vvr1-dc1 for the inner libvirt storage pool that backs the node/edge disks (created by the bootstrap step)." + type = string + default = "/var/lib/libvirt/vr1-dc1-inner" +} + +variable "opnsense_base_path" { + description = "Path ON THE EXECUTING HOST (voffice1, D-128 Plane 2) to the prepped OPNsense nano qcow2 -- the remote qemu+ssh provider UPLOADS volume content from ITS OWN filesystem (dc0, measured 2026-07-20). 26.7 nano: the DC-edge base PROVEN end to end on dc0 (boot, D-112(c) console bootstrap, D-113(a2) API, egress)." + type = string + default = "/var/lib/libvirt/vr1-dc1-inner/opnsense-26.7-nano.qcow2" +} + +variable "domain_suffix" { + description = "DNS domain suffix for the planes (mirrors the outer root)." + type = string +} + +variable "underlay_mtu" { + description = "Jumbo MTU for the internal fabric planes/mesh (9000). The wan bridge stays 1500." + type = number +} + +variable "vr1_dc1_planes" { + description = "The six vr1-dc1 planes (CIDR map), copied from the outer root's var of the same name (the committed values-of-record; D-124 amendment 2026-07-21, apex-verified). dc-planes creates them as isolated-L2 networks inside vvr1-dc1." + type = any +} diff --git a/opentofu/vr1-dc1-substrate/versions.tf b/opentofu/vr1-dc1-substrate/versions.tf new file mode 100644 index 0000000..ab929eb --- /dev/null +++ b/opentofu/vr1-dc1-substrate/versions.tf @@ -0,0 +1,18 @@ +# INNER root for vr1-dc1 (G12, 2026-07-21) -- D-123 MODEL B, mirroring +# opentofu/vr1-dc0-substrate/ (the pattern of record; its versions.tf comment is +# the authority for WHY this is a separate root + state). The vr1-dc1 substrate +# -- 6 dc-planes, the wan bridge, the OPNsense edge, an inner storage pool, and +# the 9 node VMs -- created INSIDE vvr1-dc1 via the qemu+ssh inner provider, +# NOT on vcloud. Apply order: outer (opentofu/) boots vvr1-dc1 -> bootstrap gate +# (site-headend-install.sh node-host mode) -> THIS root. Modules reused VERBATIM +# from ../modules/. Provider pin matches the outer root's versions.tf (0.9.8). +terraform { + required_version = ">= 1.6.0" + + required_providers { + libvirt = { + source = "dmacvicar/libvirt" + version = "0.9.8" + } + } +} diff --git a/scripts/carve-host-interfaces.sh b/scripts/carve-host-interfaces.sh index b3ef7b0..576141f 100644 --- a/scripts/carve-host-interfaces.sh +++ b/scripts/carve-host-interfaces.sh @@ -29,9 +29,10 @@ # # DC selector (opt-in, DOCFIX-166): set DC=vr0-dc0|vr1-dc0|vr1-dc1 to call # lib_net_select_dc/lib_hosts_select_dc explicitly. Unset (default) == DC0's -# real plane scheme + enrolled hosts, unchanged. dc1 no-ops at the network -# layer but FAILS at the host layer (no per-DC HOST_OCTET/HOST_BOOT_MAC yet); -# dc2 fails at both -- see scripts/lib-net.sh / scripts/lib-hosts.sh. +# real plane scheme + enrolled hosts, unchanged. vr1-dc0 no-ops at the +# network layer; vr1-dc1 selects its ratified planes (D-124 amendment); +# BOTH then FAIL at the host layer (no per-DC HOST_OCTET/HOST_BOOT_MAC +# yet) -- see scripts/lib-net.sh / scripts/lib-hosts.sh. # # Exit: 0 ok | 1 fatal | 2 warning diff --git a/scripts/dc-dc-radosgw-multisite.sh b/scripts/dc-dc-radosgw-multisite.sh index a66d18b..6894b92 100644 --- a/scripts/dc-dc-radosgw-multisite.sh +++ b/scripts/dc-dc-radosgw-multisite.sh @@ -144,6 +144,10 @@ DC_GATE_MSG="$(lib_net_select_dc "$DC" 2>&1 1>/dev/null)"; DC_GATE_RC=$? if [ "$DC_GATE_RC" -eq 0 ]; then DC_GATE_LINE="\$DC gate: OK ($DC)" + # The gate check ran in a subshell; select again HERE so the plane vars the + # reminder reads below belong to $DC, not the file's flat DC0 defaults + # (cross-DC value surfacing). rc 0 is already proven, safe under set -e. + lib_net_select_dc "$DC" else DC_GATE_LINE="\$DC gate: FAILED for '$DC' -- $DC_GATE_MSG" fi diff --git a/scripts/dc-dc-rbd-mirror.sh b/scripts/dc-dc-rbd-mirror.sh index 13433dd..8b7c65a 100644 --- a/scripts/dc-dc-rbd-mirror.sh +++ b/scripts/dc-dc-rbd-mirror.sh @@ -127,6 +127,10 @@ DC_GATE_MSG="$(lib_net_select_dc "$DC" 2>&1 1>/dev/null)"; DC_GATE_RC=$? if [ "$DC_GATE_RC" -eq 0 ]; then DC_GATE_LINE="\$DC gate: OK ($DC)" + # The gate check ran in a subshell; select again HERE so the plane vars the + # reminder reads below belong to $DC, not the file's flat DC0 defaults + # (cross-DC value surfacing). rc 0 is already proven, safe under set -e. + lib_net_select_dc "$DC" else DC_GATE_LINE="\$DC gate: FAILED for '$DC' -- $DC_GATE_MSG" fi diff --git a/scripts/dc-rack-net.sh b/scripts/dc-rack-net.sh index c1c6273..96f23a5 100755 --- a/scripts/dc-rack-net.sh +++ b/scripts/dc-rack-net.sh @@ -65,6 +65,22 @@ DNS_LISTEN="10.12.8.3" # MEASURED: live forwarder listen address DNS_UPSTREAM="10.10.0.20" # MEASURED: region BIND (voffice1) over transit ;; + dc1) + # MEASURED / apex-committed at the dc1 standup (2026-07-22..23): + # vr1-dc1-metal-admin 10.12.68.2/22 rack MAAS/DHCP leg -- the + # apex-committed rack IP (docs/audit/dc1-rack-import-commit-20260722.txt; + # NetBox dcim.site vr1-dc1), D-124 amendment plane map. + # vr1-dc1-metal-admin 10.12.68.3/22 node-DNS forwarder listen alias + # -- D-120 static, below the .100-.200 dynamic range (dc0 .3 analog). + # vr1-dc1-provider-public 10.12.64.2/22 edge-LAN leg -- MEASURED live on + # the provider-public bridge 2026-07-23 (edge gw 10.12.64.1; lib-net.sh + # vr1-dc1 arm, PLANE_GW ["10.12.64.0/22"]="10.12.64.1"). + LEGS="vr1-dc1-metal-admin=10.12.68.2/22 +vr1-dc1-metal-admin=10.12.68.3/22 +vr1-dc1-provider-public=10.12.64.2/22" + DNS_LISTEN="10.12.68.3" # MEASURED: forwarder listen alias (metal-admin) + DNS_UPSTREAM="10.10.0.20" # MEASURED: region BIND (voffice1) over transit + ;; *) echo "FAIL: unknown site '$SITE' -- add a MEASURED row block first" >&2; exit 2 ;; esac diff --git a/scripts/lib-net.sh b/scripts/lib-net.sh index ec326a6..6195b74 100644 --- a/scripts/lib-net.sh +++ b/scripts/lib-net.sh @@ -94,6 +94,16 @@ # exists to delete. lib_net_select_dc() { local dc="${1:?usage: lib_net_select_dc }" + # ONE selection per shell (the convention above: call ONCE, right after + # sourcing). Mechanically enforced since the vr1-dc1 arm DIVERGES (2026-07-21): + # the vr0-dc0/vr1-dc0 arms are no-ops over the file's flat defaults, so + # CHANGING selection after a diverging arm ran would leave stale cross-DC + # values in scope -- refuse instead of silently mixing DCs. Re-selecting the + # SAME DC stays a no-op; re-source the lib to genuinely switch. + if [ -n "${_LIBNET_DC_SELECTED:-}" ] && [ "$_LIBNET_DC_SELECTED" != "$dc" ]; then + echo "FAIL: DC already selected as '$_LIBNET_DC_SELECTED' in this shell -- one selection per shell; re-source lib-net.sh to switch DC" >&2 + return 1 + fi case "$dc" in vr0-dc0) : # VR0's DC0 -- the LIVE testcloud. The literals sourced above ARE its @@ -108,8 +118,36 @@ # the thing that changes (VR1 DC0 gains its own v6 per the family matrix). ;; vr1-dc1) - echo "FAIL: vr1-dc1 (VR1's SECOND DC) has no assigned network literals yet (D-101 NetBox-literals open item, tooling gap register #3) -- do not select it until NetBox assigns real CIDRs for it" >&2 - return 1 + # VR1's SECOND DC -- literals RATIFIED by the D-124 AMENDMENT (2026-07-21) + # and apex-VERIFIED against office1-netbox the same day (all six planes + # exist scoped dcim.site:vr1-dc1 -- docs/audit/dc1-apex-confirm-20260721.txt), + # which is this arm's own landing rule (NetBox assigns, then the literals + # enter the lib). DIVERGES from dc0 BY DESIGN: contiguous /22s in the + # 10.12.64.0/19 supernet (D-115), dc0's role ORDER kept -- dc0's + # 4/8/12/16/32/36 offsets cannot fit inside a /19. Values-of-record twin: + # opentofu/variables.tf `vr1_dc1_planes` (change one, change BOTH in the + # same commit). + PLANE_CIDRS=( "10.12.64.0/22" "10.12.68.0/22" "10.12.72.0/22" "10.12.76.0/22" "10.12.80.0/22" "10.12.84.0/22" ) + PLANE_NAME=( + ["10.12.64.0/22"]="provider-public" + ["10.12.68.0/22"]="metal-admin" + ["10.12.72.0/22"]="metal-internal" + ["10.12.76.0/22"]="data-tenant" + ["10.12.80.0/22"]="storage" + ["10.12.84.0/22"]="replication" + ) + # .1 site gateways on the two routed planes (D-120 convention; provider-public + # gw 10.12.64.1 is the ruled edge LAN gateway, D-124 amendment). + PLANE_GW=( ["10.12.64.0/22"]="10.12.64.1" ["10.12.68.0/22"]="10.12.68.1" ) + DATA_PLANE_CIDRS=( "10.12.72.0/22" "10.12.76.0/22" "10.12.80.0/22" "10.12.84.0/22" ) + METAL_INTERNAL_CIDR="10.12.72.0/22" + # OpenStack-layer values (VIP bands, FIP pool, keystone VIP) and the VR0 + # metal fabric facts (VLAN id, bridge iface) are NOT yet ruled/measured for + # vr1-dc1 (they arrive with its Stage-5 analog). UNSET so any use fails + # loud under `set -u` instead of silently borrowing another DC's addresses. + unset METAL_INTERNAL_VID METAL_INTERNAL_IFACE + unset VIP_PREFIX_PROVIDER VIP_PREFIX_ADMIN VIP_PREFIX_INTERNAL VIP_COUNT_EXPECT + unset FIP_POOL_START FIP_POOL_END KEYSTONE_VIP_DEFAULT ;; dc0|dc1|dc2) echo "FAIL: bare '$dc' is RETIRED (D-119). It was AMBIGUOUS ACROSS REGIONS: 'dc0' meant VR0's live DC0 here, but VR1's FIRST DC in the NetBox importer. Use the region-qualified selector: vr0-dc0 | vr1-dc0 | vr1-dc1" >&2 @@ -120,6 +158,7 @@ return 1 ;; esac + _LIBNET_DC_SELECTED="$dc" # only successful arms reach here (failed arms return 1) } # --- tiny read-only helpers --- diff --git a/scripts/phase-00-maas-standup.sh b/scripts/phase-00-maas-standup.sh index 8044462..a72d4d6 100644 --- a/scripts/phase-00-maas-standup.sh +++ b/scripts/phase-00-maas-standup.sh @@ -52,16 +52,17 @@ # runbook's Step 1: an explicit $DC env var, never an inferred default. # Unset/empty $DC changes NOTHING -- this script runs exactly as it always # has, implicitly against the D-052/D-053 plane scheme (backward compatible -# by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly (D-119); vr1-dc1 FAILS LOUD -# today (no NetBox-assigned literals yet -- gap #3) via set -e, no second -# validation layer added. This script does not source lib-hosts.sh (it never -# touches host identity), so only the network selector is called here. +# by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly +# (D-119) via set -e, no second validation layer added. This script does not +# source lib-hosts.sh (it never touches host identity), so only the network +# selector is called here. # NOTE: the PLANES table below is still a hardcoded D-052/D-053 literal, not -# derived from lib-net.sh's flat vars -- selecting dc1 is a real no-op -# (D-101: dc1 inherits dc0's layout unchanged, so the hardcoded literals are -# still correct), but this selector call alone does NOT make this script -# dc2-aware; once dc2 gets real CIDRs (gap #3), the PLANES table itself will -# still need updating for a genuinely different dc2 scheme. +# derived from lib-net.sh's vars. vr0-dc0/vr1-dc0 match it (D-101 +# inheritance). vr1-dc1's ratified scheme DIVERGES (D-124 amendment +# 2026-07-21: contiguous /22s in 10.12.64.0/19), so a parity guard below the +# table refuses any selected DC whose lib-net plane set does not match the +# table -- this script must never plan one DC's topology under another DC's +# name. DC="${DC:-}" if [ -n "$DC" ]; then lib_net_select_dc "$DC" @@ -123,6 +124,21 @@ TBL )" +# --- PLANES/lib-net parity guard (only when a $DC was explicitly selected): +# every table row's name|cidr pair must match the selected DC's PLANE_NAME +# map, else this DC0-hardcoded table would silently plan DC0 topology under +# another DC's name (exactly the cross-DC mixing lib-net's selector exists +# to prevent). Exit 2 = precondition, per the header contract. +if [ -n "$DC" ]; then + while IFS='|' read -r pname pcidr _rest; do + [ -z "$pname" ] && continue + if [ "${PLANE_NAME[$pcidr]:-}" != "$pname" ]; then + echo "FAIL: PLANES table is DC0-hardcoded ($pname $pcidr) but DC='$DC' assigns different planes (lib-net.sh) -- this script's topology target is not $DC-aware; refusing to plan another DC's scheme" >&2 + exit 2 + fi + done <<<"$PLANES" +fi + dt() { [ "$1" = "-" ] && echo "" || echo "$1"; } # decode "-" sentinel to empty hdr "MAAS stand-up mode=$MODE (D-052/D-053 target scheme)" diff --git a/scripts/reenroll-hosts.sh b/scripts/reenroll-hosts.sh index 8a835dc..214601f 100644 --- a/scripts/reenroll-hosts.sh +++ b/scripts/reenroll-hosts.sh @@ -13,8 +13,8 @@ # # DC selector (opt-in, DOCFIX-166): set DC=vr0-dc0|vr1-dc0|vr1-dc1 to call # lib_net_select_dc/lib_hosts_select_dc explicitly. Unset (default) == DC0's -# real, enrolled hosts, unchanged. dc1/dc2 currently FAIL LOUD (no per-DC host -# inventory exists yet) -- see scripts/lib-hosts.sh. +# real, enrolled hosts, unchanged. vr1-dc0/vr1-dc1 currently FAIL LOUD at the +# host layer (no per-DC host inventory exists yet) -- see scripts/lib-hosts.sh. # # Discover-assert-pin: never creates a host that already exists. Idempotent -- # a re-run after a partial run only creates the still-missing hosts. @@ -44,8 +44,9 @@ # always has, implicitly against DC0/VR0's real, enrolled hosts (backward # compatible by construction). Set DC=vr0-dc0|vr1-dc0|vr1-dc1 to select explicitly (D-119); the # selectors' own fail-loud behavior becomes this script's own exit code via -# set -e (lib_hosts_select_dc fails for BOTH dc1 and dc2 today -- no per-DC -# host inventory exists yet for either). No second validation layer is added. +# set -e (lib_hosts_select_dc fails for BOTH vr1-dc0 and vr1-dc1 today -- no +# per-DC host inventory exists yet for either). No second validation layer is +# added. DC="${DC:-}" if [ -n "$DC" ]; then lib_net_select_dc "$DC" diff --git a/tests/carve-host-interfaces/run-tests.sh b/tests/carve-host-interfaces/run-tests.sh index 9574a53..142174b 100644 --- a/tests/carve-host-interfaces/run-tests.sh +++ b/tests/carve-host-interfaces/run-tests.sh @@ -87,8 +87,9 @@ has 'no enrolled hosts yet' absent 'DO:|WOULD:' # must exit before any MAAS interaction -run_dc 1 "DC=vr1-dc1: net selector fails loud first (NetBox gap)" vr1-dc1 -has 'no assigned network literals yet' +run_dc 1 "DC=vr1-dc1: hosts selector fails loud (net arm ratified, D-124)" vr1-dc1 +has 'no enrolled hosts yet' +absent 'DO:|WOULD:' # must exit before any MAAS interaction run_dc 1 "DC=bogus: unknown token fails loud" bogus has "unknown DC 'bogus'" diff --git a/tests/cloudinit-vm/run-tests.sh b/tests/cloudinit-vm/run-tests.sh index b4ab4f1..089ff6c 100644 --- a/tests/cloudinit-vm/run-tests.sh +++ b/tests/cloudinit-vm/run-tests.sh @@ -34,6 +34,22 @@ && no "T5 data-disk volume has NO ignore_changes" \ || ok "T5 data-disk volume has NO ignore_changes" +# T5a-T5d: interface_macs MAC pinning (ported from node-vm 2026-07-22 -- the +# 2026-07-20 voffice1 Kea/MAC-regen incident class). Same regression intent as +# tests/node-vm: the var, both validations, and the conditional wiring must stay. +grep -q 'variable "interface_macs"' "$MOD/variables.tf" \ + && ok "T5a interface_macs variable present" \ + || no "T5a interface_macs variable present" +grep -q 'exactly one MAC per network_names entry' "$MOD/variables.tf" \ + && ok "T5b count validation present (all-or-nothing pinning)" \ + || no "T5b count validation present (all-or-nothing pinning)" +grep -q '0-9a-fA-F.*{2}' "$MOD/variables.tf" \ + && ok "T5c MAC format validation present" \ + || no "T5c MAC format validation present" +grep -q 'mac = length(var.interface_macs) > 0 ? { address = var.interface_macs\[i\] } : null' "$MOD/main.tf" \ + && ok "T5d interfaces wiring carries the conditional MAC pin" \ + || no "T5d interfaces wiring carries the conditional MAC pin" + # T6: module still validates when tofu is available (init -backend=false is # offline once the provider is in the plugin cache; skip cleanly otherwise) if command -v tofu >/dev/null 2>&1; then diff --git a/tests/dc-dc-dr-drill/run-tests.sh b/tests/dc-dc-dr-drill/run-tests.sh index 1bd4664..77c83ee 100644 --- a/tests/dc-dc-dr-drill/run-tests.sh +++ b/tests/dc-dc-dr-drill/run-tests.sh @@ -76,11 +76,18 @@ grep -q '11.4 SKIPPED' <<<"$FB_SKIP" && ! grep -q '11.4a' <<<"$FB_SKIP" \ && { echo " PASS T20 --skip-11-4 stops the plan after 11.2/11.3"; PASS=$((PASS+1)); } || { echo " FAIL T20"; echo "$FB_SKIP" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -# --- $DC gate: informational in dry-run (vr1-dc1 involved), blocking before --apply --- -grep -q 'gate: FAILED' <<<"$FB_OUT" && grep -q 'gate: OK (vr1-dc0)' <<<"$FB_OUT" \ - && { echo " PASS T21 failback dry-run reports vr1-dc0 OK / vr1-dc1 FAILED, still prints full plan"; PASS=$((PASS+1)); } || { echo " FAIL T21"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +# --- $DC gate: both DCs now pass (vr1-dc1 literals RATIFIED, D-124 amendment +# 2026-07-21). The gate's refusal path (exit 3) is retained in the script as +# defense-in-depth for future unassigned DC tokens, but is no longer reachable +# through the CLI (arg validation admits only the two ratified tokens). +grep -q 'gate: OK (vr1-dc1)' <<<"$FB_OUT" && grep -q 'gate: OK (vr1-dc0)' <<<"$FB_OUT" \ + && { echo " PASS T21 failback dry-run reports BOTH DC gates OK (dc1 ratified), still prints full plan"; PASS=$((PASS+1)); } || { echo " FAIL T21"; echo "$FB_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T22 failback --apply is BLOCKED by the dc2 gate (rc 3)" failback --pool glance --recovering-dc vr1-dc0 --recovering-unit rec/0 --primary-dc vr1-dc1 --primary-unit pri/0 --apply --no-prompt +if command -v juju >/dev/null 2>&1; then + echo " SKIP T22 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" +else + run 2 'juju required on PATH' "T22 failback --apply passes both DC gates, FAILS on missing juju (rc 2)" failback --pool glance --recovering-dc vr1-dc0 --recovering-unit rec/0 --primary-dc vr1-dc1 --primary-unit pri/0 --apply --no-prompt +fi # --- --apply never runs without --apply: no juju invocation attempted in dry-run --- if command -v juju >/dev/null 2>&1; then diff --git a/tests/dc-dc-radosgw-multisite/run-tests.sh b/tests/dc-dc-radosgw-multisite/run-tests.sh index e570c14..4a796f9 100644 --- a/tests/dc-dc-radosgw-multisite/run-tests.sh +++ b/tests/dc-dc-radosgw-multisite/run-tests.sh @@ -57,12 +57,21 @@ RESTART_OUT="$(bash "$SCRIPT" master-init --dc vr1-dc0 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --restart-action restart 2>&1)" grep -qE 'juju run ceph-radosgw/0 restart -m openstack' <<<"$RESTART_OUT" && { echo " PASS T15 --restart-action adds a juju run step"; PASS=$((PASS+1)); } || { echo " FAIL T15"; echo "$RESTART_OUT" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -# --- $DC gate: informational in dry-run, blocking before --apply --- -DC2_DRY="$(bash "$SCRIPT" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 2>&1)"; DC2_DRY_RC=$? -[[ "$DC2_DRY_RC" == 0 ]] && grep -q 'gate: FAILED' <<<"$DC2_DRY" && grep -q 'OK (dry-run)' <<<"$DC2_DRY" \ - && { echo " PASS T16 dc2 dry-run still prints the plan (gate is informational here)"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$DC2_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +# --- $DC gate: vr1-dc1 now passes (literals RATIFIED, D-124 amendment +# 2026-07-21); the reminder must surface DC1's OWN replication plane, never +# dc0's (cross-DC value surfacing). The gate's refusal path (exit 3) is +# retained as defense-in-depth but no longer CLI-reachable. +DC1_DRY="$(bash "$SCRIPT" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 2>&1)"; DC1_DRY_RC=$? +[[ "$DC1_DRY_RC" == 0 ]] && grep -q 'gate: OK (vr1-dc1)' <<<"$DC1_DRY" && grep -q 'OK (dry-run)' <<<"$DC1_DRY" \ + && { echo " PASS T16 dc1 dry-run prints the plan (gate OK, D-124 ratified)"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +grep -q 'REPLICATION plane (10.12.84.0/22' <<<"$DC1_DRY" \ + && { echo " PASS T16b dc1 reminder shows dc1's OWN replication CIDR (not dc0's)"; PASS=$((PASS+1)); } || { echo " FAIL T16b (cross-DC value surfaced)"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T17 dc2 --apply is BLOCKED by the gate (rc 3)" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply +if command -v juju >/dev/null 2>&1; then + echo " SKIP T17 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" +else + run 2 'juju required on PATH' "T17 dc1 --apply passes the gate, FAILS on missing juju (rc 2)" master-init --dc vr1-dc1 --unit ceph-radosgw/0 --realm R --zonegroup ZG --zone Z1 --endpoint http://x:80 --apply +fi # --- secret redaction --- JR_OUT="$(bash "$SCRIPT" join-readonly --dc vr1-dc0 --unit ceph-radosgw/0 --zonegroup ZG --zone Z2 --endpoint http://x:80 --access-key AK123 --secret TOPSECRETVALUE 2>&1)" diff --git a/tests/dc-dc-rbd-mirror/run-tests.sh b/tests/dc-dc-rbd-mirror/run-tests.sh index 302c20f..c91991b 100644 --- a/tests/dc-dc-rbd-mirror/run-tests.sh +++ b/tests/dc-dc-rbd-mirror/run-tests.sh @@ -51,12 +51,21 @@ grep -q 'peer bootstrap import --site-name vr1-dc1 --direction rx-tx glance /tmp/tok2' <<<"$BS_RXTX" \ && { echo " PASS T16 bootstrap-secondary rx-tx (two-way) plan is correct"; PASS=$((PASS+1)); } || { echo " FAIL T16"; echo "$BS_RXTX" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -# --- $DC gate: informational in dry-run, blocking before --apply --- -DC2_DRY="$(bash "$SCRIPT" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 2>&1)"; DC2_DRY_RC=$? -[[ "$DC2_DRY_RC" == 0 ]] && grep -q 'gate: FAILED' <<<"$DC2_DRY" && grep -q 'OK (dry-run)' <<<"$DC2_DRY" \ - && { echo " PASS T17 dc2 dry-run still prints the plan (gate is informational here)"; PASS=$((PASS+1)); } || { echo " FAIL T17"; echo "$DC2_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +# --- $DC gate: vr1-dc1 now passes (literals RATIFIED, D-124 amendment +# 2026-07-21); the reminder must surface DC1's OWN replication plane, never +# dc0's (cross-DC value surfacing). The gate's refusal path (exit 3) is +# retained as defense-in-depth but no longer CLI-reachable. +DC1_DRY="$(bash "$SCRIPT" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 2>&1)"; DC1_DRY_RC=$? +[[ "$DC1_DRY_RC" == 0 ]] && grep -q 'gate: OK (vr1-dc1)' <<<"$DC1_DRY" && grep -q 'OK (dry-run)' <<<"$DC1_DRY" \ + && { echo " PASS T17 dc1 dry-run prints the plan (gate OK, D-124 ratified)"; PASS=$((PASS+1)); } || { echo " FAIL T17"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } +grep -q 'REPLICATION plane (10.12.84.0/22' <<<"$DC1_DRY" \ + && { echo " PASS T17b dc1 reminder shows dc1's OWN replication CIDR (not dc0's)"; PASS=$((PASS+1)); } || { echo " FAIL T17b (cross-DC value surfaced)"; echo "$DC1_DRY" | sed 's/^/ /'; FAIL=$((FAIL+1)); } -run 3 '\$DC gate refused' "T18 dc2 --apply is BLOCKED by the gate (rc 3)" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --apply +if command -v juju >/dev/null 2>&1; then + echo " SKIP T18 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" +else + run 2 'juju required on PATH' "T18 dc1 --apply passes the gate, FAILS on missing juju (rc 2)" bootstrap-primary --dc vr1-dc1 --unit ceph-mon/0 --pool glance --site-name vr1-dc1 --apply +fi if command -v juju >/dev/null 2>&1; then echo " SKIP T19 juju-missing case (juju IS present in this environment -- can't exercise the missing-tool guard here)" diff --git a/tests/dc-rack-mgmt-import/run-tests.sh b/tests/dc-rack-mgmt-import/run-tests.sh index e91e828..5b1a742 100644 --- a/tests/dc-rack-mgmt-import/run-tests.sh +++ b/tests/dc-rack-mgmt-import/run-tests.sh @@ -29,13 +29,16 @@ NETBOX_URL= NETBOX_TOKEN= python3 "$S" --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.5 \ >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" -# NO INVENTED LITERAL -- both inputs are REQUIRED (no default CIDR/IP baked in) -NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --rack-ip 10.12.8.5 \ +# NO INVENTED LITERAL -- all three inputs are REQUIRED (no default site/CIDR/IP baked in) +NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --site vr1-dc0 --rack-ip 10.12.8.5 \ >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --transit-cidr must fail (no invented CIDR)" -NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --transit-cidr 172.31.0.0/30 \ +NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --site vr1-dc0 --transit-cidr 172.31.0.0/30 \ >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --rack-ip must fail (no invented IP)" +NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.5 \ + >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --site must fail (target DC never inferred)" grep -q 'default=os.environ.get("TRANSIT_CIDR")' "$S" && ok || bad "lost the TRANSIT_CIDR env fallback" grep -q 'default=os.environ.get("RACK_IP")' "$S" && ok || bad "lost the RACK_IP env fallback" +grep -q 'default=os.environ.get("RACK_SITE")' "$S" && ok || bad "lost the RACK_SITE env fallback" # guard against a baked-in transit /30 or /31 literal masquerading as a default grep -qE '=\s*"10\.[0-9]+\.[0-9]+\.[0-9]+/3[01]"' "$S" && bad "a /30 or /31 literal is baked in -- must be an INPUT" || ok @@ -47,7 +50,7 @@ grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" grep -q 'SANDBOX_HOSTS' "$S" && ok || bad "lost the SANDBOX_HOSTS gate" out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" \ - --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.5 --commit 2>&1)" + --site vr1-dc0 --transit-cidr 172.31.0.0/30 --rack-ip 10.12.8.5 --commit 2>&1)" printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" @@ -64,12 +67,15 @@ grep -q "outside metal-admin" "$S" && ok || bad "lost the metal-admin containment check" grep -q "must be a /30 or /31" "$S" && ok || bad "lost the /30-or-/31 shape check" -# THE D-124 SCHEME VALUES -- role, container, site scope, metal-admin band, rack dns. +# THE D-124 SCHEME VALUES -- role, container, per-site scope/metal-admin/rack dns +# (dc0 ratified 2026-07-16; dc1 per the D-124 AMENDMENT 2026-07-21). grep -qF 'ROLE_SLUG = "transit"' "$S" && ok || bad "transit role slug changed" grep -qF 'CONTAINER = "172.31.0.0/24"' "$S" && ok || bad "container is not 172.31.0.0/24 (D-124 dedicated transit supernet, operator-pinned)" -grep -qF 'SITE_SLUG = "vr1-dc0"' "$S" && ok || bad "transit site scope is not vr1-dc0" -grep -qF 'METAL_ADMIN = "10.12.8.0/22"' "$S" && ok || bad "metal-admin is not 10.12.8.0/22" -grep -qF 'RACK_DNS = "vvr1-dc0"' "$S" && ok || bad "rack dns name is not vvr1-dc0" +grep -qF '"vr1-dc0": {' "$S" && grep -qF '"vr1-dc1": {' "$S" && ok || bad "SITES map lost a DC entry" +grep -qF '"metal_admin": "10.12.8.0/22"' "$S" && ok || bad "dc0 metal-admin is not 10.12.8.0/22" +grep -qF '"metal_admin": "10.12.68.0/22"' "$S" && ok || bad "dc1 metal-admin is not 10.12.68.0/22 (D-124 amendment)" +grep -qF '"rack_dns": "vvr1-dc0"' "$S" && ok || bad "dc0 rack dns name is not vvr1-dc0" +grep -qF '"rack_dns": "vvr1-dc1"' "$S" && ok || bad "dc1 rack dns name is not vvr1-dc1" grep -q '"scope_type": "dcim.site"' "$S" && ok || bad "the transit prefix is not dcim.site-scoped" # This tool must NOT create the role/container/site -- they are preconditions. diff --git a/tests/dc-rack-mgmt-import/test_logic.py b/tests/dc-rack-mgmt-import/test_logic.py index 63f0677..fc25446 100644 --- a/tests/dc-rack-mgmt-import/test_logic.py +++ b/tests/dc-rack-mgmt-import/test_logic.py @@ -104,23 +104,30 @@ ok(label) -# Full-precondition fixture builders (transit role + transit container + vr1-dc0 site). +# Full-precondition fixture builders (transit role + transit container + DC sites). ROLE = {"slug": "transit", "name": "Transit", "id": 7} CONTAINER = {"prefix": "172.31.0.0/24", "id": 10} SITE_DC0 = {"slug": "vr1-dc0", "name": "VR1 DC0", "id": 20} +SITE_DC1 = {"slug": "vr1-dc1", "name": "VR1 DC1", "id": 21} + +# dc1 ratified values (D-124 AMENDMENT 2026-07-21): transit 172.31.0.4/30 (next /30 in +# the container), rack IP in metal-admin 10.12.68.0/22's D-120 static band. +TRANSIT_DC1 = "172.31.0.4/30" +RACK_DC1 = "10.12.68.2" +RACK_DC1_ADDR = "10.12.68.2/22" def full_fake(prefixes_extra=(), ip_extra=()): return fake_netbox.FakeNB( roles=[ROLE], prefixes=[CONTAINER, *prefixes_extra], - sites=[SITE_DC0], + sites=[SITE_DC0, SITE_DC1], ip_addresses=list(ip_extra), ) def base_args(extra=()): - return ["--transit-cidr", TRANSIT, "--rack-ip", RACK, *extra] + return ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", RACK, *extra] os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # a known sandbox (guard passes) @@ -174,7 +181,7 @@ # ----------------------------------------------------------------------------- fk = full_fake() with captured_stdout(): - rc = run_main(["--transit-cidr", TRANSIT31, "--rack-ip", RACK, "--commit"], fk) + rc = run_main(["--site", "vr1-dc0", "--transit-cidr", TRANSIT31, "--rack-ip", RACK, "--commit"], fk) check(rc == 0 and len(fk.creates) == 2, "a /31 transit is accepted and writes both objects", str((rc, len(fk.creates)))) @@ -214,7 +221,7 @@ # ----------------------------------------------------------------------------- fk = full_fake() run_dies("half-write: rack IP == .1 gateway is REJECTED before any write", - ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], fk) + ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], fk) check(len(fk.creates) == 0, "half-write: the rejected run wrote NOTHING (transit not created)") # ----------------------------------------------------------------------------- @@ -236,61 +243,70 @@ # 8. Transit CIDR shape / placement rejects. # ----------------------------------------------------------------------------- run_dies("a transit OUTSIDE the transit container is REJECTED", - ["--transit-cidr", "192.168.0.0/30", "--rack-ip", RACK, "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", "192.168.0.0/30", "--rack-ip", RACK, "--commit"], full_fake()) run_dies("a /29 transit is REJECTED (not point-to-point)", - ["--transit-cidr", "172.31.0.0/29", "--rack-ip", RACK, "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", "172.31.0.0/29", "--rack-ip", RACK, "--commit"], full_fake()) run_dies("a /32 transit is REJECTED", - ["--transit-cidr", "172.31.0.1/32", "--rack-ip", RACK, "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", "172.31.0.1/32", "--rack-ip", RACK, "--commit"], full_fake()) run_dies("a host-bits-set transit (172.31.0.1/30) is REJECTED", - ["--transit-cidr", "172.31.0.1/30", "--rack-ip", RACK, "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", "172.31.0.1/30", "--rack-ip", RACK, "--commit"], full_fake()) run_dies("a non-CIDR transit is REJECTED", - ["--transit-cidr", "not-a-cidr", "--rack-ip", RACK, "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", "not-a-cidr", "--rack-ip", RACK, "--commit"], full_fake()) # each of the above must have written nothing -- prove it once with a fresh fake _fk = full_fake() run_dies("bad transit writes nothing (proof)", - ["--transit-cidr", "172.31.0.0/29", "--rack-ip", RACK, "--commit"], _fk) + ["--site", "vr1-dc0", "--transit-cidr", "172.31.0.0/29", "--rack-ip", RACK, "--commit"], _fk) check(len(_fk.creates) == 0, "the rejected bad-transit run wrote NOTHING") # ----------------------------------------------------------------------------- # 9. Rack-IP band rejects. # ----------------------------------------------------------------------------- run_dies("rack IP OUTSIDE metal-admin 10.12.8.0/22 is REJECTED", - ["--transit-cidr", TRANSIT, "--rack-ip", "10.13.0.5", "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", "10.13.0.5", "--commit"], full_fake()) run_dies("rack IP == .1 gateway is REJECTED", - ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], full_fake()) run_dies("rack IP == .0 network address is REJECTED", - ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.0", "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.0", "--commit"], full_fake()) run_dies("rack IP in-/22-but-above-static-band (.50) is REJECTED", - ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.50", "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.50", "--commit"], full_fake()) run_dies("rack IP in-/22-but-in-a-higher-/24 (10.12.9.5) is REJECTED", - ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.9.5", "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", "10.12.9.5", "--commit"], full_fake()) run_dies("rack IP with a non-/22 mask is REJECTED", - ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.5/24", "--commit"], full_fake()) + ["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.5/24", "--commit"], full_fake()) # accepted: the boundaries of the static band (.2 low, .49 high) both write. for edge_ip in ("10.12.8.2", "10.12.8.49"): fk = full_fake() with captured_stdout(): - rc = run_main(["--transit-cidr", TRANSIT, "--rack-ip", edge_ip, "--commit"], fk) + rc = run_main(["--site", "vr1-dc0", "--transit-cidr", TRANSIT, "--rack-ip", edge_ip, "--commit"], fk) check(rc == 0 and len(fk.creates) == 2, f"rack IP band edge {edge_ip} is ACCEPTED", str((rc, len(fk.creates)))) # ----------------------------------------------------------------------------- # 10. NO INVENTED LITERAL -- missing input fails loud (does not guess a value). # ----------------------------------------------------------------------------- -run_dies("missing --transit-cidr fails loud", ["--rack-ip", RACK], full_fake()) -run_dies("missing --rack-ip fails loud", ["--transit-cidr", TRANSIT], full_fake()) +run_dies("missing --transit-cidr fails loud", ["--site", "vr1-dc0", "--rack-ip", RACK], full_fake()) +run_dies("missing --rack-ip fails loud", ["--site", "vr1-dc0", "--transit-cidr", TRANSIT], full_fake()) +run_dies("missing --site fails loud (target DC is never inferred)", + ["--transit-cidr", TRANSIT, "--rack-ip", RACK], full_fake()) -# env fallback works (args-or-env): both via env, no flags. +# env fallback works (args-or-env): all three via env, no flags. +os.environ["RACK_SITE"] = "vr1-dc0" os.environ["TRANSIT_CIDR"] = TRANSIT os.environ["RACK_IP"] = RACK fk = full_fake() with captured_stdout(): rc = run_main(["--commit"], fk) -check(rc == 0 and len(fk.creates) == 2, "env TRANSIT_CIDR/RACK_IP are honored (args-or-env)", +check(rc == 0 and len(fk.creates) == 2, "env RACK_SITE/TRANSIT_CIDR/RACK_IP are honored (args-or-env)", str((rc, len(fk.creates)))) del os.environ["TRANSIT_CIDR"] del os.environ["RACK_IP"] +# a BOGUS env site dies (argparse choices does not validate env-supplied defaults). +os.environ["RACK_SITE"] = "vr9-dc9" +run_dies("bogus RACK_SITE env value fails loud (choices bypass covered)", + ["--transit-cidr", TRANSIT, "--rack-ip", RACK], full_fake()) +del os.environ["RACK_SITE"] + # ----------------------------------------------------------------------------- # 11. Missing NETBOX_URL/TOKEN fails loud (does not guess a target). # ----------------------------------------------------------------------------- @@ -321,16 +337,64 @@ os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # ----------------------------------------------------------------------------- -# 13. Structural pins (a changed constant a behavioral test alone would miss). +# 13. Per-site behavior: --site vr1-dc1 (D-124 AMENDMENT 2026-07-21). +# ----------------------------------------------------------------------------- +# dc1 happy path: binds vr1-dc1's site id, dc1 metal-admin mask, dns vvr1-dc1. +fk = full_fake() +with captured_stdout(): + rc = run_main(["--site", "vr1-dc1", "--transit-cidr", TRANSIT_DC1, + "--rack-ip", RACK_DC1, "--commit"], fk) +check(rc == 0 and len(fk.creates) == 2, "dc1 commit writes both objects", + str((rc, len(fk.creates)))) +pfx1 = next(pl for pp, pl in fk.creates if pp == "ipam/prefixes") +ipa1 = next(pl for pp, pl in fk.creates if pp == "ipam/ip-addresses") +check(pfx1.get("prefix") == TRANSIT_DC1, "dc1 transit prefix is 172.31.0.4/30", + str(pfx1.get("prefix"))) +check(pfx1.get("scope_id") == 21, "dc1 transit prefix binds vr1-dc1's site id (21), NOT dc0's", + str(pfx1.get("scope_id"))) +check(ipa1.get("address") == RACK_DC1_ADDR, "dc1 rack IP stored with the 10.12.68.0/22 mask", + str(ipa1.get("address"))) +check(ipa1.get("dns_name") == "vvr1-dc1", "dc1 rack IP dns_name is vvr1-dc1", + str(ipa1.get("dns_name"))) +check("vr1-dc1" in ipa1.get("description", ""), "dc1 rack IP description names vr1-dc1") + +# CROSS-SITE GUARD: dc0's rack IP under --site vr1-dc1 is outside dc1's metal-admin -> die. +fk = full_fake() +run_dies("dc0 rack IP under --site vr1-dc1 is REJECTED (band is per-site)", + ["--site", "vr1-dc1", "--transit-cidr", TRANSIT_DC1, "--rack-ip", RACK, "--commit"], fk) +check(len(fk.creates) == 0, "the cross-site rejected run wrote NOTHING") +# and dc1's band edges hold on the dc1 /22. +for edge_ip in ("10.12.68.2", "10.12.68.49"): + fk = full_fake() + with captured_stdout(): + rc = run_main(["--site", "vr1-dc1", "--transit-cidr", TRANSIT_DC1, + "--rack-ip", edge_ip, "--commit"], fk) + check(rc == 0 and len(fk.creates) == 2, f"dc1 rack IP band edge {edge_ip} is ACCEPTED", + str((rc, len(fk.creates)))) +run_dies("dc1 rack IP .1 gateway (10.12.68.1) is REJECTED", + ["--site", "vr1-dc1", "--transit-cidr", TRANSIT_DC1, "--rack-ip", "10.12.68.1", + "--commit"], full_fake()) +# a dc1 run against a fake WITHOUT the vr1-dc1 site dies (site precondition is per-site). +fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[CONTAINER], sites=[SITE_DC0]) +run_dies("missing vr1-dc1 site is REJECTED for --site vr1-dc1", + ["--site", "vr1-dc1", "--transit-cidr", TRANSIT_DC1, "--rack-ip", RACK_DC1, + "--commit"], fk) +check(len(fk.creates) == 0, "missing-dc1-site run wrote nothing") + +# ----------------------------------------------------------------------------- +# 14. Structural pins (a changed constant a behavioral test alone would miss). # ----------------------------------------------------------------------------- check(T.ROLE_SLUG == "transit", "ROLE_SLUG is 'transit'") check(T.CONTAINER == "172.31.0.0/24", "CONTAINER is 172.31.0.0/24 (dedicated transit supernet)") -check(T.SITE_SLUG == "vr1-dc0", "SITE_SLUG is vr1-dc0 (transit site scope)") -check(T.METAL_ADMIN == "10.12.8.0/22", "METAL_ADMIN is 10.12.8.0/22") +check(set(T.SITES) == {"vr1-dc0", "vr1-dc1"}, "SITES covers exactly vr1-dc0 + vr1-dc1") +check(T.SITES["vr1-dc0"]["metal_admin"] == "10.12.8.0/22", "dc0 metal-admin is 10.12.8.0/22") +check(T.SITES["vr1-dc1"]["metal_admin"] == "10.12.68.0/22", + "dc1 metal-admin is 10.12.68.0/22 (D-124 amendment 2026-07-21)") check(T.STATIC_BAND_LOW == 2 and T.STATIC_BAND_HIGH == 49, "static band is .2-.49 (D-120)") check(T.STATUS == "active", "STATUS is active") -check(T.RACK_DNS == "vvr1-dc0", "RACK_DNS is vvr1-dc0") +check(T.SITES["vr1-dc0"]["rack_dns"] == "vvr1-dc0", "dc0 rack dns is vvr1-dc0") +check(T.SITES["vr1-dc1"]["rack_dns"] == "vvr1-dc1", "dc1 rack dns is vvr1-dc1") check(T.SANDBOX_HOSTS == {"localhost", "127.0.0.1", "10.10.1.10"}, "SANDBOX_HOSTS matches the sibling importers") check("get_nb" in dir(T) and callable(T.get_nb), "get_nb() injection seam exists") diff --git a/tests/dc-rack-net/run-tests.sh b/tests/dc-rack-net/run-tests.sh index e73b2c7..7fefa90 100755 --- a/tests/dc-rack-net/run-tests.sh +++ b/tests/dc-rack-net/run-tests.sh @@ -72,5 +72,26 @@ && no "T14 do_check is read-only" \ || ok "T14 do_check is read-only" +# T15-T17: dc1 identity values (measured/apex-committed 2026-07-22..23 -- +# provider-public leg measured live on the bridge, metal-admin leg = the +# apex-committed rack IP, forwarder = D-120 static; changing one silently +# breaks dc1 node DNS or rack addressing) +grep -q 'vr1-dc1-metal-admin=10.12.68.2/22' "$SCRIPT" \ + && ok "T15 dc1 rack MAAS/DHCP leg 10.12.68.2/22 on metal-admin" \ + || no "T15 dc1 rack MAAS/DHCP leg 10.12.68.2/22 on metal-admin" +grep -q 'vr1-dc1-metal-admin=10.12.68.3/22' "$SCRIPT" \ + && grep -q 'DNS_LISTEN="10.12.68.3"' "$SCRIPT" \ + && ok "T16 dc1 forwarder alias + listen 10.12.68.3" \ + || no "T16 dc1 forwarder alias + listen 10.12.68.3" +grep -q 'vr1-dc1-provider-public=10.12.64.2/22' "$SCRIPT" \ + && ok "T17 dc1 edge-LAN leg 10.12.64.2/22 on provider-public" \ + || no "T17 dc1 edge-LAN leg 10.12.64.2/22 on provider-public" + +# T18: dc1 site block carries MEASURED citations (hard rule 2) +DC1_BLOCK="$(awk '/^ dc1\)/{f=1} f{print} f&&/;;/{exit}' "$SCRIPT")" +grep -q 'MEASURED' <<<"$DC1_BLOCK" \ + && ok "T18 dc1 site rows cite MEASURED provenance" \ + || no "T18 dc1 site rows cite MEASURED provenance" + echo; echo "dc-rack-net: $PASS passed, $FAIL failed" [ "$FAIL" -eq 0 ] diff --git a/tests/dc-selector/run-tests.sh b/tests/dc-selector/run-tests.sh index b745bd2..29e66c0 100644 --- a/tests/dc-selector/run-tests.sh +++ b/tests/dc-selector/run-tests.sh @@ -6,11 +6,14 @@ # Asserts: # - sourcing either file with no further action is unaffected (backward # compatible by construction -- the flat vars populate exactly as before). -# - lib_net_select_dc: dc0/dc1 no-op (D-101: DC1 inherits DC0's v4 layout -# unchanged), dc2 fails loud, unknown token fails loud. -# - lib_hosts_select_dc: dc0 no-op, dc1 AND dc2 both fail loud (no real +# - lib_net_select_dc: vr0-dc0/vr1-dc0 no-op (D-101 inheritance); vr1-dc1 +# OVERRIDES to the D-124-amendment literals (2026-07-21: contiguous /22s in +# 10.12.64.0/19) and UNSETS the not-yet-ruled OpenStack-layer values; +# unknown/retired tokens fail loud; one selection per shell (the guard +# added with the diverging dc1 arm). +# - lib_hosts_select_dc: vr0-dc0 no-op, BOTH VR1 DCs fail loud (no real # per-DC host enrollment exists yet for either -- this is the documented -# asymmetry vs. lib-net.sh's dc0|dc1 no-op). +# asymmetry vs. lib-net.sh). # - Neither function ever silently invents/reuses a value across DCs. set -uo pipefail SD="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -37,15 +40,43 @@ chk "HOST_OCTET unaffected" "${HOST_OCTET[openstack0]}" "40" # --- lib_net_select_dc (D-119 region-qualified): vr0-dc0/vr1-dc0 no-op, -# vr1-dc1 fails loud, RETIRED bare dcN fails loud, unknown fails loud --- +# vr1-dc1 OVERRIDES (D-124 amendment 2026-07-21), RETIRED bare dcN fails +# loud, unknown fails loud --- ( lib_net_select_dc vr0-dc0 ); chk "net vr0-dc0 no-op rc" "$?" 0 ( lib_net_select_dc vr1-dc0 ); chk "net vr1-dc0 no-op rc" "$?" 0 -( lib_net_select_dc vr1-dc1 2>/dev/null ); chk "net vr1-dc1 fails-loud rc" "$?" 1 -NET_ERR="$(lib_net_select_dc vr1-dc1 2>&1 1>/dev/null || true)" -grep -q "NetBox" <<<"$NET_ERR" && ok "net vr1-dc1 error cites NetBox gap" || no "net vr1-dc1 error cites NetBox gap" +( lib_net_select_dc vr1-dc1 ); chk "net vr1-dc1 selects rc" "$?" 0 ( lib_net_select_dc bogus 2>/dev/null ); chk "net unknown-token fails-loud rc" "$?" 1 ( lib_net_select_dc 2>/dev/null ); chk "net missing-arg fails-loud rc" "$?" 1 +# vr1-dc1 arm: the D-124-amendment literals land (apex-verified 2026-07-21). +DC1_P0="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_CIDRS[0]}")" +chk "net vr1-dc1 provider-public is 10.12.64.0/22" "$DC1_P0" "10.12.64.0/22" +DC1_P5="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_CIDRS[5]}")" +chk "net vr1-dc1 replication is 10.12.84.0/22" "$DC1_P5" "10.12.84.0/22" +DC1_MA="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_NAME[10.12.68.0/22]}")" +chk "net vr1-dc1 10.12.68.0/22 is metal-admin" "$DC1_MA" "metal-admin" +DC1_GW="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${PLANE_GW[10.12.64.0/22]}")" +chk "net vr1-dc1 provider-public gw is 10.12.64.1 (ruled edge LAN gw)" "$DC1_GW" "10.12.64.1" +DC1_MI="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "$METAL_INTERNAL_CIDR")" +chk "net vr1-dc1 metal-internal CIDR is 10.12.72.0/22" "$DC1_MI" "10.12.72.0/22" +# not-yet-ruled OpenStack-layer values are UNSET (fail loud under set -u, never +# silently another DC's addresses). +DC1_KV="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${KEYSTONE_VIP_DEFAULT+still-set}")" +chk "net vr1-dc1 unsets KEYSTONE_VIP_DEFAULT" "$DC1_KV" "" +DC1_VP="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${VIP_PREFIX_PROVIDER+still-set}")" +chk "net vr1-dc1 unsets VIP_PREFIX_PROVIDER" "$DC1_VP" "" +DC1_FP="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; echo "${FIP_POOL_START+still-set}")" +chk "net vr1-dc1 unsets FIP_POOL_START" "$DC1_FP" "" + +# ONE SELECTION PER SHELL (guard added with the diverging dc1 arm): changing +# selection after dc1 is REFUSED; re-selecting the SAME DC stays a no-op. +( lib_net_select_dc vr1-dc1 >/dev/null 2>&1; lib_net_select_dc vr0-dc0 2>/dev/null ); \ + chk "net dc1->vr0-dc0 re-select REFUSED (stale cross-DC values)" "$?" 1 +SWITCH_ERR="$(lib_net_select_dc vr1-dc1 >/dev/null 2>&1; lib_net_select_dc vr0-dc0 2>&1 1>/dev/null || true)" +grep -q "already selected" <<<"$SWITCH_ERR" && ok "net re-select error says already selected" || no "net re-select error says already selected" +( lib_net_select_dc vr1-dc0 >/dev/null 2>&1; lib_net_select_dc vr1-dc0 ); \ + chk "net same-DC re-select stays a no-op rc" "$?" 0 + # D-119 REGRESSION GUARD: the bare dcN tokens are RETIRED and must be REJECTED. # Accepting them "for compatibility" would preserve the exact cross-region # ambiguity D-119 deletes -- 'dc0' meant VR0's LIVE cloud in lib-net.sh but VR1's diff --git a/tests/phase-00-maas-standup/run-tests.sh b/tests/phase-00-maas-standup/run-tests.sh index 91554a8..45412c2 100644 --- a/tests/phase-00-maas-standup/run-tests.sh +++ b/tests/phase-00-maas-standup/run-tests.sh @@ -94,8 +94,8 @@ has 'no drift' has 'OK \(dryrun\) -- topology consistent with D-052/D-053' -run_dc 1 "DC=vr1-dc1: net selector fails loud (NetBox gap)" vr1-dc1 -has 'no assigned network literals yet' +run_dc 2 "DC=vr1-dc1: PLANES parity guard refuses (diverging D-124 scheme)" vr1-dc1 +has 'PLANES table is DC0-hardcoded' absent 'DO:|WOULD:' # must exit before any MAAS interaction run_dc 1 "DC=bogus: unknown token fails loud" bogus diff --git a/tests/reenroll-hosts/run-tests.sh b/tests/reenroll-hosts/run-tests.sh index f37f9ee..d04bcfe 100644 --- a/tests/reenroll-hosts/run-tests.sh +++ b/tests/reenroll-hosts/run-tests.sh @@ -58,8 +58,8 @@ has 'no enrolled hosts yet' absent 'Current host status' # must exit before report() ever runs -run_dc 1 "DC=vr1-dc1: net selector fails loud first (NetBox gap)" vr1-dc1 -has 'no assigned network literals yet' +run_dc 1 "DC=vr1-dc1: hosts selector fails loud (net arm ratified, D-124)" vr1-dc1 +has 'no enrolled hosts yet' absent 'Current host status' # D-119 REGRESSION GUARD: the bare dcN tokens are RETIRED. 'dc0' used to mean