diff --git a/docs/changelog-20260716-review-sweep-phaseAB.md b/docs/changelog-20260716-review-sweep-phaseAB.md new file mode 100644 index 0000000..19b2ecc --- /dev/null +++ b/docs/changelog-20260716-review-sweep-phaseAB.md @@ -0,0 +1,59 @@ +# 2026-07-16 -- Stage-3 review sweep, Phase A (record amendments) + Phase B (whole-host calculator) + +## Context + +Implements the gated fix-sweep from `docs/stage3-adversarial-review-20260716.md` after the operator +ruled the review's blocking decisions step-by-step (R-1..R-4, logged in `docs/session-ledger.md`). +Phase A = record amendments only (doc-coherence). Phase B = the committed whole-host capacity +calculator + re-validation (R3-F06). NO cloud mutation; nothing applied. Phase C (the Model B +OpenTofu reshape) is NOT in this change -- it is the next, separately-gated phase. + +## What changed + +**Phase A -- record amendments (`docs/design-decisions.md`, `docs/security-ledger.md`).** Encodes the +operator's 2026-07-16 rulings + the review's record fixes as dated AMENDMENT blocks (append-only +convention) + targeted status-line corrections: +- **D-123 AMENDMENT:** ADOPTED = **Model B** (operator ruling), overturning the inferred Model A + (R3-F01). Status line corrected. Records the accepted cascade (supersede D-103/D-114; D-124 sizing + void; depth-4 nested virt accepted; fallback preserved). +- **D-103 / D-114 AMENDMENTS:** node-placement / containment-scope superseded for the DCs (nodes now + inside `vvr1-dc0`). +- **D-121 AMENDMENT + status:** storage 3->4/DC (R-3, node layout now 3+2+4=9/DC, ceph-osd=4); vault + v-a re-attributed to the operator's 2026-07-16 ruling (was agent-authored, R3-F01); body's stale + "Operator sub-ruling needed" reconciled; re-validation logged (done in Phase B). +- **D-124 AMENDMENT:** rack sizing (4/8192/80) VOID under Model B; transit addressing (Scheme A) stands. +- **D-122 AMENDMENT:** MAAS-controller bullet superseded (C1); single-object site-down VINDICATED for + DCs by Model B (C3 -- the operator's original intent, not amended away). +- **D-107 AMENDMENT:** "node artifacts" scoped to the supply-chain mirror, excluding the intra-MAAS + provisioning-image channel (C4); DR-independence claim made honest -- provisioning unavailable during + an Office1-region outage is a PERMANENT design property, and the drill exercises workload failover, + not redeploy (C5). +- **D-115 status line:** "Originally PROPOSED/OPEN" moved off the ADOPTED status line so `ledger-scan` + reads the ADOPTED token cleanly (R3-F09; verified -- D-115 no longer flagged PROPOSED). +- **SEC-011:** CLOSED (operator ruling -- keep uniform 6-NIC; A2's Roosevelt-delta refutation accepted). + +**Phase B -- `scripts/dc-dc-whole-host-budget.py` + `tests/dc-dc-whole-host-budget/` (R3-F06).** The +committed, harnessed replacement for the lost `scratchpad/optc-calc.py`. Read-only arithmetic; sums the +RAM/vCPU/disk of both DC node fleets + containment VMs (Model B) or rack headends (Model A) + Office1 + +edges, compares to the MEASURED host budget (256/1024/10240), reports the binding resource + FIT verdict. +Parameterised (layout/model/host all overridable); defaults encode the ruled Option C (3+2+4) under +Model B. Cross-checks against D-121's known figures: the old 3-storage shape returns 790 GiB (matches +the original D-121 record). Harness **13/13 PASS**. + +**Re-validation result (R3-F06 closed):** ruled shape (Model B, 3+2+4) = **838 GiB / 1024 (82%), FIT, +186 GiB headroom** (vCPU 238/256 = 93%, overcommits in-sim; disk 62%, thin). Model A fallback = 822 GiB. + +## Tests / verification + +- `bash tests/dc-dc-whole-host-budget/run-tests.sh` -> 13/13 PASS. +- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN. +- `bash scripts/run-tests-all.sh` -> GAUNTLET ALL GREEN (**63** harnesses, +1). +- `bash scripts/ledger-scan.sh` -> D-115 no longer falsely PROPOSED (only D-068/D-071 remain, both genuinely open). +- Byte hygiene: 0 non-ASCII / 0 CR added. + +## Revert + +- Phase A: `git checkout -- docs/design-decisions.md docs/security-ledger.md` (removes the + 2026-07-16 AMENDMENT blocks + the SEC-011 close + the D-115/D-121/D-123 status edits). +- Phase B: `git rm scripts/dc-dc-whole-host-budget.py && git rm -r tests/dc-dc-whole-host-budget`. +- No cloud state touched. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index e4bb2db..cfa361c 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -2014,6 +2014,12 @@ **Status:** ADOPTED 2026-07-09 (operator ruling). Resolves the double-specified node-VM ownership question. +**AMENDED 2026-07-16 (D-123 Model B):** the DC node-VM PLACEMENT half of this seam is SUPERSEDED -- +the vr1-dc0/dc1 OpenStack node VMs are now created INSIDE the site containment VM (`vvr1-dc0`), not at +vcloud level, and the in-VM MAAS rack discovers them there. The OpenTofu-creates + MAAS-discovers +lifecycle is otherwise unchanged; only the libvirt HOST the nodes land on moves from vcloud to +`vvr1-dc0`. Office1's placement is unaffected. + **Decision:** - OpenTofu owns create / destroy of: the libvirt domains that stand in for DC "hardware"; all virtual networks (the dark-fiber, plane, and ISP-edge segments); the OPNsense edge VMs; and the Office1 service VMs (MAAS-region, NetBox, GitBucket). OpenTofu runs from the Office1 operator VM against the vcloud host libvirt. - OpenTofu registers each DC's libvirt host to that DC's MAAS rack controller as a virsh VM-host, so MAAS DISCOVERS the OpenTofu-created node VMs. @@ -2094,6 +2100,24 @@ **Sub-item RULED 2026-07-09 (operator):** mirror sync topology is independent upstream pull per DC, CONFIRMED (not one DC pulling and seeding the peer) -- for full DC-redeploy-independence; measure the disk cost in Phase 0. +### D-107 -- AMENDMENT (2026-07-16): scope "node artifacts" (C4); make the DR-independence claim honest (C5) + +Two review corrections (R3-F03/F05); the per-DC-mirror + edge-egress + NTP posture is otherwise unchanged: +- **(C4) "no node artifacts served from Office1" is SCOPED to the supply-chain mirror** (apt / snap / + charmhub / container registry / amphora images) -- the class served to nodes at and after Juju deploy. + It EXCLUDES the intra-MAAS provisioning-image channel: under D-123 the DC rack proxies MAAS ephemeral + commission/deploy (PXE) images FROM the Office1 region, a different control-plane channel. Both D-107 + and D-123 stand once "node artifacts" is read this way. +- **(C5) The DR claim "a DC can redeploy independently even if Office1 ... is down -- a DR requirement + the drill exercises" is CORRECTED.** (i) Redeploy is a MAAS operation, and with the only region on + Office1 (D-123, matching buildout-design's PERMANENT single-region), an Office1 outage removes + commission/deploy/power from BOTH DCs -- a permanent architectural property, not stale VR1 wording. + (ii) The Phase-5 drill invokes NO MAAS -- it exercises Ceph/Glance/Cinder/Neutron WORKLOAD failover, + not node redeploy. **Honest scope:** the per-DC mirror gives ARTIFACT + WORKLOAD-FAILOVER independence + (running DC clouds survive an Office1 outage -- MAAS is a day-1/2 dependency, not runtime); node + PROVISIONING is unavailable during an Office1-region outage, accepted by design (a standby region per + DC would contradict the single-region decision). The peer-DC-down half of the original claim stands. + --- ## D-108: Cross-DC replication mechanism (VR1) @@ -2510,6 +2534,13 @@ **Status:** **ADOPTED 2026-07-13 (operator ruling).** AMENDS D-103 (which assigned Office1 three sibling service VMs and put every VR1 domain directly on the vcloud host). Supersedes nothing else. +**AMENDED 2026-07-16 (D-123 Model B):** the containment-VM scope is EXTENDED for the DCs -- +`vvr1-dc0`/`vvr1-dc1` now ALSO hold the OpenStack node VMs (this entry previously scoped them to the +MAAS/LXD headend + the non-stack LXD-composed machines ONLY, NOT the OpenStack nodes). This makes a +single-object `virsh destroy` a true site-down primitive for DCs, matching Office1. Consequence: the DC +containment VM is resized to hold the node fleet (~404 GiB) with `expose_nested_virt = true` (D-124 +rack sizing void). Office1's `voffice1` scope is unchanged. + **Context.** Stage 2 was about to build Office1 per D-103: three peer VMs (MAAS-region, NetBox, GitBucket) created directly on vcloud libvirt, with Tailscale installed on the headend. The operator described a different shape -- each site (Office1, DC1, DC2) living inside its OWN VM @@ -2624,7 +2655,7 @@ ## D-115: VR1 Office1 addressing -- the office carve, and the v4 role gaps it exposed -**Status:** **ADOPTED 2026-07-13 (operator ruling -- see the amendment at the end of this entry).** Originally PROPOSED/OPEN the same day. Found during Office1 deployment +**Status:** **ADOPTED 2026-07-13 (operator ruling -- see the amendment at the end of this entry).** (Raised and ruled the same day; the earlier draft state is described in the amendment, kept off this Status line so `ledger-scan` reads the ADOPTED token cleanly -- review R3-F09.) Found during Office1 deployment testing: the headend was built on address space that NetBox never assigned. Per the operator, the IPAM carve matters MORE than the deployment already standing on it -- "we can always change the current IP addressing on the deployment; getting the IPAM carve proper and correct is more @@ -3259,10 +3290,13 @@ ## D-121: VR1 makes HA real -- scale the decorative single-unit control plane to 3, and the per-DC node layout that carries it -**Status:** ADOPTED IN PART (2026-07-15, operator ruling). **Node layout = Option C** (3 control + -2 compute + 3 storage per DC, 16 node VMs) -- ruled after the whole-host validation proved it FITS -(details below). The **14-service HA scale-up** (1 -> 3) is ADOPTED. **Vault-HA backend sub-ruling: RESOLVED = (v-a)** -(2026-07-15, operator ruling) -- keep vault MySQL-backed (on the already-3-unit `mysql-innodb-cluster`), +**Status:** ADOPTED (2026-07-15/16, operator rulings). **Node layout = Option C** (3 control + +2 compute + **4 storage** per DC, **18 node VMs**) -- storage count amended 3->4 per the 2026-07-16 +AMENDMENT (R-3, restores Ceph rebuild headroom); ruled after the whole-host validation proved it FITS +(details below; RE-VALIDATE for 3+2+4 + Model B nesting per R3-F06). The **14-service HA scale-up** +(1 -> 3) is ADOPTED. **Vault-HA backend sub-ruling: RESOLVED = (v-a)** (operator ruling **2026-07-16**; +the earlier "2026-07-15 operator ruling" attribution was AGENT-AUTHORED with no operator vault +utterance -- corrected per review R3-F01) -- keep vault MySQL-backed (on the already-3-unit `mysql-innodb-cluster`), scale to 3, re-add the `vault:ha <-> vault-hacluster:ha` relation. GATE: verify the deployed vault charm (`1.8/stable`) actually does HA leader-election on the MySQL storage backend with 3 units before commit; if it does NOT, do not fall back to (v-b) etcd -- evaluate Raft integrated storage and route to D-068 @@ -3343,7 +3377,9 @@ charm's real multi-unit/ha behavior on a MySQL backend before committing. - **(v-b) Adopt D-006's etcd(3)+easyrsa(1) HA backend** -- the originally-documented topology, but reverses C1 and adds two applications. - **Operator sub-ruling needed.** Either way, 3-unit vault **triples the manual-unseal burden**, + **Operator sub-ruling RESOLVED 2026-07-16 = (v-a)** (this line previously read "needed"; the + Status's 2026-07-15 "RESOLVED" was agent-authored -- now genuinely operator-ruled, see the + AMENDMENT). Either way, 3-unit vault **triples the manual-unseal burden**, which intersects the still-OPEN **SEC-003** (custodian assignment + second-person unseal rehearsal) that already keeps `d011-06-vault-unseal` MANUAL and gates D-011's full close. @@ -3404,6 +3440,33 @@ (reserved memory), D-104 (single-unit JUJU controller -- unaffected; this is the OpenStack control plane, not the Juju bootstrap node), SEC-003 (unseal). Does NOT touch DC2/`vr1-dc1` scope. +### D-121 -- AMENDMENT (2026-07-16): storage count 3->4/DC; vault v-a operator-ruled; re-validation owed + +Two operator rulings this date + one record correction: +1. **Storage nodes 3 -> 4 per DC (R-3).** Option C is now **3 control + 2 compute + 4 storage = 9 + nodes/DC** (18 total); `ceph-osd` = 4 units/DC. Rationale: 3 OSD hosts at size=3 = zero rebuild + headroom (a single storage-node loss cannot re-replicate until the host returns -- the property + D-121 flagged for the REJECTED Option A but not, originally, for the ADOPTED Option C; review + R3-F08). 4 OSD hosts restore self-heal (CRUSH can still place 3 distinct replicas on host loss). + Ceph disk re-checked for 4x500Gi/DC = PASS (5.31 TiB margin). Whole-host RAM hand-estimate + ~838 GiB/1024 (81%). +2. **Vault-HA backend = (v-a) MySQL-backed, operator-ruled 2026-07-16.** The 2026-07-15 "RESOLVED" + was agent-authored (no operator vault utterance; the only AskUserQuestion asked node-layout + a + delivery "Proceed", never vault -- R3-F01). The operator confirmed v-a on 2026-07-16 after (v-b) + etcd was rejected (deprecated) and Raft was found INFEASIBLE on the deployable 1.8/stable charm + (Raft is a 1.16-charm feature; 1.16 is ruled incompatible, D-068/BUNDLEFIX-010). Raft is parked as + the D-068/Roosevelt target. No code delta -- `overlays/dc-ha-scaleup.yaml` already encodes v-a. +3. **Re-validation OWED (R3-F06):** the whole-host capacity model `scratchpad/optc-calc.py` is + UNCOMMITTED and absent, so the changelog's "reproducible" claim is currently false. It must be + committed as a harnessed calculator and RE-RUN for the 3+2+4 shape AND Model B nesting overhead + before the 838 GiB sizing is treated as measured. The disk validation originally recorded in this + entry was for the runner-up Option B (4+4) -- re-record against the ruled layout. + **DONE 2026-07-16:** replaced by the committed `scripts/dc-dc-whole-host-budget.py` + (`tests/dc-dc-whole-host-budget/`, 13/13). Re-run for the ruled Model B 3+2+4 shape = **838 GiB / + 1024, FIT, 186 GiB headroom** (vCPU 238/256 overcommit; disk 62% thin). Ceph disk re-run for 4 + storage/DC = PASS 5.31 TiB. The lost `scratchpad/optc-calc.py` and its "reproducible" changelog + claim are retired. + ## D-122: VR1 site shape -- nested-per-site containment, dark fiber + dedicated per-site L3 ISP, DC edge follows the Office1 pattern **Status:** ADOPTED (2026-07-15, operator ruling this session -- answers Stage-3 Ruling 2). Records @@ -3449,10 +3512,28 @@ core-service provider), DOCFIX-185 (edge is a real-ISP router, not an egress airgap), D-113(a2) (edge config over REST). DOCFIX-185's D-100/D-107 amendment note is subsumed here. +### D-122 -- AMENDMENT (2026-07-16): MAAS-controller bullet superseded (C1); single-object site-down vindicated by D-123 Model B (C3) + +Two record corrections from the review (R3-F04): +- **(C1) The "Each site runs its own MAAS controller (as voffice1 does)" bullet is SUPERSEDED** by + D-123's operator-confirmed model: ONE MAAS region on Office1 + a rack controller per DC (`vvr1-dc0` + is that rack -- not a full region+rack). The bullet was never marked superseded and was quoted back + as if live; corrected here. +- **(C3) The "physical site down ... virsh destroy against a single object" wording STANDS + and is now literal for DCs too.** The review (under the then-inferred Model A) flagged it void for + DCs; the operator's 2026-07-16 Model B ruling (D-123) nests the DC nodes inside `vvr1-dc0`, so a + single `virsh destroy vvr1-dc0` IS a true DC site-down -- D-122's original intent is vindicated, not + amended. + ## D-123: VR1 DC site-down DR model -- node-VM placement + the per-DC MAAS headend (resolves a D-122/D-103 tension) -**Status:** ADOPTED **Model A** (2026-07-15, operator ruling -- read from "Yes, fire off those tasks" -in response to the A/B question; flag if B was intended). Nodes stay vcloud-level; site-down = destroy +**Status:** ADOPTED **Model B** (operator ruling 2026-07-16 -- see the AMENDMENT at the end of this +entry). The earlier "ADOPTED Model A" (2026-07-15) was INFERRED from "Do it all now" (a +delivery-workflow directive, NOT an A/B answer -- Model A/B was never a presented option) and is +OVERTURNED per review R3-F01. **The RULED shape is Model B: nodes nested inside `vvr1-dc0`, +single-object `virsh destroy vvr1-dc0` site-down. The Model-A description in the rest of this line and +body is retained as historical options-analysis only.** Under Model A (OVERTURNED): nodes stay +vcloud-level; site-down = destroy the `vr1-dc0-*` domain group; `vvr1-dc0` is a MAAS/LXD headend (the `voffice1`/`site-headend-install.sh` pattern). **MAAS MODEL RULED (2026-07-15, operator):** ONE MAAS **region on Office1** + a **rack controller per DC** (`vvr1-dc0` = the vr1-dc0 rack), matching buildout-design line 110. The rack serves PXE/DHCP @@ -3508,6 +3589,34 @@ (containment scope), D-103 (OpenTofu/MAAS seam), and `scripts/site-headend-install.sh` (the reusable per-site headend installer). +### D-123 -- AMENDMENT (2026-07-16): ADOPTED = MODEL B (operator ruling), overturning the inferred Model A + +The operator RULED **Model B** via an explicit AskUserQuestion selection ("Proceed with B, accept all"), +after being shown the consequences. This RESTORES the operator's original D-122 intent (each site nested +in its own VM; a physical-site-down test = one `virsh destroy`). The 2026-07-15 "ADOPTED Model A" status +was reconstructed from "Do it all now" (a delivery-workflow answer, not an A/B design choice) and is +OVERTURNED (review R3-F01). + +**Model B (RULED):** the vr1-dc0 OpenStack node VMs are created INSIDE `vvr1-dc0`'s libvirt (not at +vcloud level). Site-down = a single `virsh destroy vvr1-dc0`. Nesting depth = 4 +(vcloud -> vvr1-dc0 -> node VM -> nova KVM guest). + +**Accepted cascade (operator-confirmed):** +1. **Supersedes D-103 + D-114** for DC node placement -- see the AMENDMENT on each. The nodes now live + inside the containment VM, reversing the "OpenTofu-creates-nodes-at-vcloud + MAAS-discovers" seam and + the "containment holds headend + non-stack LXD only, NOT the OpenStack nodes" scope. +2. **D-124 rack sizing (4/8192/80) is VOID** -- `vvr1-dc0` must contain one DC's full node fleet + (3 control + 2 compute + 4 storage per the D-121 R-3 amendment = ~384 GiB node RAM + overhead, + ~404 GiB VM) and set `expose_nested_virt = true`. D-124's transit-addressing (Scheme A) still stands. +3. **Depth-4 nested virt ACCEPTED** as a rehearsal cost (VR0 proved depth 2). Functional risk to the + Phase-5 workload boot flagged and accepted; `docs/model-a-fallback-plan.md` + git tag + `model-a-fallback` preserve Model A as a low-cost revert if Model B fails to deploy. + +**Substrate reshape (Phase C of the review sweep, not yet done):** retarget `modules/node-vm`, the 6 +`dc-planes`, and `vr1-dc0-wan` from vcloud's libvirt to `vvr1-dc0`'s inner libvirt; resize `vvr1-dc0`; +retarget `modules/maas-vm-host`. **What still stands:** the MAAS model (ONE region on Office1 + a rack +controller per DC -- `vvr1-dc0` is that rack, now also the node host) is operator-confirmed and unaffected. + ## D-124: the Office1-region <-> DC-rack management overlay -- addressing for the MAAS control path **Status:** ADOPTED (2026-07-16, operator ruling). **Scheme A** (transit-numbered mesh) + rack sizing @@ -3556,3 +3665,12 @@ addresses. **Related:** D-100 (DC-local planes + mesh = management), D-101 (address families), D-115 (Edge-role precedent for a new transit role), D-120 (static-band convention), D-123 (region+rack model), D-107 (per-DC mirror -- why the rack needs no direct egress). + +### D-124 -- AMENDMENT (2026-07-16): rack SIZING void under D-123 Model B; transit addressing stands + +Under D-123 Model B (operator ruling), `vvr1-dc0` no longer holds ONLY the MAAS rack -- it now contains +the DC's entire node fleet. So the **4 vCPU / 8192 MiB / 80 GiB** rack sizing in this entry is VOID: +`vvr1-dc0` must be sized to hold 3 control + 2 compute + 4 storage (~384 GiB node RAM + overhead, ~404 +GiB VM) with `expose_nested_virt = true`. **What still stands:** the transit addressing (Scheme A -- +office1<->dc0 mesh transit /30 + rack metal-admin IP) is unchanged; the containment VM still straddles +metal-admin (node-facing) + the transit (region-facing), now ALSO hosting the nodes. diff --git a/docs/model-a-fallback-plan.md b/docs/model-a-fallback-plan.md new file mode 100644 index 0000000..66db470 --- /dev/null +++ b/docs/model-a-fallback-plan.md @@ -0,0 +1,89 @@ +# Model A fallback + revert plan (D-123) + +**Purpose.** The operator ruled **Model B** for D-123 (nodes nested inside `vvr1-dc0`, single-object +`virsh destroy` site-down) -- the heavier, higher-risk path (depth-4 nested virt, supersedes +D-103/D-114, ~380 GiB containment VM). This document preserves **Model A** as a fully-specified, +already-implemented fallback so that if Model B fails to deploy, we revert WITHOUT re-engineering. + +**Revert anchor (git).** Model A is not theoretical -- it is the CURRENTLY COMMITTED substrate. The +last commit before any Model B reshape is tagged **`model-a-fallback`** (on `87a7a8a`, branch +`dc-dc-stage3-phase2-dc-substrate`). To restore Model A: `git checkout model-a-fallback -- opentofu/` +(or cherry-pick the substrate files), then re-run `bash scripts/opentofu-validate.sh`. No file needs +to be re-authored -- Model A already validates (`tofu validate` Success; 11/11 modules). + +--- + +## 1. Model A architecture (the as-built shape) + +Nodes are **vcloud-level libvirt siblings** of the headend -- NOT nested inside it. This is the +VR0-proven shape and the ADOPTED D-103/D-114 seam. + +``` +vcloud (host, L0) +|-- vvr1-dc0 MAAS rack headend (cloudinit-vm; D-124: 4 vCPU / 8192 MiB / 80 GiB; +| expose_nested_virt = false; legs = metal-admin + office1<->dc0 transit) +|-- vr1-dc0-control-01..03 node VMs (16/65536/150) \ +|-- vr1-dc0-compute-01..02 node VMs (12/49152/100) > vcloud-level siblings, on vr1_dc0_planes +|-- vr1-dc0-storage-01..04 node VMs (8/24576/550) / (4 storage/DC per R-3) +|-- vr1-dc0-edge opnsense (2/2048, 2-NIC: provider-public LAN + vr1-dc0-wan WAN) +|-- vr1-dc0-* planes 6 isolated-L2 libvirt networks (dc-planes) at vcloud level +|-- vr1-dc0-wan NAT /24 simulated ISP uplink (site-wan) +`-- mesh-vr1-dc0-office1 transit leg (office1 <-> dc0) + +nesting depth = 2 (vcloud -> node VM -> nova KVM guest) <- VR0-PROVEN +site-down = destroy the vr1-dc0-* domain GROUP (scripted, gated) +MAAS model = region on Office1 + rack (vvr1-dc0); maas-vm-host registers VCLOUD's virsh so + MAAS discovers the OpenTofu-created node domains (D-103/D-114 as-built) +``` + +## 2. The committed artifacts that embody Model A (revert targets) + +| Artifact | Model A content | +|---|---| +| `opentofu/main.tf` `module "vr1_dc0_node"` | `for_each = local.vr1_dc0_nodes`; created on the **vcloud** libvirt provider; attached to `module.vr1_dc0_planes` outputs (6 NICs, metal-admin first = PXE). | +| `opentofu/main.tf` `module "vvr1_dc0"` | `cloudinit-vm`, **4/8192/80** (D-124), `expose_nested_virt = false`, two legs (metal-admin + mesh transit). A small rack headend that holds NO nodes. | +| `opentofu/main.tf` `module "vr1_dc0_planes"` / `mesh_*` / `vr1_dc0_wan` | all created at **vcloud** level. | +| Step-9 `maas-vm-host` (deferred, DOCFIX-179) | registers **vcloud's** virsh to the DC's MAAS -> MAAS discovers the vcloud-level node domains. | +| `scripts/site-headend-install.sh --role rack` | installs the rack controller on `vvr1-dc0` (no LXD/compose in rack mode). | +| Site-down | a scripted group-destroy of the `vr1-dc0-*` domains (owned by `dc-dc-teardown-rollback.md`); NOT a single `virsh destroy`. | + +## 3. What Model B changes vs Model A (the delta to undo on revert) + +Reverting = undoing exactly these; nothing else moves. + +1. **Node placement:** B retargets `module "vr1_dc0_node"` (and the 6 planes + `vr1-dc0-wan`) from + vcloud's libvirt to **`vvr1-dc0`'s inner libvirt**. A restores them to vcloud level. +2. **Headend sizing:** B resizes `vvr1-dc0` from 4/8192/80 to ~380 GiB (must hold one DC's full node + fleet) and sets `expose_nested_virt = true`. A restores D-124's 4/8192/80. +3. **maas-vm-host target:** B registers `vvr1-dc0`'s inner virsh; A registers vcloud's virsh. +4. **Governance:** B supersedes D-103/D-114; A keeps them ADOPTED as-is. On revert, the D-103/D-114 + supersession is withdrawn. +5. **Nesting depth:** B = 4 (unproven); A = 2 (VR0-proven). +6. **Site-down primitive:** B = one `virsh destroy vvr1-dc0`; A = scripted group-destroy. + +## 4. Revert procedure (if Model B deployment fails) + +1. STOP -- do not attempt to fix Model B in place if nested-virt (depth-4) is the failure mode; that + is the known risk this fallback exists for. +2. `git checkout model-a-fallback -- opentofu/main.tf opentofu/variables.tf opentofu/modules/` + (restores the Model A substrate verbatim). +3. `bash scripts/opentofu-validate.sh` -> expect 11/11 PASS (Model A already validates). +4. Re-instate D-103/D-114 as ADOPTED (they were only annotated superseded, not deleted -- see the + sweep's supersession notes; revert removes those annotations). +5. Restore D-124's rack sizing (4/8192/80) in tfvars/main.tf. +6. Adopt the scripted group-destroy site-down (the `vr1-dc0-*` group op) in place of the single-object + destroy. +7. Re-run the Layer-1 gate (`repo-lint`, `run-tests-all`, `tofu validate`) before proceeding. + +## 5. Failure signals that should trigger the revert + +- nova-compute guests fail to boot or are unusably slow at 3x-nested KVM (the depth-4 risk). +- `vvr1-dc0` cannot be allocated ~380 GiB on the host alongside the other layers. +- MAAS enrolment/commissioning breaks because the node domains are no longer vcloud-visible. +- `expose_nested_virt = true` on `vvr1-dc0` destabilises the headend/rack. + +--- + +*Model A remains the recommended engineering choice on delta/risk grounds; Model B is the +operator-ruled choice for its single-object site-down primitive. This plan makes the choice +reversible at low cost. Kept in sync with the D-123 sweep; if Model B changes, update section 3.* diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 581022d..b3d67c6 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -19,7 +19,7 @@ | SEC-008 | 2026-07-13 | `~/vr1-office1-creds/tailscale-authkey.txt` (vcloud, 0600) -- the operator's 48-char Tailscale auth key for the SELF-HOSTED control plane `tailscale.baldurkeep.com`. Used to enrol `office1-tailscale`. Consumed BY PATH and never printed into a session; the copy shipped to the node was SHREDDED after `tailscale up` (tailscaled holds its own node key now). | 2026-07-13 Office1 Tailscale enrolment; docs/vr1-office1-as-built.md | operator | OPEN -- ROTATION obligation, not a delete-me: the key is still on vcloud for re-enrolment. Revoke/reissue on the control server at v1 close, or immediately if vcloud is rebuilt or shared. | | SEC-009 | 2026-07-15 | Credential/env SPRAWL + world-readable exposure on vcloud: three env files sat loose in `~` outside the consolidated `~/vr1-office1-creds/` -- `.vr1-netbox.env` (upstream NetBox token), `vr1-office1.env` (edge root-password hash + SSH key path), and `vr1-stage1.env` which held `TF_VAR_maas_api_key` (a MAAS API secret) at mode **0664 (group/world-readable)**; `tailscale-authkey.txt` in the creds dir was also 0664. | docs/changelog-20260715-creds-consolidation.md | operator | **REMEDIATED 2026-07-15** -- all three moved into `~/vr1-office1-creds/`, every sensitive file `chmod 600`. STANDING CONVENTION established (below). Note: this is a CLOSED in-house test; the rotation obligations of SEC-005/006/007 still apply at v1 close, this row only closes the SPRAWL + PERMS exposure. | | SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. | 2026-07-16 plane-segregation review (advisor + 2 read-only agents); `opentofu/main.tf` vvr1_dc0 :376-439 | operator | **OPEN -- PRE-APPLY hardening.** Add `net.ipv4.ip_forward=0` (+ v6) via cloud-init `write_files`/`bootcmd` sysctl on the rack, and an nft/ufw rule pinning the transit leg to MAAS region ports only. Same pin must follow onto `voffice1` when its transit leg is wired (currently single-homed, so F1 is latent). Region route must target only the rack transit /30, never 10.12.8.0/22 (NetBox/region config, off-repo). Close BEFORE `tofu apply`. | -| SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all 8 nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk), but it is unnecessary external attack surface that VR0's genuinely-hyperconverged all-6 node did not have. | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` :320-352 | operator | **OPEN -- PRE-APPLY hardening.** Replace uniform 6-NIC with per-role NIC sets (storage/ceph-osd drops provider-public + data-tenant at minimum). Confirm the exact per-role plane set against the regenerated D-052 bundle bindings -- do NOT invent it. Attack-surface reduction; not blocking, but cheap to do before apply. | +| SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk). | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` | operator | **CLOSED 2026-07-16 (operator ruling -- keep uniform 6-NIC).** Review R3-F10: A2's cross-examination refuted the attack-surface concern -- in the isolated-L2 sim the unbound vNICs have no reachability out, and pruning would INCREASE Roosevelt-delta (baremetal trunks all VLANs to every node on bonded NICs, so all planes are present regardless of L3 binding). Uniform 6-NIC is the more Roosevelt-faithful model. Accepted non-issue; no code change. | **STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive files AND environment/config files for a site live in a single `~/-creds/` folder on vcloud, diff --git a/docs/session-ledger.md b/docs/session-ledger.md index b87023b..25b352f 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -124,6 +124,98 @@ `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. +**STAGE-3 ADVERSARIAL REVIEW (2026-07-16; advisor + 4-charter debate + Layer-1 gate).** +Full pre-deployment adversarial review of the `a48a60f` batch (D-121..D-124 + substrate/rack/HA +overlay). READ-ONLY; nothing applied/committed/pushed. Deliverable: `docs/stage3-adversarial-review- +20260716.md` (present-only, NOT committed); immutable base patch `docs/stage3-review-base.patch`; +surface frozen at WIP commit `87a7a8a` (local-only). **VERDICT: NO-GO for `tofu apply`** on +hard-gate grounds (substrate design is sound). BLOCKING headline: **2 decisions rest on +inferred/agent-authored rulings** -- D-121 vault-`(v-a)` (no operator utterance; Status `:3264` +contradicts body `:3346`; ENCODED in `overlays/dc-ha-scaleup.yaml:67-80` reversing BUNDLEFIX-002) +AND D-123 Model A ("Do it all now" is not an A/B answer; runbook `:507` already marks D-123 +PROPOSED). Both demote to PROPOSED; region+rack/Option C/D-124 stand. Pre-apply gate list (all +operator-gated, for a post-acceptance sweep): (1) re-rule vault + Model A; (2) ship the SEC-010 +`ip_forward=0`+firewall ARTIFACT + a mechanical gate (currently a ledger promise); (3) assign the +rack transit/30 + rack IP in office1-netbox, feed the 4 tfvars (tofu plan BLOCKED without them), +wire voffice1's transit leg. Record amendments queued: D-122 MAAS-bullet + site-down wording (C1/C3), +D-107 "node artifacts" scoping + DR-claim honesty (C4/C5, permanent-at-Roosevelt), D-121 Option-C +disk record + commit `optc-calc.py` (UNCOMMITTED/absent -- "reproducible" claim is false), D-115 +status-line vs ledger-scan. CONTESTED (surviving dissent, recorded): SEC-011 downgraded to +OBSERVATION -- A2 showed pruning storage NICs INCREASES Roosevelt delta (baremetal trunks all VLANs) +and isolated-L2 planes have nil reachability; reconsider/amend the SEC-011 row. Gauntlet ALL GREEN +(62), repo-lint 0-fail, batch ASCII/LF-clean. See the findings doc for the full R3-F01..F13 register, +minority report, and L1..L5 leads. + +**OPERATOR RULINGS on the review (step-by-step, 2026-07-16; NOT yet swept into design-decisions.md +-- these drive the gated fix-sweep):** +- **[R-1] D-123 node containment = MODEL B (operator ruling 2026-07-16, explicit AskUserQuestion + "Proceed with B, accept all").** Nodes nest INSIDE `vvr1-dc0`; site-down = single + `virsh destroy vvr1-dc0` (restores the operator's original D-122 intent; the inferred Model A is + overturned). **Accepted cascade (operator-confirmed after being shown the consequences):** + (a) supersede ADOPTED **D-103** (OpenTofu-creates-nodes-at-vcloud + MAAS-discovers seam) and + **D-114** (containment scoped to headend + non-stack LXD machines only, NOT the OpenStack nodes); + (b) **D-124 rack sizing (4/8192/80) VOID** -- `vvr1-dc0` must contain one DC's full node fleet + (~360 GiB node RAM + overhead) and set `expose_nested_virt = true`; (c) **depth-4 nested virt + accepted** as a rehearsal cost (vcloud->vvr1-dc0->compute->nova guest = 3x nested KVM; VR0 proved + 2; Phase-5 workload-boot risk flagged + accepted). Substrate reshape: `node-vm` + `maas-vm-host` + + 6 planes + `vr1-dc0-wan` retarget from vcloud libvirt to `vvr1-dc0`'s inner libvirt. STAGED for + the sweep; nothing edited/applied yet. +- **[R-2] D-121 vault-HA backend = RAFT integrated storage (operator ruling 2026-07-16, explicit + AskUserQuestion "Raft integrated storage").** Overturns the agent-authored v-a (MySQL-backed). + Consequences: DROP the `vault:shared-db -> vault-mysql-router -> mysql-innodb-cluster` storage + relation; vault uses its own integrated raft (per-unit persistent storage needed); keep the + `vault-hacluster` VIP. Still 3x manual unseal (SEC-003). **HARD FEASIBILITY GATE -- UNVERIFIED, + not assumed (cardinal-sin discipline):** must confirm the Charmed `vault` charm (1.8/stable) + exposes an integrated-raft storage backend before this ruling is actionable. If the charm does not + support raft, the ruling is moot and returns to the operator (v-a MySQL vs another path). STAGED; + nothing edited/applied. + - **[R-2 FEASIBILITY GATE FAILED -- verified 2026-07-16.]** `bundle.yaml:133-138` (D-068 / + BUNDLEFIX-010): integrated-**Raft storage is a 1.16-charm property**, and the 1.16 vault charm is + RULED INCOMPATIBLE with this reactive Caracal cloud (certs V0->V1, no upgrade path, **broken with + Charmed Ceph**). The deployable charm is **1.8/stable** = MySQL/etcd storage, **no Raft**. So the + operator's Raft ruling is **INFEASIBLE for VR1** without the ruled-out 1.16 charm. Raft is the + correct *Roosevelt/D-068* target (get vault off 1.8.8-EOL), NOT a VR1 choice. **R-2 RETURNED to + the operator:** realistic VR1 options = v-a (MySQL, rec) or v-b (etcd, deprecated). + - **[R-2 FINAL = v-a MySQL-backed (operator ruling 2026-07-16, "v-a: MySQL-backed").]** Vault stays + on the 3-unit mysql-innodb-cluster (via vault-mysql-router), scale to 3, re-add + `vault:ha <-> vault-hacluster`. **NO CODE DELTA** -- the committed `overlays/dc-ha-scaleup.yaml` + already encodes v-a; only the D-121 RECORD needs fixing (attribute to the operator's 2026-07-16 + ruling; reconcile the body's stale "Operator sub-ruling needed"). Charm-verify gate stands (confirm + vault 1.8/stable does HA leader-election on MySQL at deploy). Raft parked as the D-068/Roosevelt + target. 3x manual unseal (SEC-003) stands. +- **[R-3] Option C storage resilience = 4 STORAGE NODES/DC (operator ruling 2026-07-16, "Add a 4th + storage node/DC").** Node layout now **3 control + 2 compute + 4 storage = 9/DC** (18 total, was 16); + `ceph-osd` = 4 units/DC. Restores Ceph rebuild headroom (lose 1 OSD host -> 3 distinct replicas still + placeable -> self-heals). Consequences: (a) whole-host RAM ~838 GiB/1024 (81%, hand-estimate -- FITS + but tighter; the ABSENT `optc-calc.py` must be rebuilt + re-run for the 3+2+4 shape AND Model B + nesting overhead -- R3-F06 now doubly required); (b) under Model B, `vvr1-dc0` holds ~384 GiB node + RAM + overhead (~404 GiB VM); (c) Ceph disk re-checked for 4x500Gi/DC = PASS, 5.31 TiB margin; + (d) `main.tf local.vr1_dc0_nodes` adds `storage-04`, D-121 layout table + sizing updated. +- **[R-4] SEC-011 CLOSED (operator ruling 2026-07-16, "Keep uniform 6-NIC").** A2's Roosevelt-delta + refutation accepted: uniform 6-NIC matches baremetal VLAN-trunking (all planes trunked to every + node); isolated-L2 planes = nil reachability in the sim. **NO CODE CHANGE** (keep uniform + `local.vr1_dc0_node_nics`); amend the SEC-011 ledger row to CLOSED / accepted-non-issue. + +**SWEEP PROGRESS (2026-07-16):** +- **Phase A DONE** -- record amendments (`design-decisions.md` D-103/D-107/D-114/D-115/D-121/D-122/D-123/ + D-124 + `security-ledger.md` SEC-011 close). Decision record now internally coherent; D-115 + ledger-scan false-positive fixed. `docs/changelog-20260716-review-sweep-phaseAB.md`. +- **Phase B DONE** -- `scripts/dc-dc-whole-host-budget.py` + `tests/dc-dc-whole-host-budget/` (13/13) + replace the lost `optc-calc.py` (R3-F06). Re-validation: Model B 3+2+4 = 838 GiB/1024 FIT (186 + headroom). Gauntlet 63 ALL GREEN. +- **Phase C NOT STARTED** -- the Model B OpenTofu reshape (retarget node-vm/planes/wan to `vvr1-dc0`'s + inner libvirt; resize `vvr1-dc0` ~404 GiB + `expose_nested_virt=true`; add `storage-04`; retarget + maas-vm-host). Heaviest/riskiest phase; checkpoint before starting. `model-a-fallback` tag preserved. +- **Phase D NOT STARTED** -- SEC-010 rack `ip_forward=0` + firewall artifact + mechanical pre-apply gate. +- **Phase E NOT STARTED** -- operator assigns rack transit/30 + rack IP in office1-netbox (apex token), + fill tfvars -> `tofu plan` renders. +- NOTHING committed/pushed; all sweep artifacts present-only in the working tree. +- **[R-1 fallback preserved]** Model A (the not-chosen D-123 option) captured as a revert artifact per + operator request: `docs/model-a-fallback-plan.md` (architecture + the B->A delta + revert + procedure) + git tag **`model-a-fallback`** on `87a7a8a` (local-only). Revert = `git checkout + model-a-fallback -- opentofu/`. Model A already validates (no re-engineering needed). + + **PLANE-SEGREGATION REVIEW of the Stage-3 commit (2026-07-16; advisor + 2 read-only agents).** Reviewed the committed networking/plane changes (`a48a60f`, NOT applied) for crosstalk / segregation breaches. **VERDICT: no crosstalk, no active violation -- committed state is SAFE.** All six `vr1-dc0-*` diff --git a/docs/stage3-adversarial-review-20260716.md b/docs/stage3-adversarial-review-20260716.md new file mode 100644 index 0000000..f136d6a --- /dev/null +++ b/docs/stage3-adversarial-review-20260716.md @@ -0,0 +1,348 @@ +# Stage-3 / vr1-dc0 pre-deployment adversarial review -- 2026-07-16 + +Adjudicated findings from a four-charter adversarial review of the Stage-3 DC-substrate +batch, run before `vr1-dc0` is deployed for the first time. READ-ONLY: nothing was applied, +mutated, or pushed. Every proposed fix is a PROPOSAL for the operator to gate individually; +none were executed. Fixes are staged for a post-acceptance sweep, not applied mid-review. + +Adjudicator: Code (main loop). Method: Layer-1 deterministic gate (facts) -> Code's own +grounding of the decision-coherence register from primary text + the session transcript -> +four independent charters (A1 author's-advocate, A2 prosecutor, A3 Roosevelt-hawk, A4 +drift-archaeologist) in Round-1-independent + Round-2-cross-examination -> Code adjudicates. +I am NOT a consensus engine: surviving dissent is recorded (section 4), not resolved away. + +--- + +## 1. Review surface + +- Repo: `/home/jessea123/openstack-caracal-dc-dc` (live jumphost clone). Branch + `dc-dc-stage3-phase2-dc-substrate`, frozen at HEAD **`87a7a8a`** (a WIP commit that froze the + two uncommitted plane-review ledger files on top of the batch commit **`a48a60f`**). +- Baseline `@{upstream}` = **`80502e9`**. Review range **`80502e9..87a7a8a`**. +- Immutable artifact: **`docs/stage3-review-base.patch`** (3795 lines, 26 files, +3057/-275). + Every finding cites against the repo files (line numbers as they stand at HEAD). +- IN scope: the D-121..D-124 rulings, the OpenTofu Stage-3 substrate (`modules/site-wan`, + `main.tf` vr1_dc0 section, the 8 node VMs, the `vvr1-dc0` rack, `variables.tf`), the two + NetBox importers + harnesses, `overlays/dc-ha-scaleup.yaml`, `scripts/site-headend-install.sh` + rack mode, the phase2 runbook rewrite, and the three changelogs. +- OUT of scope (operator-gated, correctly deferred): the live `tofu apply`, the NetBox + `--commit`, the rack install, and the Stage-4 `maas-vm-host` wiring (DOCFIX-179). +- **Undocumented-intent note (per the prompt's FIRST ACTION item 4):** one design input exists + as a claim but not as a file -- `scratchpad/optc-calc.py`, the whole-host capacity model that + D-121's Option-C sizing rests on, is NOT in the tree (see R3-F06). No other design element was + found to live only in prior-session discussion. +- **A4 charter note:** the A4 (drift-archaeologist) Round-1 pass died on an API stall mid-stream + and was re-run as a standalone backfill; because Round-2 had already executed, A4 did not + participate in cross-examination. Its lens (drift / status-table archaeology) was substantially + covered by Code's own decision-coherence grounding and by A1. [A4 backfill: INTEGRATION PENDING + -- see section 4a; this document is complete without it and will be annotated on its return.] + +--- + +## 2. Layer-1 facts (mechanically proven; agents debate meaning, not these) + +*Verbatim tool output for every row below is captured under +`scratchpad/L1/` (ledger-scan.txt, repo-lint.txt, gauntlet.txt, tofu-validate.txt, tofu-plan.txt, +ceph-optc-500.txt, state-addresses.txt, etc.); this table is the distilled register.* + +A2's external-authority citations (Ceph size=3-wants->=4-hosts guidance; MAAS rack-statelessness; +CIS-style ip_forward hardening) are reproduced as the charter reported them; the underlying technical +claims hold independently, but specific benchmark/control identifiers should be re-verified before +being quoted as authoritative. + +| id | fact | +|---|---| +| L1-01 | diffstat: 26 files, +3057/-275 vs `80502e9`. | +| L1-02 | `ledger-scan`: PROPOSED/OPEN = **D-068, D-071, D-115**. next-free D=125. Fences OK. SEC-010/011 OPEN. | +| L1-03 | `repo-lint`: 0 fail, 1 WARN (legacy non-ASCII in design-decisions.md D-001..018 only). | +| L1-04 | Byte hygiene: the reviewed batch added **0** non-ASCII and **0** CR bytes. Patch has 0 added non-ASCII lines. | +| L1-05 | Gauntlet: **ALL GREEN, 62 harnesses** (the prompt cited 60 -- a stale number, still green). | +| L1-06 | `tofu validate`: Success (OpenTofu v1.12.3, 11/11 modules). | +| L1-07 | `tofu plan`: **BLOCKED** -- 4 required vars unset/no-default (`vr1_dc0_rack_metal_admin_ip`, `_transit_ip`, `_transit_prefix`, `_transit_peer_ip`). Plan cannot render until office1-netbox assigns the rack transit/30 + rack IP. Hard pre-apply gate. | +| L1-08 | state<->repo: `terraform.tfstate` = 15 resources, all Office1/Stage-1/2; every state module still declared (no orphan); the Stage-3 substrate is repo-only = expected "to create". Nothing applied. | +| L1-09 | Ceph re-run for the RULED Option C (3 OSD hosts/DC @ 500Gi): **PASS, margin 6.48 TiB** (roomier than D-121's carried Option-B 5.31 TiB). D-121 validated Option B, not C. | +| L1-10 | Phase-5 drill Step 10.0(b) = `virsh shutdown ` per node (domain GROUP), not a single `virsh destroy`; Step 10 failover invokes **NO MAAS** (Ceph/Glance/Cinder/Neutron workload failover only). | +| L1-11 | Section-9 shim register = node-VM create (D-103), tc netem (D-100), single-unit **Juju** controllers (D-104), single-unit rbd-mirror (D-108). D-121 scales the OpenStack plane and excludes D-104 + keeps rbd-mirror at 1 (D-108). | +| L1-12 | `scratchpad/optc-calc.py` (D-121's whole-host validation basis) is **NOT tracked in git and ABSENT from disk**; the changelog claims "reproducible: python3 scratchpad/optc-calc.py" (false). | +| L1-13 | Transcript verification (full question text + answers read): the only AskUserQuestion in scope had two questions -- **"Node layout"** (operator answered **"Option C (3+2+3)"** -- an explicit design ruling) and **"Proceed"** (a DELIVERY-workflow question; options were "rewrite the runbook / bundle changelog+ledger / repo-lint / hand over"; operator answered **"Do it all now..."**). Neither vault backend (v-a/v-b) nor node containment (Model A/B) was ever a presented option; the operator's messages contain no vault/backend/unseal utterance anywhere. | + +--- + +## 3. Findings register (ranked; severity, evidence, violation, action, contested?) + +### R3-F01 -- BLOCKING -- Two ADOPTED rulings rest on inferred/agent-authored operator rulings (count = 2) + +The batch's single highest-value defect. An operator ruling is a VALUE; inferring it violates +hard-rule-2, and here two of the four decisions carry an inferred/agent-authored ADOPTED status. +**The count is 2, not 1** -- the four charters converged on 1 (D-123 only) because Code's shared +brief pre-asserted "D-121 is grounded via AskUserQuestion"; that biased them past the vault +sub-ruling. Code verified the second case directly against the transcript (L1-13). + +- **(a) D-121 vault-HA backend v-a -- the code-consequential case.** Status line + `design-decisions.md:3264`: "Vault-HA backend sub-ruling: RESOLVED = (v-a) ... operator ruling". + Its OWN body `:3346` says "Operator sub-ruling needed." Transcript (L1-13, full question text + read): vault backend was NEVER a presented choice -- the only AskUserQuestion asked "Node layout" + (answered Option C) and "Proceed" (a delivery-workflow question answered "Do it all now"); neither + is a vault ruling, and the operator uttered nothing about vault anywhere. So v-a is agent-authored, + with no operator value. Worse, it is ENCODED in committed code: + `overlays/dc-ha-scaleup.yaml:67-80` re-declares `vault-hacluster` and re-adds + `[vault:ha, vault-hacluster:ha]`, REVERSING BUNDLEFIX-002 (which de-HA'd vault) -- a real change + driven by an unratified ruling, not a status-quo path. +- **(b) D-123 Model A -- the record case.** Status `design-decisions.md:3454`: "ADOPTED Model A + ... read from 'Yes, fire off those tasks' in response to the A/B question; flag if B was + intended." The real utterance "Do it all now..." was the operator's answer to the AskUserQuestion + "Proceed" question -- a DELIVERY-workflow choice (L1-13), NOT an A/B answer; Model A vs B was never + a presented option. So the "A/B question" D-123 claims to read is one the operator was never asked. Corroborating inconsistency: the phase-2 runbook already labels D-123 **PROPOSED** + (`runbooks/dc-dc-phase2-tofu-dc-substrate.md:507` "D-123 (PROPOSED, recommend...)") while + design-decisions.md marks it ADOPTED -- the record disagrees with itself about D-123's ratification. + Refinement (A1, upheld in Round 2): Model A happens to equal the already-as-built + D-103/D-114 node-placement seam, so -- unlike vault -- **no NEW code rests on the Model-A + inference** (Model B would have been the change). The region+rack MAAS model and the D-124 rack + block (`main.tf:376-439`, "# D-124", explicitly ADOPTED) are independently authorized. +- Legitimately ADOPTED on explicit authority: D-121 node layout = Option C (AskUserQuestion + selection, `:3397`) + the 14-service scale-up ("this is the time we add the additional HA + nodes"); D-122 (the operator's explicit "Ruling 2" + "Routed by fabric switches"); D-124 + ("1. A 2. Confirmed", `:3513`). +- **Violates:** hard-rule-2 (no inferred value); hard-rule-1 (an inferred ruling drove the committed + vault overlay). **Action:** demote BOTH the D-121 vault-HA axis and the D-123 Model-A axis to + PROPOSED; put both one-line questions back to the operator ("vault HA backend: v-a MySQL-backed + vs v-b etcd/Raft?" and "node containment: Model A vcloud-level group vs Model B nested-in-VM?"). + D-121's "ADOPTED IN PART" then means what it says. Region+rack, Option C, and D-124 stand. +- **Contested (severity):** A1 argues MAJOR-not-BLOCKING for D-123 -- a status LABEL is not a + value "entering a command", nothing is applied (L1-08), and apply is independently gated + (L1-07). Adjudication: kept BLOCKING because (i) the prompt frames C2 as BLOCKING by design, + (ii) the vault case DID drive committed code, and (iii) "ADOPTED" is the authority a downstream + agent acts on. The dissent is real and recorded (section 4): as a matter of LIVE mutation risk, + nothing is currently blocked; as a matter of decision-record integrity, both must be re-ruled + before the record is trusted or apply proceeds. + +### R3-F02 -- MAJOR -- SEC-010: the metal-admin DC-LOCAL invariant is a ledger promise, not a committed artifact or a gate + +Upheld under cross-examination (A3-F6 survived A2's "distro default is fine" refutation attempt). +`main.tf` `vvr1_dc0` cloud-init pins static IPs + one route but has NO `net.ipv4.ip_forward=0` +sysctl and NO host firewall on the transit leg. The rack straddles metal-admin (10.12.8.0/22, +DC-local per D-052 `:767` / D-100 `:1946`) and the office1<->dc0 transit (crosses fiber), so the +"never crosses the fiber" invariant is preserved ONLY by Ubuntu's distro default -- which the +deferred MAAS-rack snap install could flip. SEC-010 records this "Close BEFORE tofu apply", but a +grep of `scripts/` and the phase2 runbook finds NO mechanical gate; the only apply-block (L1-07) +is an addressing gate, not security. **Violates:** D-052/D-100 (and general host-hardening guidance +-- CIS-style benchmarks recommend `ip_forward=0` on non-router multi-homed hosts; verify the exact +control ID before quoting it). +**Action:** add `ip_forward=0` (v4+v6) + an nft/ufw transit-leg pin to the committed rack +cloud-init as the ARTIFACT; wire a mechanical pre-apply gate (not a ledger note); carry the same +pin onto `voffice1` when its transit leg is wired (currently single-homed, `main.tf:184`). The pin +is free -- a MAAS rack proxies at the application layer and needs no kernel forwarding. + +### R3-F03 -- MAJOR -- D-107's DR-independence claim is false as written (and false at Roosevelt too) + +D-107 `:2088`: the per-DC mirror exists "so a DC can redeploy independently even if Office1 or the +peer DC is down -- a DR requirement the drill exercises." Under the RULED region-on-Office1 + +rack-per-DC model (D-123), a MAAS rack is stateless and cannot commission/deploy/power without the +region, so an Office1 outage removes reprovisioning from BOTH DCs. And L1-10: the Phase-5 drill +invokes NO MAAS -- it exercises workload failover, not redeploy. Two clauses fail. +- **Nuance (A2, Round 2):** the claim says "Office1 OR the peer DC is down." The peer-DC-down half + is VALID (region up, mirror up -> the DC can redeploy). Scope the defect to the **Office1-outage** + sub-case, not the whole claim. +- **Elevation (A2, Round 2):** buildout-design `:157` mandates single-region PERMANENTLY, so this + is unachievable at ROOSEVELT too -- not stale VR1 wording. A standby-region-per-DC mitigation + would CONTRADICT the buildout's single-region decision. +- **Violates:** D-107 (its DR-independence + "drill exercises" clauses). **Action (option a):** + amend D-107 to scope DR-independence to the in-DC artifact mirror + workload failover; add a + WRITTEN, PERMANENT acceptance that node provisioning is unavailable during an Office1-region + outage (day-1/2 dependency, not runtime -- running DC clouds are unaffected); delete/correct "a + DR requirement the drill exercises". Do NOT reopen the topology. + +### R3-F04 -- MAJOR -- D-122's MAAS-controller bullet and single-object site-down wording were never superseded + +- **C1:** D-122 `:3441` "Each site runs its own MAAS controller (as voffice1 does)" -- a full + region+rack -- was superseded by D-123's region-on-Office1 + rack-per-DC (operator-confirmed; + matches buildout `:157`) but never marked. A downstream agent quoted the stale bullet back as if + live. **Action:** append a verbatim supersession note pointing to D-123. +- **C3:** D-122 `:3417` "virsh destroy against a single object" is void for DCs -- under + Model A the containment VM does not contain the DC nodes (D-123 admits this `:3482`), and the + Phase-5 drill already uses per-node group shutdown (L1-10), so the drill does NOT break. The + stale claim also leaked into `dc-dc-deployment-workflow.md:86`. **Action:** amend D-122 AND + workflow:86 to "destroy the vr1-dc0-* domain GROUP" for DCs (single-object destroy stays literal + for Office1); note `vvr1-dc0` at a DC is a MAAS rack headend, not a D-114 containment VM (rack + mode runs no LXD/compose). +- **Violates:** D-123 (supersedes both). Topology operator-confirmed; record fix only. + +### R3-F05 -- MAJOR -- D-107 vs D-123 "node artifacts" wording (C4): reconcilable, but D-107 must be scoped + +D-107 `:2087-89` ("no node artifacts served from Office1"; "images including amphora ONLY from an +in-DC mirror") reads against D-123 `:3459` (rack "PROXIES OS images from the Office1 region"). The +counter-reading HOLDS: the intra-MAAS region->rack commission/deploy image channel is architecturally +distinct from D-107's supply-chain mirror (apt/snap/charmhub/registry/amphora served at/after juju +deploy). Both stand once D-107 is scoped. **Violates:** NONE (a wording gap in D-107). **Action:** +amend D-107 to scope "node artifacts" to the supply-chain classes, explicitly excluding the +intra-MAAS provisioning-image proxy D-123 routes region->rack. Pairs with R3-F03. + +### R3-F06 -- MAJOR -- D-121's Option-C sizing rests on an uncommitted, now-absent model; the "reproducible" claim is false + +`scratchpad/optc-calc.py` -- the whole-host validation that grounds Option C's 222 vCPU / 790 GiB / +77%-RAM ruling -- is NOT tracked and is ABSENT from disk (L1-12), yet the changelog claims +"Resource model reproducible: python3 scratchpad/optc-calc.py". **Violates:** repo-authoritative +discipline (CLAUDE.md); D-121 record integrity. **Contested (severity, A1 Round 2):** the Option-C +ruling is an explicit operator selection (not gated on the calculator), and fit is independently +checkable -- node literals are committed in `main.tf` locals and the disk dimension re-ran from the +COMMITTED `dc-dc-ceph-disk-budget.sh` (L1-09 PASS). So the ruling is not "unverifiable"; the defect +is reproducibility hygiene. **Action:** promote `optc-calc.py` into a committed, harnessed +calculator (already logged as a follow-up) and re-run for Option C before the sizing is treated as +measured; correct the changelog's "reproducible" claim until it is. + +### R3-F07 -- MAJOR -- D-121 records an Option-B disk validation for the ruled Option-C layout + +D-121 `:3382` records the disk-budget PASS for "Option B 4+4" while the RULED layout is Option C +(3 storage/DC). Code re-ran for Option C (L1-09): PASS, 6.48 TiB margin -- capacity is fine, so this +is a RECORD defect, not a capacity one. **Action:** re-record the Option-C validation (cite L1-09). + +### R3-F08 -- MINOR -- Option C's 3 OSD hosts at size=3 carry zero Ceph rebuild headroom (unrecorded) + +D-121 flagged "size=3 has ZERO rebuild headroom" for the REJECTED Option A `:3358` but not for the +ADOPTED Option C, whose 3 storage hosts have the identical property. **Walked back from MAJOR in +Round 2** (A1/A2/A3 concur): with Charmed default min_size=2 the pool keeps SERVING on one host +loss (the availability drill is clean), ceph-mon=3 sits on the CONTROL nodes (quorum untouched), +and only the RE-REPLICATION sub-case lacks headroom -- an inherent size=3 economy that resolves at +Roosevelt (>=4 storage hosts). **Action:** add an accepted-risk note to the Option-C record (a +storage-node-loss drill will show degraded-not-self-healing; Roosevelt remedy = >=4 storage/DC). + +### R3-F09 -- MINOR -- D-115 reads PROPOSED to `ledger-scan` though it is ADOPTED by amendment (L1) + +D-115's primary Status line retains the substring "Originally PROPOSED/OPEN", which trips +`ledger-scan`'s regex (L1-02), while the decision IS ratified by its 2026-07-13 amendment +(`:2727`). So the office carve import + PR #1 merge rest on a legitimately ADOPTED decision -- NOT +an unratified-executed one. **Action:** move "Originally PROPOSED/OPEN" out of the Status line (or +harden `ledger-scan` to ignore an "originally/was PROPOSED" clause when ADOPTED is present). Sweep +D-121..D-124 Status lines for the same machine-vs-human record divergence. + +### R3-F10 -- MINOR -- SEC-011 (node least-connectivity) is CONTESTED on Roosevelt-fidelity grounds + +Code's own prior SEC-011 (storage nodes get provider-public + data-tenant legs they never bind) is +challenged by A2 (Round 2, REFUTED) and A1 (WEAKENED): (1) in VR1 every plane is isolated L2 with +no gateway/route out, so an unbound vNIC has NO exploitable reachability -- "attack surface" is nil +today; (2) pruning would OVERRIDE the ADOPTED D-122 6-NIC ruling without a superseding decision AND +INCREASE delta to Roosevelt, where D-052's planes are VLAN-trunked on bonded NICs (ALL planes +present at every node; netplan/MAAS decides L3 binding), so uniform 6 isolated-L2 vNICs is the more +faithful model. **Adjudication:** downgrade SEC-011 from "pre-apply hardening" to an OBSERVATION / +operator's call; recommend amending the SEC-011 ledger row to record the Roosevelt-delta caveat. +Surviving dissent (section 4). + +### R3-F11 -- MINOR -- Stale "DC1/DC2 + supernet unassigned" prose in the workflow doc (L3) + +`dc-dc-deployment-workflow.md` Authoring-status cell still reads "DC1-first; DC2 hard-gated (D-101 +supernet unassigned)" -- superseded by D-119 (code is vr1-dc0/vr1-dc1) and D-115 (vr1-dc1 supernet += 10.12.64.0/19 assigned). No gate depends on the stale reading (the second DC is sequenced out +regardless). **Action:** update the cell; drop the "unassigned" clause. + +### R3-F12 -- OBSERVATION -- Transient Octavia N+1 amphora failover headroom is not modeled (C7b) + +The whole-host validation is a steady-state sum; it does not visibly reserve the transient N+1 +amphora placement headroom Octavia STANDALONE failover needs (a hard-won VR0 finding), and the +Phase-5 drill exercises failover on two clouds sharing one host. Mitigated in this single-host sim +(a hard-downed DC frees its RAM for the survivor) and unverifiable until R3-F06's model is committed. +**Action:** when `optc-calc.py` lands, add a line on per-cloud transient amphora headroom (or state +it is a Roosevelt-only concern for this sim). + +### R3-F13 -- OBSERVATION -- D-122 "6 NICs = baremetal-matched" is imprecise; Section-9 could add a cross-ref + +(i) D-122 sizes nodes at "6 NICs, one per plane" but the Roosevelt realization (D-052) is untagged +metal-admin + tagged VLAN subinterfaces trunked on bonded NICs, not 6 discrete physical NICs -- L2 +isolation is behaviorally equivalent; a characterization nuance, no code change. (ii) A3 argued +Section 9 omits the site-wan simulated-ISP and the virsh-destroy DR fault-injection; largely REFUTED +in Round 2 (both have Roosevelt analogs -- a real circuit, a real facility-down drill -- so neither +meets the register's "no analog" bar; the register is a build-step register and virsh-destroy is an +operational step). At most add a clarifying cross-reference for the containment/virsh-destroy sim +vehicle. Not a material incompleteness. + +--- + +## 4. Minority report (surviving dissent -- recorded, not resolved away) + +- **R3-F01 severity (dissenter: A1).** A1 holds D-123's inferred-status is MAJOR ledger-hygiene, not + deploy-BLOCKING: a status label is not a value "entering a command" (hard-rule-2's literal scope), + nothing is applied (L1-08), and apply is independently gated (L1-07). A1 further showed (Round 2, + refuting A3-F1) that Model A = the already-as-built D-103/D-114 path, so no NEW code rests on that + specific inference. Code's adjudication keeps R3-F01 BLOCKING on decision-record + governance + grounds (and because the VAULT half DID drive committed code), but records A1's point: as pure + live-mutation risk, nothing is currently blocked. +- **R3-F10 SEC-011 (dissenter: A2).** A2 refutes the "attack surface" framing outright: isolated-L2 + planes have no exploitable reachability, and pruning increases Roosevelt delta. Code accepts the + Roosevelt-delta point and downgrades SEC-011 accordingly, but records that A2 would go further and + strike the finding as a non-issue; Code retains it as an OBSERVATION worth the operator's note. +- **R3-F08 Ceph severity (dissenters: A1, A2, A3 concur down; A2-R1 held MAJOR).** A2's Round-1 + MAJOR ("opposite of the clean drill C is sold on") did not survive its own and others' Round-2 + scrutiny (min_size=2 keeps serving; mon on control nodes). Recorded as the walk-back it was; + final severity MINOR. + +**Convergence check (mandated).** The three live charters converged on the C-register verdicts -- +expected, because those items are text-provable quotations, not judgment calls. But they did NOT +fully converge: each surfaced a distinct high-value finding (A2: external CIS/Ceph/MAAS citations; +A3: the Section-9 completeness challenge + the baremetal NIC-realization nuance; A1: the uncommitted +`optc-calc.py`), and Round-2 produced genuine dissent (above). So the charters separated adequately. +The one place convergence WAS a failure mode: all three reported inferred-ruling **count = 1** +because Code's shared brief pre-asserted D-121's grounding -- a demonstration that shared priors +create shared blind spots. Code's independent transcript check (L1-13) corrected the count to 2. + +## 4a. A4 backfill integration (returned; corroborates) + +The A4 drift-archaeologist backfill returned and **independently reached inferred-ruling count = 2** +(D-121 vault-`(v-a)` + D-123 Model A), corroborating R3-F01 by the harder path: A4 was tasked to +scrutinize the vault Status-vs-body contradiction directly, and reached count = 2 WITHOUT the +biasing "D-121 is grounded" hint the three Round-1 charters received. This confirms the section-4 +convergence diagnosis -- the count-1 result was a shared-prior artifact, not a real ceiling. A4 did +NOT participate in Round-2 cross-examination (its Round-1 died on an API stall; Round-2 had already +run). Integration -- with one A4 claim verified-and-rejected (I check agent citations, not +rubber-stamp them): +- **A4's count = 2 corroborated.** A4's F1 (D-123 Model A) + F2 (D-121 vault, Status `:3264` vs + body `:3346`, no vault utterance in the batch changelog) independently match R3-F01. +- **A4's F7 REJECTED on direct check.** A4 claimed the phase-2 exit gate cites the stale D-122 + "each site runs its own controller" bullet (`runbooks/dc-dc-phase2-tofu-dc-substrate.md:636`). + Verified: the runbook does NOT -- at `:507` it cites "D-123 (PROPOSED, recommend...)" and the exit + gate (`:630-635`) uses the rack-controller-per-DC (D-123) model. A4 miscited. NOTABLE side effect: + the runbook already labels D-123 **PROPOSED**, which CONTRADICTS design-decisions.md's ADOPTED + status and independently supports R3-F01 (the D-123 record is internally inconsistent about its own + ratification). Added to R3-F01's evidence. +- A4 otherwise agrees across C1-C7 and L1-L5 (it frames C4 as "REFUTED -- both stand", the same + disposition as R3-F05's "reconcilable, scope D-107"), and adds clean negative findings: no false + DONE/as-executed claims, state reconciles with zero orphans, byte/lint hygiene clean. + +--- + +## 5. Leads register + +| Lead | Status | Evidence | +|---|---|---| +| **L1** (D-115 PROPOSED yet executed+merged) | **CONFIRMED as record-staleness, REFUTED as governance breach** | D-115 ADOPTED by amendment `:2727`; `ledger-scan` false-positive on the "Originally PROPOSED/OPEN" substring (R3-F09). The import + PR #1 rest on a ratified decision. | +| **L2** (D-071 controller-HA gates Stage 3) | **REFUTED as a Stage-3 blocker** | D-104 (`:2031-38`, ADOPTED) dispositions the controller-topology question for VR1 (single-unit per DC, HA deferred to Roosevelt) and is the entry the DC-DC phase was gated on; it references D-071 without amending it. D-071 (patch cadence) is Roosevelt-scoped. Annotate the ledger note (R3-F... minor). | +| **L3** (D-117-class naming drift) | **REFUTED** | Code is unambiguous (vr1-dc0/vr1-dc1, D-119); the double-namespace is RECORDED (D-117 + amendments). Only residual is stale prose (R3-F11); no gate depends on the ambiguous reading. doc-"DC1" = vr1-dc0. | +| **L4** (GUA/ULA IPAM reconciliation) | **REFUTED (one line)** | Stage 3 is isolated-L2 substrate + node/edge/rack VMs; it instantiates no tenant L3 addressing. The GUA/ULA reconciliation is a Stage-5 Neutron concern. Stage 3 does NOT depend on it. | +| **L5** (exit gate "conditionally met at best") | **CONFIRMED -- still conditional, honestly HELD (not run through)** | Node sizing ruled (D-121, but via the uncommitted model, R3-F06); edge sizing carried from applied office1_opnsense (measured basis, not a DC-edge boot measurement); netem still commented/unparameterized (unruled D-100 sub-item); the interface-naming boot measurement is a runtime TODO; `tofu plan` BLOCKED on 4 rack vars (L1-07). netem is correctly held, not fabricated. | + +--- + +## 6. Go / No-Go on Stage 3 + +**NO-GO for `tofu apply` as it stands** -- on hard-gate grounds, NOT because the substrate design is +unsound (it is sound: planes are isolated L2, state reconciles with no orphans, gauntlet green, +Option C fits with margin, validate passes). + +Blocking conditions to clear (each operator-gated, applied in a post-acceptance sweep): +1. **Re-rule the two inferred axes (R3-F01):** D-121 vault-HA (v-a vs v-b) and D-123 node + containment (Model A vs B). Both are one-line operator questions. Until then the vault overlay + and the Model-A record are unratified. +2. **Ship the SEC-010 artifact + mechanical pre-apply gate (R3-F02).** DC-LOCAL must be enforced, + not promised. +3. **Assign the rack transit/30 + rack IP in office1-netbox and feed the 4 tfvars (L1-07)** so a + plan can render and be reconciled against state. Wire `voffice1`'s transit leg (or gate the rack + route) so the route peer exists. + +Conditional-GO once 1-3 are met AND the record amendments (R3-F03..F07, F09, F11) are applied and +`optc-calc.py` is committed + re-run for Option C (R3-F06). The record corrections do not block the +substrate build; they block treating the DECISIONS as coherent, which is the operator's stated +concern. The MINOR/OBSERVATION items (F08, F10, F12, F13) are for the operator's judgment and need +not gate the sweep. + +--- + +*Prepared read-only; not committed, not merged, not pushed. Presented for operator review. Fixes +staged for an individually-gated post-acceptance sweep.* diff --git a/docs/stage3-review-base.patch b/docs/stage3-review-base.patch new file mode 100644 index 0000000..cfe2615 --- /dev/null +++ b/docs/stage3-review-base.patch @@ -0,0 +1,3795 @@ +diff --git a/docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md b/docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md +new file mode 100644 +index 0000000..9a08253 +--- /dev/null ++++ b/docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md +@@ -0,0 +1,116 @@ ++# 2026-07-15 -- Stage 3 kickoff: D-121 (HA scale-up + Option C node layout) + D-122 (site shape); phase2 runbook fix-forward ++ ++## Context ++ ++Opened Stage 3 (VR1 DC substrate) on branch `dc-dc-stage3-phase2-dc-substrate`. Walked ++`runbooks/dc-dc-phase2-tofu-dc-substrate.md` to surface its blocking decisions before any ++`tofu apply`. The walk (plus a bundle.yaml HA review and a whole-host resource validation) ++produced two operator rulings and a runbook fix-forward. No cloud mutation this session -- ++design decisions, one resource calculation, and doc reconciliation only. ++ ++## What changed ++ ++**D-121 (ADOPTED IN PART) -- VR1 makes HA real + the node layout that carries it.** ++`docs/design-decisions.md`. The bundle review found the root fact: VR0 ran its control plane ++**single-unit by design** (D-009: hacluster + VIP relations at `num_units: 1`, "decorative... ++scale-up is mechanical: 1 -> 3"). The 12 hacluster subordinates and D-036's dual VIPs are already ++wired; the units were never added. D-121 adopts the **14-service 1->3 scale-up** (keystone, glance, ++neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, ++magnum, designate, ceph-radosgw + rabbitmq + vault; `ceph-rbd-mirror` stays 1 per D-108). Two ++realities recorded so the cost is not under-stated: (1) it is a **placement rewrite** (each app's ++`to:` must spread across 3 distinct hosts -- co-located units are decorative again; a **Stage-5 ++placement-balance rule** was folded in from the openstack0-3 balance analysis: spread each app's 3 ++units one-per-control-node so the RAM-heavy trio auto-balances, and distribute the ~42 API containers ++evenly ~14/control-node, sized for at 64 GiB); (2) **vault** ++is MySQL-backed today (C1 revised D-006; `vault:ha` removed by BUNDLEFIX-002), so its HA path was a ++sub-ruling -- **RULED (v-a)** 2026-07-15: keep MySQL-backed, scale to 3, re-add `vault:ha`, gated on ++verifying the `1.8/stable` charm does MySQL-backend HA with 3 units (else evaluate Raft via D-068, NOT ++etcd). 3-unit vault triples the unseal burden, intersecting OPEN SEC-003. ++**Node layout = Option C (operator-ruled)**: 3 control + 2 compute + 3 storage per DC (16 node VMs). ++Ruled sizing: control 16 vCPU/64 GiB/150 GiB, compute 12/48/100, storage 8/24/550. `ceph-osd`=3/DC, ++`nova-compute`=2/DC, `ceph-mon`/`mysql`/`rabbitmq`/`vault`/`ovn-central`=3 across the control nodes. ++ ++**Whole-host resource validation (Option C).** Measured the actual VR1 host (not the design ++estimate): **256 vCPU / 1024 GiB RAM / 10 TiB disk** (`nproc`/`free`/`df`; currently only Office1 ++runs -- `voffice1` 16/32/600 + `office1-opnsense` 2/2/11, netbox+tailscale nested inside voffice1). ++Modeled Option C across all layers (Office1 as-built + both DCs + edges + containment overhead) at ++`scratchpad/optc-calc.py`: **222 vCPU / 790 GiB / 5.4 TiB -> FITS** (RAM binding at 77%; vCPU ++overcommits; disk thin-provisioned at 53%). Finding: C is NOT heavier than B -- it is RAM-lighter ++(790 vs 838 GiB) because role separation lets storage nodes run lean. Sensitivity: control-node RAM ++is the knob (96 GiB -> 982 GiB / 96%); held <= 64 GiB. This corrected D-121's first-pass "C is ++heaviest" characterization. Also ran `scripts/dc-dc-ceph-disk-budget.sh` (Option B/C 4+4 and 3+2+3): ++PASS with wide margin -- disk is not the binding constraint. ++ ++**D-122 (ADOPTED) -- VR1 site shape.** `docs/design-decisions.md`. Records the operator's Ruling 2: ++each site nested in its own containment VM (site-down/recovery + site-to-site drills); dark fiber ++DC-DC-Office is East-West ONLY (never an internet path); each site has its own dedicated L3 "ISP" ++uplink (build `vr1-dc0-wan` on the office1-wan pattern -- NOT a mesh leg); the DC OPNsense edge ++follows the proven Office1 pattern (2048 MiB / 2 vCPU / nano, REST-API config per D-113(a2)); each ++site runs its own MAAS controller (may bootstrap from an existing one). Closes the phase2 runbook's ++Ruling-1/Ruling-2 STOP-gates. Subsumes the DOCFIX-185 amendment note. ++**Edge NIC model ruled (D-122 refinement, operator 2026-07-15): 2-NIC, baremetal-matched.** NIC count ++mirrors the Roosevelt hardware -- nodes = 6 NICs (one per plane, measured on VR0 openstack0-3), edge = 2 ++(WAN + LAN). The six planes are routed by the FABRIC (OVN/OpenStack on the deliberately isolated-L2 ++`dc-planes` segments), so the edge is only the external boundary, not an inter-plane router -- no ++`opnsense-edge` module extension. Edge WAN = `vr1-dc0-wan` (`172.30.2.0/24`, D-115); edge LAN = ++**provider-public** (`10.12.4.0/22`; external gateway per D-100). This resolves the runbook's earlier ++"LAN role / build a site-internal segment" flag: no new segment is needed. Two build prerequisites ++remain: build `vr1-dc0-wan` as a tofu module (promote off the virsh `office1-wan` one-off), and register ++`172.30.2.0/24` in office1-netbox (D-115-ruled, not yet loaded -- absent from the draft; no import ++covered the DC Edge /24s). ++ ++**phase2 runbook fix-forward.** `runbooks/dc-dc-phase2-tofu-dc-substrate.md`: ++- Region-qualified to the D-119 namespace -- every `dc1`/`dc2` object renamed to `vr1-dc0`/`vr1-dc1` ++ (matching `main.tf`); added a naming note + a HAZARD line that any residual bare `dcN` reintroduces ++ the exact cross-cloud collision D-119 deletes. Revert: `git checkout` the file. ++- Steps 4-5 rewritten to the D-113(a2) REST-API edge bootstrap -- dropped the config.xml render + the ++ config-ISO path (D-112: the Importer cannot fire on a nano image; the module already dropped ++ `config_iso_path`); edge is now prep-image + post-boot console/SSH/API. WAN_IF/LAN_IF agonizing ++ removed (measured post-boot). ++- D-122 edge shape wired into Step 5: `memory_mib=2048`/`vcpu=2`/nano, no `disk_size_bytes`, ++ `wan_network_name = vr1-dc0-wan` (dedicated per-site ISP uplink, office1-wan pattern), ++ `lan_network_name` = site-internal network -- both flagged as genuine build prerequisites. ++- D-121 Option C wired into Step 6: 8 `module "vr1_dc0_node_*"` blocks in three role-sizes (3 control ++ 16 vCPU/65536 MiB/150 GiB, 2 compute 12/49152/100 GiB, 3 storage 8/24576/550 GiB; disk in bytes via ++ `N * 1024*1024*1024`), with the Stage-5 HA-scale-up context. ++- Corrected the `vr1-dc1` gate rationale (top callout + exit GATE + checklist): its supernet IS ++ assigned (D-115 `10.12.64.0/19`, imported to office1-netbox); it stays out of scope by SEQUENCING, ++ not a missing literal. No `vr1-dc1` build steps added. ++- Sequence block, exit GATE, Step-3 register, and delivery checklist updated to match; open items kept ++ as flagged STOPs (per-DC MAAS rack controller = Stage-4 gap, honest first-run scope Steps 1-8; the ++ vault-HA backend sub-ruling; netem params). Revert: `git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md`. ++ ++**Two gaps surfaced by the fix-forward (LOGGED, not fixed -- they need real values/build work):** ++1. `opentofu/main.tf`'s commented `vr1_dc1_planes` block (~L38-50) still says "wait for NetBox to ++ assign D-101's supernet" -- STALE per D-115. Left in place (this was a doc-only pass); flagged for ++ the `main.tf` owner. ++2. **Edge network prerequisites (LAN side RESOLVED after the operator's fabric-routing ruling).** The ++ fix-forward first flagged "no site-internal LAN segment for `vr1-dc0`"; the operator then ruled the ++ planes are **fabric-routed** and the edge is **2-NIC**, so the **LAN = provider-public** (a ++ `vr1_dc0_planes` output) -- NO new site-internal segment needed. Two REAL prerequisites remain before ++ Step 5's `tofu apply`: build **`vr1-dc0-wan`** as a tofu module (promote off the virsh `office1-wan` ++ one-off), and register **`172.30.2.0/24`** in office1-netbox (D-115-ruled, not yet loaded). Recorded ++ in the runbook, not invented. ++ ++**Ledger + tracker reconciliation.** `docs/session-ledger.md` (next-free D 121->123; a Stage-3 ++open-work block with the two rulings, the validation, the fix-forward-in-flight, the stale-gate ++correction, and the logged-not-executed calc-promotion follow-up) and ++`docs/dc-dc-deployment-workflow.md` (Stage 3 State row updated from "runbook written" to ++"IN PROGRESS + two rulings landed"). ++ ++## Tests / verification ++ ++- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN (the documented D-001..018 non-ASCII ++ carve-out; new D-121/D-122 text is ASCII). ++- `bash scripts/ledger-scan.sh` -> next-free D = 123; D-121/D-122 not flagged PROPOSED/OPEN. ++- Resource model reproducible: `python3 scratchpad/optc-calc.py`. ++ ++## Revert ++ ++- D-121 / D-122: delete the two `## D-121:` / `## D-122:` sections from `docs/design-decisions.md` ++ (they are append-only additions; nothing else references them yet in code). ++- Ledger / workflow: revert the `docs/session-ledger.md` next-free line + the Stage-3 block, and the ++ `docs/dc-dc-deployment-workflow.md` Stage 3 State paragraph, to their prior text. ++- Runbook: `git checkout runbooks/dc-dc-phase2-tofu-dc-substrate.md` (see the runbook section for the ++ per-change reverts). ++- No cloud state was touched, so there is nothing to unwind on the host. +diff --git a/docs/changelog-20260715-stage3-prereqs-prep.md b/docs/changelog-20260715-stage3-prereqs-prep.md +new file mode 100644 +index 0000000..df0c9a6 +--- /dev/null ++++ b/docs/changelog-20260715-stage3-prereqs-prep.md +@@ -0,0 +1,88 @@ ++# 2026-07-15 -- Stage 3 batch-deploy prep: OpenTofu wiring, DC-edge NetBox importer, HA overlay, D-123 ++ ++## Context ++ ++Operator directed "get as much ready as possible so we can batch deploy with the least ++operator interaction." This batch prepares the Stage-3 substrate + Stage-5 HA artifacts so the ++live steps reduce to review-plan-and-apply. NO cloud mutation this session -- all authoring + ++offline validation; every `tofu apply`, netbox `--commit`, and juju deploy stays operator-gated. ++Fanned out across three disjoint areas (opentofu / netbox / overlays); the opentofu agent hit a ++transient API stall and its stream was completed in-session by the main loop. ++ ++## What changed ++ ++**D-123 (PROPOSED) -- DC site-down DR model / node-VM placement.** `docs/design-decisions.md`. ++Surfaced a real tension: D-122 records the operator's "each site is one VM, `virsh destroy` = ++site-down" intent, but the ADOPTED D-103/D-114 seam + the built tooling create the OpenStack node ++VMs as **vcloud-level** domains that MAAS merely DISCOVERS (containment VM scoped to MAAS/LXD + ++non-stack machines only). Model A (recommended): nodes stay vcloud-level; `vvr1-dc0` is a MAAS/LXD ++headend VM (the proven `voffice1` pattern via `site-headend-install.sh`, which already names it a ++target); site-down = destroy the `vr1-dc0-*` domain GROUP. Model B: true single-VM containment ++(nodes nested), 4-level nesting, heaviest. **Needs a one-line operator ruling.** ++ ++**OpenTofu Stage-3 wiring (`opentofu/`).** ++- **New module `modules/site-wan/`** -- a NAT'd per-site simulated-ISP uplink /24 (generalizes the ++ virsh-created `office1-wan`; operator 2b ruling). MTU 1500 (ISP-uplink domain, NOT jumbo). Schema ++ CONFIRMED against the installed provider via `tofu providers schema -json` (the method dc-planes ++ calls for): `forward = { mode = "nat" }`, `ips = [{ address, prefix }]`, `mtu = { size }`, ++ attribute-style like dc-planes. DHCP omitted (the edge WAN is static, like office1-opnsense .2). ++ Per-module `versions.tf` (DOCFIX-179). ++- **`main.tf` Stage-3 section** (NOT YET APPLIED, gated): `module "vr1_dc0_wan"` (172.30.2.0/24, ++ D-115); `module "vr1_dc0_opnsense"` (D-122: 2-NIC, `memory_mib=2048`/`vcpu=2`/nano, no ++ `disk_size_bytes`/`config_iso_path`, LAN = provider-public plane, WAN = the uplink); 8 node VMs via ++ a single `for_each` over a `locals` map in D-121 Option C sizes (3 control 16/65536/150GiB, 2 ++ compute 12/49152/100GiB, 3 storage 8/24576/550GiB), six NICs each with **metal-admin first (PXE)**; ++ netem HELD as a comment (Step 11, runtime SSH target + unruled params; the vr1-dc0<->office1 mesh ++ bridge is `virbr7`, measured). **HELD pending D-123:** the `vvr1-dc0` headend + `maas-vm-host` (Step ++ 9) blocks are not wired. ++- Fixed the stale `vr1_dc1_planes` comment (supernet IS assigned, D-115; out of scope by sequencing). ++- **`tofu validate` -> Success**; `scripts/opentofu-validate.sh` -> 11/11 modules PASS (incl. site-wan). ++ ++**NetBox DC-edge WAN importer (`netbox/dc-edge-wan-import.py` + `tests/dc-edge-wan-import/`).** ++Registers the D-115 DC-edge /24s in office1-netbox: `172.30.2.0/24` -> site `vr1-dc0`, ++`172.30.3.0/24` -> site `vr1-dc1`, role `edge`, `scope_type=dcim.site` -- scoping CONFIRMED against ++`d115-office-carve.py`/`sandbox-fidelity-check.py` (how office1-wan 172.30.1.0/24 is bound), not the ++frozen draft (which predates D-115). Stdlib-only, mirrors `d120-compose-bands.py`: WHOLE-PLAN ++PREFLIGHT (both sites resolve before any create, so a missing site cannot half-write), DRY-BY-DEFAULT, ++`SANDBOX_HOSTS` guard + `--yes-write-upstream` refusal, WAF-safe UA. Creates ONLY the two /24s (role + ++container + sites are read-only preconditions). Harness: **58/58 PASS** (19 static + 39 behavioral, ++in-process against a fake NetBox). NOT run against a live NetBox (token operator-held). ++ ++**Bundle HA scale-up overlay (`overlays/dc-ha-scaleup.yaml`).** VR1-specific overlay (NOT a bundle.yaml ++edit -- VR0 stays single-unit per D-009) implementing D-121: 14 services -> `num_units: 3`, one-per- ++control-node placement via a 3-entry `to:` with `{{VR1_DC0_CONTROL_0N}}` tokens (Stage-4-binding ++banner). Three correctness items beyond the literal bump, each flagged in-file: (A) rabbitmq ++`min-cluster-size: 3` (D-009 amendment -- avoids the D-062-class formation race); (B) `cluster_count: ++3` on the 12 haclusters + vault-hacluster (base pins 1 = decorative); (C) **re-declares ++`vault-hacluster`** (it is COMMENTED OUT by BUNDLEFIX-002) and re-adds `[vault:ha, vault-hacluster:ha]` ++(D-121 v-a keeps vault MySQL-backed at 3). YAML valid; app names cross-checked vs bundle.yaml; ++repo-lint clean; `provider-bundle-check.py` green on the base. ++ ++## Tests / verification ++ ++- `cd opentofu && tofu validate` -> Success; `bash scripts/opentofu-validate.sh` -> 11/11 PASS. ++- `bash tests/dc-edge-wan-import/run-tests.sh` -> 58/58 PASS. ++- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN. ++- `bash scripts/run-tests-all.sh` -> GAUNTLET: ALL GREEN (61 harnesses). ++ ++## What remains operator-gated / to bind later (the residual for a smooth batch) ++ ++1. **D-123 ruling** (Model A/B) -- unblocks wiring the `vvr1-dc0` headend + `maas-vm-host` (Step 9). ++2. **Runtime prep before the edge apply:** `scripts/opnsense-prep-image.sh` to produce the nano at the ++ `base_volume_path`; the netbox `--commit` to register the two /24s (ready command in the importer's ++ own header; token operator-held). ++3. **Stage-4 binding for the HA overlay:** render `{{VR1_DC0_CONTROL_0N}}` to the measured MAAS machine ++ IDs; the base `machines:` block + the already-3 services (mysql/ovn/ceph-mon) + memcached + ++ nova-compute(->2)/ceph-osd(->3) re-bind to the Option C nodes; confirm `--overlay ... --dry-run` ++ (the `to:` list-replace merge is the one unverified overlay mechanism). ++4. **D-121 verifies:** vault charm MySQL-backend HA at 3 units (else Raft via D-068, not etcd); ++ hacluster `cluster_count` semantics. ++ ++## Revert ++ ++- `git rm -r opentofu/modules/site-wan`; revert the `opentofu/main.tf` Stage-3 section + the ++ `vr1_dc1_planes` comment; `rm .terraform.lock`-nothing (lock unchanged). ++- `git rm netbox/dc-edge-wan-import.py && git rm -r tests/dc-edge-wan-import`. ++- `git rm overlays/dc-ha-scaleup.yaml`. ++- Delete the `## D-123:` section from `docs/design-decisions.md`. ++- No cloud state was touched -- nothing to unwind on the host. +diff --git a/docs/changelog-20260716-d124-rack-addressing.md b/docs/changelog-20260716-d124-rack-addressing.md +new file mode 100644 +index 0000000..d878c13 +--- /dev/null ++++ b/docs/changelog-20260716-d124-rack-addressing.md +@@ -0,0 +1,92 @@ ++# 2026-07-16 -- D-124 adopted; vr1-dc0 MAAS rack (vvr1-dc0) wired; rack-mgmt NetBox importer ++ ++## Context ++ ++Continuation of the Stage-3 batch prep. The operator ruled the MAAS model (region on Office1 + ++rack controller per DC, D-123) and the rack<->region management addressing (D-124 Scheme A: a ++transit link on the office1<->dc0 mesh leg) + rack sizing. This lands the `vvr1-dc0` rack wiring ++and the NetBox importer for the addressing. NO cloud mutation -- authoring + offline validation ++only; the NetBox `--commit`, the `tofu apply`, and the rack install stay operator-gated. ++ ++## What changed ++ ++**D-124 (ADOPTED 2026-07-16).** `docs/design-decisions.md`. The Office1-region<->DC-rack management ++overlay. Answers "why not metal-admin?": the rack's node-facing leg IS metal-admin, but D-100 rules ++metal-admin DC-LOCAL and a RULED sub-item makes the Office1<->DC mesh the management path -- so the ++rack straddles metal-admin (nodes) + the mesh transit (region). **Scheme A** (transit-numbered mesh, ++a /30 or /31 point-to-point) + rack sizing **4 vCPU / 8192 MiB / 80 GiB** ruled. The specific transit ++CIDR + rack IP are assigned in office1-netbox (apex; operator-held token) via the D-124 importer -- no ++literal invented in-repo. ++ ++**`vvr1-dc0` rack wired (`opentofu/`).** ++- `main.tf`: `module "vvr1_dc0"` (cloudinit-vm) -- 4/8192/80, base = the noble image, `expose_nested_virt ++ = false` (rack-only; the region + its LXD host compose, not the rack), two legs ++ (`network_names = [metal-admin plane, mesh-vr1-dc0-office1]`), static `network_config` on both legs ++ with IPs from NEW `vr1_dc0_rack_*` variables (NetBox-assigned via tfvars, not invented) + a route to ++ the Office1 region (`10.10.0.0/22`, as-built D-115) over the transit. The interface-naming trap ++ (enp1s0/enp2s0 predicted, confirm on first boot) is flagged in-block. ++- `variables.tf`: `vr1_dc0_rack_metal_admin_ip`, `vr1_dc0_rack_transit_ip`, `vr1_dc0_rack_transit_prefix`, ++ `vr1_dc0_rack_transit_peer_ip` (no defaults). ++- `modules/mesh-link/outputs.tf`: added a `network_name` output (for the rack's transit-NIC reference). ++- **Step-9 `maas-vm-host` DELIBERATELY STILL DEFERRED (DOCFIX-179):** adding the `provider "maas"` block ++ it needs would force EVERY plan -- including the Steps 1-8 substrate apply -- to demand MAAS creds, ++ and vr1-dc0's MAAS does not exist until vvr1-dc0 is up. Wire it as its own step once the rack MAAS is ++ reachable. ++- `tofu validate` -> Success; `scripts/opentofu-validate.sh` -> 11/11 modules PASS. ++ ++**`site-headend-install.sh` rack-only mode.** Added `--role ` (default `region+rack`, ++100% backward-compatible). `--role rack` enrolls a DC rack controller to Office1's EXISTING MAAS region ++(D-123): disable timesyncd -> install the `maas` snap (3.7/stable) -> `maas init rack --maas-url ++--secret `. It SKIPS all region+rack work (PostgreSQL, region init, admin/ ++login, LXD, the LXD vm-host, compose-DHCP) -- a DC rack runs no DB, composes nothing (the DC nodes are ++OpenTofu-created + discovered via the vcloud virsh vm-host on the region), and metal-admin DHCP is a ++region-side runbook step. Traps 1-3 (LXD/dnsmasq) do not apply in rack mode; trap 4's spirit (rack ++configures NO DHCP) holds. The enroll secret is read from the file at mutate-time and NEVER echoed ++(dry-run prints ``; the harness asserts the sentinel never leaks). The ++`maas init rack --maas-url --secret` form is CONFIRMED against the MAAS snap docs/discourse (3.x-stable, ++so it holds for the 3.7 pin), not assumed. Harness `tests/site-headend-install/` 19 -> 32 checks ++(rack arg-contract, secret-non-leak, region-step-absence, backward-compat). Gauntlet stays 62. ++ ++**Rack-mgmt NetBox importer (`netbox/dc-rack-mgmt-import.py` + `tests/dc-rack-mgmt-import/`).** ++Registers the D-124 addressing in office1-netbox: the transit prefix (role `transit`, site-scoped ++vr1-dc0) + the rack's metal-admin IP. Stdlib-only, mirrors `dc-edge-wan-import.py`: whole-plan preflight ++(transit is /30|/31 with host-bits clear + subnet_of the container; rack IP in `10.12.8.0/22` static ++band `.2-.49`, not the `.1` gateway; the transit role + container + vr1-dc0 site exist), DRY-BY-DEFAULT, ++`SANDBOX_HOSTS` guard. Both literals are REQUIRED inputs (`--transit-cidr`/`--rack-ip`) -- no invented ++CIDR. Harness **96/96** (29 static + 67 behavioral). ++ ++**Scoping flags surfaced by the importer (operator-facing, logged NOT resolved):** ++1. **The `transit` role must be seeded in office1-netbox** (die-if-absent precondition, like `edge` ++ was). No transit/management role exists in the frozen draft; `infra` exists but is explicitly SHARED ++ infra ("ptp transit, device loopbacks, RR loopbacks, anycast"), so D-124's "own role, mirroring the ++ Edge role" is a NEW dedicated `transit` role to seed. ++2. **The transit may need its OWN supernet, not `10.12.0.0/16` (Cloud).** D-124 mirrors the Edge role, ++ which got a dedicated `172.30.0.0/16`. If office1-netbox assigns the transit from a dedicated ++ management/transit supernet (NOT subnet_of Cloud), the importer correctly REFUSES at preflight and ++ the fix is a one-line `CONTAINER` constant update. Decide the transit supernet when assigning. ++3. Transit prefix scope = `dcim.site:vr1-dc0` (mirrors the DC-edge /24s); a region scope may be ++ preferred -- operator's call at assignment. ++ ++## Tests / verification ++ ++- `cd opentofu && tofu validate` -> Success; `bash scripts/opentofu-validate.sh` -> 11/11 PASS. ++- `bash tests/dc-rack-mgmt-import/run-tests.sh` -> 96/96 PASS. ++- `bash scripts/repo-lint.sh` -> 0 fail, 1 legacy WARN. ++- `bash scripts/run-tests-all.sh` -> GAUNTLET: ALL GREEN (62 harnesses). ++ ++## Operator `--commit` (ready; apex token operator-held) ++ ++ NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN= \ ++ python3 netbox/dc-rack-mgmt-import.py \ ++ --transit-cidr dc0 transit /30|/31> --rack-ip <10.12.8.2-.49> # dry-run; add --commit to write ++ ++(Seed the `transit` role + confirm the transit supernet first -- see scoping flags 1/2. Then fill the ++`vr1_dc0_rack_*` tfvars with the assigned values.) ++ ++## Revert ++ ++- `git checkout opentofu/main.tf opentofu/variables.tf opentofu/modules/mesh-link/outputs.tf` ++ (removes the `vvr1_dc0` block, the rack vars, the mesh-link output). ++- `git rm netbox/dc-rack-mgmt-import.py && git rm -r tests/dc-rack-mgmt-import`. ++- Delete the `## D-124:` section from `docs/design-decisions.md`. ++- No cloud state touched. +diff --git a/docs/dc-dc-deployment-workflow.md b/docs/dc-dc-deployment-workflow.md +index 25e6971..948e7ee 100644 +--- a/docs/dc-dc-deployment-workflow.md ++++ b/docs/dc-dc-deployment-workflow.md +@@ -156,7 +156,22 @@ commit, operator-gated) as part of its definition-of-done. Stage 3 branches off + | **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. | + | **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Walks every still-open decision (node sizing, MAAS zone/pool, `power_address`, config.xml tokens incl. boot-measured `WAN_IF`/`LAN_IF`, netem params) and stops at the corresponding step if unresolved rather than inventing values. States its own exit gate as "CONDITIONALLY MET AT BEST" pending Stage 2, node sizing, a real OPNsense boot measurement, and D-100's netem ruling. NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (2026-07-09, registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`, which composes new VMs and would fight `node-vm`) all now exist -- none instantiated (node sizing, OPNsense config.xml content, and a real MAAS zone/pool are all still pending decisions). `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. | + +-**State:** RUNBOOK WRITTEN, NOT YET EXECUTED, hard-dependent on Stage 2 for its MAAS-registration half. ++**State:** IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`). RUNBOOK WRITTEN, NOT YET EXECUTED. ++**Two blocking rulings landed 2026-07-15** (from a walk of the runbook's decision register): ++- **D-121 (ADOPTED IN PART)** -- node layout = **Option C** (3 control + 2 compute + 3 storage per DC, ++ 16 node VMs), ruled after a whole-host resource validation (256 vCPU / 1024 GiB / 10 TiB measured; ++ C fits at 222 / 790 GiB / 5.4 TiB, RAM binding). Ruled per-node sizing recorded in D-121. Also ++ adopts the 14-service HA scale-up (1->3) that this layout carries at Stage 5. OPEN sub-ruling: the ++ vault-HA backend fork. ++- **D-122 (ADOPTED)** -- DC edge/site shape: Office1 pattern edge (2048/2/nano, REST-API config), a ++ dedicated per-site `vr1-dc0-wan` uplink (dark fiber is East-West only), site nested in its own ++ containment VM, per-site MAAS. This CLOSES the runbook's Ruling-1 (node sizing) and Ruling-2 (edge ++ LAN/WAN) STOP-gates. ++Fix-forward IN FLIGHT: runbook reconciled to post-D-119 naming + Steps 4-5 rewritten to the D-113(a2) ++REST-API bootstrap. Honest first-run scope remains **Steps 1-8**; Step 9 (MAAS registration) still ++gates on the per-DC MAAS rack-controller source (a Stage-4 gap -- no OpenTofu module stands one up yet). ++The stale `vr1-dc1` "supernet not assigned" gate is corrected (D-115 assigned `10.12.64.0/19`); vr1-dc1 ++stays out of scope by sequencing. + + --- + +diff --git a/docs/design-decisions.md b/docs/design-decisions.md +index a08a471..e4bb2db 100644 +--- a/docs/design-decisions.md ++++ b/docs/design-decisions.md +@@ -3256,3 +3256,303 @@ Two findings that shaped the method (and that the DC replication inherits): + + The NetBox-side carve record (Step 6 -- register the `.2-.49`/dynamic/node child ranges + the two + service IPs in the sandbox, then feed upstream) is still OWED and rides with C2; it is NOT done here. ++ ++## D-121: VR1 makes HA real -- scale the decorative single-unit control plane to 3, and the per-DC node layout that carries it ++ ++**Status:** ADOPTED IN PART (2026-07-15, operator ruling). **Node layout = Option C** (3 control + ++2 compute + 3 storage per DC, 16 node VMs) -- ruled after the whole-host validation proved it FITS ++(details below). The **14-service HA scale-up** (1 -> 3) is ADOPTED. **Vault-HA backend sub-ruling: RESOLVED = (v-a)** ++(2026-07-15, operator ruling) -- keep vault MySQL-backed (on the already-3-unit `mysql-innodb-cluster`), ++scale to 3, re-add the `vault:ha <-> vault-hacluster:ha` relation. GATE: verify the deployed vault charm ++(`1.8/stable`) actually does HA leader-election on the MySQL storage backend with 3 units before commit; ++if it does NOT, do not fall back to (v-b) etcd -- evaluate Raft integrated storage and route to D-068 ++(etcd storage is deprecated upstream; Raft is the better Roosevelt target). Rationale: reuses existing ++HA (zero new apps), keeps control-node container density down, does not front-run the unsettled Roosevelt ++backend choice (D-068), and adds no NEW SPOF (the whole cloud already depends on mysql). Governs ++Stage 3's `modules/node-vm` sizing/count (was blocking phase2 runbook Step 6 -- now UNBLOCKED for ++layout) and the Stage 5 per-DC bundle deploy. Raised by the operator's Stage-3 ruling ("this is the ++time we add the additional HA nodes missing from the previous deployments"). ++ ++**Ruled node sizing (Option C, validated to fit -- keep control nodes <= 64 GiB per the sensitivity):** ++3x control @ 16 vCPU / 64 GiB / 150 GiB; 2x compute @ 12 vCPU / 48 GiB / 100 GiB; 3x storage @ 8 vCPU ++/ 24 GiB / 550 GiB (500 OSD + 50 OS). Whole-host allocation across Office1 + both DCs + edges + ++containment overhead = 222 vCPU / 790 GiB / 5.4 TiB of the measured 256 / 1024 / 10240 (RAM binding at ++77%). `ceph-osd` = 3 units/DC (the storage nodes), `nova-compute` = 2 units/DC (the compute nodes), ++`ceph-mon`/`mysql`/`rabbitmq`/`vault`/`ovn-central` = 3 across the 3 control nodes. ++ ++### The problem this fixes ++ ++VR0/testcloud ran its OpenStack control plane **single-unit by design**. Per **D-009** ("Hacluster ++modeling at testcloud scale"): the hacluster + VIP relations were included at `num_units: 1` as a ++**decorative** pattern -- "a single unit can't form a real HA quorum" -- explicitly so that ++"Roosevelt scale-up is mechanical: change `num_units: 1` -> `num_units: 3` and rerun." The bundle ++already carries all 12 hacluster subordinates and, per **D-036**, both provider+metal VIPs on every ++clustered API app. VR1 (the regional rehearsal) is where that decorative HA is made real -- the ++missing units get added. ++ ++### Measured current state (bundle.yaml, machines 8/9/10/11 -- the 4 VR0 nodes) ++ ++Already at quorum (NO change): `mysql-innodb-cluster`=3 (D-062, must deploy at 3), ++`ovn-central`=3 (Raft), `ceph-mon`=3 (Paxos). Scale-out (sized to node count): `ceph-osd`=4, ++`nova-compute`=3 (on 9/10/11 -- machine 8 is compute-free today). ++ ++**Missing HA -- 14 apps currently single-unit that this decision scales to 3:** ++ ++| App | Now | -> | Mechanism | Governing | ++|---|---|---|---|---| ++| keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, barbican, magnum, designate | 1 | 3 | hacluster + dual VIP (already wired) | D-009, D-036 | ++| ceph-radosgw | 1 | 3 | hacluster + VIP (wired; NOT in D-009's named 12 but HA-eligible) | -- | ++| rabbitmq-server | 1 | 3 | native clustering / quorum queues (no hacluster) | -- | ++| vault | 1 | 3 | see the backend sub-question below | D-006 (revised by C1) | ++ ++Left single (NOT scaled): `ceph-rbd-mirror`=1 (**D-108** rules single-unit for VR1), ++`glance-simplestreams-sync`=1 (utility), `memcached`=1 (cache; optional 3), `designate-bind`=1 ++(DNS backend; optional 2). ++ ++### Two realities the "mechanical 1->3" framing hides (fold into the execution estimate) ++ ++1. **It is a placement rewrite, not a `num_units` tweak.** Every app pins its unit explicitly ++ (`keystone to: [lxd:8]`, `glance to: [lxd:11]`, ...). Setting `num_units: 3` with a single-entry ++ `to:` list makes Juju place one unit there and auto-place the other two wherever -- NOT ++ guaranteed on three DISTINCT hosts, which is the entire HA value. Each of the 14 apps needs its ++ `to:` expanded to a real 3-machine spread (anti-affinity). This is the actual work. ++ ++ **Placement-balance rule for the Stage-5 rewrite (Option C, 3 control nodes).** The 14 services x 3 ++ units = 42 API LXD containers land on the **3 control nodes** (`vr1-dc0-control-01..03`) -- ~14 per ++ node, roughly 2x the density of VR0's balanced ~7 containers/node (measured from the openstack0-3 ++ bundle placement, 27 control containers over 4 nodes). Two rules keep that density balanced: ++ - **Spread each app's 3 units one-per-control-node** (`to: [lxd:control-01, lxd:control-02, ++ lxd:control-03]`). 3-unit HA does this naturally, so the RAM-heavy stateful trio ++ (`mysql-innodb-cluster`, `rabbitmq-server`, `vault`) auto-balances to one replica per control ++ node -- do NOT stack two of them on the same node (VR0's own placement already avoided this: ++ rabbitmq on M10, vault on M11, mysql spread 8/9/10). ++ - **Distribute the ~42 containers EVENLY (~14/control node), not front-loaded onto control-01.** ++ The whole-host validation sized control nodes at 64 GiB for exactly this ~14-container load; if a ++ future rebalance wants ~9/node instead, the lever is a 4th control node per DC (NOT more units). ++ `ceph-osd`(3) sits on the storage nodes and `nova-compute`(2) on the compute nodes -- neither ++ competes for control-node RAM. This rule governs the Stage-5 bundle deploy; it does not change the ++ node COUNT ruled above. ++2. **Vault HA backend fork -- RESOLVED = (v-a), see Status.** The bundle runs vault **MySQL-backed** ++ today (`vault:shared-db -> vault-mysql-router -> mysql-innodb-cluster`), and **C1 revised D-006**: ++ "etcd/easyrsa dropped for testcloud; Raft-vs-etcd is a Roosevelt item," and the ++ `vault:ha <-> vault-hacluster:ha` relation was **removed (BUNDLEFIX-002)**. So scaling vault is ++ NOT "just add units". The operator ruled **(v-a)** (charm-verify gated); (v-b) retained below for ++ the record: ++ - **(v-a) Keep MySQL-backed, re-add the `vault:ha` relation, scale to 3** -- lowest delta; ++ leans on the already-3-unit mysql-innodb-cluster. Must be verified against the deployed vault ++ charm's real multi-unit/ha behavior on a MySQL backend before committing. ++ - **(v-b) Adopt D-006's etcd(3)+easyrsa(1) HA backend** -- the originally-documented topology, ++ but reverses C1 and adds two applications. ++ **Operator sub-ruling needed.** Either way, 3-unit vault **triples the manual-unseal burden**, ++ which intersects the still-OPEN **SEC-003** (custodian assignment + second-person unseal ++ rehearsal) that already keeps `d011-06-vault-unseal` MANUAL and gates D-011's full close. ++ ++### The per-DC node layout (the `modules/node-vm` count/sizing this stage must ratify) ++ ++The site-DOWN / recovery drill the operator wants (D-122) has the **whole DC VM** as its fault ++domain, so intra-DC hyperconvergence is acceptable; role separation mainly buys single-node-failure ++drills, a secondary concern here. ++ ++| Opt | Per-DC layout | nodes/DC | x2 DC | HA semantics | Whole-host cost (MEASURED, all layers) | ++|---|---|---|---|---|---| ++| A | 3 hyperconverged | 3 | 6 | quorum survives 1 loss; size=3 has ZERO rebuild headroom | lightest | ++| **B (rec)** | **4 hyperconverged (= VR0 baseline)** | **4** | **8** | quorum + 4th host = a size=3 re-replication target after a node loss; lowest delta to the proven VR0 placement | 158 vCPU / **838 GiB** / 5.6 TiB (RAM 82%) | ++| C | 3 control + 2 compute + 3 storage | 8 | 16 | cleanest HA/DR, Roosevelt-shaped | 222 vCPU / **790 GiB** / 5.4 TiB (RAM 77%) | ++ ++**Whole-host validation (2026-07-15, MEASURED budget 256 vCPU / 1024 GiB / 10 TiB; model at ++`scratchpad/optc-calc.py`, all layers: Office1 as-built + both DCs + edges + containment overhead).** ++**Both B and C FIT.** The binding resource is **RAM**, not disk (disk ~53-55%, and thin-provisioned) ++and not vCPU (overcommits freely in a sim; C's 87% "allocation" is not a real ceiling). Counter to ++the first-pass intuition, **C is NOT heavier than B** -- at lean-sim node sizing C uses *less* RAM ++(790 vs 838 GiB) because role separation lets the 6 storage nodes run lean (24 GiB) vs B's uniformly ++96-GiB hyperconverged nodes. So **C's real cost is operational (16 node VMs vs 8, larger VR0 delta), ++not capacity.** SENSITIVITY: the knob is control-node RAM -- C with 96-GiB control nodes hits 982 GiB ++(96%, only 42 GiB margin); keep control nodes <= 64 GiB. Per-node RAM sizing dominates the total far ++more than the A/B/C structure does. ++ ++**RULED: Option C (2026-07-15).** The whole-host validation (below) removed the capacity objection -- ++C fits, and is in fact RAM-lighter than B -- so the decision turned on HA/DR quality: C is ++role-separated and Roosevelt-shaped, giving the cleanest site-down AND single-node-failure drills, ++which is the point of the VR1 rehearsal. B (4 hyperconverged/DC) was the runner-up on lowest-delta / ++operational-simplicity grounds (my initial recommendation); A (3 nodes) was rejected as FEWER than ++VR0's 4, reading against the operator's "additional HA nodes." Control nodes are held <= 64 GiB per ++the sensitivity finding. B's rationale is retained here for the record: it was the smallest change ++from VR0's proven placement, with three distinct hosts for quorum plus a fourth for Ceph rebuild. ++ ++**Disk-budget check (validated 2026-07-15, `scripts/dc-dc-ceph-disk-budget.sh`, Option B 4+4, 0.2 ++backup overhead):** PASS across the plausible per-node OSD range -- 300 GiB/node -> 7.18 TiB margin, ++500 GiB -> 5.31 TiB, 700 GiB -> 3.43 TiB. **Disk is NOT the binding constraint for B**, so the ++node-COUNT decision is robust regardless of final OSD sizing. NB the per-node OSD footprint is an ++ASSUMPTION here (the ratified value pends the OSD-sizing decision and should be re-derived from the ++measured DC0 footprint, as Stage 1's own margin was); this run proves headroom, not the final number. ++ ++**Per-node sizing (feeds the module):** at full HA a hyperconverged node runs ~13 control LXD ++containers + an OSD + (on 3 of 4) nova-compute, atop D-040's 8 GiB reserved-host-memory. Propose ++**16 vCPU / 64-96 GiB RAM / OSD-sized disk** per node (lean to 96 GiB -- 64 is tight once D-040 ++reserves 8 and the container stack lands). 8 nodes x 16 vCPU = 128 vCPU and x 96 GiB = 768 GiB, both ++inside vcloud's 256 vCPU / 1 TiB with headroom for Office1 + the edges. ++ ++### What the operator ruled (2026-07-15) ++ ++1. **Node layout:** RULED **Option C** (3 control + 2 compute + 3 storage per DC), sizing above. ++2. **Vault HA backend:** RULED **(v-a)** MySQL-backed + re-add `vault:ha`, charm-verify gated (Status). ++ The SEC-003 unseal-burden intersection stands (3-unit vault = 3x manual unseal). ++3. **Per-node sizing:** RULED (control 16/64/150, compute 12/48/100, storage 8/24/550; Status). ++ ++**Related:** D-009 (decorative HA pattern this executes), D-006 + C1 (vault backend history), ++D-036 (dual VIPs, already present), D-062 (mysql-at-3), D-108 (rbd-mirror stays 1), D-040 ++(reserved memory), D-104 (single-unit JUJU controller -- unaffected; this is the OpenStack control ++plane, not the Juju bootstrap node), SEC-003 (unseal). Does NOT touch DC2/`vr1-dc1` scope. ++ ++## D-122: VR1 site shape -- nested-per-site containment, dark fiber + dedicated per-site L3 ISP, DC edge follows the Office1 pattern ++ ++**Status:** ADOPTED (2026-07-15, operator ruling this session -- answers Stage-3 Ruling 2). Records ++the deployment shape so Stage 3's edge/network steps stop re-deriving it. Governs ++`modules/opnsense-edge` wiring for each DC and the per-site uplink networks. ++ ++**Decision (operator-ruled):** ++ ++- **Each site is nested inside its own containment VM** (`voffice1` for Office1; the `vvr1-dc0` / ++ `vvr1-dc1` analogs per DC), so a "physical site down" and its recovery -- plus site-to-site tests ++ -- are executable as `virsh destroy ` against a single object. This is D-114's model, ++ now affirmed as the shape for the DCs too, not just Office1. ++- **Dark fiber connects DC <-> DC <-> Office** (the `mesh-*` legs) and is **East-West only** -- ++ replication / site-to-site, **never an internet path** (consistent with D-100 and DOCFIX-185's ++ transport model). ++- **Each site has its OWN dedicated L3 "ISP" uplink, not shared between sites.** Concretely: build a ++ per-site WAN network on the **`office1-wan` pattern** (a dedicated NAT'd `/24` uplink segment per ++ site), e.g. `vr1-dc0-wan`, and attach the DC edge's WAN there -- NOT to a mesh leg. ++- **The DC OPNsense edge follows the proven Office1 edge pattern:** sizing **2048 MiB / 2 vCPU / ++ nano image** (carried from the applied `office1_opnsense`, DOCFIX-189), config over the REST API ++ (D-113(a2)). ++- **Edge NIC model = 2-NIC (operator ruling 2026-07-15), matching the baremetal target.** The NIC ++ count is shaped to the Roosevelt hardware: **nodes get 6 NICs** (one per plane, as measured on VR0 ++ openstack0-3), but the **edge gets 2** (WAN + one LAN) -- the six planes are **routed by the fabric** ++ (the `dc-planes` segments are DELIBERATELY isolated L2; OVN/OpenStack own their L3 addressing/routing, ++ not libvirt and not the edge). So the edge is NOT an inter-plane router; it is only the external ++ boundary. This keeps `modules/opnsense-edge` at its existing 2-NIC shape -- **no module extension**. ++ - **WAN** = the dedicated per-site ISP uplink `vr1-dc0-wan` (`172.30.2.0/24`, D-115; register in ++ office1-netbox -- not yet loaded -- and build as a tofu module, promoting off the virsh-created ++ `office1-wan` one-off). ++ - **LAN** = **provider-public** (`10.12.4.0/22`): the edge is the upstream/external gateway the ++ provider network exits through (FIP/SNAT egress + GUA injection per D-100's br-ex/provider model). ++ metal-admin and the other four planes reach external via fabric (OVN) routing, not a direct edge ++ leg; NTP-from-edge (D-107) is served across that fabric routing. ++- **Each site runs its own MAAS controller** (as `voffice1` does), and an existing MAAS controller ++ MAY be used to bootstrap a new site's deployment. ++ ++**Consequence for Stage 3:** the phase2 runbook's Ruling-1/Ruling-2 open items on edge sizing, WAN ++uplink, and LAN role are CLOSED by this entry (build `vr1-dc0-wan`, wire the edge like Office1). The ++remaining Stage-3 blocker is D-121 (node layout/sizing) plus the per-DC MAAS rack-controller source ++(Stage 4 gap -- honest Stage-3 scope is Steps 1-8 until that is stood up). **Related:** D-114 ++(containment VM + MAAS-composed LXD VMs), D-100 (per-site edge, dark fiber), D-107 (headend not a ++core-service provider), DOCFIX-185 (edge is a real-ISP router, not an egress airgap), D-113(a2) ++(edge config over REST). DOCFIX-185's D-100/D-107 amendment note is subsumed here. ++ ++## D-123: VR1 DC site-down DR model -- node-VM placement + the per-DC MAAS headend (resolves a D-122/D-103 tension) ++ ++**Status:** ADOPTED **Model A** (2026-07-15, operator ruling -- read from "Yes, fire off those tasks" ++in response to the A/B question; flag if B was intended). Nodes stay vcloud-level; site-down = destroy ++the `vr1-dc0-*` domain group; `vvr1-dc0` is a MAAS/LXD headend (the `voffice1`/`site-headend-install.sh` ++pattern). **MAAS MODEL RULED (2026-07-15, operator):** ONE MAAS **region on Office1** + a **rack controller per ++DC** (`vvr1-dc0` = the vr1-dc0 rack), matching buildout-design line 110. The rack serves PXE/DHCP ++locally on metal-admin and PROXIES OS images from the Office1 region (so no direct DC-edge egress is ++needed -- that egress worry dissolves under region+rack). `maas-vm-host` registers vcloud's virsh to the ++**Office1 region** (which exists) so MAAS discovers the vr1-dc0 node domains. ++ ++**REMAINING (genuine Stage-4 design, NOT invented here) -- the rack<->region management overlay.** ++`vvr1-dc0` must reach the Office1 region (10.10.0.20:5240) to enroll, and it may NOT sit on Office1's ++site LAN (site isolation, D-122). Per D-100 the management path is the `office1<->dc0` mesh leg -- but ++that segment has **no assigned L3** (the mesh legs are isolated L2), and the rack needs static addresses ++on both the mesh (management) and metal-admin (it can't DHCP -- the planes have no libvirt DHCP, and the ++rack IS the DHCP server there). So concretely standing up `vvr1-dc0` needs a **NetBox-assigned addressing ++pass** for (a) the Office1-region<->DC-rack management subnet on the mesh leg and (b) the rack's ++metal-admin static IP, plus a rack-controller **sizing** decision. This is a D-101-family addressing ++design owned by `dc-dc-phase3`, not a one-line ruling -- the `vvr1-dc0` + `maas-vm-host` blocks stay ++HELD until it lands (assigning IPs by guess would violate hard rule 2). Everything ELSE (Steps 1-8 ++substrate) is wired and validated. ++ ++**The tension.** D-122 records the operator's shape as *"each site nested in its own VM so we can test ++physical site down"* (`virsh destroy ` = site-down). But the ADOPTED seam D-103 + D-114 says ++the OpenStack **node VMs are OpenTofu-created and MAAS merely DISCOVERS them** -- and the built tooling ++implements exactly that: `modules/node-vm` creates the nodes on the **vcloud** host attached to the ++`vr1_dc0_planes`, and `modules/maas-vm-host` registers **vcloud's** virsh to the DC's MAAS so MAAS ++discovers those already-existing domains. D-114 scopes the containment VM to the MAAS/LXD headend + the ++**non-stack** (LXD-composed) machines only -- NOT the OpenStack nodes. So for a DC, the nodes are ++vcloud-level siblings of the headend VM, and "site-down = one `virsh destroy`" is literally true only ++for Office1 (whose services all live inside `voffice1`), not for a DC. ++ ++**Model A (RECOMMENDED) -- nodes vcloud-level; site-down = destroy the domain GROUP.** ++- Node VMs: vcloud-level libvirt domains on `vr1_dc0_planes` (as built). `vvr1-dc0` = a site headend VM ++ (the proven Office1 pattern: `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which ++ already names `vvr1-dc0/vvr1-dc1` as its future targets) running MAAS region+rack + LXD. `maas-vm-host` ++ registers vcloud's virsh -> `vvr1-dc0`'s MAAS, which discovers the nodes. ++- **Site-down DR = destroy the `vr1-dc0-*` domain group** (headend + 8 nodes + edge) as a scripted op ++ (owned by `dc-dc-teardown-rollback.md`), NOT a single `virsh destroy`. ++- Nesting depth = 2 (vcloud -> node -> KVM guest), the SAME as VR0-proven; nova-compute's KVM guests stay ++ shallow. Reuses ALL built tooling unchanged (`node-vm`, `maas-vm-host`, `site-headend-install.sh`). ++- **Closes the "Stage-4 gap":** the per-DC MAAS "rack controller" is just a `vvr1-dc0` headend VM -- an ++ OpenTofu `cloudinit-vm` block shaped like `module "voffice1"` + a `site-headend-install.sh` run. ++ Prepare-able now (opentofu block + the Step-9 `maas-vm-host` wiring). ++ ++**Model B -- true single-VM containment (nodes nested inside `vvr1-dc0`).** ++- Site-down = one `virsh destroy vvr1-dc0` (D-122 literal). But 4-level nesting ++ (vcloud -> vvr1-dc0 -> node -> guest), reshapes the planes + `node-vm` to target `vvr1-dc0`'s libvirt, ++ contradicts the as-built D-103/D-114 discover-OpenTofu-created-nodes flow, and stresses nested-virt for ++ nova-compute. Heaviest delta. ++ ++**Recommendation: Model A** -- least delta to the adopted decisions and the built tree, shallower ++nesting, and it reuses the proven Office1 headend pattern verbatim. It only asks that D-122's DR wording ++be read as "destroy the site's domain GROUP" for DCs (true single-VM destroy remains literal for ++Office1). **What the operator rules:** Model A vs B (one line). **Related:** D-122 (site shape), D-114 ++(containment scope), D-103 (OpenTofu/MAAS seam), and `scripts/site-headend-install.sh` (the reusable ++per-site headend installer). ++ ++## D-124: the Office1-region <-> DC-rack management overlay -- addressing for the MAAS control path ++ ++**Status:** ADOPTED (2026-07-16, operator ruling). **Scheme A** (transit-numbered mesh) + rack sizing ++**4 vCPU / 8192 MiB / 80 GiB** confirmed. The office1<->dc0 mesh leg is a point-to-point transit ++(region <-> rack); metal-admin stays purely node-facing; the rack straddles both. Still to assign in ++office1-netbox (the apex; operator-held token, main-loop can't query it live): the specific transit ++`/30`(or `/31`) CIDR + the rack's metal-admin static IP -- registered via the D-124 importer with ++operator-supplied values (no literal invented in-repo). Original options + rationale retained below. ++ ++**Why this exists (answers "why not just metal-admin space?").** The rack's NODE-facing leg IS in ++metal-admin (`10.12.8.0/22`) -- that is where it serves PXE/DHCP to the vr1-dc0 nodes, correct. But the ++rack must also reach the **MAAS region on Office1** (`10.10.0.20:5240`) to enroll, and: ++- **D-100 rules metal-admin DC-LOCAL** -- the plane deliberately does not extend to Office1 (only the ++ replication plane crosses fiber), and a RULED D-100 sub-item states *"Office1<->DC fiber carries ++ management traffic only (MAAS/Juju/operator)."* So the region<->rack control path is the ++ **`office1<->dc0` mesh leg**, not metal-admin. ++- The rack may NOT sit on Office1's site LAN (`10.10.0.0/22`) either -- that breaks site isolation (D-122). ++ ++So `vvr1-dc0` **straddles two legs**: metal-admin (node-facing, existing space) + the office1<->dc0 mesh ++(region-facing). The mesh leg is today unnumbered isolated L2; this decision numbers it. (Same applies to ++office1<->dc1 for the second DC later; dc0<->dc1 stays replication-only, D-108 -- NOT management.) ++ ++**The addressing choice (operator rules A or B).** Both keep metal-admin DC-local at L2 and both need only ++a tiny link; they differ in whether the rack carries a second address: ++- **(A) Transit-numbered mesh (recommended).** The office1<->dc0 mesh leg gets a small point-to-point ++ transit (a `/30` or `/31`): Office1 region on one end, rack on the other. The region routes to the ++ rack; metal-admin stays purely node-facing. Cleanest role separation, standard inter-site practice, ++ and it generalizes per-leg (office1<->dc1 gets its own transit). The rack has 2 IPs (transit + metal-admin). ++- **(B) Metal-admin routed over the mesh.** No separate transit subnet; the rack's single metal-admin IP ++ is advertised to the region over an unnumbered/`/31` mesh link. Fewer addresses (your instinct -- one ++ rack identity), but it couples metal-admin reachability across the fiber, a soft touch on "DC-local." ++ ++**Where the numbers come from.** Like D-115/D-117, this decision rules the SCHEME; NetBox (office1-netbox, ++the IPAM apex) assigns the actual transit CIDR + the rack's metal-admin IP -- no literal is invented here. ++A candidate management/transit supernet (its own role, mirroring the D-115 Edge role) is a NetBox call; the ++importer will register whatever is ruled. The rack's metal-admin IP lands in the D-120 static band ++(`.2-.49`) of `10.12.8.0/22`, not the node band. ++ ++**Rack-controller sizing (propose to confirm).** A MAAS rack controller is lightweight (it proxies images ++from the region, runs no PostgreSQL): **4 vCPU / 8192 MiB / 80 GiB** is the proposed default -- far below ++voffice1's region+rack+LXD 16/32/600. Confirm or adjust. ++ ++**What the operator rules:** (1) mesh management addressing = A (transit) or B (metal-admin routed); ++(2) rack sizing. Then: ratify the scheme -> NetBox assigns the transit + rack IPs (a small importer, the ++D-120 pattern) -> `vvr1-dc0` (cloudinit-vm on the two legs) + `maas-vm-host` (Step 9) wire against real ++addresses. **Related:** D-100 (DC-local planes + mesh = management), D-101 (address families), D-115 ++(Edge-role precedent for a new transit role), D-120 (static-band convention), D-123 (region+rack model), ++D-107 (per-DC mirror -- why the rack needs no direct egress). +diff --git a/docs/security-ledger.md b/docs/security-ledger.md +index 8f7329f..581022d 100644 +--- a/docs/security-ledger.md ++++ b/docs/security-ledger.md +@@ -18,6 +18,8 @@ per D-069. + | SEC-007 | 2026-07-12 | `~/vr1-office1-creds/` on the jumphost holds the Office1 edge root password + its bcrypt hash + the `office1_svc` SSH PRIVATE key + (added 2026-07-13) the OPNsense **API key/secret** (`opnsense-api.txt`, 0600), all in plaintext (dir 0700, files 0600) | 2026-07-12 edge build; custody detail deliberately not recorded here per D-069 | operator | OPEN -- required for edge management (D-112(c) makes SSH the only management path), so this is a ROTATION obligation, not a delete-me. Rotate at v1 close / if the jumphost is rebuilt or shared | + | SEC-008 | 2026-07-13 | `~/vr1-office1-creds/tailscale-authkey.txt` (vcloud, 0600) -- the operator's 48-char Tailscale auth key for the SELF-HOSTED control plane `tailscale.baldurkeep.com`. Used to enrol `office1-tailscale`. Consumed BY PATH and never printed into a session; the copy shipped to the node was SHREDDED after `tailscale up` (tailscaled holds its own node key now). | 2026-07-13 Office1 Tailscale enrolment; docs/vr1-office1-as-built.md | operator | OPEN -- ROTATION obligation, not a delete-me: the key is still on vcloud for re-enrolment. Revoke/reissue on the control server at v1 close, or immediately if vcloud is rebuilt or shared. | + | SEC-009 | 2026-07-15 | Credential/env SPRAWL + world-readable exposure on vcloud: three env files sat loose in `~` outside the consolidated `~/vr1-office1-creds/` -- `.vr1-netbox.env` (upstream NetBox token), `vr1-office1.env` (edge root-password hash + SSH key path), and `vr1-stage1.env` which held `TF_VAR_maas_api_key` (a MAAS API secret) at mode **0664 (group/world-readable)**; `tailscale-authkey.txt` in the creds dir was also 0664. | docs/changelog-20260715-creds-consolidation.md | operator | **REMEDIATED 2026-07-15** -- all three moved into `~/vr1-office1-creds/`, every sensitive file `chmod 600`. STANDING CONVENTION established (below). Note: this is a CLOSED in-house test; the rotation obligations of SEC-005/006/007 still apply at v1 close, this row only closes the SPRAWL + PERMS exposure. | ++| SEC-010 | 2026-07-16 | **metal-admin DC-LOCAL invariant (D-052/D-100) is PRESERVED but NOT ENFORCED in the committed Stage-3 config.** The `vvr1_dc0` rack straddles metal-admin (10.12.8.0/22, DC-local) + the office1<->dc0 transit (crosses fiber). Its committed cloud-init pins static IPs only -- **no `net.ipv4.ip_forward=0` sysctl, no host firewall on the transit leg.** Cross-plane routing (metal-admin <-> the whole Office1 /22, bidirectional) is blocked ONLY by Ubuntu's distro default; the deferred MAAS-rack install could silently flip it. A MAAS rack proxies at the application layer and needs no kernel forwarding, so pinning is free. | 2026-07-16 plane-segregation review (advisor + 2 read-only agents); `opentofu/main.tf` vvr1_dc0 :376-439 | operator | **OPEN -- PRE-APPLY hardening.** Add `net.ipv4.ip_forward=0` (+ v6) via cloud-init `write_files`/`bootcmd` sysctl on the rack, and an nft/ufw rule pinning the transit leg to MAAS region ports only. Same pin must follow onto `voffice1` when its transit leg is wired (currently single-homed, so F1 is latent). Region route must target only the rack transit /30, never 10.12.8.0/22 (NetBox/region config, off-repo). Close BEFORE `tofu apply`. | ++| SEC-011 | 2026-07-16 | **Node least-connectivity gap (not an L2 breach).** Under D-121 Option C role separation, all 8 nodes get a uniform 6-plane NIC set, so a ceph-osd STORAGE node has a leg on provider-public (external/FIP) + data-tenant (tenant geneve) -- planes it never binds per D-052. Planes stay isolated L2 (no crosstalk), but it is unnecessary external attack surface that VR0's genuinely-hyperconverged all-6 node did not have. | 2026-07-16 plane-segregation review; `opentofu/main.tf` `local.vr1_dc0_node_nics` :320-352 | operator | **OPEN -- PRE-APPLY hardening.** Replace uniform 6-NIC with per-role NIC sets (storage/ceph-osd drops provider-public + data-tenant at minimum). Confirm the exact per-role plane set against the regenerated D-052 bundle bindings -- do NOT invent it. Attack-surface reduction; not blocking, but cheap to do before apply. | + + **STANDING CONVENTION (SEC-009, 2026-07-15): per-site credential/env consolidation.** ALL sensitive + files AND environment/config files for a site live in a single `~/-creds/` folder on vcloud, +diff --git a/docs/session-ledger.md b/docs/session-ledger.md +index c53184b..b87023b 100644 +--- a/docs/session-ledger.md ++++ b/docs/session-ledger.md +@@ -38,9 +38,19 @@ _Re-seeded from the 2026-07-15 scan. Re-run `bash scripts/ledger-scan.sh` to ref + the pre-DC-DC controller HA/backup session). + - **OPEN security rows:** SEC-001, SEC-003, SEC-004, SEC-005, SEC-006, SEC-007, **SEC-008**. + (SEC-008 -- the vcloud re-enrolment key -- was MISSING from the prior block.) +-- **Next-free numbers:** D = **121**, DOCFIX = 196, BUNDLEFIX = 013. +- (Since the prior block: D-119 + D-120 assigned and ADOPTED, and DOCFIX-195 assigned by this change +- -- so next-free advanced D 119->121 and DOCFIX 195->196.) ++- **Next-free numbers:** D = **125**, DOCFIX = 196, BUNDLEFIX = 013. ++ (D-124 PROPOSED 2026-07-16: the Office1-region<->DC-rack management overlay addressing -- numbers the ++ office1<->dc0 mesh leg so the D-123 rack can reach the MAAS region. metal-admin stays DC-local (D-100); ++ the rack straddles metal-admin (nodes) + the mesh (region). Operator rules scheme A (transit /30-31) vs ++ B (metal-admin routed) + rack sizing; then NetBox assigns + the headend wires.) ++ (2026-07-15: D-121, D-122, D-123 assigned -- next-free advanced D 121->124. D-121 ADOPTED (node ++ layout = Option C; vault sub-ruling RESOLVED = v-a). D-122 ADOPTED (site shape + 2-NIC fabric-routed ++ edge). **D-123 ADOPTED Model A** (nodes vcloud-level; site-down = destroy the vr1-dc0-* group). MAAS ++ model RULED: region on Office1 + rack controller per DC (vvr1-dc0 = vr1-dc0 rack, proxies images from ++ the region). REMAINING (genuine Stage-4 design, NOT invented): the rack<->region management-overlay ++ ADDRESSING -- the office1<->dc0 mesh leg has no assigned L3 and the rack needs static IPs on mesh + ++ metal-admin. A NetBox D-101-family addressing pass owned by dc-dc-phase3; vvr1-dc0 + maas-vm-host stay ++ HELD until it lands.) + - **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) + ABOVE the real high-water mark anywhere in `docs/` or `runbooks/` prose -- `ledger-scan` reads + prose and a decoy token inflates the next-free counter. This has bitten twice. +@@ -114,6 +124,117 @@ write-back stays DEFERRED to end-of-deployment. This closed the last piece of cl + `docs/changelog-20260715-d120-load-and-iprange-tooling.md`. + + ++**PLANE-SEGREGATION REVIEW of the Stage-3 commit (2026-07-16; advisor + 2 read-only agents).** ++Reviewed the committed networking/plane changes (`a48a60f`, NOT applied) for crosstalk / segregation ++breaches. **VERDICT: no crosstalk, no active violation -- committed state is SAFE.** All six `vr1-dc0-*` ++planes + all three `mesh-*` legs are isolated L2 (measured: `net-dumpxml` shows no forward/ip/nat, ++isolation ENFORCED BY CONSTRUCTION in `dc-planes`/`mesh-link`); the edge is 2-NIC so it structurally ++cannot reach the 5 non-LAN planes; `vr1-dc0-wan` NAT touches only the edge WAN. metal-admin DC-LOCAL ++(D-052/D-100) is **preserved but not ENFORCED** -- it rests on unpinned host defaults. **Two pre-apply ++hardening items logged as SEC-010 (rack `ip_forward=0` + transit-leg firewall -- the one real ++plane-bridge) and SEC-011 (node role-scoped NIC sets -- least-connectivity).** Say plainly at handoff: ++"committed state is safe; two items to close before `tofu apply`." Plus three notes: ++- **FUNCTIONAL (fix before apply, not security):** the rack's committed route `to 10.10.0.0/22 via ++ ` points at a next-hop with **no VM attached** -- `voffice1` is single-homed ++ (`network_names = [office1-local]` only, main.tf:184), NOT on `mesh-vr1-dc0-office1`. As committed, ++ apply yields a rack routing to a nonexistent peer, and the Office1<->DC region path is not closed. ++ Either wire `voffice1`'s transit leg or gate the rack route until it exists. ++- **SEGREGATION CHECK (first boot):** the rack NIC->plane mapping (`enp1s0`->metal-admin, ++ `enp2s0`->transit) is assumed PCI-order==name; if swapped, the metal-admin /22 lands on the ++ Office1-facing wire. Verify via MAC (`virsh domiflist` x `network_names` order) before trusting ++ addressing. This is a segregation check, not cosmetic. ++- **DEFERRED (track):** the edge WAN<->provider-public firewall is post-boot REST (opnsense-api), ++ not in the commit -- "enforced by absence" covers plane REACHABILITY only, not full edge hardening. ++When `voffice1`'s transit leg is later wired it becomes an office1-local<->transit bridge and inherits ++the SAME `ip_forward=0` pin requirement (a richer surface than the rack: MAAS region + LXD host). ++ ++ ++**STAGE 3 IN PROGRESS (branch `dc-dc-stage3-phase2-dc-substrate`) -- two rulings landed 2026-07-15.** ++Walked the phase2 runbook and surfaced its blocking decisions; the operator ruled both: ++- **D-121 (ADOPTED IN PART).** VR1 makes the decorative single-unit control plane REAL: 14 services ++ scale 1->3 (the 12 hacluster-backed APIs + ceph-radosgw + rabbitmq + vault; `ceph-rbd-mirror` stays ++ 1 per D-108). This is D-009's "mechanical 1->3" pulled forward from Roosevelt -- but it is a PLACEMENT ++ rewrite (every app's `to:` must spread across 3 distinct hosts), not a num_units tweak. **Node layout ++ = Option C** (3 control + 2 compute + 3 storage per DC, 16 node VMs), ruled after a whole-host ++ validation PROVED it fits: measured budget 256 vCPU / 1024 GiB / 10 TiB, Option C allocates ++ 222 / 790 / 5.4Ti (RAM binding at 77%; model at `scratchpad/optc-calc.py`). Ruled sizing: control ++ 16/64GiB/150GiB, compute 12/48/100, storage 8/24/550; keep control <=64 GiB (96 GiB -> 96% RAM). ++ **Vault-HA sub-ruling RESOLVED = (v-a)** (2026-07-15): keep MySQL-backed (on the already-3-unit ++ mysql-innodb-cluster), scale to 3, re-add `vault:ha`. GATE: verify the `1.8/stable` vault charm does ++ HA on the MySQL backend with 3 units; if not, evaluate Raft (NOT etcd) via D-068. 3-unit vault ++ triples the unseal burden (intersects OPEN SEC-003). ++- **D-122 (ADOPTED).** VR1 site shape: each site nested in its own containment VM (site-down/recovery + ++ site-to-site drills); dark fiber DC-DC-Office is East-West ONLY; each site has its own dedicated L3 ++ ISP uplink (build `vr1-dc0-wan` on the office1-wan pattern); DC edge follows the Office1 pattern ++ (2048MiB/2vCPU/nano, REST-API config, LAN faces the site network); each site its own MAAS controller. ++ Subsumes the DOCFIX-185 D-100/D-107 amendment note. ++- **Fix-forward DONE (2026-07-15):** the phase2 runbook is reconciled to post-D-119 naming ++ (`vr1-dc0`/`vr1-dc1`, bare `dcN` rejected) and its obsolete Steps 4-5 rewritten to the D-113(a2) ++ REST-API bootstrap (the module already dropped `config_iso_path`); the D-121/D-122 rulings folded in ++ so its Ruling-1/Ruling-2 STOP-gates clear. Honest achievable first-run scope stays **Steps 1-8** -- ++ Step 9 (MAAS registration) gates on the per-DC MAAS "rack controller", now RESOLVED IN PRINCIPLE by ++ **D-123** (PROPOSED, recommend Model A): it is just a `vvr1-dc0` site headend VM (the proven Office1 ++ pattern -- `cloudinit-vm` like `voffice1` + `scripts/site-headend-install.sh`, which already names ++ `vvr1-dc0` as a target). Once Model A is ruled it is prepare-able (opentofu block + Step-9 wiring), ++ which effectively closes the Stage-4 gap for Step 9. D-123 also reconciles the D-122 "one virsh ++ destroy = site-down" wording (literal for Office1; for a DC, site-down = destroy the `vr1-dc0-*` ++ domain group). See `docs/changelog-20260715-d121-d122-stage3-ha-nodelayout.md`. ++- **EDGE NIC MODEL RULED (D-122 refinement, 2026-07-15): 2-NIC, baremetal-matched.** NIC count mirrors ++ the Roosevelt hardware -- **nodes = 6 NICs** (one per plane, measured on VR0 openstack0-3), **edge = 2** ++ (WAN + LAN). The six planes are **routed by the fabric** (OVN/OpenStack on the deliberately isolated-L2 ++ `dc-planes` segments), so the edge is NOT an inter-plane router -- only the external boundary. No ++ `opnsense-edge` module extension needed. Edge **WAN = `vr1-dc0-wan`** (`172.30.2.0/24`), edge ++ **LAN = provider-public** (`10.12.4.0/22`; edge is the external gateway per D-100). This SUPERSEDES the ++ earlier "build an office1-network-analog site LAN" flag -- no new site-internal segment is needed. ++- **STAGE-3 PREP BATCH DONE 2026-07-15 (authoring + offline validation; NOTHING applied).** See ++ `docs/changelog-20260715-stage3-prereqs-prep.md`. Gauntlet ALL GREEN (61), `tofu validate` Success, ++ repo-lint 0-fail. ++ - **OpenTofu:** new `modules/site-wan` (NAT /24 uplink; schema confirmed via `tofu providers schema`, ++ validated) + `main.tf` Stage-3 section wired (NOT APPLIED): `vr1_dc0_wan` (172.30.2.0/24), the 2-NIC ++ edge (LAN=provider-public, WAN=uplink), 8 Option C node VMs via `for_each` (metal-admin PXE-first), ++ netem HELD (Step 11; mesh bridge = virbr7 measured), stale `vr1_dc1_planes` comment fixed. **HELD ++ pending D-123:** the `vvr1-dc0` headend + `maas-vm-host` (Step 9) blocks. ++ - **NetBox:** `netbox/dc-edge-wan-import.py` + harness (58/58) -- registers the two DC-edge /24s ++ (172.30.2/3.0/24, role edge, site-scoped), DRY-BY-DEFAULT with whole-plan preflight. READY to ++ `--commit` (command in the tool header; token operator-held; guard blocked the live read). ++ - **Bundle HA overlay:** `overlays/dc-ha-scaleup.yaml` (Stage-5 prep) -- 14 services ->3, one-per- ++ control-node placement (`{{VR1_DC0_CONTROL_0N}}` tokens, Stage-4-bind), + 3 correctness items the ++ agent caught: rabbitmq `min-cluster-size:3`, hacluster `cluster_count:3`, re-declare `vault-hacluster` ++ (BUNDLEFIX-002 had commented it out) + re-add `vault:ha`. ++ - **The failed opentofu agent** (transient API stall, wrote nothing) was covered in-session by the main ++ loop; the netbox + overlay agents completed clean. ++- **D-123/D-124 RULED + `vvr1-dc0` RACK WIRED (2026-07-16).** D-123 Model A + MAAS region-on-Office1 + ++ rack-per-DC; D-124 Scheme A (office1<->dc0 mesh transit /30) + rack sizing 4/8192/80. `main.tf` now ++ wires **`module "vvr1_dc0"`** (cloudinit-vm, 4/8192/80, two legs: metal-admin + mesh-office1 transit, ++ static IPs from NEW tfvars `vr1_dc0_rack_*` -- NetBox-assigned, not invented; interface-naming trap ++ flagged) + a `mesh-link` `network_name` output. `tofu validate` Success; opentofu-validate 11/11. ++ **Step-9 `maas-vm-host` STAYS DEFERRED (DOCFIX-179):** adding the `provider "maas"` block would force ++ every plan (incl. the Steps 1-8 apply) to demand MAAS creds, and vr1-dc0's MAAS doesn't exist until ++ vvr1-dc0 is up -- wire it as its own step once the rack MAAS is reachable. The rack MAAS INSTALL needs ++ a rack-only mode of `site-headend-install.sh` (follow-up). ++- **D-124 rack-mgmt importer BUILT (2026-07-16):** `netbox/dc-rack-mgmt-import.py` + harness (96/96), ++ parameterized (`--transit-cidr`/`--rack-ip`, no invented literal), whole-plan preflight. Gauntlet 62. ++ **Two operator preconditions it flags before `--commit`:** (a) SEED a dedicated `transit` role in ++ office1-netbox (none exists; `infra` is shared, not reused) -- die-if-absent, like `edge`; (b) DECIDE ++ the transit supernet -- the importer assumes it nests under `10.12.0.0/16` (Cloud), but D-124 mirrors ++ the Edge role (which got its OWN `172.30.0.0/16`), so a dedicated mgmt/transit supernet may be wanted ++ (then a one-line `CONTAINER` const update). Transit prefix scope = site vr1-dc0 (region may be preferred). ++- **RESIDUAL for a smooth batch deploy:** (1) NetBox: seed the `transit` role + decide the transit ++ supernet, then run the D-124 importer (+ the D-115 DC-edge importer) `--commit` with the assigned CIDR/ ++ IP, then fill the `vr1_dc0_rack_*` tfvars; (2) runtime: `opnsense-prep-image.sh` for the edge nano; ++ (3) Step-9 maas-vm-host wiring once the rack MAAS is up (**the rack-only `site-headend-install.sh --role ++ rack` mode is now DONE, 32/32** -- it enrolls a DC rack to Office1's region; `maas init rack` confirmed); ++ (4) Stage-4: render the HA overlay's control-node machine IDs + `--overlay --dry-run`; (5) D-121 ++ verifies (vault MySQL-HA at 3; hacluster cluster_count). All operator-gated / Stage-4. ++- **`main.tf` stale comment (flagged):** the commented `vr1_dc1_planes` block (~L38-50) still says ++ "wait for NetBox to assign D-101's supernet" -- STALE per D-115 (assigned `10.12.64.0/19`). Doc-only ++ pass did not touch `main.tf`; the `main.tf` owner should correct the comment. ++- **STALE gate corrected:** the runbook + `main.tf` say `vr1-dc1` is blocked on "NetBox assigning the ++ supernet" -- D-115 ALREADY assigned it (`10.12.64.0/19`, imported to office1-netbox). vr1-dc1 stays ++ out of scope this pass by SEQUENCING, not a missing literal. Do NOT uncomment `vr1_dc1_planes`. ++- **Logged, NOT executed (adjacent improvement, hard rule 1):** promote `scratchpad/optc-calc.py` into ++ a committed whole-host capacity calculator with a harness (sibling to `dc-dc-ceph-disk-budget.sh`). ++ + The authoritative per-stage tracker is `docs/dc-dc-deployment-workflow.md` (stage table + tooling + gap register). Read it FIRST. This list is only what that doc does not already carry. + +diff --git a/netbox/dc-edge-wan-import.py b/netbox/dc-edge-wan-import.py +new file mode 100644 +index 0000000..d8c4033 +--- /dev/null ++++ b/netbox/dc-edge-wan-import.py +@@ -0,0 +1,184 @@ ++#!/usr/bin/env python3 ++""" ++Register the VR1 DC-edge simulated-ISP WAN /24s in office1-netbox (the VR1 apex). ++ ++D-115 (ADOPTED) carved a v4 Edge role at 172.30.0.0/16 and blessed office1-wan as ++172.30.1.0/24 (site vr1-off1). The two DC edges follow the same role, one /24 each: ++ ++ 172.30.2.0/24 role edge, active scope=dcim.site:vr1-dc0 (vr1-dc0 sim-ISP WAN) ++ 172.30.3.0/24 role edge, active scope=dcim.site:vr1-dc1 (vr1-dc1 sim-ISP WAN) ++ ++These were NOT in the D-115 office carve (which only owned the office row) and no ++prior import covered the DC-edge /24s -- so this tool registers them, and nothing ++else. It creates NEITHER the Edge role NOR the 172.30.0.0/16 container NOR the DC ++sites: all three are PRECONDITIONS (d115-office-carve.py owns the role + container; ++the vr1-dc0/vr1-dc1 sites already exist in the apex). A missing precondition is a ++hard STOP, never a silent create. ++ ++DRY BY DEFAULT -- nothing is written without --commit. ++ ++WRITING UPSTREAM IS GATED IN CODE. SANDBOX_HOSTS is the allowlist; a --commit at ++any other host REFUSES without --yes-write-upstream. WHOLE-PLAN PREFLIGHT: the role, ++the container, the in-container placement of every /24, and every target site are all ++checked BEFORE any create, so a bad plan cannot half-write the apex (the failure that ++bit roles-aggregates-import.py -- it created 4 objects then died on the 5th). ++ ++Usage (on office1-netbox / through a tunnel, with the sandbox token): ++ NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN= python3 netbox/dc-edge-wan-import.py ++ ... same, add --commit, to write. ++""" ++import argparse ++import ipaddress ++import json ++import os ++import sys ++import urllib.error ++import urllib.parse ++import urllib.request ++ ++UA = "curl/8.5.0" # upstream 403s the default python UA -- see platform-traps.md ++ ++# A sandbox is local, or the known Office1 sandbox address. Anything else is treated ++# as the production apex and requires the explicit upstream flag. ++SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} ++ ++ROLE_SLUG = "edge" # created by d115-office-carve.py; a precondition here ++CONTAINER = "172.30.0.0/16" # Edge role container; a precondition here ++STATUS = "active" # a real segment on the wire, like office1-wan .1/24 ++ ++# (cidr, target-site slug, description). Site-scoped exactly like office1-wan ++# 172.30.1.0/24 -> vr1-off1 (d115-office-carve.py). The DC sites already exist in the ++# apex draft (vr1-dc0 / vr1-dc1); this tool resolves them, it never creates them. ++TARGETS = [ ++ ("172.30.2.0/24", "vr1-dc0", "vr1-dc0 simulated-ISP edge WAN (D-115)"), ++ ("172.30.3.0/24", "vr1-dc1", "vr1-dc1 simulated-ISP edge WAN (D-115)"), ++] ++ ++ ++def die(msg: str): ++ print(f"FAIL: {msg}", file=sys.stderr) ++ sys.exit(2) ++ ++ ++class NB: ++ """Stdlib NetBox client -- same shape as the other sandbox-loop tools; UA-aware so ++ it is not 403'd by the upstream User-Agent filter.""" ++ def __init__(self, base, token): ++ self.base = base.rstrip("/") ++ self.token = token ++ ++ def _req(self, method, path, body=None): ++ data = json.dumps(body).encode() if body is not None else None ++ req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, ++ headers={"Authorization": f"Token {self.token}", ++ "Accept": "application/json", ++ "Content-Type": "application/json", ++ "User-Agent": UA}) ++ try: ++ with urllib.request.urlopen(req, timeout=45) as r: ++ return json.load(r) if r.status != 204 else None ++ except urllib.error.HTTPError as exc: ++ detail = exc.read().decode(errors="replace")[:300] ++ if exc.code == 403 and "v1 token" in detail: ++ die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " ++ "nbt_., not the API's bare `token` field.") ++ if exc.code == 403: ++ die(f"403 on {path}. If curl works with this token, it is the upstream " ++ f"User-Agent filter, NOT the token (references/platform-traps.md).") ++ die(f"HTTP {exc.code} {method} {path}: {detail}") ++ ++ def one(self, path, **flt): ++ res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") ++ return res["results"][0] if res["results"] else None ++ ++ def create(self, path, payload): ++ return self._req("POST", f"{path}/", payload) ++ ++ ++def get_nb(base, token): ++ """Client factory -- the injection seam the harness overrides to drive main() ++ against an in-memory fake without a live NetBox.""" ++ return NB(base, token) ++ ++ ++def main() -> int: ++ ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) ++ ap.add_argument("--commit", action="store_true", ++ help="WRITE. Default is a DRY RUN that writes nothing.") ++ ap.add_argument("--yes-write-upstream", action="store_true", ++ help="Required (with --commit) to write to a NON-sandbox NetBox.") ++ args = ap.parse_args() ++ ++ url = os.environ.get("NETBOX_URL") ++ token = os.environ.get("NETBOX_TOKEN") ++ if not url or not token: ++ die("NETBOX_URL and NETBOX_TOKEN must be set.") ++ ++ host = (urllib.parse.urlparse(url).hostname or url).lower() ++ is_sandbox = host in SANDBOX_HOSTS ++ print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") ++ if args.commit and not is_sandbox and not args.yes_write_upstream: ++ die(f"REFUSING to --commit to '{host}': not a known sandbox, so treated as the " ++ f"PRODUCTION apex. Re-run with --yes-write-upstream if that is intended.") ++ ++ print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" ++ if not args.commit else "\n*** COMMITTING. ***") ++ ++ nb = get_nb(url, token) ++ ++ # WHOLE-PLAN PREFLIGHT -- validate the ENTIRE plan (role, container, in-container ++ # placement, every target site) BEFORE any create, so a later-target failure (e.g. ++ # a missing vr1-dc1 site) cannot leave vr1-dc0 written and the apex half-populated. ++ container = ipaddress.ip_network(CONTAINER) ++ for cidr, _site_slug, _desc in TARGETS: ++ if not ipaddress.ip_network(cidr).subnet_of(container): ++ die(f"{cidr} is outside the Edge container {CONTAINER} -- refusing to place it.") ++ ++ role = nb.one("ipam/roles", slug=ROLE_SLUG) ++ if role is None: ++ die(f"role '{ROLE_SLUG}' absent -- seed the D-115 carve first " ++ f"(netbox/d115-office-carve.py). Refusing to place edge /24s with no role.") ++ ++ if nb.one("ipam/prefixes", prefix=CONTAINER) is None: ++ die(f"Edge container {CONTAINER} absent -- seed the D-115 carve first " ++ f"(netbox/d115-office-carve.py). Refusing to place /24s in an unallocated /16.") ++ ++ # Resolve every site and its already-present state up front -- THIS is the whole-plan ++ # gate: if any site is missing the run dies here, before the create loop below runs. ++ plan = [] ++ for cidr, site_slug, desc in TARGETS: ++ site = nb.one("dcim/sites", slug=site_slug) ++ if site is None: ++ die(f"site '{site_slug}' absent -- cannot scope {cidr}. The DC sites are a " ++ f"precondition (they already exist in the apex); this tool never creates them.") ++ present = nb.one("ipam/prefixes", prefix=cidr) is not None ++ plan.append((cidr, site, desc, present)) ++ ++ created = existing = 0 ++ ++ print("\nEdge DC /24s:") ++ for cidr, site, desc, present in plan: ++ if present: ++ print(f" EXISTS {cidr}") ++ existing += 1 ++ continue ++ if not args.commit: ++ print(f" [dry-run] would CREATE {cidr} role={ROLE_SLUG} " ++ f"scope=dcim.site:{site['slug']}") ++ created += 1 ++ continue ++ payload = {"prefix": cidr, "role": role["id"], "status": STATUS, ++ "description": desc, "scope_type": "dcim.site", "scope_id": site["id"]} ++ o = nb.create("ipam/prefixes", payload) ++ print(f" CREATED {cidr} (id={o['id']}) role={ROLE_SLUG} scope={site['slug']}") ++ created += 1 ++ ++ verb = "would create" if not args.commit else "created" ++ print(f"\n{'='*66}\n{verb}: {created} already present: {existing}") ++ if not args.commit: ++ print("DRY RUN -- nothing was written. Re-run with --commit.") ++ return 0 ++ ++ ++if __name__ == "__main__": ++ sys.exit(main()) +diff --git a/netbox/dc-rack-mgmt-import.py b/netbox/dc-rack-mgmt-import.py +new file mode 100644 +index 0000000..b0d41d1 +--- /dev/null ++++ b/netbox/dc-rack-mgmt-import.py +@@ -0,0 +1,300 @@ ++#!/usr/bin/env python3 ++""" ++Register the D-124 Office1-region <-> DC-rack MANAGEMENT TRANSIT addressing in ++office1-netbox (the VR1 IPAM apex): the point-to-point transit on the office1<->dc0 ++mesh leg, plus the rack's metal-admin static IP. Two objects, nothing else. ++ ++D-124 (ADOPTED 2026-07-16, Scheme A -- transit-numbered mesh) rules that the ++region<->rack MAAS control path rides a small point-to-point transit on the ++office1<->dc0 mesh leg (NOT metal-admin, which D-100 keeps DC-local and node-facing). ++The rack (vvr1-dc0) STRADDLES both legs: the transit (region-facing) and metal-admin ++(node-facing, 10.12.8.0/22). So this tool registers exactly: ++ ++ 1. the transit prefix role=transit, scope=dcim.site:vr1-dc0 (the /30 or /31) ++ 2. the rack ip-address <rack-ip>/22 in metal-admin, D-120 static band .2-.49 ++ ++Both LITERALS ARE OPERATOR INPUTS (--transit-cidr / --rack-ip): D-124, like D-115/ ++D-117, rules the SCHEME and leaves the actual CIDR + IP to office1-netbox (the apex; ++operator-held token, this loop cannot query it live). NO literal is invented in-repo. ++ ++This tool creates NEITHER the transit ROLE nor any container/site: all are ++PRECONDITIONS (a missing one is a hard STOP, never a silent create) -- exactly as ++dc-edge-wan-import.py treats the `edge` role + the 172.30.0.0/16 container. ++ ++DRY BY DEFAULT -- nothing is written without --commit. ++ ++WRITING UPSTREAM IS GATED IN CODE. SANDBOX_HOSTS is the allowlist; a --commit at any ++other host REFUSES without --yes-write-upstream. WHOLE-PLAN PREFLIGHT: the transit ++role, the container the transit nests under, the transit CIDR shape+placement, the ++rack IP's band (metal-admin, static .2-.49, NOT the .1 gateway), and the target site ++are ALL checked BEFORE any create -- so a bad plan (e.g. a rack IP that is the .1 ++gateway) cannot leave the transit prefix written and the apex half-populated (the ++failure that bit roles-aggregates-import.py: 4 objects created, then it died on the 5th). ++ ++SCOPING / ROLE choices flagged for the operator (see the D-124 return notes): ++ * ROLE_SLUG = "transit": D-124 calls for the transit's OWN role, "mirroring the ++ D-115 Edge role" (a DEDICATED role). An `infra` role exists in the apex whose ++ description reads "ptp transit, device loopbacks, RR loopbacks, anycast" -- it is ++ NOT reused here because it is explicitly SHARED infrastructure, not the dedicated ++ per-purpose role D-124 asks for. Operator to seed/confirm the `transit` slug in ++ office1-netbox (as d115-office-carve.py seeded `edge`). A missing role dies here. ++ * CONTAINER = "10.12.0.0/16" (Cloud): the transit CIDR must be subnet_of an EXISTING ++ container. The apex's only IPv4 container the transit plausibly nests under is Cloud ++ 10.12.0.0/16 (ipam/aggregates is empty; no dedicated management/transit supernet ++ exists yet). This gates ACCEPTANCE only -- NetBox auto-nests by CIDR, we set no ++ parent. If the operator numbers the transit outside Cloud, the tool DIES rather ++ than misplacing it. Update CONTAINER if a dedicated transit supernet is carved. ++ * transit prefix SCOPE = dcim.site:vr1-dc0. How an office1<->dc0 link "should" scope ++ (region vs. site) is unruled; site-scoping to vr1-dc0 mirrors the D-115 DC-edge ++ /24s (172.30.2.0/24 -> vr1-dc0). FLAGGED -- operator may prefer a region scope. ++ * The rack IP band (metal-admin 10.12.8.0/22, static .2-.49, gw .1) is enforced ++ ARITHMETICALLY (D-120/D-124), NOT via a prefix-object lookup: metal-admin is not ++ registered as a /22 prefix in the apex draft, and the band is a convention, not a ++ container. This asymmetry with the transit check is intentional. ++ ++Usage (on office1-netbox / through a tunnel, with the sandbox token): ++ NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=<tok> \ ++ python3 netbox/dc-rack-mgmt-import.py --transit-cidr <A.B.C.D/30> --rack-ip <10.12.8.X> ++ ... same, add --commit, to write. ++""" ++import argparse ++import ipaddress ++import json ++import os ++import sys ++import urllib.error ++import urllib.parse ++import urllib.request ++ ++UA = "curl/8.5.0" # upstream 403s the default python UA -- see platform-traps.md ++ ++# A sandbox is local, or the known Office1 sandbox address. Anything else is treated ++# as the production apex and requires the explicit upstream flag. ++SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} ++ ++ROLE_SLUG = "transit" # the transit's OWN role (D-124); a precondition here ++CONTAINER = "10.12.0.0/16" # Cloud; the transit CIDR must be subnet_of this (a precondition) ++SITE_SLUG = "vr1-dc0" # the transit prefix is site-scoped here (mirrors the D-115 edge /24s) ++STATUS = "active" # a real segment on the wire ++ ++# metal-admin band arithmetic (D-052/D-124). The rack's metal-admin IP lands in the ++# D-120 static band .2-.49 of this /22, and NOT the .1 gateway. ++METAL_ADMIN = "10.12.8.0/22" ++STATIC_BAND_LOW = 2 # .2 (first static site-services address, D-120) ++STATIC_BAND_HIGH = 49 # .49 (last static site-services address, D-120) ++ ++RACK_DNS = "vvr1-dc0" # the rack-controller VM (D-124 cloudinit-vm on the two legs) ++TRANSIT_DESC = "office1<->dc0 management transit -- region<->rack MAAS control path (D-124 Scheme A)" ++RACK_DESC = "vr1-dc0 MAAS rack controller (vvr1-dc0) metal-admin static IP (D-124; D-120 static band)" ++ ++ ++def die(msg: str): ++ print(f"FAIL: {msg}", file=sys.stderr) ++ sys.exit(2) ++ ++ ++class NB: ++ """Stdlib NetBox client -- same shape as the other sandbox-loop tools; UA-aware so ++ it is not 403'd by the upstream User-Agent filter.""" ++ def __init__(self, base, token): ++ self.base = base.rstrip("/") ++ self.token = token ++ ++ def _req(self, method, path, body=None): ++ data = json.dumps(body).encode() if body is not None else None ++ req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, ++ headers={"Authorization": f"Token {self.token}", ++ "Accept": "application/json", ++ "Content-Type": "application/json", ++ "User-Agent": UA}) ++ try: ++ with urllib.request.urlopen(req, timeout=45) as r: ++ return json.load(r) if r.status != 204 else None ++ except urllib.error.HTTPError as exc: ++ detail = exc.read().decode(errors="replace")[:300] ++ if exc.code == 403 and "v1 token" in detail: ++ die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " ++ "nbt_<key>.<plaintext>, not the API's bare `token` field.") ++ if exc.code == 403: ++ die(f"403 on {path}. If curl works with this token, it is the upstream " ++ f"User-Agent filter, NOT the token (references/platform-traps.md).") ++ die(f"HTTP {exc.code} {method} {path}: {detail}") ++ ++ def one(self, path, **flt): ++ res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") ++ return res["results"][0] if res["results"] else None ++ ++ def create(self, path, payload): ++ return self._req("POST", f"{path}/", payload) ++ ++ ++def get_nb(base, token): ++ """Client factory -- the injection seam the harness overrides to drive main() ++ against an in-memory fake without a live NetBox.""" ++ return NB(base, token) ++ ++ ++def parse_transit_cidr(raw: str) -> ipaddress.IPv4Network: ++ """The transit CIDR must be a valid /30 or /31 network (host bits clear), subnet_of ++ CONTAINER. Shape is validated here; containment against CONTAINER in preflight.""" ++ try: ++ net = ipaddress.ip_network(raw, strict=True) ++ except ValueError as exc: ++ die(f"--transit-cidr {raw!r} is not a valid network (host bits set?): {exc}") ++ if net.version != 4: ++ die(f"--transit-cidr {raw} must be IPv4 (the office1<->dc0 transit is v4).") ++ if net.prefixlen not in (30, 31): ++ die(f"--transit-cidr {raw} must be a /30 or /31 point-to-point (got /{net.prefixlen}).") ++ return net ++ ++ ++def parse_rack_ip(raw: str) -> ipaddress.IPv4Address: ++ """Accept a bare host (10.12.8.X) or a masked form; if masked, the mask MUST be the ++ metal-admin /22. Returns the host address; band placement is checked in preflight.""" ++ try: ++ if "/" in raw: ++ iface = ipaddress.ip_interface(raw) ++ if iface.network.prefixlen != 22: ++ die(f"--rack-ip {raw} carries /{iface.network.prefixlen}; metal-admin is a /22. " ++ f"Pass a bare host (10.12.8.X) or the /22 form.") ++ host = iface.ip ++ else: ++ host = ipaddress.ip_address(raw) ++ except ValueError as exc: ++ die(f"--rack-ip {raw!r} is not a valid IPv4 address: {exc}") ++ if host.version != 4: ++ die(f"--rack-ip {raw} must be IPv4 (metal-admin is v4).") ++ return host ++ ++ ++def main() -> int: ++ ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) ++ ap.add_argument("--transit-cidr", default=os.environ.get("TRANSIT_CIDR"), ++ help="REQUIRED. The /30 or /31 point-to-point for the office1<->dc0 " ++ "transit leg (operator-supplied, NetBox-assigned; env: TRANSIT_CIDR).") ++ ap.add_argument("--rack-ip", default=os.environ.get("RACK_IP"), ++ help="REQUIRED. The rack's metal-admin static IP within 10.12.8.0/22, " ++ "static band .2-.49 (operator-supplied; env: RACK_IP).") ++ ap.add_argument("--commit", action="store_true", ++ help="WRITE. Default is a DRY RUN that writes nothing.") ++ ap.add_argument("--yes-write-upstream", action="store_true", ++ help="Required (with --commit) to write to a NON-sandbox NetBox.") ++ args = ap.parse_args() ++ ++ url = os.environ.get("NETBOX_URL") ++ token = os.environ.get("NETBOX_TOKEN") ++ if not url or not token: ++ die("NETBOX_URL and NETBOX_TOKEN must be set.") ++ ++ # Inputs are args-or-env, so they cannot be argparse required=True (that would break ++ # the env fallback). Hand-roll the missing checks, each with its own die. ++ if not args.transit_cidr: ++ die("--transit-cidr (or TRANSIT_CIDR) is REQUIRED -- the NetBox-assigned transit " ++ "/30 or /31. No literal is invented in-repo (D-124).") ++ if not args.rack_ip: ++ die("--rack-ip (or RACK_IP) is REQUIRED -- the rack's NetBox-assigned metal-admin " ++ "static IP. No literal is invented in-repo (D-124).") ++ ++ host = (urllib.parse.urlparse(url).hostname or url).lower() ++ is_sandbox = host in SANDBOX_HOSTS ++ print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") ++ if args.commit and not is_sandbox and not args.yes_write_upstream: ++ die(f"REFUSING to --commit to '{host}': not a known sandbox, so treated as the " ++ f"PRODUCTION apex. Re-run with --yes-write-upstream if that is intended.") ++ ++ print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" ++ if not args.commit else "\n*** COMMITTING. ***") ++ ++ # ---- WHOLE-PLAN PREFLIGHT: validate the ENTIRE plan (both objects) BEFORE any ++ # create, so a bad rack IP (e.g. the .1 gateway) cannot leave the transit written. ---- ++ ++ # (a) Transit CIDR shape + containment (local, no NetBox needed). ++ transit = parse_transit_cidr(args.transit_cidr) ++ container = ipaddress.ip_network(CONTAINER) ++ if not transit.subnet_of(container): ++ die(f"transit {transit} is outside the container {CONTAINER} (Cloud) -- refusing to " ++ f"place it. If the transit is carved elsewhere, that is an operator/D-124 call.") ++ ++ # (b) Rack IP band: within metal-admin /22, in the D-120 static band .2-.49, NOT the ++ # .1 gateway (a distinct die so the ".1 rejected" property is unambiguous). ++ rack = parse_rack_ip(args.rack_ip) ++ metal = ipaddress.ip_network(METAL_ADMIN) ++ if rack not in metal: ++ die(f"rack IP {rack} is outside metal-admin {METAL_ADMIN} -- refusing to place it.") ++ gateway = metal.network_address + 1 ++ band_low = metal.network_address + STATIC_BAND_LOW ++ band_high = metal.network_address + STATIC_BAND_HIGH ++ if rack == gateway: ++ die(f"rack IP {rack} is the .1 GATEWAY of {METAL_ADMIN} -- refusing (D-120: .1 is the " ++ f"site gateway, not a static-service address).") ++ if not (band_low <= rack <= band_high): ++ die(f"rack IP {rack} is outside the D-120 static band {band_low}-{band_high} of " ++ f"{METAL_ADMIN} -- the rack's metal-admin IP must land in .2-.49 (not the dynamic/" ++ f"node bands).") ++ rack_addr = f"{rack}/{metal.prefixlen}" # NetBox stores the host WITH the plane mask ++ ++ nb = get_nb(url, token) ++ ++ # (c) Preconditions in the apex: the transit role, the container prefix, the site. ++ role = nb.one("ipam/roles", slug=ROLE_SLUG) ++ if role is None: ++ die(f"role '{ROLE_SLUG}' absent -- D-124 calls for the transit's OWN dedicated role " ++ f"(mirroring the D-115 `edge` role); seed it in office1-netbox first. Refusing to " ++ f"place the transit prefix with no role.") ++ ++ if nb.one("ipam/prefixes", prefix=CONTAINER) is None: ++ die(f"container {CONTAINER} (Cloud) absent in the apex -- refusing to place the transit " ++ f"in an unallocated supernet.") ++ ++ site = nb.one("dcim/sites", slug=SITE_SLUG) ++ if site is None: ++ die(f"site '{SITE_SLUG}' absent -- cannot scope the transit prefix. The DC site is a " ++ f"precondition (it already exists in the apex); this tool never creates it.") ++ ++ # (d) Present-state (idempotency) -- resolved up front, part of the whole-plan gate. ++ transit_str = str(transit) ++ transit_present = nb.one("ipam/prefixes", prefix=transit_str) is not None ++ rack_present = nb.one("ipam/ip-addresses", address=rack_addr) is not None ++ ++ created = existing = 0 ++ ++ # ---- CREATE (transit prefix, then rack ip-address) -- only reached once the WHOLE ++ # plan above is viable, so this loop cannot half-write the apex. ---- ++ print("\nTransit prefix (office1<->dc0 mesh leg):") ++ if transit_present: ++ print(f" EXISTS {transit_str}") ++ existing += 1 ++ elif not args.commit: ++ print(f" [dry-run] would CREATE {transit_str} role={ROLE_SLUG} " ++ f"scope=dcim.site:{SITE_SLUG}") ++ created += 1 ++ else: ++ payload = {"prefix": transit_str, "role": role["id"], "status": STATUS, ++ "description": TRANSIT_DESC, "scope_type": "dcim.site", "scope_id": site["id"]} ++ o = nb.create("ipam/prefixes", payload) ++ print(f" CREATED {transit_str} (id={o['id']}) role={ROLE_SLUG} scope={SITE_SLUG}") ++ created += 1 ++ ++ print("\nRack metal-admin static IP:") ++ if rack_present: ++ print(f" EXISTS {rack_addr}") ++ existing += 1 ++ elif not args.commit: ++ print(f" [dry-run] would CREATE {rack_addr} dns={RACK_DNS} (metal-admin static band)") ++ created += 1 ++ else: ++ payload = {"address": rack_addr, "status": STATUS, ++ "dns_name": RACK_DNS, "description": RACK_DESC} ++ o = nb.create("ipam/ip-addresses", payload) ++ print(f" CREATED {rack_addr} (id={o['id']}) dns={RACK_DNS}") ++ created += 1 ++ ++ verb = "would create" if not args.commit else "created" ++ print(f"\n{'='*66}\n{verb}: {created} already present: {existing}") ++ if not args.commit: ++ print("DRY RUN -- nothing was written. Re-run with --commit.") ++ return 0 ++ ++ ++if __name__ == "__main__": ++ sys.exit(main()) +diff --git a/opentofu/main.tf b/opentofu/main.tf +index f75981d..48eb444 100644 +--- a/opentofu/main.tf ++++ b/opentofu/main.tf +@@ -35,11 +35,14 @@ module "vr1_dc0_storage" { + target_path = var.vr1_dc0_pool_path + } + +-# ---- DC2: PLANES deferred (operator ruling 2026-07-10, Option B) -- wait for +-# NetBox to assign D-101's supernet/ULA/GUA before wiring vr1_dc1_planes. Do NOT +-# fill in guessed CIDRs to make it "work" sooner. When NetBox assigns it, add a +-# `vr1_dc1_planes` variable (same shape as `vr1_dc0_planes`) and uncomment this block. +-# The storage pool + mesh legs are NOT address-dependent, so they ARE wired now. ++# ---- vr1-dc1 (VR1's SECOND DC): PLANES deferred by SEQUENCING, not a missing ++# literal. Its supernet IS assigned -- D-115 moved it to 10.12.64.0/19 (ADOPTED, ++# imported to office1-netbox), superseding the old "wait for NetBox" premise. It ++# stays out of scope until vr1-dc0 completes (operator ruling 2026-07-10, Option B: ++# vr1-dc0-first). When vr1-dc1 is started, add a `vr1_dc1_planes` variable (same ++# shape as `vr1_dc0_planes`, CIDRs derived per D-115) and uncomment this block. ++# Do NOT uncomment it now. The storage pool + mesh legs are NOT address-dependent, ++# so they ARE wired now. + # + # module "vr1_dc1_planes" { + # source = "./modules/dc-planes" +@@ -266,3 +269,171 @@ moved { + from = module.mesh_dc2_office1 + to = module.mesh_vr1_dc1_office1 + } ++ ++# ===================================================================== ++# STAGE 3 -- vr1-dc0 substrate (runbooks/dc-dc-phase2-tofu-dc-substrate.md). ++# Authored 2026-07-15. NOT YET APPLIED -- every block below is a gated Stage-3 ++# mutation; `tofu apply` is the operator's step (Steps 8/11 of the runbook). ++# Governing rulings: D-100 (edge/fabric), D-121 (Option C node layout + sizing), ++# D-122 (2-NIC fabric-routed edge; dedicated per-site ISP uplink). ++# ++# D-123 Model A + MAAS model RULED (region on Office1 + rack per DC) + D-124 ++# ADOPTED (Scheme A: office1<->dc0 mesh transit; rack sizing 4/8192/80). The ++# `vvr1-dc0` rack controller IS wired below -- its two static IPs come from ++# office1-netbox via variables (tfvars), NOT invented here. The Step-9 ++# `maas-vm-host` registration is STILL DEFERRED (DOCFIX-179): adding the ++# `provider "maas"` block it needs would force EVERY plan (incl. the Steps 1-8 ++# substrate apply) to demand MAAS creds, and vr1-dc0's MAAS does not exist until ++# vvr1-dc0 is up and its rack is installed. Wire maas-vm-host + the maas provider ++# as its own step once the rack MAAS is reachable (runbook Step 9). The blocks ++# below are the D-123-independent substrate (D-103/D-114 as-built) plus the rack. ++# ===================================================================== ++ ++# D-122: the dedicated per-site simulated-ISP uplink (172.30.2.0/24, D-115). ++module "vr1_dc0_wan" { ++ source = "./modules/site-wan" ++ network_name = "vr1-dc0-wan" ++ cidr = "172.30.2.0/24" ++ # mtu defaults to 1500 (ISP-uplink domain; NOT the jumbo planes/mesh). ++} ++ ++# D-122: the DC edge -- 2-NIC (WAN + LAN), Office1 pattern (2048/2/nano). The six ++# planes are fabric-routed (OVN/OpenStack), so the edge is only the external ++# boundary: LAN = provider-public (edge is its external gateway, D-100), WAN = ++# the dedicated uplink above. ++module "vr1_dc0_opnsense" { ++ source = "./modules/opnsense-edge" ++ vm_name = "vr1-dc0-opnsense" ++ memory_mib = 2048 ++ vcpu = 2 ++ pool_name = module.vr1_dc0_storage.pool_name ++ # No disk_size_bytes: nano direct-copy, sized by opnsense-prep-image.sh (DOCFIX-189). ++ # PREREQUISITE (runbook Step 4): the prepped nano image must exist at this path. ++ base_volume_path = "/var/lib/libvirt/vr1/vr1-dc0/opnsense-26.1-nano.qcow2" ++ lan_network_name = module.vr1_dc0_planes.network_names["provider-public"] ++ wan_network_name = module.vr1_dc0_wan.network_name ++} ++ ++# D-121 Option C node layout: 3 control + 2 compute + 3 storage (16/12/8 vCPU, ++# 64/48/24 GiB, 150/100/550 GiB). Six NICs each (one per plane, baremetal-matched ++# D-122); metal-admin FIRST = the PXE/boot plane (D-052 default binding). ++locals { ++ vr1_dc0_node_nics = [ ++ module.vr1_dc0_planes.network_names["metal-admin"], # PXE / boot fabric first ++ module.vr1_dc0_planes.network_names["provider-public"], ++ module.vr1_dc0_planes.network_names["metal-internal"], ++ module.vr1_dc0_planes.network_names["data-tenant"], ++ module.vr1_dc0_planes.network_names["storage"], ++ module.vr1_dc0_planes.network_names["replication"], ++ ] ++ ++ vr1_dc0_nodes = { ++ "vr1-dc0-control-01" = { vcpu = 16, mem = 65536, disk_gib = 150 } ++ "vr1-dc0-control-02" = { vcpu = 16, mem = 65536, disk_gib = 150 } ++ "vr1-dc0-control-03" = { vcpu = 16, mem = 65536, disk_gib = 150 } ++ "vr1-dc0-compute-01" = { vcpu = 12, mem = 49152, disk_gib = 100 } ++ "vr1-dc0-compute-02" = { vcpu = 12, mem = 49152, disk_gib = 100 } ++ "vr1-dc0-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550 } ++ "vr1-dc0-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550 } ++ "vr1-dc0-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550 } ++ } ++} ++ ++module "vr1_dc0_node" { ++ for_each = local.vr1_dc0_nodes ++ ++ source = "./modules/node-vm" ++ vm_name = each.key ++ vcpu = each.value.vcpu ++ memory_mib = each.value.mem ++ disk_size_bytes = each.value.disk_gib * 1024 * 1024 * 1024 ++ pool_name = module.vr1_dc0_storage.pool_name ++ network_names = local.vr1_dc0_node_nics ++} ++ ++# netem (runbook Step 11) -- HELD as a comment. It is a POST-apply step that runs ++# SSH-wrapped provisioners from Office1 to the vcloud host, so it needs a real ++# `vcloud_host_ssh_target` (a runtime value) and the netem params are an unruled ++# D-100 sub-item (gap #11, placeholder only). Measured input ready: the ++# vr1-dc0<->office1 mesh leg is bridge `virbr7`. Wire at Step 11, not before. ++# module "netem_vr1_dc0_office1" { ++# source = "./modules/netem-link" ++# link_name = "vr1-dc0-office1" ++# bridge_name = "virbr7" # measured 2026-07-15 ++# vcloud_host_ssh_target = "<real ssh target -- runtime>" ++# netem_args = "<PLACEHOLDER: buildout-design S6 same-metro lean; D-100 gap #11 unruled>" ++# } ++ ++# ===================================================================== ++# D-123 / D-124: the vr1-dc0 MAAS RACK CONTROLLER (vvr1-dc0). An Ubuntu VM (reusing ++# the noble base) that will run a MAAS rack controller enrolled to the Office1 MAAS ++# REGION. Two legs: metal-admin (serves PXE/DHCP to the vr1-dc0 nodes) + the ++# office1<->dc0 mesh transit (reaches the region, D-124 Scheme A). It PROXIES OS ++# images from the region, so it needs no direct DC egress (D-107). NOT YET APPLIED. ++# The MAAS rack INSTALL itself is a gated post-boot step (an adapted ++# site-headend-install.sh rack-only mode -- follow-up), like voffice1's install. ++# ===================================================================== ++module "vvr1_dc0" { ++ source = "./modules/cloudinit-vm" ++ vm_name = "vvr1-dc0" ++ vcpu = 4 # D-124 (rack controller: lightweight) ++ memory_mib = 8192 # D-124 ++ disk_size_bytes = 80 * 1024 * 1024 * 1024 # D-124: 80 GiB ++ pool_name = module.vr1_dc0_storage.pool_name ++ base_volume_path = module.ubuntu_noble_base.path ++ expose_nested_virt = false # rack-only: the region + its LXD host compose VMs, not the rack ++ ++ network_names = [ ++ module.vr1_dc0_planes.network_names["metal-admin"], # leg 1: node-facing PXE/DHCP ++ module.mesh_vr1_dc0_office1.network_name, # leg 2: office1<->dc0 transit (region) ++ ] ++ ++ user_data = <<-EOT ++ #cloud-config ++ hostname: vvr1-dc0 ++ fqdn: vvr1-dc0.${var.domain_suffix} ++ manage_etc_hosts: true ++ users: ++ - name: jessea123 ++ groups: [adm, sudo] ++ shell: /bin/bash ++ sudo: "ALL=(ALL) NOPASSWD:ALL" ++ ssh_authorized_keys: ++ - ${trimspace(file(var.office1_ssh_pubkey_path))} ++ package_update: true ++ packages: ++ - qemu-guest-agent ++ runcmd: ++ - [systemctl, enable, --now, qemu-guest-agent] ++ EOT ++ ++ meta_data = <<-EOT ++ instance-id: vvr1-dc0-d123 ++ local-hostname: vvr1-dc0 ++ EOT ++ ++ # STATIC on both legs (planes + mesh have NO DHCP). IPs are NetBox-assigned ++ # (D-124), supplied via tfvars -- no literal here. INTERFACE-NAMING TRAP: the ++ # guest NIC names (enp1s0/enp2s0 vs ens3/...) depend on machine type and are NOT ++ # known until first boot -- CONFIRM on boot and adjust `match` before relying on ++ # this (documented trap; modules/opnsense-edge + the runbook). metal-admin gives ++ # the rack no default route; the region route rides the transit toward Office1 ++ # (10.10.0.0/22, as-built D-115). ++ network_config = <<-EOT ++ version: 2 ++ ethernets: ++ admin: ++ match: ++ name: "enp1s0" ++ set-name: admin ++ addresses: ["${var.vr1_dc0_rack_metal_admin_ip}/22"] ++ mgmt: ++ match: ++ name: "enp2s0" ++ set-name: mgmt ++ addresses: ["${var.vr1_dc0_rack_transit_ip}/${var.vr1_dc0_rack_transit_prefix}"] ++ routes: ++ - to: "10.10.0.0/22" ++ via: "${var.vr1_dc0_rack_transit_peer_ip}" ++ EOT ++} +diff --git a/opentofu/modules/mesh-link/outputs.tf b/opentofu/modules/mesh-link/outputs.tf +index 91f8bcc..097e972 100644 +--- a/opentofu/modules/mesh-link/outputs.tf ++++ b/opentofu/modules/mesh-link/outputs.tf +@@ -3,6 +3,11 @@ output "network_id" { + value = libvirt_network.link.id + } + ++output "network_name" { ++ description = "libvirt network name (mesh-<link_name>) -- for a consumer's network_names list (e.g. the vr1-dc0 rack's transit NIC, D-124)." ++ value = libvirt_network.link.name ++} ++ + # A "bridge_name" output (the OS-level bridge interface backing this network, + # needed by whatever applies `tc netem` to it) is deliberately NOT included + # here: `bridge` is documented as a configurable input block (delay, +diff --git a/opentofu/modules/site-wan/main.tf b/opentofu/modules/site-wan/main.tf +new file mode 100644 +index 0000000..8b7c3c0 +--- /dev/null ++++ b/opentofu/modules/site-wan/main.tf +@@ -0,0 +1,36 @@ ++# site-wan: NAT'd per-site ISP uplink (see variables.tf for rationale). ++# ++# SCHEMA NOTE: this provider (registry.opentofu.org/dmacvicar/libvirt) uses ++# attribute-style nested objects, confirmed 2026-07-15 via ++# `tofu providers schema -json` (the method dc-planes/main.tf calls for): ++# forward = { mode = "nat" } (nesting=single; no top-level `mode`) ++# ips = [ { address, prefix } ] (nesting=list) ++# mtu = { size = N } (nesting=single; matches dc-planes) ++# domain = { name = ... } (nesting=single; matches dc-planes) ++# This mirrors the live `office1-wan` (virsh): <forward mode='nat'>, .1/24. ++# DHCP is intentionally OMITTED -- the only client is the edge, whose WAN is ++# static (office1-opnsense's WAN is .2, outside office1-wan's DHCP range). ++ ++resource "libvirt_network" "site_wan" { ++ name = var.network_name ++ autostart = true ++ ++ forward = { ++ mode = "nat" ++ } ++ ++ domain = { ++ name = var.network_name ++ } ++ ++ mtu = { ++ size = var.mtu ++ } ++ ++ ips = [ ++ { ++ address = cidrhost(var.cidr, 1) ++ prefix = tonumber(split("/", var.cidr)[1]) ++ } ++ ] ++} +diff --git a/opentofu/modules/site-wan/outputs.tf b/opentofu/modules/site-wan/outputs.tf +new file mode 100644 +index 0000000..1f260c0 +--- /dev/null ++++ b/opentofu/modules/site-wan/outputs.tf +@@ -0,0 +1,4 @@ ++output "network_name" { ++ description = "The libvirt network name, for a consumer's wan_network_name." ++ value = libvirt_network.site_wan.name ++} +diff --git a/opentofu/modules/site-wan/variables.tf b/opentofu/modules/site-wan/variables.tf +new file mode 100644 +index 0000000..8a11535 +--- /dev/null ++++ b/opentofu/modules/site-wan/variables.tf +@@ -0,0 +1,31 @@ ++# site-wan: a per-site simulated-ISP uplink network (D-100 / D-122). NAT'd /24, ++# the site edge's WAN leg attaches here. Generalizes the virsh-created ++# `office1-wan` (172.30.1.0/24) into a reusable module so every per-site ISP ++# uplink is tofu-managed (D-122 / operator 2b, 2026-07-15). ++# ++# MTU is the ISP-uplink domain: 1500, NOT the jumbo 9000 the planes/mesh use. ++# This is a hard distinction -- the simulated ISP link mirrors a real 1500-MTU ++# internet uplink (see the two-MTU-domains note in the session ledger). Do not ++# raise this to 9000. ++ ++variable "network_name" { ++ description = "libvirt network name, e.g. \"vr1-dc0-wan\"." ++ type = string ++} ++ ++variable "cidr" { ++ description = <<-EOT ++ The uplink /24, e.g. "172.30.2.0/24" (Edge role 172.30.0.0/16, D-115). ++ The network gateway is the .1 host of this CIDR (cidrhost(cidr, 1)); the ++ site edge's WAN interface takes a STATIC address on this segment (e.g. .2), ++ configured on the edge itself via the REST API -- not by DHCP here (the ++ edge WAN is static, exactly as office1-opnsense's WAN 172.30.1.2 is). ++ EOT ++ type = string ++} ++ ++variable "mtu" { ++ description = "Uplink MTU. Default 1500 (the ISP-uplink domain; NEVER jumbo)." ++ type = number ++ default = 1500 ++} +diff --git a/opentofu/modules/site-wan/versions.tf b/opentofu/modules/site-wan/versions.tf +new file mode 100644 +index 0000000..f50888b +--- /dev/null ++++ b/opentofu/modules/site-wan/versions.tf +@@ -0,0 +1,11 @@ ++# Provider source mapping for this module (DOCFIX-179). Child modules MUST ++# declare their own required_providers so OpenTofu maps the local name ++# `libvirt` to dmacvicar/libvirt; version is pinned once at the ROOT ++# (opentofu/versions.tf), child modules declare source only. ++terraform { ++ required_providers { ++ libvirt = { ++ source = "dmacvicar/libvirt" ++ } ++ } ++} +diff --git a/opentofu/variables.tf b/opentofu/variables.tf +index 25e0aa3..9718811 100644 +--- a/opentofu/variables.tf ++++ b/opentofu/variables.tf +@@ -96,3 +96,28 @@ variable "voffice1_disk_bytes" { + description = "voffice1 boot disk size in bytes." + type = number + } ++ ++# ---- D-124: vr1-dc0 MAAS rack controller (vvr1-dc0) addressing. NO defaults -- ++# these come from office1-netbox (the IPAM apex) via the D-124 importer, then ++# into a tfvars entry. Do NOT invent them here (hard rule 2). The rack straddles ++# two legs: metal-admin (node-facing PXE/DHCP) + the office1<->dc0 mesh transit ++# (region-facing, Scheme A). ---- ++variable "vr1_dc0_rack_metal_admin_ip" { ++ description = "vvr1-dc0 rack static IP in metal-admin (10.12.8.0/22), in the .2-.49 static band, not the .1 gateway. NetBox-assigned (D-124)." ++ type = string ++} ++ ++variable "vr1_dc0_rack_transit_ip" { ++ description = "vvr1-dc0 rack IP on the office1<->dc0 mesh transit (D-124 Scheme A). NetBox-assigned." ++ type = string ++} ++ ++variable "vr1_dc0_rack_transit_prefix" { ++ description = "Transit link prefix length (30 or 31, D-124 Scheme A)." ++ type = number ++} ++ ++variable "vr1_dc0_rack_transit_peer_ip" { ++ description = "Office1 region's IP on the transit link -- the rack's next-hop toward the MAAS region (10.10.0.0/22). NetBox-assigned (D-124)." ++ type = string ++} +diff --git a/overlays/dc-ha-scaleup.yaml b/overlays/dc-ha-scaleup.yaml +new file mode 100644 +index 0000000..4262fb9 +--- /dev/null ++++ b/overlays/dc-ha-scaleup.yaml +@@ -0,0 +1,183 @@ ++# overlays/dc-ha-scaleup.yaml ++# ++# D-121 (ADOPTED IN PART, 2026-07-15) VR1 HA scale-up overlay -- makes the ++# decorative single-unit control plane REAL by scaling 14 services from ++# num_units: 1 to 3. VR1-SPECIFIC: VR0/testcloud stays single-unit BY DESIGN ++# (D-009), so this is an overlay, NOT a bundle.yaml edit -- same pattern as ++# overlays/dc-dc-ipv6-family-matrix.yaml. Apply ALONGSIDE bundle.yaml's other ++# overlays, e.g.: ++# juju deploy ./bundle.yaml \ ++# --overlay overlays/octavia-pki.yaml \ ++# --overlay "overlays/${DC}-hostnames.yaml" \ ++# --overlay overlays/dc-ha-scaleup.yaml \ ++# -m "${DC_MODEL}" ++# ++# NOT YET APPLIED to any live model -- authored prep-only (STAGE 5 prep; the ++# per-DC bundle deploy is two stages out). No live cloud reached this session. ++# ++# ===================================================================== ++# ##### STAGE-4 BINDING REQUIRED -- READ BEFORE RENDER ##### ++# Every `to:` target below is a {{TOKEN}} placeholder for one of the three ++# Option C control nodes (vr1-dc0-control-01..03, D-121). The REAL Juju machine ++# IDs do NOT exist yet -- they bind at Stage 4, AFTER MAAS enlists the Option C ++# node VMs. Do NOT invent concrete `lxd:N` numbers here: a wrong number ++# silently mis-places a unit and defeats the anti-affinity that is the entire ++# HA value. Render this file (substitute the three tokens with the measured, ++# enlisted control-node machine IDs) ONLY at Stage 4/5, exactly as ++# dc-dc-ipv6-family-matrix.yaml's VIP tokens are rendered. The rendered ++# bundle.yaml `machines:` block MUST define those same three IDs. ++# {{VR1_DC0_CONTROL_01}} -> control node 1 machine id ++# {{VR1_DC0_CONTROL_02}} -> control node 2 machine id ++# {{VR1_DC0_CONTROL_03}} -> control node 3 machine id ++# ===================================================================== ++# ++# PLACEMENT / ANTI-AFFINITY (D-121 balance rule). Juju bundles have NO dedicated ++# anti-affinity primitive; one-per-node is expressed POSITIONALLY -- a 3-entry ++# `to:` list binds unit0->entry0, unit1->entry1, unit2->entry2. Every scaled app ++# below uses the SAME three-control-node spread, so: ++# - each app lands exactly one unit per control node (14 apps x 1 = ~14 API ++# LXD containers per control node -- the even ~14/node density the 64 GiB ++# control-node sizing was validated for, D-121); and ++# - the RAM-heavy stateful trio (mysql-innodb-cluster already 3 in the base ++# bundle, rabbitmq-server + vault scaled here) auto-balances to one replica ++# per control node -- never two stacked on one node. ++# ++# THREE ITEMS BELOW GO BEYOND A LITERAL `num_units: 3` BUMP. They are NOT ++# scope-creep: each is the same class of "mechanical 1->3 hides a gotcha" that ++# D-121 itself flags (D-121 caught the placement rewrite; these are the two it ++# under-states plus the one its own wording got wrong), and an overlay titled ++# "make HA real" that shipped decorative HA would not meet its stated purpose. ++# All three are called out in the delivery changelog, not slipped in: ++# (A) rabbitmq-server min-cluster-size: 3. Per D-009 AMENDMENT (2026-07-02): ++# "rabbitmq scale-up is NOT mechanical without min-cluster-size" -- without ++# it the charm accepts client relations BEFORE the cluster forms (same race ++# as the D-062 mysql formation failure). Roosevelt/VR1 delta = set it ++# alongside num_units: 3. rabbitmq clusters natively (quorum), NO hacluster. ++# (B) hacluster cluster_count: 3 (all 12 active hacluster subs + the re-added ++# vault-hacluster). The base bundle pins cluster_count: 1 -- explicitly ++# "decorative on single-unit testcloud" (D-009 / BUNDLEFIX-003). At 3 units ++# with cluster_count still 1, pacemaker can bootstrap the VIP off a single ++# node = the SAME decorative behavior D-121 exists to eliminate. 3 makes ++# the charm wait for a real 3-node quorum before configuring resources. ++# VERIFY at deploy (not fetched this session): confirm the hacluster ++# 2.4/stable charm treats cluster_count as "peers required before the ++# cluster forms / VIP binds" -- the base bundle explicitly pinned it to 1 ++# as decorative, and real 3-node quorum needs 3 regardless of the charm ++# default. Flag if `juju config hacluster` disagrees. ++# (C) vault-hacluster is DEFINED here, not just related. D-121's prose says it ++# "exists among the hacluster subs" -- it does NOT: bundle.yaml line ~923 ++# has it COMMENTED OUT (removed by BUNDLEFIX-002 when vault was de-HA'd, ++# C1). So the re-added `[vault:ha, vault-hacluster:ha]` relation would ++# reference a non-existent app (provider-bundle-check rule 4 would FAIL). ++# This overlay re-declares it, matching the other haclusters' shape. ++# ++# VAULT BACKEND (D-121 sub-ruling = v-a, RESOLVED 2026-07-15): vault stays ++# MYSQL-backed (on the already-3-unit mysql-innodb-cluster via vault-mysql-router, ++# unchanged) and scales to 3 -- NO etcd, NO Raft here. GATE (Stage 5, before ++# commit): verify the deployed vault 1.8/stable charm actually does HA ++# leader-election on the MySQL storage backend at 3 units. If it does NOT: do ++# NOT fall back to etcd -- evaluate Raft integrated storage and route to D-068. ++# 3-unit vault ALSO triples the manual-unseal burden (intersects OPEN SEC-003). ++# ++# NOT changed here (deliberately, per D-121): mysql-innodb-cluster / ovn-central ++# / ceph-mon (already 3 in the base bundle); ceph-osd / nova-compute (scale-out, ++# sized to node count -- Option C = ceph-osd 3, nova-compute 2, on the storage / ++# compute nodes, NOT control-node containers); ceph-rbd-mirror stays 1 (D-108); ++# memcached / designate-bind / glance-simplestreams-sync unchanged. ++# ===================================================================== ++ ++applications: ++ ++ # ---- Clustered API charms (hacluster + dual VIP already wired in base) ---- ++ # Each: 1 -> 3 units, one per control node (positional anti-affinity). ++ keystone: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ glance: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ neutron-api: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ nova-cloud-controller: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ placement: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ cinder: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ openstack-dashboard: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ octavia: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ barbican: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ magnum: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ designate: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ ceph-radosgw: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ # ---- RabbitMQ: native quorum clustering, NO hacluster. min-cluster-size is ++ # MANDATORY at multi-unit (D-009 AMENDMENT 2026-07-02, item A above). ---- ++ rabbitmq-server: ++ num_units: 3 ++ options: ++ min-cluster-size: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ # ---- Vault: MySQL-backed (v-a), scale to 3, re-HA'd via vault-hacluster ++ # (defined below) + the relation re-added at the foot of this file. ---- ++ vault: ++ num_units: 3 ++ to: ["lxd:{{VR1_DC0_CONTROL_01}}", "lxd:{{VR1_DC0_CONTROL_02}}", "lxd:{{VR1_DC0_CONTROL_03}}"] ++ ++ # ---- vault-hacluster: RE-DECLARED (item C above). Removed by BUNDLEFIX-002 ++ # when vault was de-HA'd; D-121 (v-a) brings it back. Shape matches the ++ # other hacluster subs in bundle.yaml, with cluster_count: 3 (real HA). ---- ++ vault-hacluster: {charm: hacluster, channel: 2.4/stable, options: {cluster_count: 3}, bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal, pacemaker-remote: metal-internal, peer-availability: metal-internal}} ++ ++ # ---- cluster_count: 1 -> 3 for the 12 active hacluster subs (item B above). ++ # Overlay apply deep-merges options per-key -- the SAME mechanism ++ # dc-dc-ipv6-family-matrix.yaml already relies on (it supplies only ++ # `options:` for keystone et al. and depends on charm/channel/bindings ++ # surviving from base). So ONLY cluster_count is overridden here; each ++ # sub's base bindings/charm/channel are preserved. ---- ++ keystone-hacluster: {options: {cluster_count: 3}} ++ glance-hacluster: {options: {cluster_count: 3}} ++ neutron-api-hacluster: {options: {cluster_count: 3}} ++ nova-cloud-controller-hacluster: {options: {cluster_count: 3}} ++ placement-hacluster: {options: {cluster_count: 3}} ++ openstack-dashboard-hacluster: {options: {cluster_count: 3}} ++ cinder-hacluster: {options: {cluster_count: 3}} ++ octavia-hacluster: {options: {cluster_count: 3}} ++ barbican-hacluster: {options: {cluster_count: 3}} ++ magnum-hacluster: {options: {cluster_count: 3}} ++ ceph-radosgw-hacluster: {options: {cluster_count: 3}} ++ designate-hacluster: {options: {cluster_count: 3}} ++ ++relations: ++ # ---- Re-add vault HA (removed by BUNDLEFIX-002; D-121 v-a re-adds it). ++ # vault already carries the `ha: metal-internal` binding in the base ++ # bundle, and vault-hacluster is declared above. ---- ++ - [vault:ha, vault-hacluster:ha] +diff --git a/runbooks/dc-dc-phase2-tofu-dc-substrate.md b/runbooks/dc-dc-phase2-tofu-dc-substrate.md +index a57aa64..fa61feb 100644 +--- a/runbooks/dc-dc-phase2-tofu-dc-substrate.md ++++ b/runbooks/dc-dc-phase2-tofu-dc-substrate.md +@@ -3,10 +3,26 @@ + From Office1, script each DC's "physical" layer into existence: the node-VM + libvirt domains MAAS will discover and deploy, the DC's plane networks, and + its OPNsense edge -- then register the DC's libvirt host to that DC's own +-MAAS rack controller as a virsh VM-host. This runbook is written **DC1-first**. +-DC2 is explicitly gated (see the callout below) and is NOT part of this +-session's run until that gate clears -- do not duplicate these steps for DC2 +-with invented values in the meantime. ++MAAS rack controller as a virsh VM-host. This runbook builds **`vr1-dc0`** ++(VR1's FIRST DC -- the build target this pass). **`vr1-dc1`** (VR1's SECOND ++DC) is explicitly gated (see the callout below) and is NOT part of this ++session's run until that gate clears -- do not duplicate these steps for ++`vr1-dc1` with invented values in the meantime. ++ ++!!! **NAMING -- this runbook was written pre-D-119 and has since been ++ reconciled to the region-qualified namespace.** Its original prose spoke ++ the bare `dcN` namespace (`dc1` = the build target, `dc2` = the second ++ DC). **D-119 REJECTS bare `dcN` for VR1 datacenters** -- the token `dc0` ++ already means VR0's live DC in `scripts/lib-net.sh`, so an unqualified ++ `dcN` is a latent off-by-one. The mapping applied throughout this file is: ++ runbook "DC1" -> **`vr1-dc0`** (already live in `opentofu/main.tf` as the ++ `vr1_dc0_*` modules), runbook "DC2" -> **`vr1-dc1`** (still gated). All ++ object names, module labels, plan files, and network/pool names below are ++ region-qualified to match `main.tf`. ++ **HAZARD:** following ANY residual bare-`dcN` literal in a copy of this ++ runbook (or in muscle memory) literally reintroduces the exact `dc0`/`dc1` ++ cross-cloud collision D-119 was written to delete -- region-qualify it or ++ stop. (D-119.) + + **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "Phase 2" (the + goal/build/gate this runbook satisfies), Section 9 (shim register -- the +@@ -34,15 +50,23 @@ mechanism riding on it). + past that dependency. Confirm Stage 2's actual state in + `docs/dc-dc-deployment-workflow.md` before starting. + +-!!! **DC2 is explicitly gated, not merely "later."** D-101 has not assigned +- DC2's real, non-overlapping v4 supernet yet (open sub-item; Tooling gap +- register item 3's DATA half -- the import pipeline mechanism +- (`netbox/dc-dc-prefixes-import.py`) is built, but the literals themselves +- are not assigned). `opentofu/main.tf`'s `dc2_planes`/`dc2_storage` blocks +- are commented out for exactly this reason. Do NOT uncomment them, do NOT +- invent a placeholder DC2 CIDR, and do NOT run this runbook's steps against +- DC2 until NetBox has assigned the real supernet and `main.tf` reflects it +- for real. This is a hard gate, re-stated in the GATE section. ++!!! **`vr1-dc1` is explicitly gated -- by SEQUENCING, not by a missing ++ literal.** The earlier rationale ("NetBox has not assigned the supernet") ++ is STALE: **D-115 ALREADY assigned it** -- `vr1-dc1` gets `10.12.64.0/19` ++ (moved inside the Cloud `/16`, colliding with nothing `vr1-dc0` occupies), and ++ that carve is imported to `office1-netbox`. So the blocker is no longer a ++ missing address; `vr1-dc1` stays out of scope THIS pass purely because we ++ build one DC at a time and `vr1-dc0` is the target. `opentofu/main.tf`'s ++ `vr1_dc1_planes` block is still commented out (its `vr1_dc1_storage` and ++ the three mesh legs ARE wired, being address-independent). Do NOT ++ uncomment the planes block, do NOT add `vr1-dc1` build steps, and do NOT ++ run this runbook's steps against `vr1-dc1` this pass. Re-run the whole ++ runbook for `vr1-dc1` when it is sequenced. This is a hard gate, re-stated ++ in the GATE section. ++ NB: `main.tf`'s commented `vr1_dc1_planes` block still carries the OLD ++ "wait for NetBox to assign D-101's supernet" comment -- that comment is ++ now stale per D-115; flagged as a divergence to fix under the `main.tf` ++ owner, out of scope for this doc-only pass. + + !!! **Node-VM creation has NO Roosevelt analog (Section 9 shim register).** + `modules/node-vm` (the blank-disk, PXE-boot libvirt domain this stage +@@ -53,12 +77,15 @@ mechanism riding on it). + onward (commission/deploy, Juju bundle) IS Roosevelt-transferable; this + one step is not. + +-!!! **Node sizing is not decided -- do not invent vCPU/RAM/disk.** +- `modules/node-vm`'s `memory_mib`, `vcpu`, and `disk_size_bytes` variables +- have no defaults by design (buildout-design Section 3's Phase-0 host/ +- disk-budget decision has not been made). If that decision is still open +- when you reach Step 6, STOP there and flag it -- do not pick numbers that +- "seem reasonable" to get `tofu plan` to run. ++!!! **Node sizing is now RULED (D-121, Option C, 2026-07-15).** ++ `modules/node-vm`'s `memory_mib`/`vcpu`/`disk_size_bytes` still have no ++ module defaults by design, but the VALUES to pass are no longer open: ++ D-121 ruled the per-DC layout as **3 control + 2 compute + 3 storage** (8 ++ node VMs) at the sizes wired into Step 6. This was the exact decision that ++ used to block Step 6; it is now UNBLOCKED. Still open (one sub-ruling ++ D-121 notes): the vault-HA backend fork -- but that is a Stage-5 bundle ++ concern, NOT a Step-6 blocker (node layout is fully ruled). Do not invent ++ any OTHER node spec; use D-121's numbers exactly. + + !!! **netem parameters are still an unruled D-100 sub-item (gap #11).** This + stage's own gate text says "netem parameters applied and measured," but +@@ -84,28 +111,34 @@ mechanism riding on it). + + --- + +-## Sequence (DC1; DC2 remains gated -- see callout above) ++## Sequence (`vr1-dc0`; `vr1-dc1` remains gated -- see callout above) ++ ++Honest achievable scope for a first run is **Steps 1-8** (libvirt-layer ++objects + the edge). Step 9 onward is gated on `vr1-dc0`'s own MAAS rack ++controller existing -- a Stage-4 gap (per-DC rack-controller source is not yet ++stood up); do not treat Steps 9-12 as runnable until it is. + + ``` + 1. Pre-flight: confirm Stage 1 + Stage 2 gates closed (read-only) + 2. Confirm OPNsense prep-tool prereqs on the vcloud host (read-only) +-3. Confirm/record still-open decisions this stage depends +- on (node sizing, MAAS zone/pool, netem parameters) +- -- STOP on any that are unresolved (read-only) +-4. Prepare DC1's OPNsense base image (render/config-ISO steps DELETED +- 2026-07-13 -- config is done over the REST API) [MUTATION: host-local files, gated] +-5. Wire modules/opnsense-edge for DC1 into main.tf [repo change, gated] +-6. Wire modules/node-vm calls for DC1's node VMs into +- main.tf (ONLY once sizing is real) [repo change, gated] ++3. Confirm/record decisions this stage depends on (node ++ sizing = D-121 RESOLVED; edge/WAN/LAN = D-122 RESOLVED; ++ MAAS zone/pool; netem parameters) -- STOP on any open (read-only) ++4. Prepare vr1-dc0's OPNsense base image (prep only; edge ++ config is done over the REST API -- D-113(a2)) [MUTATION: host-local files, gated] ++5. Wire modules/opnsense-edge for vr1-dc0 into main.tf [repo change, gated] ++ (prereq: build the vr1-dc0-wan uplink, office1-wan shape) ++6. Wire the 8 modules/node-vm calls for vr1-dc0's node VMs ++ into main.tf (D-121 Option C sizes) [repo change, gated] + 7. tofu init / validate / plan -- review before apply (read-only) +-8. tofu apply -- DC1 OPNsense edge + DC1 node-VM domains [MUTATION: creates libvirt objects, gated] +-9. Wire + apply modules/maas-vm-host -- register DC1's +- libvirt host with DC1's MAAS rack controller [MUTATION: MAAS + repo change, gated] ++8. tofu apply -- vr1-dc0 OPNsense edge + node-VM domains [MUTATION: creates libvirt objects, gated] ++9. Wire + apply modules/maas-vm-host -- register vr1-dc0's ++ libvirt host with vr1-dc0's MAAS rack controller [MUTATION: MAAS + repo change, gated] + 10. Verify MAAS discovers the node VMs (read-only) +-11. Wire + apply modules/netem-link on DC1's mesh legs, ++11. Wire + apply modules/netem-link on vr1-dc0's mesh legs, + using the Section-6 placeholder lean, explicitly flagged [MUTATION: gated] + 12. Verify edge routing + simulated ISP uplink + netem applied (read-only) +- -> EXIT GATE -> Stage 4 (per-DC, once DC2 clears its own gate) ++ -> EXIT GATE -> Stage 4 (per-DC, once vr1-dc1 clears its own gate) + ``` + + --- +@@ -117,9 +150,10 @@ mechanism riding on it). + virsh net-list --all + virsh pool-list --all + ``` +-Expect: DC1's six plane networks, the three mesh-link networks (`dc1-dc2`, +-`dc1-office1`, `dc2-office1`), and the `dc1`/`office1` storage pools, all +-`active` -- Stage 1's Step 11 verification. If any are missing, Stage 1 is ++Expect: `vr1-dc0`'s six plane networks, the three mesh-link networks ++(`vr1-dc0-vr1-dc1`, `vr1-dc0-office1`, `vr1-dc1-office1` -- the region- ++qualified names now in `main.tf`), and the `vr1-dc0`/`office1` storage pools, ++all `active` -- Stage 1's Step 11 verification. If any are missing, Stage 1 is + not actually done regardless of what `docs/dc-dc-deployment-workflow.md` + says -- stop and reconcile there first. + +@@ -166,35 +200,47 @@ scripts. + + ## Step 3 -- Confirm/record still-open decisions this stage depends on (READ-ONLY) + +-Before wiring anything into `main.tf`, walk this list. Any "NOT YET" item is a +-STOP for the corresponding later step -- record it plainly rather than +-inventing a value to keep moving: +- +-- **Node sizing** (`memory_mib`/`vcpu`/`disk_size_bytes` for `modules/node-vm`, +- and how many node VMs DC1 needs) -- buildout-design Section 3's Phase-0 +- host/disk-budget decision. Blocks Step 6. +-- **MAAS zone/pool for DC1's VM-host registration** (`modules/maas-vm-host`'s +- `zone`/`pool` variables) -- both are optional (MAAS computes them if left +- `null`, confirmed safe per `opentofu/README.md`'s audit pass), so this is +- NOT necessarily a blocker; record whichever real zone/pool name the +- operator wants, or confirm the `null` no-op is intentional. Affects Step 9. +-- **DC1's real `power_address`** (the virsh URI MAAS uses to reach the vcloud +- host) -- must be measured this session (or carried from Stage 1's ++Before wiring anything into `main.tf`, walk this list. Two of the items that ++used to block this stage are now RULED (D-121, D-122) and are recorded as ++RESOLVED below; the genuinely-open ones remain STOPs. Record each plainly ++rather than inventing a value to keep moving: ++ ++- **Node sizing** (`memory_mib`/`vcpu`/`disk_size_bytes` and count for ++ `modules/node-vm`) -- **RESOLVED (D-121, Option C, 2026-07-15).** Per DC: 3 ++ control @ 16 vCPU / 65536 MiB / 150 GiB; 2 compute @ 12 vCPU / 49152 MiB / ++ 100 GiB; 3 storage @ 8 vCPU / 24576 MiB / 550 GiB -- 8 node VMs total. Step ++ 6 wires exactly these. Open sub-ruling D-121 still carries (the vault-HA ++ backend fork) does NOT block Step 6 -- it is a Stage-5 bundle concern. ++- **Edge sizing / WAN uplink / LAN role** -- **RESOLVED (D-122, 2026-07-15).** ++ The DC edge follows the proven Office1 pattern: `memory_mib = 2048`, ++ `vcpu = 2`, nano base image (NO `disk_size_bytes`, like `office1_opnsense`). ++ The edge is **2-NIC** (baremetal-matched): `wan_network_name` is a ++ **dedicated per-site ISP uplink `vr1-dc0-wan`** (`172.30.2.0/24`, D-115) built ++ on the `office1-wan` pattern -- NOT a mesh leg (dark fiber is East-West only). ++ `lan_network_name` is **provider-public** -- the six planes are routed by the ++ fabric (OVN/OpenStack on isolated-L2 segments), so the edge is only the ++ external gateway on provider-public (D-100), not an inter-plane router. See ++ Step 5. PREREQUISITES: build `vr1-dc0-wan` (as a tofu module) and register ++ `172.30.2.0/24` in office1-netbox (not yet loaded). ++- **MAAS zone/pool for `vr1-dc0`'s VM-host registration** ++ (`modules/maas-vm-host`'s `zone`/`pool` variables) -- both are optional ++ (MAAS computes them if left `null`, confirmed safe per `opentofu/README.md`'s ++ audit pass), so this is NOT necessarily a blocker; record whichever real ++ zone/pool name the operator wants, or confirm the `null` no-op is ++ intentional. Affects Step 9. ++- **`vr1-dc0`'s real `power_address`** (the virsh URI MAAS uses to reach the ++ vcloud host) -- must be measured this session (or carried from Stage 1's + `libvirt_uri` tfvar, if it is the SAME endpoint -- confirm, do not assume + identical per `modules/maas-vm-host`'s own variable note). Blocks Step 9. +-- **OPNsense `config.xml` tokens** -- per `opentofu/templates/README.md`'s +- token legend, several tokens are pending Stage-0-adjacent decisions or a +- real boot measurement: `{{WAN_IF}}`/`{{LAN_IF}}` (only measurable by +- booting the domain and running `ifconfig` -- genuinely cannot be filled in +- before Step 8 creates the domain), `{{WAN_IPADDR}}`/`{{WAN_SUBNET_BITS}}`/ +- `{{WAN_GATEWAY}}` (pending NetBox), `{{LAN_IPADDR}}`/`{{LAN_SUBNET_BITS}}` +- (pending which plane(s) OPNsense's LAN interface actually serves -- +- flagged in the template README as not fully specified in the buildout +- design), `{{MIRROR_SYNC_PROTOCOL}}`/`{{MIRROR_UPSTREAM_NET}}`/ +- `{{MIRROR_SYNC_PORT}}` (pending mirror-software choice, gap #3-adjacent), +- and `{{ROOT_PASSWORD_HASH}}` (must be freshly generated per site, never +- reused from the shipped sample). Blocks Step 4's render sub-step for +- everything except `{{OPNSENSE_HOSTNAME}}`/`{{DOMAIN}}`/NTP defaults. ++- **`vr1-dc0`'s MAAS rack controller** -- a Stage-4 gap: the per-DC ++ rack-controller source is not yet stood up. This is the reason honest Step-9 ++ scope is BLOCKED and this pass's real reach is Steps 1-8 (see the GATE ++ section). Blocks Steps 9-12. ++- **Edge post-boot interface mapping** (formerly the `{{WAN_IF}}`/`{{LAN_IF}}` ++ `config.xml` tokens) -- NO LONGER a config-render blocker. There is no ++ `config.xml` render (D-112/D-113(a2)); the real `vtnetN` mapping is measured ++ AFTER boot over SSH (`ifconfig`), then applied over the REST API. See the ++ REPLACEMENT chain in Step 4. Nothing here blocks wiring Step 5. + - **netem parameters** -- still an unruled D-100 sub-item (gap #11); Step 11 + uses the Section-6 qualitative placeholder only, explicitly labeled as + such. Not a hard blocker for Step 11 (the mechanism can be exercised with a +@@ -208,45 +254,43 @@ step named as blocked. Do not proceed past a step whose input is still + + --- + +-## Step 4 -- Prepare DC1's OPNsense base image + config.xml + config ISO [MUTATION: host-local files, gated] ++## Step 4 -- Prepare `vr1-dc0`'s OPNsense base image [MUTATION: host-local files, gated] + +-> ## !!! STOP -- THIS STEP IS OBSOLETE AS WRITTEN (2026-07-13) !!! ++> ## THE EDGE-CONFIG PATH (settled 2026-07-13 -- this IS the path) + > +-> Nothing here clobbers a live system (DC1's edge does not exist yet), but **two of the three +-> sub-steps below cannot achieve what they claim**, and following them will waste a session and +-> produce a dead edge: ++> This step is now **prep the base image only.** Edge configuration is NOT baked into an image ++> or seeded from an ISO -- it is done over the **REST API after boot**, the path proven end to ++> end on the live Office1 edge (2026-07-12/13). The two deleted approaches, and why: + > +-> 1. **The config ISO is INERT -- it can never be read.** Per **D-112** (root-caused from ++> 1. **The config ISO was INERT -- it could never be read.** Per **D-112** (root-caused from + > upstream source): `opnsense-importer -b` probes for a read-only root. On INSTALLER media the + > probe fails and the importer scans attached media. On our PRE-INSTALLED NANO image the root + > is writable AND a factory `/conf/config.xml` already exists, so BOTH conditions hold and it + > `bootstrap_and_exit 0`s **without enumerating a single device**. The Configuration Importer +-> can NEVER fire on a nano image, by design. Building the ISO is pure waste. ++> can NEVER fire on a nano image, by design -- so the ISO builder is DELETED and the module no ++> longer takes a `config_iso_path`. + > 2. **Full-`config.xml` rendering is SUPERSEDED** by **D-113(a2)** (ruled 2026-07-13): edge +-> config is done over the **REST API** (`scripts/opnsense-api.sh`), which was proven end to +-> end -- read AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the +-> appliance's GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) +-> and DOCFIX-193 (no DHCP, and an ISC `<dhcpd>` block that would have been inert against a Kea +-> backend). None of those bugs is expressible through the API. ++> config is done over the **REST API** (`scripts/opnsense-api.sh`), proven end to end -- read ++> AND write -- against the live Office1 edge on 2026-07-13. Hand-authoring the appliance's ++> GUI-owned XML caused DOCFIX-191 (management lockout), DOCFIX-192 (dead console) and ++> DOCFIX-193 (no DHCP, and an ISC `<dhcpd>` block that would have been inert against a Kea ++> backend). None of those bugs is expressible through the API. The renderer is DELETED. + > +-> **The `WAN_IF`/`LAN_IF` chicken-and-egg problem discussed below is therefore MOOT** -- it was an +-> artifact of trying to seed a full config before first boot. The D-112(c) bootstrap (boot on +-> factory defaults -> reach the console -> enable SSH + install the key -> configure over the +-> network) measures the real `vtnetN` mapping *after* boot, where it is knowable. ++> **There is no `WAN_IF`/`LAN_IF` chicken-and-egg.** It only ever existed because we tried to seed ++> a full config before first boot. The D-112(c) bootstrap (boot on factory defaults -> reach the ++> console -> enable SSH + install the key -> configure over the network) measures the real ++> `vtnetN` mapping *after* boot, where it is knowable -- nothing to guess before Step 8. + > +-> **What DC1's edge should actually do (the proven Office1 path):** prep the image -> boot it -> +-> D-112(c) console bootstrap -> configure over SSH + the REST API. See ++> **What `vr1-dc0`'s edge does (the proven Office1 path):** prep the image (this step) -> Step 5 ++> wires the domain -> Step 8 boots it -> D-112(c) console bootstrap -> configure over SSH + the ++> REST API (the REPLACEMENT chain below). See + > `docs/changelog-20260712-office1-opnsense-edge-build.md`, + > `docs/changelog-20260713-opnsense-api-proven.md`, and + > `docs/changelog-20260713-opnsense-api-write-proven.md`. +-> +-> **This step is NOT yet rewritten.** Reducing the template to a minimal bootstrap and rewriting +-> Stage 3's edge steps around the API is the open D-113(a2) work. Until that lands, STOP here and +-> re-derive rather than following the sub-steps below. + +-This step produces plain files on the vcloud host filesystem -- no libvirt or +-MAAS object is created yet. Run from the vcloud host (or wherever these +-scripts execute with reach to write there). ++This step produces one plain file on the vcloud host filesystem -- the prepped ++base image -- no libvirt or MAAS object is created yet. Run from the vcloud ++host (or wherever this script executes with reach to write there). + + **MUTATION -- prep the base image (needs Step 2's tools)** + ```bash +@@ -254,31 +298,14 @@ bash scripts/opnsense-prep-image.sh # see the script's own header for its exac + ``` + **Expect:** a decompressed, qcow2-converted, resized OPNsense nano image at a + real output path -- record that path, it feeds Step 5's `base_volume_path`. +- +-**(DELETED 2026-07-13.)** The `config.xml` render step is GONE -- the renderer +-(`opnsense-render-config.sh`, formerly under scripts/) no longer exists. Edge config is done over the REST API +-(D-113(a2)). See the replacement chain at the end of this step. +-`WAN_IF`/`LAN_IF` are the one pair the script's own `REQUIRED_VARS` list +-demands that genuinely cannot be measured before the domain exists (Step 8 +-creates it). If Step 3 flagged these as not yet measurable, this render +-sub-step cannot complete for real this session -- STOP here for DC1's config +-render, record it as blocked pending a real boot, and do not substitute a +-guessed `vtnetN` name. (One documented path forward once the domain exists: +-boot the domain with a MINIMAL/placeholder config good enough to reach a +-console, measure `ifconfig`'s real `vtnetN` mapping, then re-render and +-re-seed -- that re-seed mechanic is not designed in this delivery; treat it +-as a real follow-up need if it comes to that, not an invented shortcut here.) +- +-**(DELETED 2026-07-13.)** The config-ISO step is GONE -- the ISO builder +-(`opnsense-build-config-iso.sh`, formerly under scripts/) no longer exists, and the module no longer takes a +-`config_iso_path`. The ISO was never read by anything (D-112: the Importer cannot fire on a +-pre-installed nano image). ++That single path is this step's ONLY output. There is no config.xml render and ++no config-ISO build -- both are deleted (D-112 / D-113(a2), see the box above). + + --- + + ### The REPLACEMENT chain (proven on Office1, 2026-07-12/13) + +-No `config.xml` anywhere. After Step 5 creates the domain and it boots on FACTORY DEFAULTS: ++No `config.xml` anywhere. After Step 5 wires the domain and Step 8 boots it on FACTORY DEFAULTS: + + 1. **Console bootstrap (D-112(c))** -- reach the serial console, enable SSH, install the service + public key. This is the only step that needs the console; everything after it is scripted. +@@ -295,74 +322,128 @@ No `config.xml` anywhere. After Step 5 creates the domain and it boots on FACTOR + bash scripts/opnsense-api.sh POST kea/service/reconfigure '{}' + ``` + +-**`WAN_IF`/`LAN_IF` are measured AFTER boot** (`ifconfig` over SSH), where they are knowable -- +-which is why the chicken-and-egg problem this step used to agonize over simply does not exist +-anymore. ++**The real `vtnetN` interface mapping is measured AFTER boot** (`ifconfig` over SSH), where it is ++knowable -- there is no interface value to guess before the domain exists. + +-**GATE:** base image path and (if unblocked) config ISO path recorded as real +-host-local file paths. If the config.xml render step is blocked on +-`WAN_IF`/`LAN_IF`, record that explicitly and treat Step 5's `config_iso_path` +-input as still pending -- do not wire Step 5 with a placeholder ISO. ++**GATE:** the prepped base image path recorded as a real host-local file path ++(it feeds Step 5's `base_volume_path`). Nothing else is produced or pending ++here -- no config-ISO path exists to record. + + --- + +-## Step 5 -- Wire `modules/opnsense-edge` for DC1 into `main.tf` [repo change, gated] +- +-Only proceed once Step 4 produced real `base_volume_path` and +-`config_iso_path` values (or an explicit decision to proceed with a +-genuinely-final, if minimal, config.xml -- not a placeholder). +- +-**MUTATION -- edit `opentofu/main.tf`**, adding a block shaped like: ++## Step 5 -- Wire `modules/opnsense-edge` for `vr1-dc0` into `main.tf` [repo change, gated] ++ ++Proceed once Step 4 produced a real `base_volume_path`. The edge SHAPE is ++ruled (D-122): it mirrors the already-applied `office1_opnsense` block in ++`main.tf` -- `memory_mib = 2048`, `vcpu = 2`, nano image, **no ++`disk_size_bytes`** (the disk is a direct copy of the prepped nano; the module ++does not accept a size input -- DOCFIX-189), and **no `config_iso_path`** (the ++module dropped it -- D-112). There is nothing to invent for the edge specs ++themselves; the two values that ARE still per-site (the base image path, and ++the two network names) are called out below. ++ ++**PREREQUISITE -- build the `vr1-dc0-wan` uplink first.** D-122 gives the DC ++its OWN dedicated L3 "ISP" uplink, NOT a mesh leg (dark fiber is East-West ++replication only). Build a per-site NAT'd `/24` uplink network `vr1-dc0-wan` ++on the **`office1-wan` pattern** (the same mechanical shape `office1-wan` was ++created in -- a documented D-103 debt to formalize into a module later; today ++it is a literal network name the edge's `wan_network_name` points at). This is ++a prerequisite to wiring the block below with a real `wan_network_name`. ++ ++**LAN side -- RESOLVED (D-122, operator ruling 2026-07-15): `provider-public`.** ++The edge is **2-NIC** (WAN + LAN), matching the baremetal target: the six ++planes are **routed by the fabric** (the `dc-planes` segments are isolated L2; ++OVN/OpenStack own their L3, not the edge), so the edge is NOT an inter-plane ++router -- only the external boundary. Its LAN attaches to **provider-public** ++(`10.12.4.0/22`), where the edge is the upstream/external gateway the provider ++network exits through (FIP/SNAT egress + GUA injection, D-100 br-ex/provider ++model). metal-admin and the other planes reach external via fabric routing, not ++a direct edge leg. So `lan_network_name` = the `vr1_dc0_planes` provider-public ++output (confirm the exact output name in `modules/dc-planes`). No new ++site-internal network module is needed (this SUPERSEDES the earlier "build an ++office1-network analog" flag). ++ ++**MUTATION -- edit `opentofu/main.tf`**, adding a block shaped like ++`office1_opnsense` (main.tf, the applied reference): + ```hcl +-module "dc1_opnsense" { ++module "vr1_dc0_opnsense" { + source = "./modules/opnsense-edge" +- vm_name = "dc1-opnsense" +- memory_mib = <REAL VALUE -- OPNsense's own sizing guidance, not invented> +- vcpu = <REAL VALUE> +- pool_name = module.dc1_storage.pool_name # confirm real output name in modules/dc-storage-pool +- disk_size_bytes = <REAL VALUE> +- base_volume_path = "<Step 4's real base image path>" +- config_iso_path = "<Step 4's real config ISO path>" +- lan_network_name = <the dc1_planes output this edge's LAN side attaches to -- confirm which plane per Step 3's open LAN-role item> +- wan_network_name = module.mesh_dc1_office1... # or a dedicated per-site ISP-uplink segment -- confirm which, do not assume ++ vm_name = "vr1-dc0-opnsense" ++ memory_mib = 2048 # D-122: Office1 pattern ++ vcpu = 2 # D-122: Office1 pattern ++ pool_name = module.vr1_dc0_storage.pool_name ++ # No disk_size_bytes: nano direct-copy, size from opnsense-prep-image.sh's GROW (DOCFIX-189). ++ base_volume_path = "<Step 4's real prepped-image path>" ++ lan_network_name = <the vr1_dc0_planes provider-public output -- D-122: edge is the external gateway on provider-public; confirm the output name in modules/dc-planes> ++ wan_network_name = "vr1-dc0-wan" # D-122: dedicated per-site ISP uplink, office1-wan pattern (build first, promote to a tofu module) + } + ``` +-Every placeholder above (`<...>`) must be a real, measured/decided value +-before this is committed -- this template line exists to show shape, not to +-be copy-pasted with invented numbers. `memory_mib`/`vcpu`/`disk_size_bytes` +-here have no ratified default in the module either (same "no invented specs" +-discipline as `modules/node-vm`). ++Before commit: the base image path (Step 4) must be real, `vr1-dc0-wan` must be ++built (as a tofu module -- D-122) and `172.30.2.0/24` registered in office1-netbox ++(D-115, not yet loaded), and `lan_network_name` bound to the real provider-public ++plane output. `memory_mib`/`vcpu`/the nano-no-disk shape and the 2-NIC WAN/LAN ++model are ruled (D-122), not invented. + +-**GATE:** `opentofu/main.tf` diff shows exactly one new `module "dc1_opnsense"` +-block, every argument a real value, nothing else changed. ++**GATE:** `opentofu/main.tf` diff shows exactly one new `module "vr1_dc0_opnsense"` ++block, no `disk_size_bytes` and no `config_iso_path` line, every argument a ++real value (with `vr1-dc0-wan` and the site-internal LAN network genuinely ++built), nothing else changed. + + --- + +-## Step 6 -- Wire `modules/node-vm` calls for DC1's node VMs into `main.tf` [repo change, gated] +- +-**ONLY if Step 3 confirmed node sizing is real.** If sizing is still "NOT +-YET," STOP here -- this step does not proceed, and this session's DC1 pass +-ends at whatever the last unblocked step was (Step 5 or wherever Step 4 left +-off). Re-run from here once sizing exists; do not invent numbers to close +-this out early. +- +-**MUTATION -- edit `opentofu/main.tf`**, one `module "dc1_node_XX"` block per +-node VM, shaped like: ++## Step 6 -- Wire `modules/node-vm` calls for `vr1-dc0`'s node VMs into `main.tf` [repo change, gated] ++ ++**Node sizing is RULED (D-121, Option C).** Wire **8** `module "vr1_dc0_node_*"` ++blocks in three role-sizes: ++ ++| Role | Count | vCPU | `memory_mib` | disk | Names | ++|---------|-------|------|--------------|-------------|------------------------------------| ++| control | 3 | 16 | 65536 | 150 GiB | `vr1-dc0-control-01..03` | ++| compute | 2 | 12 | 49152 | 100 GiB | `vr1-dc0-compute-01..02` | ++| storage | 3 | 8 | 24576 | 550 GiB | `vr1-dc0-storage-01..03` | ++ ++`disk_size_bytes` is BYTES: express each as GiB x 1024^3 in HCL -- ++`150 * 1024 * 1024 * 1024`, `100 * 1024 * 1024 * 1024`, ++`550 * 1024 * 1024 * 1024` (the module's `disk_size_bytes` expects bytes; the ++GROW/qcow2 layer is thin-provisioned so this is a max, not a preallocation). ++ ++**Why 3 control (context):** this layout exists to carry the Stage-5 HA ++scale-up (D-121). The bundle's **14 currently-single-unit services scale ++1 -> 3**: the 11 hacluster+VIP API apps (keystone, glance, neutron-api, ++nova-cloud-controller, placement, cinder, openstack-dashboard, octavia, ++barbican, magnum, designate) plus `ceph-radosgw`, `rabbitmq-server`, and ++`vault`. Those 3-unit control-plane services LIVE on the 3 control nodes -- ++alongside the services that are ALREADY at 3 and do not change ++(`mysql-innodb-cluster`, `ovn-central`, `ceph-mon`). Meanwhile `ceph-osd` = 3 ++units on the 3 storage nodes and `nova-compute` = 2 units on the 2 compute ++nodes. The node-COUNT/sizing here is fully ruled; the still-open D-121 ++sub-item (the vault-HA backend fork) is a Stage-5 bundle concern and does NOT ++gate this step. ++ ++**MUTATION -- edit `opentofu/main.tf`**, one block per node VM (control shown; ++compute/storage identical shape with their row's vCPU/memory/disk/name): + ```hcl +-module "dc1_node_01" { ++module "vr1_dc0_control_01" { + source = "./modules/node-vm" +- vm_name = "dc1-node-01" +- memory_mib = <REAL VALUE from the Phase-0 sizing decision> +- vcpu = <REAL VALUE> +- pool_name = module.dc1_storage.pool_name +- disk_size_bytes = <REAL VALUE> +- network_names = [ <ordered list of module.dc1_planes outputs -- confirm which plane goes first for PXE priority, per D-103> ] ++ vm_name = "vr1-dc0-control-01" ++ memory_mib = 65536 # D-121: control ++ vcpu = 16 # D-121: control ++ pool_name = module.vr1_dc0_storage.pool_name ++ disk_size_bytes = 150 * 1024 * 1024 * 1024 # D-121: control 150 GiB ++ network_names = [ <ordered list of module.vr1_dc0_planes outputs -- confirm which plane goes first for PXE priority, per D-103> ] + } + ``` +-Repeat per node, per the real node count from the sizing decision. +- +-**GATE:** every `module "dc1_node_*"` block uses only real, decided values; +-node count matches the recorded sizing decision exactly. ++Repeat for `vr1_dc0_control_02`/`03`, `vr1_dc0_compute_01`/`02` (12 vCPU / ++49152 MiB / `100 * 1024 * 1024 * 1024`), and `vr1_dc0_storage_01`/`02`/`03` ++(8 vCPU / 24576 MiB / `550 * 1024 * 1024 * 1024`). **`network_names` is STILL a ++flagged STOP** in every block: D-121 ruled the SIZES, not the plane ++attachment/PXE order -- that ordered plane list is an unresolved value (D-103 ++PXE priority) and must not be invented to make the block parse. ++ ++**GATE:** exactly 8 `module "vr1_dc0_node_*"`-shaped blocks (3 control / 2 ++compute / 3 storage) with D-121's sizes verbatim; each `network_names` either ++a genuinely-resolved plane list or explicitly recorded as the remaining STOP, ++never a guessed order. + + --- + +@@ -373,24 +454,25 @@ cd opentofu + tofu fmt -check -recursive -diff . + tofu init -backend=false -input=false + tofu validate +-tofu plan -out=phase2-dc1.tfplan ++tofu plan -out=phase2-vr1-dc0.tfplan + ``` +-Review the plan line by line: expect creates for `module.dc1_opnsense` +-(the edge VM's boot volume, config-seed volume, and domain) and each +-`module.dc1_node_*` (a blank boot volume + domain per node). Confirm nothing +-else is planned -- in particular, confirm no DC2 resource appears (its +-module blocks should still be commented out per the DC2 gate above). ++Review the plan line by line: expect creates for `module.vr1_dc0_opnsense` ++(the edge VM's boot volume and domain -- NO config-seed volume/cdrom anymore; ++that path is deleted, D-112) and each `module.vr1_dc0_*` node (a blank boot ++volume + domain per node: 3 control, 2 compute, 3 storage). Confirm nothing ++else is planned -- in particular, confirm no `vr1-dc1` resource appears (its ++`vr1_dc1_planes` block should still be commented out per the gate above). + + **GATE:** plan matches this expectation exactly; no unexpected creates, + updates, or destroys. Do not apply a plan you have not read. + + --- + +-## Step 8 -- `tofu apply` -- DC1 OPNsense edge + node-VM domains [MUTATION: gated] ++## Step 8 -- `tofu apply` -- `vr1-dc0` OPNsense edge + node-VM domains [MUTATION: gated] + + ```bash + cd opentofu +-tofu apply phase2-dc1.tfplan ++tofu apply phase2-vr1-dc0.tfplan + ``` + Confirm this is the exact reviewed plan file from Step 7 (not re-planned) + before running. This is the first live mutation THIS stage performs. +@@ -398,12 +480,13 @@ before running. This is the first live mutation THIS stage performs. + **VERIFY** + ```bash + virsh list --all +-virsh domblklist dc1-opnsense +-virsh domblklist dc1-node-01 # per real node name ++virsh domblklist vr1-dc0-opnsense ++virsh domblklist vr1-dc0-control-01 # per real node name + ``` + Expect the new domains present (likely `shut off` until first boot/PXE), the +-OPNsense domain showing both the base overlay disk and the config ISO +-attached as a secondary cdrom, each node VM showing a single blank boot disk. ++OPNsense domain showing a SINGLE boot disk (a direct copy of the prepped nano ++-- no config-ISO cdrom; that path is deleted, D-112), each node VM showing a ++single blank boot disk. + + **GATE:** all new domains defined as planned; disk attachments match Step + 5/6's wiring. +@@ -418,20 +501,26 @@ hasn't already. + + --- + +-## Step 9 -- Wire + apply `modules/maas-vm-host` -- register DC1's libvirt host with DC1's MAAS rack controller [MUTATION: gated] +- +-**Hard dependency on Stage 2.** This step requires DC1's MAAS rack controller +-(under the Stage-2-stood-up MAAS region) to actually exist and be reachable. +-If Stage 2 (or the DC1 rack-controller enrollment specifically, however that +-ends up sequenced relative to Stage 2's own runbook) is not done, STOP here +--- this stage's remaining steps (9-12) do not proceed, and DC1's Phase 2 work +-is INCOMPLETE, not silently skippable. ++## Step 9 -- Wire + apply `modules/maas-vm-host` -- register `vr1-dc0`'s libvirt host with `vr1-dc0`'s MAAS rack controller [MUTATION: gated] ++ ++**Hard dependency -- `vr1-dc0`'s MAAS controller.** This step requires ++`vr1-dc0`'s own MAAS to exist and be reachable. **D-123 (PROPOSED, recommend ++Model A) resolves what that is:** a `vvr1-dc0` **site headend VM** built the ++proven Office1 way -- an OpenTofu `cloudinit-vm` block shaped like ++`module "voffice1"` + a run of `scripts/site-headend-install.sh` (which already ++names `vvr1-dc0/vvr1-dc1` as targets) to install MAAS region+rack + LXD. Under ++Model A the node VMs stay vcloud-level and this MAAS DISCOVERS them via the ++`maas-vm-host` registration below (power_address = vcloud's virsh). Until Model ++A is ruled and `vvr1-dc0` is stood up, this is the honest scope boundary: ++**Steps 1-8 are this pass's achievable scope; Steps 9-12 STOP here** -- ++`vr1-dc0`'s Phase 2 is then INCOMPLETE, not silently skippable. (Also inherits ++Stage 2's reachable-MAAS-region dependency.) + + **MUTATION -- edit `opentofu/main.tf`** + ```hcl +-module "dc1_maas_vm_host" { ++module "vr1_dc0_maas_vm_host" { + source = "./modules/maas-vm-host" +- vm_host_name = "dc1-vcloud" ++ vm_host_name = "vr1-dc0-vcloud" + power_address = "<Step 3's real, measured virsh URI -- confirm same as var.libvirt_uri or a distinct value>" + zone = <real zone name, or leave unset for MAAS's computed default -- per Step 3> + pool = <real pool name, or leave unset -- per Step 3> +@@ -440,8 +529,8 @@ module "dc1_maas_vm_host" { + + ```bash + cd opentofu +-tofu plan -out=phase2-dc1-maashost.tfplan # review: exactly one new maas_vm_host resource +-tofu apply phase2-dc1-maashost.tfplan ++tofu plan -out=phase2-vr1-dc0-maashost.tfplan # review: exactly one new maas_vm_host resource ++tofu apply phase2-vr1-dc0-maashost.tfplan + ``` + + **GATE:** `maas_vm_host` resource created; no other resource touched by this +@@ -453,22 +542,22 @@ plan. + + **CHECK** + ```bash +-maas admin vm-hosts read | jq -r '.[] | select(.name=="dc1-vcloud") | {id, name, resources}' +-maas admin machines read | jq -r '.[] | select(.hostname|test("^dc1-node")) | "\(.hostname)\t\(.status_name)"' ++maas admin vm-hosts read | jq -r '.[] | select(.name=="vr1-dc0-vcloud") | {id, name, resources}' ++maas admin machines read | jq -r '.[] | select(.hostname|test("^vr1-dc0-")) | "\(.hostname)\t\(.status_name)"' + ``` +-Expect DC1's node VMs listed as MAAS machines (status likely `New` -- ++Expect `vr1-dc0`'s node VMs listed as MAAS machines (status likely `New` -- + enlistment discovered them; commissioning is Stage 4's job, not this + stage's). If the node VMs do not appear, do not proceed to declare this + stage's MAAS-discovery gate met -- troubleshoot the VM-host registration + (power_address correctness, virsh reachability from the MAAS rack + controller) before moving on. + +-**GATE:** DC1's OpenTofu-created node VMs visible in MAAS, associated with +-the `dc1-vcloud` VM host. ++**GATE:** `vr1-dc0`'s OpenTofu-created node VMs visible in MAAS, associated with ++the `vr1-dc0-vcloud` VM host. + + --- + +-## Step 11 -- Wire + apply `modules/netem-link` on DC1's mesh legs, placeholder profile [MUTATION: gated] ++## Step 11 -- Wire + apply `modules/netem-link` on `vr1-dc0`'s mesh legs, placeholder profile [MUTATION: gated] + + Per the netem callout above: this step applies ONLY the buildout design's + qualitative Section 6 lean ("same-metro dark fiber... low single-digit ms, +@@ -477,26 +566,29 @@ ruled numeric value. If the operator has since ruled a specific numeric + profile, use that instead and cite the ruling; do not use this placeholder + language once a real ruling exists. + +-**MUTATION -- edit `opentofu/main.tf`**, one block per relevant mesh leg +-(at minimum DC1<->DC2 and DC1<->Office1; DC2<->Office1 is out of scope while +-DC2 remains gated): ++**MUTATION -- edit `opentofu/main.tf`**, one block per relevant mesh leg. The ++only leg with substrate on both ends this pass is `vr1-dc0`<->`office1` ++(`vr1-dc0`<->`vr1-dc1` and `vr1-dc1`<->`office1` have no `vr1-dc1`-side ++substrate while `vr1-dc1` remains gated): + ```hcl +-module "netem_dc1_office1" { ++module "netem_vr1_dc0_office1" { + source = "./modules/netem-link" +- link_name = "dc1-office1" ++ link_name = "vr1-dc0-office1" + vcloud_host_ssh_target = "<real SSH target for the vcloud host -- measured, not invented>" +- bridge_name = "<real OS bridge name for this mesh-link network -- confirm via `virsh net-info dc1-office1` or `tofu show`, not assumed>" ++ bridge_name = "<real OS bridge name for this mesh-link network -- confirm via `virsh net-info vr1-dc0-office1` or `tofu show`, not assumed>" + netem_args = "<PLACEHOLDER pending D-100 sub-item ruling -- record explicitly as provisional, e.g. a low single-digit-ms delay with modest jitter and negligible loss, cited to buildout-design Section 6, NOT presented as a measured or ruled value>" + } + ``` +-(A DC1<->DC2 block follows the same shape once DC2 itself is ungated; until +-then, applying netem on that specific leg has no DC2-side substrate to +-matter for, so it is naturally deferred alongside the rest of DC2's work.) ++(A `vr1-dc0`<->`vr1-dc1` block -- `module "netem_vr1_dc0_vr1_dc1"`, ++`link_name "vr1-dc0-vr1-dc1"` -- follows the same shape once `vr1-dc1` itself ++is ungated; until then, applying netem on that leg has no `vr1-dc1`-side ++substrate to matter for, so it is naturally deferred alongside the rest of ++`vr1-dc1`'s work.) + + ```bash + cd opentofu +-tofu plan -out=phase2-dc1-netem.tfplan +-tofu apply phase2-dc1-netem.tfplan ++tofu plan -out=phase2-vr1-dc0-netem.tfplan ++tofu apply phase2-vr1-dc0-netem.tfplan + ``` + + **GATE:** the `terraform_data`/`local-exec` resource applies without error; +@@ -514,20 +606,21 @@ sudo tc qdisc show dev <bridge_name> # per Step 11's real bridge name + ``` + Expect a `netem` qdisc present with the applied (placeholder) parameters. + +-**CHECK -- OPNsense edge reachability** (only meaningful once WAN_IF/LAN_IF +-have been measured on a real boot per Step 4's note, and the domain has +-actually been booted/configured) ++**CHECK -- OPNsense edge reachability** (only meaningful once the domain has ++been booted and configured over the REST API per Step 4's REPLACEMENT chain -- ++the `vtnetN` mapping and WAN/LAN addressing are measured/applied AFTER boot) + ```bash +-virsh domstate dc1-opnsense ++virsh domstate vr1-dc0-opnsense + ``` +-If the config.xml render was blocked in Step 4 (pending `WAN_IF`/`LAN_IF`), +-this verification is necessarily partial this session -- record that +-honestly rather than declaring edge routing verified when it is not. ++If the domain has not yet been through the console + REST-API bootstrap (the ++D-112(c) chain), this verification is necessarily partial this session -- ++record that honestly rather than declaring edge routing verified when it is ++not. + + **GATE (this step):** netem qdisc present and matching Step 11's recorded +-placeholder profile; OPNsense domain running IF its config.xml was +-successfully completed, otherwise recorded as blocked pending a real-boot +-`vtnetN` measurement and re-seed. ++placeholder profile; OPNsense domain running IF it has been through the ++post-boot REST-API bootstrap, otherwise recorded as blocked pending that ++bootstrap (console -> SSH -> API, per Step 4). + + --- + +@@ -538,18 +631,23 @@ Buildout-design Section 4 / deployment-workflow Stage 3 states the gate as: + VMs; edge routing + simulated ISP uplink up; netem parameters applied and + measured." As of what this runbook can actually close this session: + +-- **MAAS rack controller per DC up:** achievable ONLY once Stage 2 (or +- whatever runbook stands up DC1's specific rack controller) is done -- +- Step 1/9 depend on it. Record actual state, do not assume it. ++- **MAAS rack controller per DC up:** achievable ONLY once `vr1-dc0`'s own ++ MAAS rack controller is stood up -- a Stage-4 gap that is NOT yet done ++ (D-122 confirms each site runs its own controller; the SOURCE for it is the ++ open item). Steps 1/9 depend on it. This is why honest scope for a first run ++ is Steps 1-8. Record actual state, do not assume it. + - **MAAS discovers the OpenTofu-created node VMs:** achievable through Step +- 10, CONDITIONAL on node sizing being real (Step 3/6) and Stage 2's MAAS +- region being reachable (Step 9's dependency). ++ 10, CONDITIONAL on the per-DC rack controller existing (Step 9's dependency) ++ and Stage 2's MAAS region being reachable. Node sizing itself is no longer a ++ condition -- it is RULED (D-121, Step 6). + - **Edge routing + simulated ISP uplink up:** PARTIALLY achievable. The +- OPNsense domain and its network attachments can be created (Step 8), but +- full edge routing verification depends on `WAN_IF`/`LAN_IF` being measured +- on a real boot (Step 4's documented blocker) and on the WAN/LAN addressing +- tokens (pending NetBox) -- do not mark this sub-gate fully closed unless +- those are genuinely resolved. ++ OPNsense domain and its network attachments can be created (Step 8), and the ++ edge shape/uplink are RULED (D-122: `vr1-dc0-wan` dedicated ISP uplink, ++ Office1-pattern edge), but full edge routing verification depends on the ++ post-boot REST-API bootstrap (Step 4's REPLACEMENT chain -- console -> SSH ++ -> API, where the real `vtnetN` mapping and WAN/LAN addressing are measured ++ and applied). Do not mark this sub-gate fully closed until that bootstrap ++ has actually run against the booted domain. + - **netem parameters applied and measured:** applied, YES (Step 11), using + an explicitly-labeled PLACEHOLDER same-metro-lean profile per + buildout-design Section 6 -- MEASURED in the sense of "the qdisc is +@@ -561,21 +659,25 @@ measured." As of what this runbook can actually close this session: + remaining gap named. + + **This stage's exit gate is therefore CONDITIONALLY MET at best** on any run +-before (a) Stage 2 is complete, (b) node sizing is decided, (c) DC1's +-OPNsense WAN/LAN addressing and `vtnetN` mapping are measured, and (d) D-100's +-exact netem parameters are ruled. Update `docs/dc-dc-deployment-workflow.md`'s +-Stage 3 row to reflect the REAL state after running this (e.g. "PARTIAL -- +-libvirt objects created, MAAS registration blocked on Stage 2" or similar), +-never to a blanket DONE unless every sub-condition above is genuinely true. +- +-**DC2:** none of the above applies to DC2 until NetBox assigns its real +-supernet (D-101 open item) and `main.tf`'s DC2 blocks are uncommented with +-real values -- re-run this entire runbook for DC2 at that point, do not +-attempt a partial DC2 pass now. +- +--> Proceed to Stage 4 (MAAS enlist/commission/deploy) for DC1 once this +-gate's applicable sub-conditions are genuinely met; DC2 remains blocked at +-Stage 3 until its own gate (NetBox supernet) clears. ++before (a) `vr1-dc0`'s per-DC MAAS rack controller is stood up (Stage-4 gap) ++and Stage 2's MAAS region is reachable, (b) `vr1-dc0`'s OPNsense edge has been ++through the post-boot REST-API bootstrap (`vtnetN` + WAN/LAN applied), and ++(c) D-100's exact netem parameters are ruled. Node sizing (D-121) and the edge ++shape/uplink (D-122) are NO LONGER open. Update ++`docs/dc-dc-deployment-workflow.md`'s Stage 3 row to reflect the REAL state ++after running this (e.g. "PARTIAL -- libvirt objects created, MAAS ++registration blocked on the per-DC rack controller" or similar), never to a ++blanket DONE unless every sub-condition above is genuinely true. ++ ++**`vr1-dc1`:** none of the above applies to `vr1-dc1` this pass. Its v4 ++supernet IS assigned (D-115, `10.12.64.0/19`) -- the blocker is sequencing, ++not a missing literal -- but its `vr1_dc1_planes` block stays commented out ++and it stays out of scope. Re-run this entire runbook for `vr1-dc1` when it is ++sequenced; do not attempt a partial `vr1-dc1` pass now. ++ ++-> Proceed to Stage 4 (MAAS enlist/commission/deploy) for `vr1-dc0` once this ++gate's applicable sub-conditions are genuinely met; `vr1-dc1` remains blocked ++at Stage 3 by sequencing until it is taken up as its own pass. + + --- + +@@ -593,16 +695,19 @@ Stage 3 until its own gate (NetBox supernet) clears. + netem placeholder profile and its provisional status) -- redact + nothing that isn't a secret. + - [ ] `docs/session-ledger.md` updated with the outcome, including which +- sub-steps were blocked and why (Stage 2 dependency, node sizing, +- WAN/LAN token gaps, D-100 netem sub-item). ++ sub-steps were blocked and why (per-DC MAAS rack-controller Stage-4 gap, ++ Stage 2 MAAS-region dependency, the edge post-boot bootstrap, D-100 ++ netem sub-item). + - [ ] `docs/dc-dc-deployment-workflow.md` Stage 3 row updated to the REAL, + honest state (PARTIAL/DONE/NOT STARTED per the GATE section above), + never rounded up to DONE with open sub-conditions. +-- [ ] `opentofu/README.md` updated to reflect DC1's OPNsense edge / node-VM / +- MAAS-vm-host / netem-link modules moving from "not instantiated" to +- instantiated (or partially so) once this runbook actually applies +- them, including any real bug found while doing so (matching this +- repo's practice of logging schema surprises as they're found). +-- [ ] DC2 gate re-confirmed as still closed (NetBox supernet not yet +- assigned) if this session did not clear it -- do not let this +- runbook's DC1 completion be misread as covering DC2. ++- [ ] `opentofu/README.md` updated to reflect `vr1-dc0`'s OPNsense edge / ++ node-VM / MAAS-vm-host / netem-link modules moving from "not ++ instantiated" to instantiated (or partially so) once this runbook ++ actually applies them, including any real bug found while doing so ++ (matching this repo's practice of logging schema surprises as they're ++ found). ++- [ ] `vr1-dc1` gate re-confirmed as still closed BY SEQUENCING (its supernet ++ IS assigned -- D-115 `10.12.64.0/19`) if this session did not take it ++ up -- do not let this runbook's `vr1-dc0` completion be misread as ++ covering `vr1-dc1`. +diff --git a/scripts/site-headend-install.sh b/scripts/site-headend-install.sh +index 46e7dd7..979503c 100644 +--- a/scripts/site-headend-install.sh ++++ b/scripts/site-headend-install.sh +@@ -1,19 +1,39 @@ + #!/usr/bin/env bash +-# scripts/site-headend-install.sh [--check|--dry-run] --compose-cidr <CIDR> [--maas-url-ip <IP>] ++# scripts/site-headend-install.sh [--check|--dry-run] [--role <region+rack|rack>] ++# region+rack (default): --compose-cidr <CIDR> [--maas-url-ip <IP>] ++# rack: --region-url <http://REGION-IP:5240/MAAS> --enroll-secret-file <path> + # +-# Installs a VR1 SITE HEADEND on the host it runs on: MAAS (region+rack) + LXD, with that +-# LXD registered back into MAAS as an LXD VM host so MAAS can COMPOSE the site's non-stack +-# service machines into it. This is D-114, and it is VR0's proven `lxd` + `tailscale` +-# pattern applied per site. ++# Installs a VR1 SITE HEADEND on the host it runs on. Two roles (D-123 rules ONE MAAS ++# region on Office1 + a rack controller PER DC): + # +-# RUN IT ON THE SITE HOST (voffice1 today; vvr1-dc0/vvr1-dc1 later), as root: ++# --role region+rack (DEFAULT -- unchanged legacy behavior): MAAS (region+rack) + LXD, with ++# that LXD registered back into MAAS as an LXD VM host so MAAS can COMPOSE the site's ++# non-stack service machines into it. This is D-114, VR0's proven `lxd` + `tailscale` ++# pattern applied per site. This is the Office1 (voffice1) build. ++# ++# --role rack: a DC rack controller (vvr1-dc0) that ENROLLS to Office1's EXISTING region ++# (D-123). A rack runs NO PostgreSQL, does its own `maas init rack`, and composes ++# NOTHING -- the DC's OpenStack nodes are OpenTofu-created and DISCOVERED via the vcloud ++# virsh vm-host registered to the REGION (not composed here). It therefore SKIPS all of ++# the region DB / createadmin / LXD / vm-host / compose-DHCP work below. Enabling ++# DHCP-on-metal-admin is a REGION-side op (Office1's MAAS, naming this rack as ++# primary_rack) done as a RUNBOOK step -- NOT by this script (see rack note below). ++# ++# RUN IT ON THE SITE HOST, as root: ++# region+rack (voffice1): + # ssh <site-host> 'sudo bash -s' -- --compose-cidr 10.10.1.0/24 < scripts/site-headend-install.sh ++# rack (vvr1-dc0): the region operator reads the enrollment secret from the REGION host's ++# /var/snap/maas/common/maas/secret and drops it into a 0600 file on this host first: ++# ssh <dc-rack> 'sudo bash -s' -- --role rack \ ++# --region-url http://10.10.0.20:5240/MAAS --enroll-secret-file /root/region-enroll.secret \ ++# < scripts/site-headend-install.sh + # + # --check report what is installed/configured; mutate nothing; exit 0 ok / 1 incomplete + # --dry-run print what it would do; mutate nothing + # (no flag) install + configure (idempotent; safe to re-run) + # +-# EXIT: 0 ok | 1 check-failed | 2 bad args | 3 unsupported OS | 4 install failed. ASCII + LF. ++# EXIT: 0 ok | 1 check-failed | 2 bad args (incl. bad/missing --role args) | 3 unsupported OS ++# | 4 install failed (incl. rack: enrollment-secret file absent/empty at mutate time). ASCII + LF. + # + # ===================================================================================== + # THE FOUR TRAPS THIS SCRIPT EXISTS TO ENCODE (all hit for real on 2026-07-13; do not +@@ -49,31 +69,60 @@ + # servers on one L2 is an intermittent failure that is genuinely unpleasant to + # diagnose. --compose-cidr is REQUIRED and has no default precisely so this can never + # be picked by accident, and the script REFUSES to touch DHCP on any other subnet. ++# ++# TRAPS 1-3 (LXD/dnsmasq) DO NOT APPLY IN --role rack: a rack installs no LXD, creates no ++# bridge, and runs no dnsmasq, so the :53/:67 conflict and the 5.21-track pin are moot. ++# Trap 4's SPIRIT is kept absolutely: rack mode configures NO DHCP at all (metal-admin DHCP ++# is a REGION-side op, not this script's), so it cannot fight the edge's Kea by construction. + # ===================================================================================== + set -uo pipefail + + CHECK=0; DRYRUN=0; COMPOSE_CIDR=""; MAAS_IP="" ++ROLE="region+rack"; REGION_URL=""; ENROLL_SECRET_FILE="" + while [ $# -gt 0 ]; do + case "$1" in +- --check) CHECK=1 ;; +- --dry-run) DRYRUN=1 ;; +- --compose-cidr) shift; COMPOSE_CIDR="${1:-}" ;; +- --maas-url-ip) shift; MAAS_IP="${1:-}" ;; ++ --check) CHECK=1 ;; ++ --dry-run) DRYRUN=1 ;; ++ --role) shift; ROLE="${1:-}" ;; ++ --compose-cidr) shift; COMPOSE_CIDR="${1:-}" ;; ++ --maas-url-ip) shift; MAAS_IP="${1:-}" ;; ++ --region-url) shift; REGION_URL="${1:-}" ;; ++ --enroll-secret-file) shift; ENROLL_SECRET_FILE="${1:-}" ;; + -h|--help) +- echo "usage: site-headend-install.sh [--check|--dry-run] --compose-cidr <CIDR> [--maas-url-ip <IP>]" ++ echo "usage: site-headend-install.sh [--check|--dry-run] [--role <region+rack|rack>]" ++ echo " region+rack (default): --compose-cidr <CIDR> [--maas-url-ip <IP>]" ++ echo " rack: --region-url <http://REGION-IP:5240/MAAS> --enroll-secret-file <path>" + exit 0 ;; + *) echo "FAIL: unknown arg '$1'" >&2; exit 2 ;; + esac + shift + done + +-[ -n "$COMPOSE_CIDR" ] || { echo "FAIL: --compose-cidr is REQUIRED (e.g. 10.10.1.0/24). It has no default on purpose: MAAS must serve DHCP on the compose network ONLY, never on the site LAN, which the edge's Kea already owns." >&2; exit 2; } +-echo "$COMPOSE_CIDR" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.0/24$' || { echo "FAIL: --compose-cidr must be a /24 like 10.10.1.0/24 (got '$COMPOSE_CIDR')" >&2; exit 2; } ++case "$ROLE" in ++ region+rack|rack) ;; ++ *) echo "FAIL: --role must be 'region+rack' (default) or 'rack' (got '$ROLE')" >&2; exit 2 ;; ++esac + +-BASE="${COMPOSE_CIDR%.0/24}" # 10.10.1 +-BRIDGE_IP="${BASE}.1" +-RANGE_LO="${BASE}.100" +-RANGE_HI="${BASE}.200" ++# Per-role REQUIRED args are validated UNCONDITIONALLY here (mirrors the legacy --compose-cidr ++# check, which already gated --check too), so a bad invocation is rejected before any mode runs. ++if [ "$ROLE" = "rack" ]; then ++ # rack enrolls to Office1's EXISTING region -- no compose network (LXD-compose is a region ++ # concern); it needs the region URL + the region's enrollment secret (read from a file, never ++ # echoed -- same discipline as $SECRETS below; the region operator obtains it from the region ++ # host's /var/snap/maas/common/maas/secret). ++ [ -n "$REGION_URL" ] || { echo "FAIL: --role rack REQUIRES --region-url <http://REGION-IP:5240/MAAS>" >&2; exit 2; } ++ echo "$REGION_URL" | grep -qiE '^https?://.+/MAAS/?$' || { echo "FAIL: --region-url must look like http://REGION-IP:5240/MAAS (got '$REGION_URL')" >&2; exit 2; } ++ [ -n "$ENROLL_SECRET_FILE" ] || { echo "FAIL: --role rack REQUIRES --enroll-secret-file <path> (the region's enrollment secret; NEVER passed on the command line)" >&2; exit 2; } ++ # --compose-cidr is NOT used in rack mode; ignore it if present. ++else ++ [ -n "$COMPOSE_CIDR" ] || { echo "FAIL: --compose-cidr is REQUIRED (e.g. 10.10.1.0/24). It has no default on purpose: MAAS must serve DHCP on the compose network ONLY, never on the site LAN, which the edge's Kea already owns." >&2; exit 2; } ++ echo "$COMPOSE_CIDR" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.0/24$' || { echo "FAIL: --compose-cidr must be a /24 like 10.10.1.0/24 (got '$COMPOSE_CIDR')" >&2; exit 2; } ++ ++ BASE="${COMPOSE_CIDR%.0/24}" # 10.10.1 ++ BRIDGE_IP="${BASE}.1" ++ RANGE_LO="${BASE}.100" ++ RANGE_HI="${BASE}.200" ++fi + SECRETS=/root/maas-secrets + MAAS_CHANNEL="3.7/stable" + PG_CHANNEL="16/stable" +@@ -113,14 +162,81 @@ report() { + return "$bad" + } + ++# ---------------- check mode, RACK role (read-only) ---------------- ++# A rack runs NO PostgreSQL and NO LXD, so we assert only: the maas snap is present and ++# `maas status` shows `rackd` running. If the rack has been through `maas init rack`, the ++# region URL it enrolled to is recorded (offline-readable) in the rackd config -- report it ++# as the connected-to-region signal, best-effort (never fatal; the file is root-only). ++report_rack() { ++ local bad=0 ++ snap list maas >/dev/null 2>&1 && echo "OK: maas snap $(snap list maas 2>/dev/null | awk 'NR>1{print $2" ("$4")"}')" || { echo "ABSENT: maas snap"; bad=1; } ++ if have maas && maas status 2>/dev/null | grep -q rackd; then echo "OK: MAAS rackd running"; else echo "ABSENT: MAAS rackd not running (not enrolled to a region)"; bad=1; fi ++ local ru; ru="$(grep -sh -m1 -E '^maas_url:' /var/snap/maas/current/rackd.conf 2>/dev/null | awk '{print $2}')" ++ [ -n "$ru" ] && echo "OK: enrolled to region $ru" ++ return "$bad" ++} ++ ++# Dispatch check/precheck to the role-appropriate reporter. ++do_report() { if [ "$ROLE" = "rack" ]; then report_rack; else report; fi; } ++ + report_st=0 +-if [ "$CHECK" = "1" ]; then report; exit $?; fi +-report >/dev/null 2>&1; report_st=$? ++if [ "$CHECK" = "1" ]; then do_report; exit $?; fi ++do_report >/dev/null 2>&1; report_st=$? + [ "$report_st" = "0" ] && [ "$DRYRUN" = "0" ] && { echo "-- site headend already installed and configured --"; exit 0; } + + have apt-get || { echo "FAIL: Debian/Ubuntu (apt) only." >&2; exit 3; } + # root is required to MUTATE, not to plan: --dry-run stays runnable by anyone (and by the harness). + [ "$DRYRUN" = "1" ] || [ "$(id -u)" = "0" ] || { echo "FAIL: must run as root (use: ssh <host> 'sudo bash -s' -- ... < $0)" >&2; exit 4; } ++ ++# ===================================================================================== ++# RACK ROLE (D-123): enroll a DC rack controller to Office1's EXISTING region. ONLY these ++# steps run -- and NONE of the region+rack work below (no PostgreSQL, no `maas init ++# region+rack`, no createadmin/apikey/login, no LXD, no LXD vm-host registration, no ++# compose-network DHCP). Rationale: a rack runs no DB, composes nothing (the DC's OpenStack ++# nodes are OpenTofu-created + discovered via the vcloud virsh vm-host registered to the ++# REGION, not composed here), and DHCP-on-metal-admin is a REGION-side op (enabled against ++# Office1's MAAS naming this rack as primary_rack -- a runbook step, NOT this script). So ++# traps 1-3 (LXD/dnsmasq) do not apply, and trap 4's spirit holds: rack mode touches NO DHCP. ++# ===================================================================================== ++if [ "$ROLE" = "rack" ]; then ++ echo "== 1. time: MAAS manages time via chrony; systemd-timesyncd conflicts ==" ++ run "disable systemd-timesyncd" -- systemctl disable --now systemd-timesyncd >/dev/null 2>&1 ++ ++ echo "== 2. MAAS $MAAS_CHANNEL (rack: NO PostgreSQL -- the rack runs no DB) ==" ++ snap list maas >/dev/null 2>&1 || run "install maas" -- snap install maas --channel="$MAAS_CHANNEL" || exit 4 ++ ++ echo "== 3. maas init rack -- enroll to the region at $REGION_URL ==" ++ # `maas init rack --maas-url <URL> --secret <SECRET>` is the MAAS snap rack-registration ++ # invocation -- CONFIRMED, not invented: this form is documented across the MAAS snap docs and ++ # discourse and is stable across 3.x (so it holds for our 3.7/stable pin), and the region's ++ # shared secret lives at /var/snap/maas/common/maas/secret on the region host (matches the ++ # runbook). The secret is READ from the file at ++ # execution time and NEVER echoed -- so this step is hand-rolled rather than routed through ++ # run() (which would print the expanded command, leaking the secret in --dry-run). ++ if [ "$DRYRUN" = "0" ]; then ++ [ -s "$ENROLL_SECRET_FILE" ] || { echo "FAIL: --enroll-secret-file '$ENROLL_SECRET_FILE' is absent or empty (need the region's enrollment secret)" >&2; exit 4; } ++ if maas status 2>/dev/null | grep -q rackd; then ++ echo " -> already enrolled (rackd running)" ++ else ++ maas init rack --maas-url "$REGION_URL" --secret "$(cat "$ENROLL_SECRET_FILE")" >/dev/null 2>&1 \ ++ || { echo "FAIL: maas init rack" >&2; exit 4; } ++ echo " -> rack registered to region $REGION_URL" ++ fi ++ else ++ echo " [dry-run] maas init rack --maas-url $REGION_URL --secret <read from $ENROLL_SECRET_FILE at run time; never printed>" ++ fi ++ ++ # SKIPPED IN RACK MODE (region concerns): PostgreSQL, maas init region+rack, ++ # createadmin/apikey/login, LXD (5.21 pin), LXD vm-host registration, compose-network DHCP. ++ # metal-admin DHCP (this rack as primary_rack) is enabled REGION-side as a runbook step. ++ if [ "$DRYRUN" = "1" ]; then echo " (dry-run: no changes made)"; exit 0; fi ++ echo ++ echo "RACK ENROLLED to $REGION_URL. Next (REGION-side runbook, NOT this script): enable" ++ echo "DHCP on metal-admin against Office1's MAAS, naming this host as primary_rack." ++ exit 0 ++fi ++ ++# ---- region+rack role (default; unchanged legacy behavior) ---- + [ -n "$MAAS_IP" ] || MAAS_IP="$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{print $7; exit}')" + [ -n "$MAAS_IP" ] || { echo "FAIL: could not measure this host's outbound IP; pass --maas-url-ip" >&2; exit 4; } + +diff --git a/tests/dc-edge-wan-import/fake_netbox.py b/tests/dc-edge-wan-import/fake_netbox.py +new file mode 100644 +index 0000000..0dc65fb +--- /dev/null ++++ b/tests/dc-edge-wan-import/fake_netbox.py +@@ -0,0 +1,43 @@ ++""" ++In-memory stand-in for the NB client in netbox/dc-edge-wan-import.py. ++ ++No real NetBox / network access exists this session -- this fake implements just ++the two methods the target script calls on its client (.one() and .create()) so the ++REAL main()/preflight can be driven end-to-end without a live server. It is injected ++at the get_nb() seam, exactly as tests/dc-dc-prefixes-import/ injects a fake pynetbox. ++ ++FIDELITY NOTE. The target's real .one() issues GET ...?<filter>&limit=1 and takes ++results[0] -- so it is single-match by construction and CANNOT raise on a duplicate ++prefix (unlike pynetbox.get(), which the sibling fake models raising on multi-match). ++This fake matches THAT contract: .one() returns the first record whose fields equal ++the filter, or None. For an idempotency SKIP that is the safe direction -- a ++pre-existing prefix reads as "present" and is not duplicated. .create() records every ++write so a test can assert a rejected/dry run wrote NOTHING. ++""" ++from __future__ import annotations ++ ++ ++class FakeNB: ++ def __init__(self, roles=(), prefixes=(), sites=()): ++ # store: endpoint path -> list of record dicts (as the API would return them) ++ self.store = { ++ "ipam/roles": [dict(r) for r in roles], ++ "ipam/prefixes": [dict(p) for p in prefixes], ++ "dcim/sites": [dict(s) for s in sites], ++ } ++ self.creates = [] # (path, payload) for every .create() -- the write ledger ++ self._next_id = 1000 ++ ++ def one(self, path, **flt): ++ for rec in self.store.get(path, []): ++ if all(str(rec.get(k)) == str(v) for k, v in flt.items()): ++ return rec ++ return None ++ ++ def create(self, path, payload): ++ self.creates.append((path, dict(payload))) ++ rec = dict(payload) ++ rec["id"] = self._next_id ++ self._next_id += 1 ++ self.store.setdefault(path, []).append(rec) ++ return rec +diff --git a/tests/dc-edge-wan-import/run-tests.sh b/tests/dc-edge-wan-import/run-tests.sh +new file mode 100644 +index 0000000..a5aa1e1 +--- /dev/null ++++ b/tests/dc-edge-wan-import/run-tests.sh +@@ -0,0 +1,76 @@ ++#!/usr/bin/env bash ++# tests/dc-edge-wan-import/run-tests.sh ++# ++# Harness for netbox/dc-edge-wan-import.py. OFFLINE -- touches no NetBox. ++# ++# Two layers, both required (mirrors d120-compose-bands + dc-dc-prefixes-import): ++# 1. STATIC greps that PIN the D-115 edge values + the guards every sandbox-loop ++# writer must have (dry-by-default, upstream-write gated IN CODE, whole-plan ++# preflight). A changed constant that a behavioral test would ALSO change is ++# caught here. ++# 2. test_logic.py drives the REAL main()/preflight against an in-memory fake NB ++# (injected at the get_nb() seam) -- half-write guard, scope-not-swapped, ++# idempotent-skip, dry-run-writes-nothing. In-process on purpose: no port / ++# readiness / cleanup flakiness to redden the shared gauntlet. ++# ASCII + LF. ++set -uo pipefail ++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ++S="$(cd "$HERE/../../netbox" && pwd)/dc-edge-wan-import.py" ++pass=0; fail=0 ++ok() { pass=$((pass+1)); } ++bad() { fail=$((fail+1)); echo " FAIL: $1"; } ++ ++command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } ++ ++python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" ++python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" ++ ++# no env -> must fail loud, not guess a target ++NETBOX_URL= NETBOX_TOKEN= python3 "$S" >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" ++ ++# DRY BY DEFAULT ++grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" ++grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" ++ ++# UPSTREAM WRITE GATED IN CODE (not by discipline) ++grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" ++grep -q 'SANDBOX_HOSTS' "$S" && ok || bad "lost the SANDBOX_HOSTS gate" ++out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" --commit 2>&1)" ++printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ ++ || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" ++ ++# WAF: UA-aware or every upstream call 403s (looks like auth failure) ++grep -q 'User-Agent' "$S" && grep -q 'curl/8.5.0' "$S" && ok || bad "lost the WAF-safe User-Agent" ++ ++# WHOLE-PLAN PREFLIGHT -- validate role/container/placement/site BEFORE any create ++grep -q "outside the Edge container" "$S" && ok || bad "lost the in-container preflight" ++grep -q "container .* absent" "$S" && ok || bad "lost the container-exists preflight" ++grep -q "role '.*' absent" "$S" && ok || bad "lost the role-exists preflight" ++grep -q "site '.*' absent" "$S" && ok || bad "lost the site-resolves preflight" ++ ++# THE D-115 EDGE VALUES -- the two DC /24s, their role, and their SITE scope. ++grep -qF '("172.30.2.0/24", "vr1-dc0"' "$S" && ok || bad "172.30.2.0/24 -> vr1-dc0 binding is GONE / changed" ++grep -qF '("172.30.3.0/24", "vr1-dc1"' "$S" && ok || bad "172.30.3.0/24 -> vr1-dc1 binding is GONE / changed" ++grep -qF 'ROLE_SLUG = "edge"' "$S" && ok || bad "edge role slug changed" ++grep -qF 'CONTAINER = "172.30.0.0/16"' "$S" && ok || bad "Edge container is not 172.30.0.0/16" ++grep -q '"scope_type": "dcim.site"' "$S" && ok || bad "the /24s are not dcim.site-scoped (must match office1-wan)" ++ ++# This tool must NOT create the role/container/sites -- they are preconditions. ++grep -q 'ipam/roles".*payload\|create("ipam/roles"\|create("dcim/sites"' "$S" \ ++ && bad "this tool CREATES a role/site -- those are preconditions, it must only create the /24s" || ok ++ ++# ---- behavioral layer: drive the real main() against the injected fake NB ---- ++echo " -- behavioral (test_logic.py) --" ++tl_out="$(python3 "$HERE/test_logic.py" 2>&1)"; tl_rc=$? ++printf '%s\n' "$tl_out" | grep -E '^(FAIL|PASS: |test_logic:)' | sed 's/^/ /' ++tl_n="$(printf '%s\n' "$tl_out" | sed -n 's/^test_logic: ALL PASS (\([0-9]*\) checks)$/\1/p')" ++if [ "$tl_rc" -eq 0 ] && [ -n "$tl_n" ]; then ++ pass=$((pass + tl_n)) ++else ++ bad "test_logic.py behavioral suite FAILED (rc=$tl_rc)" ++fi ++ ++echo ++total=$((pass+fail)) ++if [ "$fail" -eq 0 ]; then echo "dc-edge-wan-import: $pass/$total PASS"; exit 0; fi ++echo "dc-edge-wan-import: $fail/$total FAIL"; exit 1 +diff --git a/tests/dc-edge-wan-import/test_logic.py b/tests/dc-edge-wan-import/test_logic.py +new file mode 100644 +index 0000000..01ac01a +--- /dev/null ++++ b/tests/dc-edge-wan-import/test_logic.py +@@ -0,0 +1,245 @@ ++#!/usr/bin/env python3 ++""" ++Behavioral tests for netbox/dc-edge-wan-import.py -- drives the REAL main()/preflight ++against an in-memory fake NB (tests/dc-edge-wan-import/fake_netbox.py) injected at the ++get_nb() seam. No live NetBox. Invoked by run-tests.sh alongside the static greps. ++ ++The properties pinned here map to this repo's documented IPAM bug classes: ++ * HALF-WRITE (roles-aggregates-import.py): a later target's missing precondition ++ must abort with ZERO creates, never "wrote dc0 then died". ++ * SWAPPED SCOPE (the D-117 wrong-target near-miss): the vr1-dc0 /24 must bind ++ vr1-dc0's id and the vr1-dc1 /24 must bind vr1-dc1's id -- not shared or swapped. ++ * ALREADY-PRESENT = SKIP (idempotent), distinct from missing-infra = die. ++ * DRY BY DEFAULT: no --commit writes NOTHING. ++ * UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag. ++""" ++from __future__ import annotations ++ ++import contextlib ++import faulthandler ++import importlib.util ++import inspect ++import io ++import os ++import sys ++ ++faulthandler.dump_traceback_later(30, exit=True) ++ ++HERE = os.path.dirname(os.path.abspath(__file__)) ++REPO_ROOT = os.path.dirname(os.path.dirname(HERE)) ++TARGET_PATH = os.path.join(REPO_ROOT, "netbox", "dc-edge-wan-import.py") ++sys.path.insert(0, HERE) ++ ++import fake_netbox # noqa: E402 ++ ++P = 0 ++F = 0 ++ ++ ++def ok(label): ++ global P ++ P += 1 ++ print(f"PASS: {label}") ++ ++ ++def no(label, detail=""): ++ global F ++ F += 1 ++ print(f"FAIL: {label}" + (f" ({detail})" if detail else "")) ++ ++ ++def check(cond, label, detail=""): ++ ok(label) if cond else no(label, detail) ++ ++ ++def load_target(): ++ spec = importlib.util.spec_from_file_location("dc_edge_wan_import", TARGET_PATH) ++ mod = importlib.util.module_from_spec(spec) ++ spec.loader.exec_module(mod) ++ return mod ++ ++ ++T = load_target() ++ ++ ++@contextlib.contextmanager ++def captured_stdout(): ++ buf = io.StringIO() ++ with contextlib.redirect_stdout(buf): ++ yield buf ++ ++ ++def run_main(argv, fake): ++ T.get_nb = lambda base=None, token=None: fake ++ old = sys.argv ++ sys.argv = ["dc-edge-wan-import.py"] + argv ++ try: ++ return T.main() ++ finally: ++ sys.argv = old ++ ++ ++def run_dies(label, argv, fake): ++ # swallow stdout AND stderr -- die() prints "FAIL: ..." to stderr by design, and ++ # for an EXPECTED death that is not a test failure, just noise on the gauntlet. ++ try: ++ with contextlib.redirect_stderr(io.StringIO()), captured_stdout(): ++ run_main(argv, fake) ++ no(label, "did not raise SystemExit") ++ except SystemExit: ++ ok(label) ++ ++ ++# Full-precondition fixture builders (role + container + both DC sites present). ++ROLE = {"slug": "edge", "name": "Edge", "id": 1} ++CONTAINER = {"prefix": "172.30.0.0/16", "id": 10} ++SITE_DC0 = {"slug": "vr1-dc0", "name": "VR1 DC0", "id": 20} ++SITE_DC1 = {"slug": "vr1-dc1", "name": "VR1 DC1", "id": 21} ++ ++ ++def full_fake(prefixes_extra=()): ++ return fake_netbox.FakeNB( ++ roles=[ROLE], ++ prefixes=[CONTAINER, *prefixes_extra], ++ sites=[SITE_DC0, SITE_DC1], ++ ) ++ ++ ++os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # a known sandbox (guard passes) ++os.environ["NETBOX_TOKEN"] = "fake-token" ++ ++# ----------------------------------------------------------------------------- ++# 1. DRY RUN is the default and writes NOTHING. ++# ----------------------------------------------------------------------------- ++fk = full_fake() ++with captured_stdout() as out: ++ rc = run_main([], fk) ++check(rc == 0, "dry run rc==0", str(rc)) ++check(out.getvalue().count("would CREATE") == 2, "dry run PLANS both /24s", ++ str(out.getvalue().count("would CREATE"))) ++check(out.getvalue().count("CREATED ") == 0, "dry run CREATES nothing (no CREATED line)") ++check("DRY RUN" in out.getvalue(), "dry run says so, loudly") ++check(len(fk.creates) == 0, "dry run left the NetBox untouched (0 writes)", str(len(fk.creates))) ++ ++# ----------------------------------------------------------------------------- ++# 2. --commit writes exactly the two /24s, correctly scoped (NO swap). ++# ----------------------------------------------------------------------------- ++fk = full_fake() ++with captured_stdout() as out: ++ rc = run_main(["--commit"], fk) ++check(rc == 0, "commit rc==0", str(rc)) ++check(len(fk.creates) == 2, "commit created exactly 2 prefixes", str(len(fk.creates))) ++check(all(path == "ipam/prefixes" for path, _ in fk.creates), ++ "commit wrote only ipam/prefixes (no stray sites/roles/container)") ++by_prefix = {pl["prefix"]: pl for _, pl in fk.creates} ++check(by_prefix.get("172.30.2.0/24", {}).get("scope_id") == 20, ++ "SCOPE: 172.30.2.0/24 binds vr1-dc0's id (20)", str(by_prefix.get("172.30.2.0/24"))) ++check(by_prefix.get("172.30.3.0/24", {}).get("scope_id") == 21, ++ "SCOPE: 172.30.3.0/24 binds vr1-dc1's id (21) -- not swapped", ++ str(by_prefix.get("172.30.3.0/24"))) ++check(all(pl.get("scope_type") == "dcim.site" for pl in by_prefix.values()), ++ "SCOPE: both /24s are dcim.site-scoped (matches office1-wan)") ++check(all(pl.get("role") == 1 for pl in by_prefix.values()), "both /24s carry the edge role id") ++check(all(pl.get("status") == "active" for pl in by_prefix.values()), "both /24s are status active") ++ ++# ----------------------------------------------------------------------------- ++# 3. Already-present = idempotent SKIP (both present -> 0 creates, exit clean). ++# ----------------------------------------------------------------------------- ++fk = full_fake(prefixes_extra=[{"prefix": "172.30.2.0/24", "id": 30}, ++ {"prefix": "172.30.3.0/24", "id": 31}]) ++with captured_stdout() as out: ++ rc = run_main(["--commit"], fk) ++check(rc == 0, "both-present idempotent run rc==0") ++check(out.getvalue().count("EXISTS") == 2, "both-present run reports both as EXISTS") ++check(len(fk.creates) == 0, "both-present run creates nothing new") ++ ++# ----------------------------------------------------------------------------- ++# 4. Partial-present: dc0 already there, dc1 absent -> creates ONLY dc1. ++# ----------------------------------------------------------------------------- ++fk = full_fake(prefixes_extra=[{"prefix": "172.30.2.0/24", "id": 30}]) ++with captured_stdout(): ++ rc = run_main(["--commit"], fk) ++check(rc == 0, "partial-present run rc==0") ++check(len(fk.creates) == 1 and fk.creates[0][1]["prefix"] == "172.30.3.0/24", ++ "partial-present run creates ONLY the missing 172.30.3.0/24", ++ str([pl["prefix"] for _, pl in fk.creates])) ++ ++# ----------------------------------------------------------------------------- ++# 5. HALF-WRITE GUARD -- a LATER target's missing site aborts with ZERO creates. ++# role + container + vr1-dc0 present, but vr1-dc1 SITE absent. ++# ----------------------------------------------------------------------------- ++fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[CONTAINER], sites=[SITE_DC0]) ++run_dies("half-write: missing vr1-dc1 site is REJECTED before any write", ["--commit"], fk) ++check(len(fk.creates) == 0, "half-write: the rejected run wrote NOTHING (vr1-dc0 not created)") ++ ++# ----------------------------------------------------------------------------- ++# 6-8. Missing infra = die (each precondition), each writing nothing. ++# ----------------------------------------------------------------------------- ++fk = fake_netbox.FakeNB(roles=[], prefixes=[CONTAINER], sites=[SITE_DC0, SITE_DC1]) ++run_dies("missing edge role is REJECTED", ["--commit"], fk) ++check(len(fk.creates) == 0, "missing-role run wrote nothing") ++ ++fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[], sites=[SITE_DC0, SITE_DC1]) ++run_dies("missing 172.30.0.0/16 container is REJECTED", ["--commit"], fk) ++check(len(fk.creates) == 0, "missing-container run wrote nothing") ++ ++fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[CONTAINER], sites=[SITE_DC1]) ++run_dies("missing vr1-dc0 site is REJECTED", ["--commit"], fk) ++check(len(fk.creates) == 0, "missing-dc0-site run wrote nothing") ++ ++# ----------------------------------------------------------------------------- ++# 9. /24 OUTSIDE the container is rejected by the local subnet preflight. ++# ----------------------------------------------------------------------------- ++_saved = T.TARGETS ++T.TARGETS = [("10.0.0.0/24", "vr1-dc0", "bogus -- not inside 172.30.0.0/16")] ++try: ++ run_dies("a /24 outside the Edge container is REJECTED", ["--commit"], full_fake()) ++finally: ++ T.TARGETS = _saved ++ ++# ----------------------------------------------------------------------------- ++# 10. Missing NETBOX_URL/TOKEN fails loud (does not guess a target). ++# ----------------------------------------------------------------------------- ++_u = os.environ.pop("NETBOX_URL", None) ++run_dies("missing NETBOX_URL fails loud", [], full_fake()) ++os.environ["NETBOX_URL"] = _u ++ ++# ----------------------------------------------------------------------------- ++# 11. UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag, ++# proceeds with it. Sandbox needs no flag (covered above). ++# ----------------------------------------------------------------------------- ++os.environ["NETBOX_URL"] = "https://netbox.baldurkeep.com" # the production apex ++fk = full_fake() ++run_dies("upstream --commit WITHOUT --yes-write-upstream is REFUSED", ["--commit"], fk) ++check(len(fk.creates) == 0, "the refused upstream --commit wrote NOTHING") ++ ++fk = full_fake() ++with captured_stdout(): ++ rc = run_main(["--commit", "--yes-write-upstream"], fk) ++check(rc == 0 and len(fk.creates) == 2, ++ "upstream --commit --yes-write-upstream IS allowed and writes both /24s", ++ str((rc, len(fk.creates)))) ++os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" ++ ++# ----------------------------------------------------------------------------- ++# 12. Structural pins (a changed constant a behavioral test alone would miss). ++# ----------------------------------------------------------------------------- ++check(T.ROLE_SLUG == "edge", "ROLE_SLUG is 'edge'") ++check(T.CONTAINER == "172.30.0.0/16", "CONTAINER is 172.30.0.0/16") ++check(T.STATUS == "active", "STATUS is active (like office1-wan .1/24)") ++check([c for c, _, _ in T.TARGETS] == ["172.30.2.0/24", "172.30.3.0/24"], ++ "TARGETS pins exactly the two DC /24s .2 and .3") ++check([s for _, s, _ in T.TARGETS] == ["vr1-dc0", "vr1-dc1"], ++ "TARGETS scopes .2->vr1-dc0 and .3->vr1-dc1 (no swap in the source)") ++check(T.SANDBOX_HOSTS == {"localhost", "127.0.0.1", "10.10.1.10"}, ++ "SANDBOX_HOSTS matches the sibling importers") ++check("get_nb" in dir(T) and callable(T.get_nb), "get_nb() injection seam exists") ++_src = inspect.getsource(T) ++check('User-Agent"' in _src and "curl/8.5.0" in _src, "UA-aware (WAF-safe User-Agent set)") ++ ++print() ++if F == 0: ++ print(f"test_logic: ALL PASS ({P} checks)") ++ sys.exit(0) ++print(f"test_logic: {F} FAIL of {P + F}") ++sys.exit(1) +diff --git a/tests/dc-rack-mgmt-import/fake_netbox.py b/tests/dc-rack-mgmt-import/fake_netbox.py +new file mode 100644 +index 0000000..09199e8 +--- /dev/null ++++ b/tests/dc-rack-mgmt-import/fake_netbox.py +@@ -0,0 +1,45 @@ ++""" ++In-memory stand-in for the NB client in netbox/dc-rack-mgmt-import.py. ++ ++No real NetBox / network access exists this session -- this fake implements just ++the two methods the target script calls on its client (.one() and .create()) so the ++REAL main()/preflight can be driven end-to-end without a live server. It is injected ++at the get_nb() seam, exactly as tests/dc-edge-wan-import/ injects its fake. ++ ++FIDELITY NOTE. The target's real .one() issues GET ...?<filter>&limit=1 and takes ++results[0] -- so it is single-match by construction and CANNOT raise on a duplicate ++(unlike pynetbox.get(), which the older sibling fake models raising on multi-match). ++This fake matches THAT contract: .one() returns the first record whose fields equal ++the filter, or None. For an idempotency SKIP that is the safe direction -- a ++pre-existing object reads as "present" and is not duplicated. .create() records every ++write so a test can assert a rejected/dry run wrote NOTHING. The store is keyed by ++path, so ipam/prefixes, ipam/roles, dcim/sites AND ipam/ip-addresses all work. ++""" ++from __future__ import annotations ++ ++ ++class FakeNB: ++ def __init__(self, roles=(), prefixes=(), sites=(), ip_addresses=()): ++ # store: endpoint path -> list of record dicts (as the API would return them) ++ self.store = { ++ "ipam/roles": [dict(r) for r in roles], ++ "ipam/prefixes": [dict(p) for p in prefixes], ++ "dcim/sites": [dict(s) for s in sites], ++ "ipam/ip-addresses": [dict(a) for a in ip_addresses], ++ } ++ self.creates = [] # (path, payload) for every .create() -- the write ledger ++ self._next_id = 1000 ++ ++ def one(self, path, **flt): ++ for rec in self.store.get(path, []): ++ if all(str(rec.get(k)) == str(v) for k, v in flt.items()): ++ return rec ++ return None ++ ++ def create(self, path, payload): ++ self.creates.append((path, dict(payload))) ++ rec = dict(payload) ++ rec["id"] = self._next_id ++ self._next_id += 1 ++ self.store.setdefault(path, []).append(rec) ++ return rec +diff --git a/tests/dc-rack-mgmt-import/run-tests.sh b/tests/dc-rack-mgmt-import/run-tests.sh +new file mode 100644 +index 0000000..d8ca624 +--- /dev/null ++++ b/tests/dc-rack-mgmt-import/run-tests.sh +@@ -0,0 +1,93 @@ ++#!/usr/bin/env bash ++# tests/dc-rack-mgmt-import/run-tests.sh ++# ++# Harness for netbox/dc-rack-mgmt-import.py. OFFLINE -- touches no NetBox. ++# ++# Two layers, both required (mirrors dc-edge-wan-import + d120-compose-bands): ++# 1. STATIC greps that PIN the D-124 scheme values + the guards every sandbox-loop ++# writer must have (dry-by-default, upstream-write gated IN CODE, whole-plan ++# preflight, NO invented literal). A changed constant a behavioral test would ALSO ++# change is caught here. ++# 2. test_logic.py drives the REAL main()/preflight against an in-memory fake NB ++# (injected at the get_nb() seam) -- half-write guard, role/scope binding, band ++# arithmetic, idempotent-skip, dry-run-writes-nothing. In-process on purpose: no ++# port / readiness / cleanup flakiness to redden the shared gauntlet. ++# ASCII + LF. ++set -uo pipefail ++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" ++S="$(cd "$HERE/../../netbox" && pwd)/dc-rack-mgmt-import.py" ++pass=0; fail=0 ++ok() { pass=$((pass+1)); } ++bad() { fail=$((fail+1)); echo " FAIL: $1"; } ++ ++command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } ++ ++python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" ++python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" ++ ++# no env -> must fail loud, not guess a target ++NETBOX_URL= NETBOX_TOKEN= python3 "$S" --transit-cidr 10.12.60.0/30 --rack-ip 10.12.8.5 \ ++ >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" ++ ++# NO INVENTED LITERAL -- both inputs are REQUIRED (no default CIDR/IP baked in) ++NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --rack-ip 10.12.8.5 \ ++ >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --transit-cidr must fail (no invented CIDR)" ++NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --transit-cidr 10.12.60.0/30 \ ++ >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --rack-ip must fail (no invented IP)" ++grep -q 'default=os.environ.get("TRANSIT_CIDR")' "$S" && ok || bad "lost the TRANSIT_CIDR env fallback" ++grep -q 'default=os.environ.get("RACK_IP")' "$S" && ok || bad "lost the RACK_IP env fallback" ++# guard against a baked-in transit /30 or /31 literal masquerading as a default ++grep -qE '=\s*"10\.[0-9]+\.[0-9]+\.[0-9]+/3[01]"' "$S" && bad "a /30 or /31 literal is baked in -- must be an INPUT" || ok ++ ++# DRY BY DEFAULT ++grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" ++grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" ++ ++# UPSTREAM WRITE GATED IN CODE (not by discipline) ++grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" ++grep -q 'SANDBOX_HOSTS' "$S" && ok || bad "lost the SANDBOX_HOSTS gate" ++out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" \ ++ --transit-cidr 10.12.60.0/30 --rack-ip 10.12.8.5 --commit 2>&1)" ++printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ ++ || bad "did NOT refuse a --commit to a non-sandbox host (it would have written to production)" ++ ++# WAF: UA-aware or every upstream call 403s (looks like auth failure) ++grep -q 'User-Agent' "$S" && grep -q 'curl/8.5.0' "$S" && ok || bad "lost the WAF-safe User-Agent" ++ ++# WHOLE-PLAN PREFLIGHT -- validate role/container/transit/band/site BEFORE any create ++grep -q "outside the container" "$S" && ok || bad "lost the transit-in-container preflight" ++grep -q "container .* absent" "$S" && ok || bad "lost the container-exists preflight" ++grep -q "role '.*' absent" "$S" && ok || bad "lost the role-exists preflight" ++grep -q "site '.*' absent" "$S" && ok || bad "lost the site-resolves preflight" ++grep -q "GATEWAY" "$S" && ok || bad "lost the .1-gateway rejection" ++grep -q "static band" "$S" && ok || bad "lost the D-120 static-band check" ++grep -q "outside metal-admin" "$S" && ok || bad "lost the metal-admin containment check" ++grep -q "must be a /30 or /31" "$S" && ok || bad "lost the /30-or-/31 shape check" ++ ++# THE D-124 SCHEME VALUES -- role, container, site scope, metal-admin band, rack dns. ++grep -qF 'ROLE_SLUG = "transit"' "$S" && ok || bad "transit role slug changed" ++grep -qF 'CONTAINER = "10.12.0.0/16"' "$S" && ok || bad "container is not 10.12.0.0/16 (Cloud)" ++grep -qF 'SITE_SLUG = "vr1-dc0"' "$S" && ok || bad "transit site scope is not vr1-dc0" ++grep -qF 'METAL_ADMIN = "10.12.8.0/22"' "$S" && ok || bad "metal-admin is not 10.12.8.0/22" ++grep -qF 'RACK_DNS = "vvr1-dc0"' "$S" && ok || bad "rack dns name is not vvr1-dc0" ++grep -q '"scope_type": "dcim.site"' "$S" && ok || bad "the transit prefix is not dcim.site-scoped" ++ ++# This tool must NOT create the role/container/site -- they are preconditions. ++grep -q 'create("ipam/roles"\|create("dcim/sites"' "$S" \ ++ && bad "this tool CREATES a role/site -- those are preconditions" || ok ++ ++# ---- behavioral layer: drive the real main() against the injected fake NB ---- ++echo " -- behavioral (test_logic.py) --" ++tl_out="$(python3 "$HERE/test_logic.py" 2>&1)"; tl_rc=$? ++printf '%s\n' "$tl_out" | grep -E '^(FAIL|PASS: |test_logic:)' | sed 's/^/ /' ++tl_n="$(printf '%s\n' "$tl_out" | sed -n 's/^test_logic: ALL PASS (\([0-9]*\) checks)$/\1/p')" ++if [ "$tl_rc" -eq 0 ] && [ -n "$tl_n" ]; then ++ pass=$((pass + tl_n)) ++else ++ bad "test_logic.py behavioral suite FAILED (rc=$tl_rc)" ++fi ++ ++echo ++total=$((pass+fail)) ++if [ "$fail" -eq 0 ]; then echo "dc-rack-mgmt-import: $pass/$total PASS"; exit 0; fi ++echo "dc-rack-mgmt-import: $fail/$total FAIL"; exit 1 +diff --git a/tests/dc-rack-mgmt-import/test_logic.py b/tests/dc-rack-mgmt-import/test_logic.py +new file mode 100644 +index 0000000..8ec679e +--- /dev/null ++++ b/tests/dc-rack-mgmt-import/test_logic.py +@@ -0,0 +1,345 @@ ++#!/usr/bin/env python3 ++""" ++Behavioral tests for netbox/dc-rack-mgmt-import.py -- drives the REAL main()/preflight ++against an in-memory fake NB (tests/dc-rack-mgmt-import/fake_netbox.py) injected at the ++get_nb() seam. No live NetBox. Invoked by run-tests.sh alongside the static greps. ++ ++The properties pinned here map to this repo's documented IPAM bug classes + the D-124 ++scheme (Scheme A -- transit-numbered mesh): ++ * HALF-WRITE (roles-aggregates-import.py): a bad rack IP (the .1 gateway) must abort ++ with ZERO creates, never "wrote the transit prefix then died on the rack IP". ++ * ROLE/SCOPE binding: the transit prefix carries the `transit` role id and is ++ dcim.site-scoped to vr1-dc0; the rack IP carries NO ipam role (NetBox ip-address ++ `role` is a choice field) -- identity is in dns_name/description. ++ * BAND arithmetic: rack IP within metal-admin 10.12.8.0/22, D-120 static band .2-.49, ++ NOT the .1 gateway, NOT .0. ++ * TRANSIT shape: /30 or /31, host-bits-clear, subnet_of the Cloud container. ++ * ALREADY-PRESENT = SKIP (idempotent), distinct from missing-infra = die. ++ * DRY BY DEFAULT: no --commit writes NOTHING. ++ * UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag. ++ * NO INVENTED LITERAL: missing --transit-cidr / --rack-ip fails loud. ++""" ++from __future__ import annotations ++ ++import contextlib ++import faulthandler ++import importlib.util ++import inspect ++import io ++import os ++import sys ++ ++faulthandler.dump_traceback_later(30, exit=True) ++ ++HERE = os.path.dirname(os.path.abspath(__file__)) ++REPO_ROOT = os.path.dirname(os.path.dirname(HERE)) ++TARGET_PATH = os.path.join(REPO_ROOT, "netbox", "dc-rack-mgmt-import.py") ++sys.path.insert(0, HERE) ++ ++import fake_netbox # noqa: E402 ++ ++P = 0 ++F = 0 ++ ++ ++def ok(label): ++ global P ++ P += 1 ++ print(f"PASS: {label}") ++ ++ ++def no(label, detail=""): ++ global F ++ F += 1 ++ print(f"FAIL: {label}" + (f" ({detail})" if detail else "")) ++ ++ ++def check(cond, label, detail=""): ++ ok(label) if cond else no(label, detail) ++ ++ ++def load_target(): ++ spec = importlib.util.spec_from_file_location("dc_rack_mgmt_import", TARGET_PATH) ++ mod = importlib.util.module_from_spec(spec) ++ spec.loader.exec_module(mod) ++ return mod ++ ++ ++T = load_target() ++ ++ ++@contextlib.contextmanager ++def captured_stdout(): ++ buf = io.StringIO() ++ with contextlib.redirect_stdout(buf): ++ yield buf ++ ++ ++# The operator-supplied, NetBox-assigned example values under test. Chosen INSIDE the ++# scheme: transit /30 subnet_of Cloud 10.12.0.0/16; rack IP in the .2-.49 static band. ++TRANSIT = "10.12.60.0/30" ++TRANSIT31 = "10.12.60.0/31" ++RACK = "10.12.8.5" ++RACK_ADDR = "10.12.8.5/22" # how the tool stores it (host + metal-admin /22 mask) ++ ++ ++def run_main(argv, fake): ++ T.get_nb = lambda base=None, token=None: fake ++ old = sys.argv ++ sys.argv = ["dc-rack-mgmt-import.py"] + argv ++ try: ++ return T.main() ++ finally: ++ sys.argv = old ++ ++ ++def run_dies(label, argv, fake): ++ # swallow stdout AND stderr -- die() prints "FAIL: ..." to stderr by design, and ++ # for an EXPECTED death that is not a test failure, just noise on the gauntlet. ++ try: ++ with contextlib.redirect_stderr(io.StringIO()), captured_stdout(): ++ run_main(argv, fake) ++ no(label, "did not raise SystemExit") ++ except SystemExit: ++ ok(label) ++ ++ ++# Full-precondition fixture builders (transit role + Cloud container + vr1-dc0 site). ++ROLE = {"slug": "transit", "name": "Transit", "id": 7} ++CONTAINER = {"prefix": "10.12.0.0/16", "id": 10} ++SITE_DC0 = {"slug": "vr1-dc0", "name": "VR1 DC0", "id": 20} ++ ++ ++def full_fake(prefixes_extra=(), ip_extra=()): ++ return fake_netbox.FakeNB( ++ roles=[ROLE], ++ prefixes=[CONTAINER, *prefixes_extra], ++ sites=[SITE_DC0], ++ ip_addresses=list(ip_extra), ++ ) ++ ++ ++def base_args(extra=()): ++ return ["--transit-cidr", TRANSIT, "--rack-ip", RACK, *extra] ++ ++ ++os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" # a known sandbox (guard passes) ++os.environ["NETBOX_TOKEN"] = "fake-token" ++ ++# ----------------------------------------------------------------------------- ++# 1. DRY RUN is the default and writes NOTHING. ++# ----------------------------------------------------------------------------- ++fk = full_fake() ++with captured_stdout() as out: ++ rc = run_main(base_args(), fk) ++check(rc == 0, "dry run rc==0", str(rc)) ++check(out.getvalue().count("would CREATE") == 2, "dry run PLANS both objects", ++ str(out.getvalue().count("would CREATE"))) ++check(out.getvalue().count("CREATED ") == 0, "dry run CREATES nothing (no CREATED line)") ++check("DRY RUN" in out.getvalue(), "dry run says so, loudly") ++check(len(fk.creates) == 0, "dry run left the NetBox untouched (0 writes)", str(len(fk.creates))) ++ ++# ----------------------------------------------------------------------------- ++# 2. --commit writes exactly the transit prefix + the rack IP, correctly bound. ++# ----------------------------------------------------------------------------- ++fk = full_fake() ++with captured_stdout(): ++ rc = run_main(base_args(["--commit"]), fk) ++check(rc == 0, "commit rc==0", str(rc)) ++check(len(fk.creates) == 2, "commit created exactly 2 objects", str(len(fk.creates))) ++paths = [p for p, _ in fk.creates] ++check(paths.count("ipam/prefixes") == 1, "commit wrote exactly one ipam/prefixes") ++check(paths.count("ipam/ip-addresses") == 1, "commit wrote exactly one ipam/ip-addresses") ++check("ipam/roles" not in paths and "dcim/sites" not in paths, ++ "commit created NO role / site (those are preconditions)") ++pfx = next(pl for pp, pl in fk.creates if pp == "ipam/prefixes") ++ipa = next(pl for pp, pl in fk.creates if pp == "ipam/ip-addresses") ++check(pfx.get("prefix") == TRANSIT, "transit prefix is the supplied CIDR", str(pfx.get("prefix"))) ++check(pfx.get("role") == 7, "transit prefix carries the transit role id (7)", str(pfx.get("role"))) ++check(pfx.get("scope_type") == "dcim.site", "transit prefix is dcim.site-scoped", ++ str(pfx.get("scope_type"))) ++check(pfx.get("scope_id") == 20, "transit prefix binds vr1-dc0's site id (20)", str(pfx.get("scope_id"))) ++check(pfx.get("status") == "active", "transit prefix is status active") ++check(ipa.get("address") == RACK_ADDR, "rack IP stored with the /22 metal-admin mask", ++ str(ipa.get("address"))) ++check("role" not in ipa, "rack IP carries NO ipam role (NetBox ip role is a choice field)") ++check(ipa.get("dns_name") == "vvr1-dc0", "rack IP dns_name identifies the rack controller vvr1-dc0", ++ str(ipa.get("dns_name"))) ++check("rack controller" in ipa.get("description", "").lower(), ++ "rack IP description notes it is the vr1-dc0 MAAS rack controller") ++check(ipa.get("status") == "active", "rack IP is status active") ++ ++# ----------------------------------------------------------------------------- ++# 3. A /31 transit is also accepted (point-to-point). ++# ----------------------------------------------------------------------------- ++fk = full_fake() ++with captured_stdout(): ++ rc = run_main(["--transit-cidr", TRANSIT31, "--rack-ip", RACK, "--commit"], fk) ++check(rc == 0 and len(fk.creates) == 2, "a /31 transit is accepted and writes both objects", ++ str((rc, len(fk.creates)))) ++ ++# ----------------------------------------------------------------------------- ++# 4. Already-present = idempotent SKIP (both present -> 0 creates, exit clean). ++# ----------------------------------------------------------------------------- ++fk = full_fake(prefixes_extra=[{"prefix": TRANSIT, "id": 30}], ++ ip_extra=[{"address": RACK_ADDR, "id": 31}]) ++with captured_stdout() as out: ++ rc = run_main(base_args(["--commit"]), fk) ++check(rc == 0, "both-present idempotent run rc==0") ++check(out.getvalue().count("EXISTS") == 2, "both-present run reports both as EXISTS") ++check(len(fk.creates) == 0, "both-present run creates nothing new") ++ ++# ----------------------------------------------------------------------------- ++# 5. Partial-present: transit already there, rack IP absent -> creates ONLY the rack IP. ++# ----------------------------------------------------------------------------- ++fk = full_fake(prefixes_extra=[{"prefix": TRANSIT, "id": 30}]) ++with captured_stdout(): ++ rc = run_main(base_args(["--commit"]), fk) ++check(rc == 0 and len(fk.creates) == 1 and fk.creates[0][0] == "ipam/ip-addresses", ++ "transit-present run creates ONLY the missing rack IP", ++ str([p for p, _ in fk.creates])) ++ ++# 5b. Partial-present the other way: rack IP present, transit absent -> creates ONLY transit. ++fk = full_fake(ip_extra=[{"address": RACK_ADDR, "id": 31}]) ++with captured_stdout(): ++ rc = run_main(base_args(["--commit"]), fk) ++check(rc == 0 and len(fk.creates) == 1 and fk.creates[0][0] == "ipam/prefixes", ++ "rack-IP-present run creates ONLY the missing transit prefix", ++ str([p for p, _ in fk.creates])) ++ ++# ----------------------------------------------------------------------------- ++# 6. HALF-WRITE GUARD -- a bad rack IP (the .1 gateway) aborts with ZERO creates, ++# even though the transit prefix itself is perfectly valid. The transit must NOT ++# be written before the rack-IP check fails. ++# ----------------------------------------------------------------------------- ++fk = full_fake() ++run_dies("half-write: rack IP == .1 gateway is REJECTED before any write", ++ ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], fk) ++check(len(fk.creates) == 0, "half-write: the rejected run wrote NOTHING (transit not created)") ++ ++# ----------------------------------------------------------------------------- ++# 7. Missing infra = die (each precondition), each writing nothing. ++# ----------------------------------------------------------------------------- ++fk = fake_netbox.FakeNB(roles=[], prefixes=[CONTAINER], sites=[SITE_DC0]) ++run_dies("missing transit role is REJECTED", base_args(["--commit"]), fk) ++check(len(fk.creates) == 0, "missing-role run wrote nothing") ++ ++fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[], sites=[SITE_DC0]) ++run_dies("missing Cloud container 10.12.0.0/16 is REJECTED", base_args(["--commit"]), fk) ++check(len(fk.creates) == 0, "missing-container run wrote nothing") ++ ++fk = fake_netbox.FakeNB(roles=[ROLE], prefixes=[CONTAINER], sites=[]) ++run_dies("missing vr1-dc0 site is REJECTED", base_args(["--commit"]), fk) ++check(len(fk.creates) == 0, "missing-site run wrote nothing") ++ ++# ----------------------------------------------------------------------------- ++# 8. Transit CIDR shape / placement rejects. ++# ----------------------------------------------------------------------------- ++run_dies("a transit OUTSIDE the Cloud container is REJECTED", ++ ["--transit-cidr", "192.168.0.0/30", "--rack-ip", RACK, "--commit"], full_fake()) ++run_dies("a /29 transit is REJECTED (not point-to-point)", ++ ["--transit-cidr", "10.12.60.0/29", "--rack-ip", RACK, "--commit"], full_fake()) ++run_dies("a /32 transit is REJECTED", ++ ["--transit-cidr", "10.12.60.1/32", "--rack-ip", RACK, "--commit"], full_fake()) ++run_dies("a host-bits-set transit (10.12.60.1/30) is REJECTED", ++ ["--transit-cidr", "10.12.60.1/30", "--rack-ip", RACK, "--commit"], full_fake()) ++run_dies("a non-CIDR transit is REJECTED", ++ ["--transit-cidr", "not-a-cidr", "--rack-ip", RACK, "--commit"], full_fake()) ++# each of the above must have written nothing -- prove it once with a fresh fake ++_fk = full_fake() ++run_dies("bad transit writes nothing (proof)", ++ ["--transit-cidr", "10.12.60.0/29", "--rack-ip", RACK, "--commit"], _fk) ++check(len(_fk.creates) == 0, "the rejected bad-transit run wrote NOTHING") ++ ++# ----------------------------------------------------------------------------- ++# 9. Rack-IP band rejects. ++# ----------------------------------------------------------------------------- ++run_dies("rack IP OUTSIDE metal-admin 10.12.8.0/22 is REJECTED", ++ ["--transit-cidr", TRANSIT, "--rack-ip", "10.13.0.5", "--commit"], full_fake()) ++run_dies("rack IP == .1 gateway is REJECTED", ++ ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.1", "--commit"], full_fake()) ++run_dies("rack IP == .0 network address is REJECTED", ++ ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.0", "--commit"], full_fake()) ++run_dies("rack IP in-/22-but-above-static-band (.50) is REJECTED", ++ ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.50", "--commit"], full_fake()) ++run_dies("rack IP in-/22-but-in-a-higher-/24 (10.12.9.5) is REJECTED", ++ ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.9.5", "--commit"], full_fake()) ++run_dies("rack IP with a non-/22 mask is REJECTED", ++ ["--transit-cidr", TRANSIT, "--rack-ip", "10.12.8.5/24", "--commit"], full_fake()) ++# accepted: the boundaries of the static band (.2 low, .49 high) both write. ++for edge_ip in ("10.12.8.2", "10.12.8.49"): ++ fk = full_fake() ++ with captured_stdout(): ++ rc = run_main(["--transit-cidr", TRANSIT, "--rack-ip", edge_ip, "--commit"], fk) ++ check(rc == 0 and len(fk.creates) == 2, f"rack IP band edge {edge_ip} is ACCEPTED", ++ str((rc, len(fk.creates)))) ++ ++# ----------------------------------------------------------------------------- ++# 10. NO INVENTED LITERAL -- missing input fails loud (does not guess a value). ++# ----------------------------------------------------------------------------- ++run_dies("missing --transit-cidr fails loud", ["--rack-ip", RACK], full_fake()) ++run_dies("missing --rack-ip fails loud", ["--transit-cidr", TRANSIT], full_fake()) ++ ++# env fallback works (args-or-env): both via env, no flags. ++os.environ["TRANSIT_CIDR"] = TRANSIT ++os.environ["RACK_IP"] = RACK ++fk = full_fake() ++with captured_stdout(): ++ rc = run_main(["--commit"], fk) ++check(rc == 0 and len(fk.creates) == 2, "env TRANSIT_CIDR/RACK_IP are honored (args-or-env)", ++ str((rc, len(fk.creates)))) ++del os.environ["TRANSIT_CIDR"] ++del os.environ["RACK_IP"] ++ ++# ----------------------------------------------------------------------------- ++# 11. Missing NETBOX_URL/TOKEN fails loud (does not guess a target). ++# ----------------------------------------------------------------------------- ++_u = os.environ.pop("NETBOX_URL", None) ++run_dies("missing NETBOX_URL fails loud", base_args(), full_fake()) ++os.environ["NETBOX_URL"] = _u ++ ++# ----------------------------------------------------------------------------- ++# 12. UPSTREAM-WRITE guard: --commit to a non-sandbox host refuses without the flag, ++# proceeds with it. Sandbox needs no flag (covered above). ++# ----------------------------------------------------------------------------- ++os.environ["NETBOX_URL"] = "https://netbox.baldurkeep.com" # the production apex ++fk = full_fake() ++run_dies("upstream --commit WITHOUT --yes-write-upstream is REFUSED", base_args(["--commit"]), fk) ++check(len(fk.creates) == 0, "the refused upstream --commit wrote NOTHING") ++ ++fk = full_fake() ++with captured_stdout(): ++ rc = run_main(base_args(["--commit", "--yes-write-upstream"]), fk) ++check(rc == 0 and len(fk.creates) == 2, ++ "upstream --commit --yes-write-upstream IS allowed and writes both objects", ++ str((rc, len(fk.creates)))) ++# a DRY upstream run needs no flag and writes nothing (the guard is a --commit gate). ++fk = full_fake() ++with captured_stdout(): ++ rc = run_main(base_args(), fk) ++check(rc == 0 and len(fk.creates) == 0, "a DRY upstream run is fine and writes nothing") ++os.environ["NETBOX_URL"] = "http://10.10.1.10:8000" ++ ++# ----------------------------------------------------------------------------- ++# 13. Structural pins (a changed constant a behavioral test alone would miss). ++# ----------------------------------------------------------------------------- ++check(T.ROLE_SLUG == "transit", "ROLE_SLUG is 'transit'") ++check(T.CONTAINER == "10.12.0.0/16", "CONTAINER is 10.12.0.0/16 (Cloud)") ++check(T.SITE_SLUG == "vr1-dc0", "SITE_SLUG is vr1-dc0 (transit site scope)") ++check(T.METAL_ADMIN == "10.12.8.0/22", "METAL_ADMIN is 10.12.8.0/22") ++check(T.STATIC_BAND_LOW == 2 and T.STATIC_BAND_HIGH == 49, ++ "static band is .2-.49 (D-120)") ++check(T.STATUS == "active", "STATUS is active") ++check(T.RACK_DNS == "vvr1-dc0", "RACK_DNS is vvr1-dc0") ++check(T.SANDBOX_HOSTS == {"localhost", "127.0.0.1", "10.10.1.10"}, ++ "SANDBOX_HOSTS matches the sibling importers") ++check("get_nb" in dir(T) and callable(T.get_nb), "get_nb() injection seam exists") ++_src = inspect.getsource(T) ++check('User-Agent"' in _src and "curl/8.5.0" in _src, "UA-aware (WAF-safe User-Agent set)") ++ ++print() ++if F == 0: ++ print(f"test_logic: ALL PASS ({P} checks)") ++ sys.exit(0) ++print(f"test_logic: {F} FAIL of {P + F}") ++sys.exit(1) +diff --git a/tests/site-headend-install/run-tests.sh b/tests/site-headend-install/run-tests.sh +index 4aa5c18..c954a6a 100644 +--- a/tests/site-headend-install/run-tests.sh ++++ b/tests/site-headend-install/run-tests.sh +@@ -68,6 +68,36 @@ grep -q 'is ON for a subnet that is NOT the compose network' "$S" \ + # has no egress, and the first apt install on it hangs. Cost a redeploy on 2026-07-13. + grep -q 'subnet update .*gateway_ip' "$S" && ok || bad "trap 5 LOST: gateway_ip is never set on the compose subnet -- deployed machines will have NO default route" + ++# 6. RACK ROLE (D-123: ONE region on Office1 + a rack controller per DC). --role rack enrolls ++# to the EXISTING region and must SKIP every region+rack concern (DB/LXD/compose). These ++# tests are the point of the rack mode: a regression that quietly ran a region step, or that ++# leaked the enrollment secret, would not surface until a live DC build. ++RURL="http://10.10.0.20:5240/MAAS" ++t "bad --role -> 2" 2 bash "$S" --role bogus --dry-run ++# rack REQUIRES --region-url + --enroll-secret-file (missing -> 2), mirroring --compose-cidr. ++t "rack missing --region-url -> 2" 2 bash "$S" --role rack --dry-run --enroll-secret-file /nonexistent ++t "rack missing --enroll-secret-file -> 2" 2 bash "$S" --role rack --dry-run --region-url "$RURL" ++t "rack bad --region-url -> 2" 2 bash "$S" --role rack --dry-run --region-url ftp://x --enroll-secret-file /nonexistent ++# --compose-cidr is NOT required in rack mode (LXD-compose is a region concern). ++SECFILE="$(mktemp)"; SENTINEL="RACK_ENROLL_SENTINEL_9x7"; printf '%s\n' "$SENTINEL" > "$SECFILE" ++trap 'rm -f "$SECFILE"' EXIT ++t "rack --dry-run (no --compose-cidr) -> 0" 0 bash "$S" --role rack --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" ++ ++rackout="$(bash "$S" --role rack --dry-run --region-url "$RURL" --enroll-secret-file "$SECFILE" 2>&1)" ++printf '%s' "$rackout" | grep -q 'maas init rack' && ok || bad "rack --dry-run must print the 'maas init rack' step" ++# The enrollment secret must NEVER appear in output (read from the file at run time, never echoed). ++printf '%s' "$rackout" | grep -q "$SENTINEL" && bad "rack --dry-run LEAKED the enrollment secret into output" || ok ++# rack mode must NOT run any region+rack / LXD / compose step. ++for m in 'lxdbr0' 'region+rack' 'compose' 'vm-host'; do ++ printf '%s' "$rackout" | grep -qi "$m" && bad "rack --dry-run printed a region+rack-only step: '$m'" || ok ++done ++ ++# 7. BACKWARD COMPAT: with no --role, the default is region+rack and --compose-cidr is still ++# REQUIRED (a rack-mode regression must not relax the legacy contract). ++t "default role still needs --compose-cidr -> 2" 2 bash "$S" --dry-run ++defout="$(bash "$S" --dry-run --compose-cidr 10.10.1.0/24 2>&1)" ++printf '%s' "$defout" | grep -q 'region+rack' && ok || bad "default (no --role) must still run the region+rack flow" ++ + echo + total=$((pass+fail)) + if [ "$fail" -eq 0 ]; then echo "site-headend-install: $pass/$total PASS"; exit 0; fi diff --git a/opentofu/main.tf b/opentofu/main.tf index 48eb444..36a72de 100644 --- a/opentofu/main.tf +++ b/opentofu/main.tf @@ -314,9 +314,10 @@ wan_network_name = module.vr1_dc0_wan.network_name } -# D-121 Option C node layout: 3 control + 2 compute + 3 storage (16/12/8 vCPU, -# 64/48/24 GiB, 150/100/550 GiB). Six NICs each (one per plane, baremetal-matched -# D-122); metal-admin FIRST = the PXE/boot plane (D-052 default binding). +# D-121 Option C node layout (R-3 AMENDMENT 2026-07-16): 3 control + 2 compute + 4 storage +# (16/12/8 vCPU, 64/48/24 GiB, 150/100/550 GiB) = 9 nodes/DC. Storage 3->4 restores Ceph +# rebuild headroom (R3-F08). Six NICs each (one per plane, baremetal-matched D-122); +# metal-admin FIRST = the PXE/boot plane (D-052 default binding). locals { vr1_dc0_node_nics = [ module.vr1_dc0_planes.network_names["metal-admin"], # PXE / boot fabric first @@ -336,6 +337,7 @@ "vr1-dc0-storage-01" = { vcpu = 8, mem = 24576, disk_gib = 550 } "vr1-dc0-storage-02" = { vcpu = 8, mem = 24576, disk_gib = 550 } "vr1-dc0-storage-03" = { vcpu = 8, mem = 24576, disk_gib = 550 } + "vr1-dc0-storage-04" = { vcpu = 8, mem = 24576, disk_gib = 550 } # R-3 (2026-07-16): 4th OSD host } } diff --git a/scripts/dc-dc-whole-host-budget.py b/scripts/dc-dc-whole-host-budget.py new file mode 100755 index 0000000..a39df89 --- /dev/null +++ b/scripts/dc-dc-whole-host-budget.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +"""Whole-host capacity budget for the VR1 nested deployment (D-121 / R-3 / Model B). + +Replaces the lost, never-committed `scratchpad/optc-calc.py` (review R3-F06) with a +committed, harnessed, read-only calculator -- the sibling of `dc-dc-ceph-disk-budget.sh`. +It SUMS the RAM / vCPU / disk allocation of every VR1 layer (both DCs' node fleets, the +site containment VMs, Office1, the edges) and compares it to the MEASURED host budget, +reporting the binding resource and a FIT / NO-FIT verdict. + +Mutates nothing. Every number is an operator-supplied or ruled input -- no live host is +queried and nothing is inferred. Defaults encode the ruled layout (Option C = 3 control + +2 compute + 4 storage/DC, D-121 R-3) under D-123 **Model B** (the DC node fleet lives INSIDE +the site containment VM), but every value is overridable so the tool outlives the ruling. + +RAM is the binding resource in a nested sim (vCPU overcommits; disk is thin-provisioned), +so RAM fit is the gate; vCPU/disk are reported as ratios, not hard ceilings. + +Usage: + python3 dc-dc-whole-host-budget.py [--model A|B] [--dcs N] + [--control N,VCPU,MEM_GIB,DISK_GIB] [--compute ...] [--storage ...] + [--containment-overhead-mem-gib G] [--containment-overhead-vcpu V] + [--office1 VCPU,MEM_GIB,DISK_GIB] [--edge-count N] [--edge MEM_GIB,VCPU] + [--host VCPU,MEM_GIB,DISK_GIB] + +Exit: 0 = FIT (RAM within budget), 1 = NO-FIT (RAM over budget), 2 = usage error. +""" +import argparse +import sys + + +def triple(s, n=3): + parts = s.split(",") + if len(parts) != n: + raise argparse.ArgumentTypeError(f"expected {n} comma-separated numbers, got '{s}'") + return [float(p) for p in parts] + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + # Node roles per DC: count, vCPU, mem GiB, disk GiB (ruled Option C / R-3 defaults) + ap.add_argument("--control", type=lambda s: triple(s, 4), default=[3, 16, 64, 150], + help="control nodes/DC: count,vcpu,mem_gib,disk_gib (default 3,16,64,150)") + ap.add_argument("--compute", type=lambda s: triple(s, 4), default=[2, 12, 48, 100], + help="compute nodes/DC: count,vcpu,mem_gib,disk_gib (default 2,12,48,100)") + ap.add_argument("--storage", type=lambda s: triple(s, 4), default=[4, 8, 24, 550], + help="storage nodes/DC: count,vcpu,mem_gib,disk_gib (default 4,8,24,550 -- R-3)") + ap.add_argument("--dcs", type=int, default=2, help="number of DCs (default 2)") + ap.add_argument("--model", choices=["A", "B"], default="B", + help="D-123 model: A=nodes vcloud-level (+ small rack); B=nodes inside containment VM (default B)") + # Containment VM overhead (Model B: the VM's own OS + MAAS rack + LXD, ON TOP of the node RAM it holds). + ap.add_argument("--containment-overhead-mem-gib", type=float, default=16.0, + help="per-DC containment-VM overhead RAM GiB, Model B (default 16)") + ap.add_argument("--containment-overhead-vcpu", type=float, default=4.0, + help="per-DC containment-VM overhead vCPU, Model B (default 4)") + # Model A rack headend (small, holds no nodes) -- D-124 original 4/8/80. + ap.add_argument("--rack", type=lambda s: triple(s, 3), default=[4, 8, 80], + help="Model A rack headend/DC: vcpu,mem_gib,disk_gib (default 4,8,80)") + # Office1 (voffice1) + edges. + ap.add_argument("--office1", type=lambda s: triple(s, 3), default=[16, 32, 600], + help="Office1 voffice1: vcpu,mem_gib,disk_gib (default 16,32,600)") + ap.add_argument("--edge-count", type=int, default=3, help="OPNsense edges total (office1 + per DC; default 3)") + ap.add_argument("--edge", type=lambda s: triple(s, 2), default=[2, 2], + help="per-edge: mem_gib,vcpu (default 2,2)") + # Measured host budget. + ap.add_argument("--host", type=lambda s: triple(s, 3), default=[256, 1024, 10240], + help="MEASURED host budget: vcpu,mem_gib,disk_gib (default 256,1024,10240)") + args = ap.parse_args() + + def role_sum(role): + cnt, vcpu, mem, disk = role + return cnt * vcpu, cnt * mem, cnt * disk + + # Per-DC node fleet. + dc_vcpu = dc_mem = dc_disk = 0.0 + for role in (args.control, args.compute, args.storage): + v, m, d = role_sum(role) + dc_vcpu += v; dc_mem += m; dc_disk += d + + lines = [] + lines.append(f"Per-DC node fleet: {dc_vcpu:.0f} vCPU / {dc_mem:.0f} GiB / {dc_disk:.0f} GiB") + + tot_vcpu = tot_mem = tot_disk = 0.0 + # Both DCs' nodes. + tot_vcpu += args.dcs * dc_vcpu + tot_mem += args.dcs * dc_mem + tot_disk += args.dcs * dc_disk + + if args.model == "B": + # Nodes live INSIDE the containment VM; add per-DC containment overhead on top. + tot_vcpu += args.dcs * args.containment_overhead_vcpu + tot_mem += args.dcs * args.containment_overhead_mem_gib + per_dc_vm = dc_mem + args.containment_overhead_mem_gib + lines.append(f"Model B: each containment VM (vvr1-dcN) holds one DC's fleet -> ~{per_dc_vm:.0f} GiB RAM each") + lines.append(f" + containment overhead {args.dcs} x ({args.containment_overhead_vcpu:.0f} vCPU / " + f"{args.containment_overhead_mem_gib:.0f} GiB)") + else: + # Model A: nodes at vcloud level, plus a small rack headend per DC. + rv, rm, rd = args.rack + tot_vcpu += args.dcs * rv; tot_mem += args.dcs * rm; tot_disk += args.dcs * rd + lines.append(f"Model A: nodes vcloud-level + a {rv:.0f}/{rm:.0f}/{rd:.0f} rack headend x{args.dcs}") + + # Office1 + edges (both models). + ov, om, od = args.office1 + tot_vcpu += ov; tot_mem += om; tot_disk += od + em, ev = args.edge + tot_vcpu += args.edge_count * ev + tot_mem += args.edge_count * em + lines.append(f"Office1 voffice1: {ov:.0f}/{om:.0f}/{od:.0f}; edges: {args.edge_count} x ({ev:.0f} vCPU / {em:.0f} GiB)") + + hv, hm, hd = args.host + print("=== dc-dc-whole-host-budget: VR1 nested capacity (D-121 / R-3 / Model {}) ===".format(args.model)) + for ln in lines: + print(" " + ln) + print() + print(f" TOTAL allocation : {tot_vcpu:.0f} vCPU / {tot_mem:.0f} GiB / {tot_disk:.0f} GiB") + print(f" MEASURED host : {hv:.0f} vCPU / {hm:.0f} GiB / {hd:.0f} GiB") + print(f" RAM : {tot_mem:.0f}/{hm:.0f} GiB = {100*tot_mem/hm:.0f}% <- BINDING resource") + print(f" vCPU : {tot_vcpu:.0f}/{hv:.0f} = {100*tot_vcpu/hv:.0f}% (overcommits in a sim; not a hard ceiling)") + print(f" disk : {tot_disk:.0f}/{hd:.0f} GiB = {100*tot_disk/hd:.0f}% (thin-provisioned)") + print() + + ram_fit = tot_mem <= hm + if ram_fit: + print(f"VERDICT: FIT -- RAM within budget ({hm - tot_mem:.0f} GiB headroom).") + print(f"WHOLE-HOST: model={args.model} ram={tot_mem:.0f}/{hm:.0f}GiB verdict=FIT " + f"headroom={hm - tot_mem:.0f}GiB vcpu={tot_vcpu:.0f}/{hv:.0f} disk={tot_disk:.0f}/{hd:.0f}GiB") + return 0 + else: + print(f"VERDICT: NO-FIT -- RAM OVER budget by {tot_mem - hm:.0f} GiB. Reduce per-node RAM or node count.") + print(f"WHOLE-HOST: model={args.model} ram={tot_mem:.0f}/{hm:.0f}GiB verdict=NO-FIT over={tot_mem - hm:.0f}GiB") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/dc-dc-whole-host-budget/run-tests.sh b/tests/dc-dc-whole-host-budget/run-tests.sh new file mode 100755 index 0000000..9dfe7fa --- /dev/null +++ b/tests/dc-dc-whole-host-budget/run-tests.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# Harness for scripts/dc-dc-whole-host-budget.py (R3-F06: the committed replacement +# for the lost scratchpad/optc-calc.py). Read-only arithmetic; asserts the ruled +# Model B / R-3 shape FITS, the model math, and the failure/usage paths. +set -u +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$HERE/../../scripts/dc-dc-whole-host-budget.py" +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); } +bad() { FAIL=$((FAIL+1)); echo " FAIL: $1"; } + +# assert_contains <desc> <expected-substring> <command...> +assert_contains() { + local desc="$1" want="$2"; shift 2 + local out; out="$("$@" 2>&1)" + if grep -qF -- "$want" <<<"$out"; then ok; else bad "$desc (want '$want')"; fi +} +# assert_exit <desc> <expected-code> <command...> +assert_exit() { + local desc="$1" want="$2"; shift 2 + "$@" >/dev/null 2>&1; local got=$? + if [ "$got" -eq "$want" ]; then ok; else bad "$desc (want exit $want, got $got)"; fi +} + +# 1. Ruled shape: Model B, defaults (3+2+4) -> FIT at 838 GiB, exit 0. +assert_contains "model B ram=838" "ram=838/1024GiB verdict=FIT" python3 "$SCRIPT" --model B +assert_contains "model B headroom" "headroom=186GiB" python3 "$SCRIPT" --model B +assert_exit "model B fits (0)" 0 python3 "$SCRIPT" --model B +assert_contains "model B binds RAM" "BINDING resource" python3 "$SCRIPT" --model B + +# 2. Model A (fallback) -> FIT at 822 GiB (16 GiB less: no containment overhead, has small rack). +assert_contains "model A ram=822" "ram=822/1024GiB verdict=FIT" python3 "$SCRIPT" --model A +assert_exit "model A fits (0)" 0 python3 "$SCRIPT" --model A + +# 3. Model math: B adds containment overhead on top of A's node RAM; B RAM > A RAM by 2*16 - (2*rackmem). +# (B: +2*16 overhead; A: +2*8 rack) => B - A = 32 - 16 = 16 GiB. Assert B==838, A==822. +assert_contains "B minus A = 16 GiB" "ram=838/1024GiB" python3 "$SCRIPT" --model B + +# 4. Old 3-storage shape (pre-R-3) is LIGHTER: 3 storage/DC -> 838 - 2*24 = 790 GiB (matches D-121's original). +assert_contains "3-storage = 790" "ram=790/1024GiB verdict=FIT" python3 "$SCRIPT" --model B --storage 3,8,24,550 + +# 5. NO-FIT path: oversized control nodes push RAM over 1024 -> exit 1, NO-FIT. +assert_exit "oversize -> exit 1" 1 python3 "$SCRIPT" --model B --control 3,16,160,150 +assert_contains "oversize NO-FIT" "verdict=NO-FIT" python3 "$SCRIPT" --model B --control 3,16,160,150 + +# 6. Usage error: malformed triple -> exit 2. +assert_exit "bad arg -> exit 2" 2 python3 "$SCRIPT" --host 256,1024 + +# 7. Determinism: two identical runs produce identical output. +A_OUT="$(python3 "$SCRIPT" --model B 2>&1)"; B_OUT="$(python3 "$SCRIPT" --model B 2>&1)" +if [ "$A_OUT" = "$B_OUT" ]; then ok; else bad "non-deterministic output"; fi + +# 8. Single-DC scaling: --dcs 1 halves the node RAM contribution. +assert_contains "dcs=1 lighter" "verdict=FIT" python3 "$SCRIPT" --model B --dcs 1 + +TOTAL=$((PASS+FAIL)) +echo "dc-dc-whole-host-budget: $PASS/$TOTAL PASS" +[ "$FAIL" -eq 0 ]