# DC-DC (VR1) deployment workflow overview

**Purpose.** A single followable checklist across the whole VR1 (two-DC virtual
regional) buildout, spanning many sessions. `docs/dc-dc-buildout-design.md` is the
WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT
sequence, cross-referenced to that plan, to D-100..D-110, and to what already
exists in this repo vs what still needs to be authored. STATUS AUTHORITY: stage and
gate status live ONLY in `docs/CURRENT-STATE.md` (GA-R1); this doc keeps stage
IDENTITY, links, and dated history -- it is no longer a status tracker.

**Governing constraint (unchanged throughout):** minimize delta to Roosevelt. Reuse
VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog
steps are the Section 9 shim register in the buildout design (OpenTofu node-VM
creation, `tc netem` WAN simulation, single-unit controllers/rbd-mirror).

---

## Stage 0 -- Decision ratification (CLEARED 2026-07-09)

**RATIFIED.** D-100..D-110 are ALL ADOPTED (`docs/design-decisions.md`;
D-102 MERGED into D-101 per the operator's own ruling, not deleted --
retained for numbering/history; D-109 confirmed standalone). All 6
redline items ruled (`docs/dc-dc-buildout-design.md` Section 10 is now a
ruling record, not an open list): Ceph size=3-by-default (size=2 only as an
explicit Phase-0 fallback); netem same-metro lean; D-102/D-101 merge +
D-109 standalone; metal-admin GAINS a ULA leg (a real reversal of the
stated lean, not a rubber-stamp); COS per-DC-only; mirror sync independent
per-DC. Verified via `bash scripts/ledger-scan.sh`: none of D-100..D-110
appear under "PROPOSED / OPEN decisions." Full ruling record:
`docs/archive/changelogs/changelog-20260709-stage0-ratification.md`.

**State:** cleared 2026-07-09 (history; status authority: docs/CURRENT-STATE.md).

---

## Stage 1 -- Phase 0: vcloud host preparation

| | |
|---|---|
| **Goal** | Turn the bare vcloud host into a substrate Office1 can deploy from. |
| **Build** | Nested KVM enabled; libvirt storage pools + thin-provisioning; dark-fiber + per-site ISP-edge virtual networks; the six per-DC plane segments as virtual networks; MTU plan set on the underlay. |
| **Gate** | Nested KVM verified; disk budget computed + Ceph size decision recorded; measured MTU recorded; virtual networks present and isolated as designed. |
| **Owns** | D-100 (fabric), D-101 (MTU -- folds in former D-102), buildout-design Section 3 (host/disk/MTU gates). |
| **Reuse vs new** | NEW. No VR0-DC0 analog -- that testcloud IS the single vcloud host, already prepared; this is preparing the host to carry TWO independent DC substrates plus Office1. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase0-vcloud-prep.md`** -- command-level steps (host identify -> measure CPU/RAM/disk/nested-KVM/MTU -> enable nested KVM -> prepare pool paths -> install/confirm OpenTofu -> tfvars -> init/validate/plan/apply -> post-apply gate verify), each MUTATION individually gated per this repo's discipline. Flags a real open structural question up front (the `provider "maas"` block in `main.tf` is unconditional even though this stage only touches libvirt resources -- may force `maas_api_url`/`maas_api_key` to be set at `plan` time with nothing yet using them; noted as a possible DOCFIX candidate, not silently worked around). NOT YET EXECUTED against real infrastructure -- this is the first runbook that will be, tomorrow morning, per the operator's own stated plan. `opentofu/` SCAFFOLD started 2026-07-09: `modules/dc-planes` (six per-DC planes) + `modules/mesh-link` (dark-fiber legs) + `modules/dc-storage-pool` wired for DC1 + Office1 -- see `opentofu/README.md` for what's built vs deliberately deferred (node-VM/domain resources, DC2 CIDRs, netem). UNVALIDATED: no `tofu` binary available to run `scripts/opentofu-validate.sh` yet -- this runbook's Step 8 is the first real run. |

**State:** executed 2026-07-10 (first DC-DC runbook run against real
infrastructure; vcloud host -- tofu pin authority: `docs/CURRENT-STATE.md` section 7). As-built: nested KVM already-on;
`underlay_mtu=9000` (jumbo, operator ruling -- the isolated planes/mesh are
host-internal virtio bridges, jumbo-capable regardless of the 1500 ISP uplink);
Ceph size=3 disk-budget plausibility PASS (4.60 TiB margin); 13 libvirt objects
applied (6 dc1 planes + dc1/office1/dc2 pools + office1-local + 3 mesh legs, all
MTU 9000). Prior VR0 topology torn down (`wan` kept as a gap-#17 model). DC2
PLANES deferred (Option B -- awaits NetBox supernet); DC2 storage pool wired.
Delivered DOCFIX-178 (prereq installers), DOCFIX-179 (scaffold: per-module
required_providers + fmt + maas-provider deferral). See those changelogs.

---

## Stage 2 -- Phase 1: Office1 SITE standup (RESHAPED 2026-07-13 by D-114)

> **!!! MODEL CHANGE (D-114, ADOPTED 2026-07-13). The three-sibling-service-VM model is
> SUPERSEDED.** Stage 2 no longer builds three peer VMs (MAAS-region, NetBox, GitBucket) on the
> vcloud host with Tailscale on the headend host. It builds **ONE Office1 site containment VM,
> `voffice1`** (Ubuntu 24.04, OpenTofu-created), which IS the facility: **MAAS-region runs ON it,
> LXD (pinned to the 5.21 LTS track) runs ON it**, that LXD is **registered BACK INTO MAAS as an
> LXD KVM host**, and the non-stack service machines (**NetBox, Tailscale** -- GitBucket removed per D-116) are VMs
> **COMPOSED BY MAAS** into it -- MAAS-visible: enlisted, commissioned, deployed, powered. This
> is VR0's own already-proven `lxd` + `tailscale` pattern, applied per site.
>
> **Explicitly NOT affected:** the Juju-deployed OpenStack services on the DC nodes. They stay in
> Juju-created LXD **CONTAINERS** (bundle `lxd:N` placements), invisible to MAAS, exactly as in
> VR0. D-114 grants MAAS a composition right scoped ONLY to the LXD VM host and the non-stack
> machines; `modules/node-vm` still pre-creates the OpenStack node VMs and MAAS merely DISCOVERS
> them (D-103 unchanged on that point).
>
> **Sequencing is an operator ruling:** Office1 FIRST, fully up on this model. **DC1 is GATED
> behind it** and is not started until Office1 is complete; DC2 follows DC1. Stages 3-7 below are
> therefore gated behind THIS stage, and their own content is otherwise unchanged by D-114.

| | |
|---|---|
| **Goal** | Stand up the Office1 SITE first, before any DC -- and prove the D-114 containment-VM model at its cheapest point. |
| **Build** | `voffice1` site containment VM (OpenTofu, `modules/cloudinit-vm` + `modules/base-image`, on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`); MAAS region controller ON `voffice1`; LXD ON `voffice1` (5.21 LTS track); that LXD registered into MAAS as an LXD KVM host; NetBox + Tailscale as MAAS-COMPOSED LXD VMs inside it (**GitBucket REMOVED from scope -- D-116**); Office1's own OPNsense simulated-ISP edge (gap #16, Stage 2 owns it). |
| **Gate** | Office1 OPNsense edge up (**MET** -- routing, NAT, egress, Kea DHCP; gap #17 closed for Office1 via `office1-wan`); `office1-local` network created (gap #12 CLOSED); OpenTofu reaches vcloud host libvirt (**MET**); **`voffice1` exists and runs (MET 2026-07-13, measured)**; `voffice1` has a reserved address + SSH + nested KVM exposed; MAAS region reachable ON `voffice1`; LXD on the 5.21 track with `lxdbr0` OFF `office1-local`; LXD registered as a MAAS KVM host; **the non-stack machines COMPOSED BY MAAS and deployed -- i.e. an LXD VM actually BOOTS at L3, which is D-114's DC1 entry gate**; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); Tailscale confirmed to Office1 only. **GitBucket is NO LONGER A GATE (D-116): no Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record.** |
| **Owns** | **D-114** (site containment VM + MAAS-composed LXD VMs, incl. the LXD 5.21 pin), D-103 as AMENDED by D-114 (OpenTofu owns the containment VM and the site networks; MAAS composes INSIDE the site), D-107 (headend is not a core-service provider). |
| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but LESS new than it looks: the MAAS-composed-LXD-VM half is **exactly what VR0 runs today** (its `lxd` KVM host at LXD 5.21.4, with `tailscale` composed into it at 2 cores / 2 GiB / 25 GB) and carries **NO Roosevelt delta**. The containment VM itself IS a deliberate simulation-only delta -- it is what makes a "hard-down an entire facility" DR drill (`virsh destroy vvr1-dc0`) executable at all. NetBox-as-IPAM-apex discipline and the `netbox/*-import.py` pattern transfer unchanged. |
| **Authoring status** | **Runbook REWRITTEN 2026-07-13 to the D-114 model: `runbooks/dc-dc-phase1-office1-standup.md`.** The old Option A / Option B provisioning fork is **CLOSED** -- OpenTofu (Option A) is the path and `module "voffice1"` is a real, applied `cloudinit-vm` instantiation (real image source, real `user_data`/`meta_data`/`network_config`, plus the new `expose_nested_virt` variable). **PENDING VERIFICATION:** the MAAS/LXD install + LXD-KVM-host registration + VM-compose command lines are NOT yet confirmed against Canonical's current docs -- the runbook marks each unconfirmed flag/channel/subcommand rather than fabricating one, and a separate verification pass owns filling them in. |

### STAGE 2 WAS THE BRANCH MERGE POINT -- MERGED 2026-07-15 (PR #1, `e468d2e`)

Branch `dc-dc-stage1-phase0-first-exec` merged to `main` at Stage 2 CLOSE, once every close-out
gate bullet (C1..C5) was MET, the gauntlet was ALL GREEN (60) and repo-lint 0-fail. This was the
FIRST VR1 feature merge -- `main` had never taken one (its prior tip was ad-hoc uploads) -- so it
brought D-112..D-120, the Office1 headend, and the NetBox tooling onto trunk. **`main` is now
trunk-of-record for VR1 through Stage 2.** The merged branch was retired.

This established the STANDING **branch-per-stage + merge-at-close** discipline (see "Cross-cutting
discipline" above): every subsequent stage develops on its own branch and merges to `main` (merge
commit, operator-gated) as part of its definition-of-done. Stage 3 branches off the post-merge `main`.

---

**State (history; status authority: docs/CURRENT-STATE.md):** substrate + site both built by 2026-07-14; stage closed and merged to main (PR #1 `e468d2e`, 2026-07-15).

> The 2026-07-13 "not done" list that stood here is **SUPERSEDED and was already stale when
> written** -- MAAS-region, LXD, the LXD-KVM-host registration and both composed service
> machines all landed the same day. This doc's status table has now drifted three times; the
> correction below is measured, not assumed.

- **done (substrate):** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is
  BUILT and LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on
  udp/67). Config is REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt. **`voffice1`
  EXISTS and is running** (`module.voffice1` is in `opentofu/terraform.tfstate`).
- **done (site) -- D-114 IS PROVEN END TO END:** MAAS 3.7.2 region+rack ON `voffice1`; LXD 5.21.5
  ON `voffice1`; that LXD **registered back into MAAS as an LXD VM host**; and both service
  machines **COMPOSED BY MAAS** and deployed -- `office1-netbox` (10.10.1.10, NetBox 4.6.4) and
  `office1-tailscale` (10.10.1.11, subnet router for `10.10.0.0/22`, route now ENABLED on the
  self-hosted control server). GitBucket is out of scope (D-116), so Office1's composed machines
  are exactly TWO. Codified in `scripts/site-headend-install.sh` (harness 18/18) -- reusable for
  the DCs.
- **The L3 nesting proof is NO LONGER an unknown -- it is DEFINITIVE.** A guest booted, PXE'd and
  COMMISSIONED three levels deep. DC1 inherits the proof; D-114's model stands.
- **NetBox is populated in `office1-netbox` -- the VR1 IPAM APEX, a working draft (DOCFIX-195)** (D-115
  office carve + the six-plane roles + both DCs, 90 -> 134 prefixes). The apex literals that blocked it
  are now assigned: **D-117** (site naming) and **D-118** (`ORG_ULA_48 = fd50:840e:74e2::/48`).
  **Operator ruling 2026-07-15 (DOCFIX-195):** during VR1, `office1-netbox` (10.10.1.10) takes ALL
  NetBox writes and HOLDS THE IPAM APEX ROLE -- every consuming system (OpenTofu, MAAS, overlays, the
  Juju bundle) derives its NetBox values from it. It is a *working draft* (edited as the buildout
  proceeds), but it is the authoritative NetBox. `netbox.baldurkeep.com` is the production apex, held
  as a READ-ONLY v1 reference draft and NOT consulted by VR1 systems; the write-back to it is DEFERRED
  to end-of-deployment (may not happen) -- it is NOT a Stage 2 gate. So the "NetBox authoritative + populated" gate is met when `office1-netbox` holds
  the complete validated dataset, NOT by any upstream write. See C2 for what "complete" and "verified"
  require (the D-120 additions, plus a fidelity-check re-run -- the earlier "fidelity check clean"
  verdict predates the 2026-07-14 hardening and must be re-established).

**Stage 2 close-out checklist (what must be true before the merge):**

| # | Item | State |
|---|---|---|
| C1 | Office1 LAN IPv6 deployed (`2602:f3e2:f01:100::1/64` + RA on the edge LAN). WAN v6 leg DEFERRED: v6 does not egress the lab. | **done 2026-07-14** -- live and PROVEN ON THE WIRE: address on the kernel, radvd advertising the prefix (`mode=unmanaged`), `voffice1` autoconfigured `...:5054:ff:fe6a:87e5` by SLAAC, edge->voffice1 GUA ping 0.0% loss. NOTE the D-113 AMENDMENT: the ADDRESS half is **not** REST-API-doable (`scripts/opnsense-set-interface-v6.sh`); only the RA half is. `docs/archive/changelogs/changelog-20260714-office1-lan-ipv6-executed.md` |
| C2 | `office1-netbox` (10.10.1.10) is the VR1 IPAM apex -- a working draft (DOCFIX-195) -- and holds the complete validated dataset: D-115 carve + six-plane roles + both DCs' prefixes/aggregates/RIRs, PLUS the D-120 additions -- `10.10.1.0/24`'s child ranges (`ip-ranges`: static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments (`ip-addresses`: office1-netbox 10.10.1.10, office1-tailscale 10.10.1.11). `netbox.baldurkeep.com` stays a READ-ONLY v1 reference draft; the write-back to it is DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15) -- NOT a Stage 2 gate. | **done 2026-07-15.** (i) The HARDENED fidelity check was re-run against office1-netbox -> exit 0 (faithful); it also surfaced + fixed a checker scope false-positive (`docs/archive/changelogs/changelog-20260715-fidelity-check-scope-correctness.md`). (ii) The tooling gap was CLOSED -- `prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover `ip-ranges`/`ip-addresses` -- and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox via `netbox/d120-compose-bands.py --commit` and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned delta (D-115/D-117/D-118/D-120)." `docs/archive/changelogs/changelog-20260715-d120-load-and-iprange-tooling.md`. Gauntlet ALL GREEN (60). |
| C3 | The `$DC` shell-selector collision resolved. | **done 2026-07-14 (D-119)** -- selectors are REGION-QUALIFIED (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`) across `lib-net.sh`, `lib-hosts.sh`, `opentofu/`, the importer and every runbook call site. Bare `dcN` is REJECTED. The importer's DC->site map is now an IDENTITY, so the off-by-one is structurally impossible. Gap #19 CLOSED. |
| C4 | The NetBox sandbox loop documented. | **done 2026-07-14** -- `docs/dc-dc-netbox-buildout-scope.md` section 8 (rationale + write-guard table + WAF trap) and `runbooks/dc-dc-phase1-office1-standup.md` Step 10b (operational sequence). |
| C5 | This doc's Stage 2 row reconciled against reality (it has drifted three times). | **done 2026-07-14** |

---

## Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

| | |
|---|---|
| **Goal** | From Office1, script each DC's "physical" layer into existence. |
| **Build** | Per DC, D-123 **Model B** (TWO OpenTofu roots + a bootstrap gate between): the OUTER root (vcloud) boots + sizes the containment VM `vvr1-dc0` (~416 GiB, `expose_nested_virt=true`) + the transit + the D-125 vcloud ISP uplink; a gated BOOTSTRAP (`site-headend-install.sh --host-nodes`) makes it a nested libvirt/KVM host AND the MAAS rack; the INNER root (qemu+ssh to `vvr1-dc0`, R-5) creates the 9 node-VM domains (3+2+4, R-3; SHIM, Section 9), the 6 plane networks, and the OPNsense edge INSIDE `vvr1-dc0`. `maas-vm-host` registers `vvr1-dc0`'s OWN (inner) virsh to the Office1 REGION. |
| **Gate** | `vvr1-dc0` up + sized + nested-KVM on (the rack IS `vvr1-dc0`, D-123); inner node VMs boot at depth-4 (D-114/D-123 boot gate); MAAS discovers them; edge routing + the D-125 per-DC ISP egress up (bridge-in isolation test passes); SEC-010 DC-LOCAL enforced; netem parameters applied and measured (D-100 gap #11, still unruled). |
| **Owns** | D-103 (seam, as AMENDED by D-123 Model B), D-100 (edges), D-121 (node layout, R-3 = 9/DC), D-122 (site/edge shape), D-123 (Model B site-down + two-root), D-124 (rack transit), D-125 (per-DC ISP egress). |
| **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** vr1-dc0-first (D-119 naming); vr1-dc1 explicitly hard-gated (HELD, gate G12: transit/rack addressing unruled -- its v4 supernet 10.12.64.0/19 IS assigned by D-115, R3-F11 fix) rather than run with placeholders. Originally walked a set of STOP-gate decisions; most are now RULED (see the rulings summary below): node sizing = D-121/R-3, edge/WAN shape = D-122, site-down/placement = D-123 Model B, rack transit = D-124, ISP egress = D-125, edge-config delivery = D-113(a2) REST (config.xml/ISO retired -- no more `config.xml` tokens). STILL open at execution: MAAS zone/pool + `power_address` (Stage-4 concerns), boot-measured `WAN_IF`/`LAN_IF` (real OPNsense boot), and netem params (D-100 gap #11). NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`; under Model B it targets `vvr1-dc0`'s inner virsh -> Office1 region, DOCFIX-179-deferred) all now exist AND are instantiated across the two Model B roots (D-123): the outer root (`opentofu/`) + the inner root (`opentofu/vr1-dc0-substrate/`), plus the new `modules/wan-bridge` (D-125). Both roots `tofu validate`. Remaining un-instantiated is deploy-time (MAAS zone/pool, `power_address`) not authoring. `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. |

**State:** CLOSED 2026-07-21 for the vr1-dc0 scope (operator-ruled; status authority:
docs/CURRENT-STATE.md sections 1 and 6; stage record
docs/archive/stage-records/vr1-stage3-record.md). vr1-dc1 remains the stage's designed HELD
remainder (gate G12) and returns when its addressing is ruled. Branch merged to `main` and
retired. NOTE the Gate line above predates two recorded corrections: `maas-vm-host`
registration was REFUTED for DCs (per-machine virsh power replaced it -- D-103/D-123
amendments 2026-07-20) and the netem target is the dc0<->dc1 leg (DOCFIX-197).

**Rulings landed (through 2026-07-16; supersedes the earlier "two blocking rulings landed 2026-07-15" note):**
- **D-121 (ADOPTED) + R-3** -- node layout = **Option C** = 3 control + 2 compute + **4 storage = 9
  nodes/DC** (R-3 added the 4th storage node; the earlier "3 storage / 16 VMs" is superseded). Whole-host
  fit re-validated under Model B: **FIT at 870/1024 GiB (154 GiB headroom)** via
  `scripts/dc-dc-whole-host-budget.py`. Vault-HA fork RULED (**R-2**: v-a MySQL-backed; Raft infeasible on
  the 1.8 charm, D-068/BUNDLEFIX-010).
- **D-122 (ADOPTED)** -- Office1-pattern edge (2048/2/nano, REST-API config), dedicated per-site ISP
  uplink, site nested in its own containment VM, per-site MAAS. Its WAN-uplink REALIZATION is superseded by
  **D-125** (bridge-in) and node PLACEMENT by **D-123**; D-122's intent stands.
- **D-123 (ADOPTED Model B)** -- DC nodes nest INSIDE `vvr1-dc0`; a single `virsh destroy vvr1-dc0` =
  site-down. Forces the TWO-ROOT split (a libvirt provider can't target a same-apply VM). Overturns the
  earlier inferred Model A (preserved as `docs/archive/model-a-fallback-plan.md` + git tag `model-a-fallback`).
- **D-124 (ADOPTED)** -- rack transit addressing (office1<->dc0 mesh /30, Scheme A); the entry's rack
  SIZING is VOID under Model B (`vvr1-dc0` now holds the fleet).
- **D-125 (ADOPTED)** -- per-DC ISP egress via bridge-in (resolves OBS-3's design gap; egress efficacy is a
  deploy-time gate).

Steps 4-5 are rewritten to the D-113(a2) REST-API bootstrap; naming is post-D-119. **First-run scope is
now the full outer -> bootstrap -> inner -> maas-register sequence** (the runbook's "D-123 MODEL B RESHAPE"
section). The per-DC MAAS rack source is NO LONGER a missing-module gap: `vvr1-dc0` IS the rack, stood up
by `site-headend-install.sh --host-nodes` (a bootstrap SCRIPT, not a tofu module) between the outer and
inner applies. The stale `vr1-dc1` "supernet not assigned" gate is corrected (D-115 assigned
`10.12.64.0/19`); vr1-dc1 stays out of scope by sequencing.

**Blocking to EXECUTE (all operator-gated):** the HELD NetBox addressing -- **D-124's rack transit/30 +
rack metal-admin IP** (via office1-netbox, apex token; D-125 needs NO new NetBox object -- its /24 is the
ruled `172.30.2.0/24`, OPNsense WAN stays `.2`); then the gated applies; then the deploy-time gates
(depth-4 nova boot D-114/D-123; D-125 foreign-MAC egress isolation
test; SEC-010 packet-drop).

---

## Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

| | |
|---|---|
| **Goal** | Bring the discovered node VMs to a deployed OS, ready for Juju. |
| **Build** | Per DC: MAAS commissions + deploys the node VMs; provider NIC left RAW (br-ex discipline, D-100); VLAN-103 metal-internal bridge stack pre-built by MAAS. |
| **Gate** | Nodes READY + carved + tagged (NOT "deployed" -- MAAS-deploy is SKIPPED per the 2026-07-23 READY-handoff ruling / DOCFIX-200; Juju provisions at Stage 5); six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC artifact source answering on its own address with an attested-current sync (dc0 = D-135 item-1 mirror, dc1 = the D-135-AMENDED apt caching proxy -- the two DCs differ BY RULING). Node-side reachability and the node time source are gate **G17**, split out 2026-07-27 (operator ruling, GA-R5) because powered-off nodes cannot be probed in this stage. `NTP from edge` was STRUCK -- superseded by D-129(iv) 2026-07-21, which keeps the MAAS hierarchy as time authority and gives the edge no NTP role (DOCFIX-204). |
| **Owns** | D-103, D-100 (br-ex), D-107 (mirror + NTP). |
| **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing now RULED (D-121/R-3 = 9/DC), so that sub-gap is closed; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md).

---

## Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

| | |
|---|---|
| **Goal** | A running independent OpenStack cloud per DC. |
| **Build** | Per DC: bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle; Vault init with a per-DC independent root, issuing v4 + IPv6-SAN certs; apply the family matrix (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider). |
| **Gate** | `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC; controller backup taken + stored; Ceph-over-v6 and geneve-over-v6 verified. |
| **Owns** | D-104 (controllers), D-109 (Vault), D-101 (families). |
| **Reuse vs new** | **This is the ground-truth/copy point** (`docs/dc-dc-buildout-design.md` line 12: seeded from `openstack-caracal-ipv4` at 8813efc). `bundle.yaml` and `runbooks/phase-01..08-*.md` are the per-DC template -- run essentially the existing VR0 DC0 deploy loop TWICE (once per DC), adapted for: independent Keystones (no cross-DC federation), the IPv6 family matrix (D-101/D-102), and per-DC Vault roots (D-109) instead of the single shared root. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`.** Does not duplicate `phase-01..08`'s content -- describes what changes to run that loop twice (once per DC): DC1 needs NO VIP/CIDR edits at all (D-101 inherits DC0 unchanged); DC2 is hard-blocked pending its NetBox literals. Requires every adapted `phase-01..08` command block that reads `lib-net.sh`/`lib-hosts.sh` to call `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` first (DOCFIX-151). Flags a real, unresolved design gap: the D-101 IPv6 family-matrix overlay (dual-stack/ULA-only per-plane charm config) does not exist yet -- the exact charm config option names for v6 addressing have not been confirmed against any charm's real schema, so no overlay file is invented; Step 6 names this as blocking work rather than gliding past it. Also flags D-109's IPv6-SAN cert-issuance support as an unverified charm-behavior assumption. NOT YET EXECUTED. |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md). **This is where the existing repo tooling (`preflight.sh`,
`cloud-assert.sh`, `scripts/lib-net.sh`/`lib-hosts.sh`, `run-logged.sh`) does the
most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they
apply to either DC's clone/session without modification.**

---

## Stage 6 -- Phase 5: DR wiring and failover drill

| | |
|---|---|
| **Goal** | Prove Tier-A recoverability, one-way first, then two-way. |
| **Build** | Replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones); rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency groups where needed. Run the failover/failback drill. |
| **Gate** | One-way drill clean (RTO/RPO measured); then two-way; controller restore drill (D-104) exercised; RTO recorded as a measured output, not an SLA. |
| **Owns** | D-108 (replication), D-104 (control-plane backup/restore in the drill). |
| **Reuse vs new** | Ground truth: `docs/archive/dc-dc-replication-DR-seed.md` (the original Tier-A/bidirectional/cinder-backup+rbd-mirror decision) -- D-108 corrected its carrier from the seed's v4 replication plane to IPv6-only ULA. The failover/failback drill skeleton is already drafted (buildout design Section 8): confirm DC1 down -> promote Glance -> restore Cinder -> rebuild instances -> record RTO/RPO; failback is split-brain-safe (no auto-resume, controlled window to flip primary back). |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase5-dr-failover-drill.md`.** Command-level implementation of Section 8's failover/failback skeleton, staged one-way-then-two-way per D-108's own requirement, plus the D-104 controller backup/restore drill. Found and flagged three real bundle gaps while writing it (grepped `bundle.yaml`): no `cinder-backup` charm deployed, no `ceph-rbd-mirror` charm deployed, and `ceph-mon`'s existing `rbd-mirror` endpoint binding is on `storage` instead of `replication` (inherited from the VR0/DC0 seed, predates D-108) -- all three become this runbook's own first gated mutation (Step 2), not silently routed around. Every realm/zone/pool/endpoint name is an explicit `<UPPER_SNAKE_TBD>` placeholder. Includes a negative-control sub-drill (sever only the replication link first, confirm the split-brain check correctly refuses to promote) before the real hard-down drill. NOT YET EXECUTED -- no live Ceph cluster this session (harness for gap #5's scripts is future work, logged, not built). |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md).

---

## Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

| | |
|---|---|
| **Goal** | Complete the service set in both DCs. |
| **Build** | Per DC: Designate (designate-bind) with per-DC zones + A/AAAA per the D-008 bootstrap order; COS observability; Magnum/CAPI with in-cloud single-homed capi-mgmt (`domain-setup` re-run per D-046). |
| **Gate** | Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); COS scraping; a CAPI workload cluster comes up per DC. |
| **Owns** | D-106 (Designate), D-105 (CAPI/COS). |
| **Reuse vs new** | HIGH REUSE for CAPI/Magnum: `runbooks/phase-06-incloud-mgmt-cluster.md` + `phase-07-conductor-graft.md` + `phase-08-workload-cluster-acceptance.md` are direct precedent, per-DC. Designate is a REVERSAL, not new build: VR0 DC0 deferred it (D-019, "DNS scope reduction"); D-106 reactivates it (supersedes D-019, reactivates the original D-008 bootstrap order) -- read D-008 and D-019 together before writing this stage's runbook, the static-`/etc/hosts` workaround they document goes away. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase6-designate-cos-magnum.md`.** Quotes `bundle.yaml`'s and `phase-01-bundle-deploy.md`'s actual current "NO designate (D-019)" text verbatim rather than paraphrasing, then the D-106 reversal against it. Flags that the reversal is bigger than "add designate" -- D-106's bootstrap order also reactivates `os-public-hostname` per API charm, reversing the bundle's separate B5 IP-only posture; proposes (not silently adopts) pushing per-DC hostname literals into new `overlays/dc1-hostnames.yaml`/`dc2-hostnames.yaml` files to keep `bundle.yaml` itself DC-agnostic, mirroring the existing octavia-pki overlay pattern. D-046's domain-setup trap quoted verbatim in a top callout and again inline. COS deployment mechanism explicitly left as an operator decision point (D-105 rules only the scope) rather than invented. Reuses `phase-06/07/08` by filename with concrete DC-parameterization notes (confirmed via tracing which scripts source `lib-net.sh`). NOT YET EXECUTED. |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md).

---

## Cross-cutting discipline (applies at every stage, every session)

- **Session bootstrap, every session:** `git pull` -> `bash scripts/repo-lint.sh`
  (0 fail / 1 documented legacy WARN expected) -> `bash scripts/ledger-scan.sh`
  to reconcile open decisions/next-free numbers against `docs/session-ledger.md`.
- **Any session that will mutate the cloud:** `bash scripts/run-logged.sh <label>`
  first (as-executed log).
- **Operating model (D-128):** Claude runs ON the vcloud jumphost -- do not relocate
  the session to a workstation. **Plane 1** (substrate -- outer `tofu` `qemu:///system`
  + local `virsh`, building the DC *host* VMs) runs on vcloud (no Roosevelt analog);
  **Plane 2** (MAAS / NetBox / the inner `tofu` root `qemu+ssh`-from-Office1 / tenant
  tests) EXECUTES on `voffice1` via `ssh voffice1 '...'`, so runbooks read "from the
  Office1 headend" and transfer to Roosevelt. Durable jumphost->Office1 reach =
  `scripts/site-baseleg.sh` (D-126 amendment); run `check office1` if `ssh voffice1`
  is dead after a reboot. The workstation tailnet (D-107) is the human GUI path only.
- **Before any add-model-like mutation:** `bash scripts/preflight.sh` PASS.
- **After any deploy/restart/change, and as an incident-triage first step:**
  `bash scripts/cloud-assert.sh` (add `--capture` at real deploy-completion
  milestones for the committed BOM).
- **Before touching ANY built surface:** grep `docs/design-decisions.md` for the
  governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present
  options, do not implement.
- **Every deliverable (script, runbook, doc):** `repo-lint` clean, the touched
  script's harness green (write one if missing), a changelog entry with a
  per-item revert, and a `docs/session-ledger.md` update.
- **Branch-per-stage + MERGE AT STAGE CLOSE (standing, from Stage 2 on):** each
  stage is developed on its own branch (`dc-dc-stageN-...`). Merging that branch to
  `main` is PART OF the stage's definition-of-done, not an afterthought: when every
  close-out gate bullet is MET, the gauntlet is ALL GREEN and repo-lint is 0-fail,
  the branch is merged to `main` via a PR with a **merge commit** (NOT squash --
  preserve the per-item revertable history + changelogs), then the merged branch is
  RETIRED (local + remote). **The stage close ALSO SWEEPS THE OPERATING SKILL**
  (`.claude/skills/openstack-cloud-ops/`): fold in any new INVARIANT the stage
  established and reconcile any divergence from repo HEAD, in the same close (facts
  delegate to the repo and go stale; discipline/invariants live in the skill and must
  be kept current). The merge is an operator-gated action (present it, do not run it
  unasked). `main` is trunk-of-record only for stages that have merged; the next stage
  branches off the post-merge `main`. Precedent: Stage 2 -> PR #1 (`e468d2e`), 2026-07-15.
- **Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN:** `bash
  scripts/ledger-scan.sh` for next-free (it is the numbering authority, not
  repo-lint's informational count).
- **Hard rule reminder:** no inferred values -- every IP/ID/name/scope is
  measured this session or carried from confirmed as-built; verify before
  mutate, always; destructive/secret-handling steps are individually gated,
  never batched.

---

## Tooling gap register (audited 2026-07-09; update as items close)

> ### !!! STALE-CONTENT WARNING (2026-07-13) -- READ BEFORE ACTING ON ANY OPNSENSE ITEM BELOW !!!
>
> Everything in this document describing OPNsense edge config delivery via a rendered
> `config.xml` and/or a **config ISO** is **OBSOLETE and, against the live Office1 edge,
> DANGEROUS**:
>
> - **The config ISO is INERT (D-112).** The OPNsense Configuration Importer can NEVER fire on a
>   pre-installed nano image -- it probes for a read-only root, finds a writable one with a
>   factory `/conf/config.xml`, and exits without enumerating any device. Nothing ever reads it.
> - **Full-`config.xml` delivery is SUPERSEDED (D-113(a2), ruled + PROVEN 2026-07-13).** Edge
>   config is done over the **REST API** (`scripts/opnsense-api.sh`), proven end to end -- read
>   AND write -- against the live Office1 edge.
> - **Office1's DHCP is now API-MANAGED.** A full rendered `config.xml` push to `10.10.0.1` would
>   CLOBBER live state (it replaces `/conf/config.xml` wholesale, dropping ~667
>   migration-populated elements including the only 2 firewall pass rules).
>
> The as-built path that actually worked: prep image -> boot on factory defaults -> **D-112(c)**
> console bootstrap -> enable SSH + install key -> configure over SSH and the REST API. See
> `docs/archive/changelogs/changelog-20260713-opnsense-api-proven.md` and
> `docs/archive/changelogs/changelog-20260713-opnsense-api-write-proven.md`.
>
> **done (D-113(a2)) -- reconciled 2026-07-16; the repo won over this register's stale "OPEN".** The
> config-ISO path and the module's `config_seed`/cdrom wiring are RETIRED: `modules/opnsense-edge` no
> longer takes `config_iso_path` (verified -- no `config_seed`/`config_iso_path` anywhere in the module),
> `opentofu/templates/opnsense-config.xml.tmpl` is DELETED, and Stage 3's edge steps (phase-2 Steps 4-5)
> are REWRITTEN around the REST API (`scripts/opnsense-api.sh`, harnessed). So the OPNsense steps in the
> phase-1/phase-2 RUNBOOKS are the API path and are NOT reference-only. **Caveat:** THIS document's own
> prose below (and other gap items) still describe the old `config.xml`/ISO delivery in places -- treat
> **this workflow doc** as the stale surface and the runbooks + module as authoritative until a full
> gap-register reconciliation lands.

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not
inferred from the design doc alone. Ordered roughly by "you'll hit this first,"
not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;
#2 is the single largest build.

1. **`$DC` selector convention -- CLOSED 2026-07-09 (DOCFIX-151).**
   `scripts/lib-net.sh` gained `lib_net_select_dc()` (dc0/dc1 no-op, since
   D-101 rules DC1 inherits DC0's v4 layout unchanged; dc2 fails loud --
   NetBox literals not yet assigned, gap #3) and `scripts/lib-hosts.sh`
   gained `lib_hosts_select_dc()` (dc0 no-op; dc1 AND dc2 BOTH fail loud --
   no real per-DC host enrollment exists yet for either, an intentional
   asymmetry vs. the net selector, documented in both files' comments and
   `docs/archive/changelogs/changelog-20260709-dc-selector-convention.md`). Backward compatible
   by construction -- every existing caller of either library is unaffected.
   New harness `tests/dc-selector/run-tests.sh`: 21/21 PASS. The mechanism
   exists; Stage 5's runbook still needs to actually CALL
   `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` when it's
   authored -- that call-site wiring is part of writing that runbook, not
   this delivery.
2. **OpenTofu** -- **STATUS CORRECTED 2026-07-13. The "UNVALIDATED / no tofu
   binary" framing below is now FALSE and must not be relied on:**
   - **A `tofu` binary EXISTS on the jumphost** (version pin authority:
     `docs/CURRENT-STATE.md` section 7 ONLY). The tree can self-check. The whole "authored in a session with no binary" caveat that
     rides along with every OpenTofu instruction in this repo is now closed as
     a *capability* statement.
   - **The tree VALIDATES: 10/10 modules**, root + every module standalone
     (`bash scripts/opentofu-validate.sh`).
   - **It has also been APPLIED for real.** `opentofu/terraform.tfstate` holds
     11 resources: the DC1 planes, the pools, the three mesh links, the
     `office1-network`, and the Office1 OPNsense edge domain + volumes. This is
     no longer a paper tree.
   - **BUT two modules were BROKEN and had NEVER been parsed by any tool**
     (DOCFIX-194, 2026-07-13): `node-vm` (`libvirt_volume` with a bogus
     top-level `format` -- it nests under `target`) and `netem-link` (a
     destroy-time provisioner referencing `var.*`, which OpenTofu rejects at
     INIT -- the module could not even initialize). Root's `main.tf` does not
     call either one, and `tofu validate` on the root module never parses an
     uncalled module -- so the gate printed **PASS** over both. Both are fixed;
     the gate now validates EVERY module standalone (S3). **`node-vm` builds
     every DC node and `netem-link` is the Stage 6 failover-drill mechanism** --
     they would have detonated mid-deploy. See
     `docs/archive/changelogs/changelog-20260713-docfix194-opentofu-module-validation.md`.
   - **`modules/cloudinit-vm` IS NOW INSTANTIATED FOR REAL (2026-07-13, D-114).**
     The long-standing "the mechanism exists but no image source is chosen and no
     `user_data`/`meta_data`/`network_config` has been designed for it" caveat --
     which is what forced Stage 2's old Option A / Option B fork -- is **CLOSED
     for `voffice1`**: `module "voffice1"` in root `main.tf` carries a real image
     source (`modules/base-image` -> the official Ubuntu 24.04 noble cloud
     image), real cloud-init content, and a `network_config` that DHCPs off the
     Office1 edge's Kea (matching the NIC by glob, not by a guessed kernel name).
     The module also **gained an `expose_nested_virt` variable** -- load-bearing,
     because LXD virtual machines are qemu/KVM guests and D-114's composed
     service machines cannot boot without the host CPU feature passed through.
     Applied; the domain is running. **The three Office1 service VMs are NOT
     cloudinit-vm targets any more at all** -- under D-114 they are MAAS-COMPOSED
     LXD VMs, so no further `cloudinit-vm` design work is owed for them.
   - **Still true:** `tofu plan`/`apply` has NOT been exercised for the Stage 3
     DC substrate. Validation is not application.

   *Original 2026-07-09 text follows (kept for the module inventory; its status
   claims are superseded by the block above).*

   **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded
   2026-07-09 (was: zero files).** `opentofu/` now has: `modules/dc-planes`
   (six per-DC planes), `modules/mesh-link` (the D-100 dark-fiber triangle
   legs), `modules/dc-storage-pool`, `modules/node-vm` (blank disk, PXE-boot
   MAAS-managed node VM, the D-103 pattern for Stage 3), `modules/base-image`
   + `modules/cloudinit-vm` (the OTHER VM pattern -- pre-built image +
   cloud-init, for Office1's own service VMs), `modules/opnsense-edge` (see
   gap #4), `modules/maas-vm-host` (registers the vcloud host with MAAS as a
   virsh VM host -- via the official `canonical/maas` provider, confirmed
   against its real schema; deliberately NOT `maas_vm_host_machine`, which
   composes/creates new VMs itself and would fight `node-vm` for VM
   creation -- D-103 explicitly rules out MAAS composing), and
   `modules/netem-link` (`terraform_data` + SSH-wrapped `local-exec`
   provisioners applying `tc qdisc ... netem` to a mesh-link bridge --
   OpenTofu runs from Office1, not the vcloud host, so a bare local command
   can't reach the bridge). DC1 + Office1 pool/plane wiring is instantiated
   in root `main.tf`; everything else is NOT (node/VM sizing, MAAS zone/pool,
   netem parameters are all pending decisions -- neither invented here).
   Provider schema was confirmed against the actual current docs AND real
   example `.tf` files (not memory) -- it differs materially from older
   common examples: nested structures are attribute-style objects (`key =
   { ... }`), not classic HCL blocks. **A real bug from the first pass was
   found and fixed:** `dc-planes`/`mesh-link`/`dc-storage-pool` originally
   used block syntax for `libvirt_network`'s `domain`/`mtu` and
   `libvirt_pool`'s `target`; corrected once real provider examples surfaced
   the actual convention -- discovered precisely BECAUSE building the domain
   module forced fetching real example code instead of doc-summaries alone.
   Still missing, deliberately deferred (see `opentofu/README.md`):
   OPNsense's own bootability via `cloudinit-vm` is NOT confirmed (FreeBSD,
   not the same cloud-init/NoCloud guarantee as Linux images); the
   per-device `boot`-order attribute's exact internal shape (flagged
   UNVERIFIED in `node-vm/main.tf`, an inference by pattern not a confirmed
   field); and whether `devices.interfaces` list order reliably maps to
   LAN/WAN NIC enumeration (flagged in `opnsense-edge/main.tf`).
   **UNVALIDATED** -- no `tofu` binary was available to run
   `scripts/opentofu-validate.sh` this session; run it on a connected
   machine before trusting any of this.
3. **NetBox multi-DC/dual-stack import pipeline -- MECHANISM CLOSED
   2026-07-09 (DOCFIX-152), literals still pending.** `netbox/dc-dc-prefixes-
   import.py` extends the v1 pattern to VR1's two-DC dual-stack model:
   per-DC (`--dc dc1`/`dc2`) creates the six-plane IPv4 (DC1 inherited,
   hardcoded per D-101; DC2 via required `DC2_V4_SUPERNET`, no default) plus
   the D-101 family-matrix IPv6 legs (ULA/GUA), carved from two REQUIRED
   env vars (`ORG_ULA_48`, `DC_GUA_PREFIX`) with no defaults -- fails loud
   rather than inventing either. 40/40 tests (`tests/dc-dc-prefixes-
   import/`, fake-NetBox end-to-end). UNVALIDATED against a real NetBox (no
   instance reachable this session). The literals themselves (org ULA /48,
   per-DC GUA carve, DC2 supernet) still don't exist -- that operator/NetBox
   assignment step is unchanged by this delivery; this closes the TOOLING
   gap, not the DATA gap. Still blocks Stage 2/3 until those literals are
   assigned and this script is actually run against a live NetBox.
4. **OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) done
   2026-07-09, (d) is the only real gap left.** Each site (Office1, DC1, DC2)
   gets its own independent OPNsense edge with a simulated ISP uplink (D-100);
   OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge)
   and Stage 3 (per-DC edges) gates, which require netem parameters "applied
   and measured."
   - **(a) Network substrate -- done.** `opentofu/modules/mesh-link` creates
     the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an
     OPNsense VM's interfaces would eventually attach to.
   - **(b)+(c) The OPNsense VM/image AND its config -- BUILT
     (`opentofu/modules/opnsense-edge`), UNVALIDATED.** Cloud-init is
     confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum
     consensus, not just a hedge. The real native mechanism is the
     Configuration Importer (official docs): scans an attached volume for
     `/conf/config.xml` in a 2-3 second boot-time window; ISO9660 support was
     added to it specifically for VM/cloud automation (attach a config ISO as
     a secondary CD-ROM) -- mechanically IDENTICAL to `modules/cloudinit-vm`'s
     shape (a secondary volume attached as `cdrom`), just a different payload.
     Built as `opentofu/modules/opnsense-edge` + two prep scripts
     (`scripts/opnsense-prep-image.sh` for the nano image's decompress+
     convert, `scripts/opnsense-build-config-iso.sh` for the config.xml ISO).
     The real `config.xml` DESIGN is now also done:
     `opentofu/templates/opnsense-config.xml.tmpl` (a `{{TOKEN}}`-parameterized
     template built from OPNsense's own real shipped sample, covering
     interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall
     rules) + `scripts/opnsense-render-config.sh` (the renderer -- fully
     tested end-to-end, 8/8, unlike the other opnsense-* scripts, since it
     needs no external tool). UNVALIDATED regardless: no `tofu` binary, no
     ISO-building tool, and no real OPNsense boot were available to confirm
     the whole chain end-to-end. Several template tokens still can't be
     filled with real per-site values -- pending Stage 0 ratification
     (D-100/D-101/D-107) or only measurable on a real boot (which `vtnetN`
     device a network becomes). Full sourcing in `opentofu/README.md` and
     `opentofu/templates/README.md`'s token legend.
   - **(d) `tc netem` WAN-simulation -- mechanism BUILT (`modules/netem-link`),
     only the real parameters remain.** No Terraform/OpenTofu-native
     equivalent exists (confirmed against the `libvirt_network` schema, gap
     #2); built instead as a `terraform_data` + SSH-wrapped `local-exec`
     provisioner (OpenTofu runs from Office1, not the vcloud host where the
     bridge lives -- a bare local command can't reach it). UNVALIDATED, not
     instantiated. The exact parameters (latency/jitter/loss/rate) are
     themselves still an unruled D-100 open sub-item (gap #11) -- **a
     concrete PROPOSAL now exists** (`docs/dc-dc-netem-and-ula-gua-
     proposal.md`, DOCFIX-168, Part 1: 1ms/0.2ms jitter/0.01% loss/uncapped
     rate, with reasoning, presented for operator ratification, not
     silently applied) -- still needs an actual ruling before it can be
     applied for real.
5. **Ceph replication tooling (radosgw multisite + rbd-mirror) -- MECHANISM
   CLOSED 2026-07-09 (DOCFIX-165), data half still open.** `runbooks/dc-
   dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and 11.2-11.4's
   `radosgw-admin`/`rbd mirror` command sequences (verbatim upstream Ceph
   administration, not invented) are now callable, parameterized scripts:
   `scripts/dc-dc-radosgw-multisite.sh` (subcommands `master-init`/`join-
   readonly`/`enable-two-way` -- Step 4 one-way staging + Step 8's radosgw
   half), `scripts/dc-dc-rbd-mirror.sh` (subcommands `bootstrap-primary`/
   `bootstrap-secondary --direction rx-only|rx-tx` -- Step 5 one-way staging
   + Step 8's rbd-mirror half), and `scripts/dc-dc-dr-drill.sh` (subcommands
   `failover` -- Step 10.2 force-promote + a flagged, never-auto-executed
   Glance re-registration reminder -- and `failback` -- Steps 11.2-11.4, with
   the demote-CURRENT-PRIMARY-before-promote-RECOVERING safety ordering
   HARD-CODED as fixed call order, not left to documentation). Every
   realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument, no
   invented defaults; default mode is a `--dry-run` plan print, `--apply`
   executes one command at a time via `juju ssh`/`juju run`, aborting on
   first failure. New harnesses `tests/dc-dc-radosgw-multisite/`,
   `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23
   PASS) test ONLY these scripts' own argument parsing, guard clauses,
   dry-run output, and (critically) the failback step ORDER -- NOT real
   radosgw/rbd-mirror behavior, since no live Ceph cluster exists this
   session to test against (same UNVALIDATED posture as gap #3's NetBox
   pipeline). The MECHANISM exists; the DATA half (real DC1/DC2 Ceph
   cluster, real names/endpoints, the three `bundle.yaml` gaps in the Stage
   6 runbook's own "Known gaps" section) is unchanged and still blocks Stage
   6 for real. See `docs/archive/changelogs/changelog-20260709-ceph-replication-tooling.md`.
6. **Designate reactivation is a bundle change -- CLOSED 2026-07-10
   (DOCFIX-167).** `bundle.yaml` now carries `designate`/`designate-
   bind`/`designate-mysql-router`/`designate-hacluster` + 8 relations,
   researched against designate's real metadata.yaml (a real mistake
   caught during authoring: its provides-side endpoint is `dnsaas`, not
   `public` like the other API charms -- corrected). `os-public-hostname`
   re-activation (D-106's OTHER coupled change) deliberately NOT done in
   the shared bundle -- pushed to a proposed per-DC overlay instead, to
   keep `bundle.yaml` DC-agnostic. `python3 scripts/provider-bundle-
   check.py` PASS, all 6 existing invariants unaffected. See
   `docs/archive/changelogs/changelog-20260709-designate-cinderbackup-rbdmirror.md`. NOT
   applied to any live model -- repo-source change only.
7. **MTU/geneve budget + Ceph disk-budget calculators -- CLOSED 2026-07-09
   (DOCFIX-162).** D-102 and buildout-design Section 3 both describe
   measurement-then-computation gates (host L2 MTU -> geneve-over-v6
   overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback)
   that were manual arithmetic in prose, not a script, blocking the Stage 1
   Phase-0 gates. `scripts/dc-dc-mtu-geneve-budget.sh` takes the MEASURED
   underlay MTU via required `--underlay-mtu` (no default) and reproduces
   D-101's own worked example exactly (1500 - 56 = 1444), with a jumbo
   (>=9000) path leaving tenant MTU at 1500, plus an added IPv6-minimum-MTU
   (1280, RFC 8200) sanity floor clearly separated from D-101's own text.
   `scripts/dc-dc-ceph-disk-budget.sh` takes MEASURED total disk, per-DC
   OSD node counts and per-node footprints, and a required
   `--backup-overhead-fraction` (deliberately NO default -- not specified
   as a hard number anywhere in this repo, must come from real
   Ceph/radosgw-admin measurement once DC1 exists) and computes whether
   size=3 across both DCs plus overhead fits the measured total disk;
   if not, it names size=2 as the ONLY documented fallback (D-101 /
   Section 3) without silently applying it -- explicitly an operator-logged
   decision. New harnesses `tests/dc-dc-mtu-geneve-budget/run-tests.sh`
   (19/19 PASS) and `tests/dc-dc-ceph-disk-budget/run-tests.sh` (16/16
   PASS). Both scripts are read-only/mutate nothing and have not yet been
   run against the real vcloud host (prep-only session, no live
   infrastructure reachable) -- that first real run is part of executing
   `runbooks/dc-dc-phase0-vcloud-prep.md` Step 3. Changelog:
   `docs/archive/changelogs/changelog-20260709-mtu-ceph-budget-calculators.md`.
8. **The "developed IPv6 plan" D-101 cites does not exist in this repo.** D-101's
   context line references "the developed family-follows-reachability IPv6 plan
   (in-project, not yet committed)" as prior art -- grepped the whole repo, it is
   not present as a standalone doc anywhere. If the exact Phase A/B/C detail
   behind D-101/D-102 is needed by a future session, it currently is NOT
   retrievable from the repo alone. Not a blocker for Stage 0 ratification of
   what IS written, but a real content gap if finer detail is needed later.
9. **Runbook debt -- CLOSED 2026-07-09.** All seven Stage 1-7 runbooks are now
   written (`runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-
   designate-cos-magnum.md`), Stage 1 authored directly, Stages 2/3/4/6/7
   delegated to parallel subagents and Stage 5 authored directly, all
   reviewed and repo-lint clean. NONE have been executed against real
   infrastructure yet -- this was a prep-only session (the operator was away
   from the workstation, not the vcloud host). Each runbook is explicit about
   what it can and cannot close given today's real state (NetBox literals,
   node sizing, and several other decisions remain genuinely open -- see
   each runbook's own GATE section for the honest per-stage breakdown).
10. **Harness debt -- LARGELY CLOSED 2026-07-09/10.** This repo's own delivery
    rule -- no script change ships without its `tests/<name>/run-tests.sh`
    harness -- has been honored for every new script tonight: `dc-selector`
    (21/21), `dc-dc-prefixes-import` (40/40), `dc-dc-mtu-geneve-budget`
    (19/19), `dc-dc-ceph-disk-budget` (16/16), `dc-dc-radosgw-multisite`
    (19/19), `dc-dc-rbd-mirror` (19/19), `dc-dc-dr-drill` (23/23), plus new
    `$DC` test cases added to `carve-host-interfaces`/`phase-00-maas-
    standup` and a new `reenroll-hosts` harness. Gap #2 (OpenTofu modules)
    remains the one area still lacking a harness beyond `scripts/opentofu-
    validate.sh` itself (no `tofu` binary this session to validate against).
11. **Decision sub-items still genuinely open** (the big items are RULED per
    Stage 0; these specific literals/params are not -- but see
    `docs/archive/dc-dc-netem-and-ula-gua-proposal.md` (DOCFIX-168) for a concrete
    netem proposal and exact, safe ULA-generation guidance, drafted
    2026-07-10): exact netem parameters
    (D-100), exact NetBox literals for DC2/ULA/GUA (D-101).
12. **No Office1-local virtual network is modeled anywhere in `opentofu/` --
    CLOSED 2026-07-09.** `main.tf` previously wired only `office1_pool_path`
    (storage only) and the three inter-site mesh links (management-traffic-
    only per D-100's own sub-item ruling) -- nothing the three Office1
    service VMs (MAAS-region, NetBox, GitBucket), or the vcloud host's own
    management interface, could actually attach to as a shared local network.
    Resolved by a NEW `opentofu/modules/office1-network` module (a
    `dc-planes`-shaped isolated `libvirt_network`, sized for ONE network
    rather than six planes since Office1 is a headend, not an OpenStack DC),
    deliberately NOT a reused host bridge -- a hand-configured bridge would
    be the one network in this topology living outside OpenTofu's D-103-
    mandated inventory, silently breaking this repo's own `virsh net-list
    --all` completeness assumption. INSTANTIATED for real in root `main.tf`
    (`module "office1_network"`) -- unlike most of gap #2's other scaffolding,
    this needed no unmeasured value beyond `domain_suffix`/`underlay_mtu`,
    already real required inputs elsewhere in the same file. See
    `opentofu/README.md` and `docs/archive/changelogs/changelog-20260709-office1-network-edge.md`
    (DOCFIX-163) for the full design rationale. **Follow-on work also done
    2026-07-13 (D-114):** the "designing the service VMs' actual `cloudinit-vm`
    `network_config` against this network" tail of this gap is closed, but not
    the way it was written -- there are no longer three service VMs to attach.
    ONE VM attaches to `office1-local`: the `voffice1` containment VM, whose
    `network_config` DHCPs from the OPNsense edge's Kea. The service machines
    live INSIDE `voffice1`'s LXD (MAAS-composed) and never touch this network
    directly. **DHCP hazard to carry forward:** LXD's `lxdbr0` runs its OWN
    dnsmasq -- it must stay on LXD's internal NAT bridge and must NEVER be
    bridged onto `office1-local`, where Kea is authoritative.
13. **The D-101 IPv6 family-matrix overlay -- DRAFTED 2026-07-10, one real
    risk still open.** `overlays/dc-dc-ipv6-family-matrix.yaml` now exists,
    sourced against real charm config/source (`docs/dc-dc-ipv6-charm-
    research.md` has full citations): `prefer-ipv6` (confirmed real option,
    genuinely additive dual-stack HAProxy binding per `charm-nova-cloud-
    controller`'s actual commit, not an either/or switch) + appended v6/GUA
    VIPs for the API charms; `ceph-mon`'s own separate `prefer-ipv6` +
    `ceph-public-network`/`ceph-cluster-network` for the ULA-only planes;
    confirmed NO charm-config option exists for OVN (geneve family follows
    the bound interface automatically). D-109's IPv6-SAN cert support is
    CONFIRMED (read `charm-vault`'s actual `sort_sans()` code -- no v4/v6
    distinction). **Genuinely unresolved:** Octavia's `lb-mgmt-net` IPv6
    support is excluded from the overlay on purpose -- two real, still-
    referenced upstream Launchpad bugs (#1911788, #1913409) document IPv6
    lb-mgmt-net failures; needs an explicit operator decision (accept the
    risk and test for real, or keep Octavia's lb-mgmt-net as a deliberate
    v4-only D-101 exception), not silently resolved here. Also unconfirmed:
    8 of the 9 dual-stack charms' `prefer-ipv6` presence is inferred by
    shared-framework pattern, not individually fetched. UNVALIDATED against
    any live model.
14. **Three real `bundle.yaml` gaps for D-108's DR mechanism -- CLOSED
    2026-07-10 (DOCFIX-167).** `cinder-backup` (a SUBORDINATE charm,
    confirmed via real metadata.yaml -- the first draft wrongly gave it
    principal-style `num_units`/`to`/`amqp`, corrected) and `ceph-rbd-
    mirror` (a principal with `ceph-local`/`ceph-remote` requires
    endpoints, confirmed via real metadata.yaml -- the first draft
    invented a wrong binding key, corrected) are now in `bundle.yaml`;
    `ceph-mon`'s `rbd-mirror` binding corrected from `storage` to
    `replication`. **Genuinely unresolved, documented not invented:**
    `ceph-remote` (the peer DC's ceph-mon) is a cross-Juju-controller
    relation D-104's per-DC-independent-controller design means cannot be
    a same-bundle relation line -- only this DC's own `ceph-local` leg is
    wired; the cross-DC peer relationship needs a `juju offer`/`consume`
    or the manual `rbd mirror pool peer bootstrap` CLI flow (Stage 6's
    runbook already documents the latter). `python3 scripts/provider-
    bundle-check.py` PASS. See `docs/changelog-20260709-designate-
    cinderbackup-rbdmirror.md`. NOT applied to any live model.
15. **Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the
    Stage 4 runbook** (full detail in that runbook's own "Known gaps"
    section). **Sub-item 2 (script `$DC`-parameterization) CLOSED
    2026-07-10 (DOCFIX-166)** -- `reenroll-hosts.sh`,
    `carve-host-interfaces.sh`, and `phase-00-maas-standup.sh` now accept an
    opt-in `$DC` env var and call `lib_net_select_dc`/`lib_hosts_select_dc`
    immediately after sourcing (unset `$DC` is byte-for-byte unchanged
    behavior; see `docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md`). This
    closes the CLI-invocation half of the gap only -- it does NOT create any
    new per-DC data (no dc1/dc2 hostnames, octets, boot MACs, or node-count
    figures were invented), and `phase-00-maas-standup.sh`'s own `PLANES`
    table remains hardcoded to D-052/D-053's literals rather than driven by
    `lib-net.sh`'s flat vars, so it is not truly dc2-topology-aware yet even
    though it now gates dc2 out cleanly. The other six sub-items remain
    OPEN: no OpenTofu module stands up a per-DC MAAS rack controller VM
    itself (only registration, `modules/maas-vm-host`, exists); per-DC node
    count/sizing is an unmade Phase-0 decision; DC2 is still blocked at the
    network-selector layer (gap #3, NetBox literals not assigned); the
    per-DC artifact mirror (D-107) has no identified owning stage/module;
    the MAAS status a VM-host-discovered (vs. PXE-enlisted) machine lands in
    is unconfirmed; `HOST_TAG`/boot-fabric-name literals are DC0-specific.
16. **No stage explicitly owns creating Office1's own OPNsense edge VM --
    CLOSED 2026-07-09.** The topology (buildout design Section 1) is
    explicit: "full dark-fiber triangle... plus a **per-site** OPNsense
    simulated ISP edge" -- three sites, three edges (DC1, DC2, AND Office1).
    Stage 3's Build list already covered DC1/DC2's edges
    (`modules/opnsense-edge` per DC); Stage 2's Build list (MAAS region,
    OpenTofu, NetBox, GitBucket, Tailscale) did not mention an Office1 edge
    at all, and neither runbook instantiated one. **Decision: YES, Office1
    gets its own `modules/opnsense-edge` call, a fourth site alongside
    DC1/DC2; Stage 2's runbook
    (`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it.**
    Rationale: Stage 2's entire scope is already "Office1 headend standup"
    (the OPNsense edge IS part of that headend, not DC substrate), and
    Stage 1 already creates the mesh-link networks this edge's neighbors
    depend on -- Stage 3 has no reason to reach back and create a piece of
    Office1's own infrastructure. Wired as a commented-out `module
    "office1_opnsense"` skeleton in root `main.tf` (mirroring the identical
    NOT-YET-REAL status of DC1/DC2's own edges -- neither is instantiated
    either) plus a new gated step in
    `runbooks/dc-dc-phase1-office1-standup.md` (Step 4b) and its own GATE
    bullet. Related to, but distinct from, gap #12 (Office1-local network,
    now also CLOSED) -- that gap was the service VMs' own attach point;
    this one was whether Office1 gets a simulated-ISP-edge boundary at all,
    and who builds it. The edge's LAN side now resolves cleanly to gap
    #12's new network; its WAN side does not -- see NEW gap #17
    immediately below, a genuinely deeper, cross-site gap surfaced while
    closing this one, not invented as a substitute for a real answer.
    **D-114 follow-on (2026-07-13):** the edge is now BUILT and LIVE, and it is
    a load-bearing PREREQUISITE of the rest of Stage 2, not just a gate bullet
    -- `voffice1` takes its DHCP lease from the edge's Kea and reaches the
    internet through it. Office1's edge is also the only one of the three that
    is live; DC1/DC2's edges are Stage 3's problem and are still blocked on gap
    #17's DC half.
17. **PARTIALLY CLOSED 2026-07-13 -- the DECISION is made and PROVEN for
    Office1; DC1/DC2 still have no uplink network.**
    - **Office1: CLOSED.** The answer was the one this gap proposed -- a
      dedicated per-site ISP-uplink `libvirt_network`, **not** a mesh leg.
      `office1-wan` exists and is live (measured 2026-07-13: `virbr11`, NAT
      forward, `172.30.1.0/24`). The Office1 edge's WAN sits on it at
      `172.30.1.2` with default route `172.30.1.1`, and egress to the internet
      is verified (0.0% loss). So the topology question is ANSWERED and the
      pattern is PROVEN in production, not merely designed.
    - **DC1/DC2: STILL OPEN.** Neither has an ISP-uplink network. Measured:
      `dc1-provider-public` has **no IP and no forward** (isolated), and there
      is no `dc1-wan`/`dc2-wan` at all. Stage 3's `dc1_opnsense` still carries
      the unresolved `wan_network_name` placeholder described below.
    - **The fix is now mechanical, not a decision:** replicate the
      `office1-wan` shape per DC. The D-100 reasoning below (mesh legs are
      MANAGEMENT-TRAFFIC-ONLY and must NOT carry an edge's WAN side) stands
      unchanged and is what the Office1 build honoured.

    *Original 2026-07-09 text follows (the reasoning is still correct; the "for
    ANY site / nowhere" status claim is superseded by the block above).*

    **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in
    this repo, for ANY site** (found 2026-07-09 while closing gap #16 for
    Office1). Every `modules/opnsense-edge` caller needs a
    `wan_network_name` distinct from BOTH its LAN-side network (a
    `dc-planes` plane, or gap #12's new `office1-network`) AND the three
    D-100 mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC-
    ONLY by D-100's own sub-item ruling (Office1<->DC fiber carries MAAS/
    Juju/operator traffic only), not a simulated ISP uplink, so wiring an
    edge's WAN side to a mesh leg would misrepresent the topology rather
    than merely placeholder it. This is NOT an Office1-specific gap:
    `runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 5's own `dc1_opnsense`
    template carries the IDENTICAL unresolved placeholder
    (`wan_network_name = module.mesh_dc1_office1... # or a dedicated
    per-site ISP-uplink segment -- confirm which, do not assume`) --
    confirmed by reading that runbook directly, not assumed. Deliberately
    NOT invented here for Office1 alone (that would contradict D-100 and
    leave DC1/DC2 with the same unresolved gap under a different name).
    Needs a decision: a fourth network shape per site (one more
    `dc-planes`/`mesh-link`/`office1-network`-style dedicated
    `libvirt_network` standing in for that site's own "internet" uplink),
    or some other mechanism not yet considered. Blocks instantiating ANY of
    the three sites' `opnsense-edge` calls (`dc1_opnsense`, `dc2_opnsense`,
    `office1_opnsense`) end-to-end -- all three are otherwise ready
    (image/config mechanism per gap #4, LAN-side network per gap #12/this
    stage's plane modules) but for this one missing piece.

18. **OpenTofu state file had no documented protection -- CLOSED 2026-07-10
    (DOCFIX-175).** Found while reviewing what's missing from the buildout,
    before Stage 1's own first real `tofu apply` (`dc-dc-phase0-vcloud-
    prep.md` Step 10 -- confirmed by reading the runbook directly, not
    assumed from filenames; an earlier pass wrongly guessed Stage 3):
    `terraform.tfstate` stores `sensitive = true` variables like
    `var.maas_api_key` in plaintext (verified against HashiCorp's own docs,
    not assumed -- `sensitive` only suppresses CLI output, not state
    storage), and nothing gitignored it, documented backup/permission
    handling, or noted the local-backend's lack of locking. Fixed:
    `.gitignore` now excludes `opentofu/**/*.tfstate*` and related local
    artifacts, `opentofu/README.md` gained a full "State file handling"
    section, `variables.tf`'s `maas_api_key` description carries the caveat
    inline, and `dc-dc-phase0-vcloud-prep.md` Step 10 gained a "SECURE THE
    STATE FILE" callout (`chmod`, gitignore verification, out-of-band
    backup) as a standing discipline from the first real apply onward (a
    short pointer added at Stage 3 Step 8 too, the first step needing a REAL
    MAAS credential). NOT fixed (documented, not invented): a remote backend
    with real locking -- production-grade, Roosevelt-scope, not a rehearsal
    decision.

19. **No teardown/rollback runbook existed for the VR1 OpenTofu-provisioned
    infrastructure itself -- CLOSED 2026-07-10 (DOCFIX-176).**
    `runbooks/phase-00-teardown-*.sh` (D-061) covers tearing down the VR0
    juju/MAAS layer, a different pairing of tools; nothing documented how to
    `tofu destroy` DC1/DC2/Office1's networks, storage pools, node-VM
    domains, and OPNsense edges in safe dependency order, or what MAAS-side
    cleanup has to happen alongside it. Fixed:
    `runbooks/dc-dc-teardown-rollback.md` -- two paths (scoped single-site
    via `-target`, or full VR1 destroy), the D-061 principle carried over
    (MAAS-side cleanup before destroying the libvirt resource underneath),
    a rollback decision tree favoring fix-forward over destroy for the
    common partial-apply-failure case, and explicit non-scope for mesh-link
    legs (shared between two sites, only destroyed when both ends are
    going away). **NOT resolved (flagged, not invented):** the exact
    MAAS-side release/delete commands for machines enrolled under a VR1
    `vm_host`, since this repo has never reached Stage 4 for any VR1 DC and
    has no real machines to test against yet -- the runbook states the
    principle and points at MAAS's own docs rather than guessing flags.

19. **NEW 2026-07-14 -- THE `dc1`/`dc0` NAME NOW MEANS TWO DIFFERENT DATACENTERS.
    This BLOCKS Stage 3 and is the same off-by-one D-117 just fixed, one layer down.**

    D-117 (ADOPTED, Option B) moved the repo onto the NetBox apex's 0-indexed
    convention: VR1's **first** DC is `vr1-dc0`, its **second** is `vr1-dc1`.
    That rename was executed on the IMPORT PATH ONLY (`netbox/dc-dc-prefixes-
    import.py` + the scope doc). It was **NOT** executed on the two other
    surfaces that speak `$DC`, and they now disagree with the apex:

    | Surface | says `dc1` means | says `dc0` means |
    |---|---|---|
    | NetBox apex (authority, D-117) | VR1's **SECOND** DC | VR1's **FIRST** DC |
    | `opentofu/main.tf` (`dc1_planes`, `dc2_storage`, `mesh_dc1_dc2`) | VR1's **FIRST** DC | -- (no `dc0`) |
    | `scripts/lib-net.sh` / `lib-hosts.sh` (DOCFIX-151 selectors) | VR1's first DC | **VR0's** DC0 -- a different REGION |

    So "the next DC we deploy" is, simultaneously: apex `vr1-dc0`, OpenTofu
    `dc1_*`, and a `lib-net.sh` `$DC` value that ALSO names VR0's live testcloud.
    Three names, one machine, and one of them collides with a **running cloud**.

    **Why a mechanical rename is a trap (D-117 TRAP 1/TRAP 2, now live here):**
    `sed s/dc1/dc0/` on `opentofu/main.tf` would hand VR1's first DC the string
    `dc0`, which `lib-net.sh` already resolves to **VR0's deployed literals**. The
    fix is a SEMANTIC remap through placeholders, and the shell selectors must be
    region-qualified (`vr0-dc0` / `vr1-dc0` / `vr1-dc1`), not merely renumbered.

    **Consequence if ignored:** `module "dc1_nodes"` gets authored against the
    wrong site's prefixes, and the first VR1 DC deploys with the second DC's
    addressing -- discovered, at best, at `juju deploy`.

    **Status: OPEN. It is Stage 2 close-out item C3 and the FIRST step of any
    Stage 3 OpenTofu work.** Nothing in `opentofu/` should gain a new DC-keyed
    module until it is closed.

20. **NEW 2026-07-17 -- durable host->site (isolated-net) reachability had no owning
    tool -- CLOSED for Office1; DC path templated.**

    `office1-local` is an ISOLATED libvirt net (no `<ip>`/`<forward>`), so the vcloud
    host has no L3 presence on `10.10.0.0/24`: its base leg (`10.10.0.10/24` on the
    bridge) + compose route (`10.10.1.0/24 via 10.10.0.20`) were re-added BY HAND each
    session and dropped on every reboot. The vcloud OS-patch+reboot (2026-07-17) proved
    the cost -- it stranded `ssh voffice1` and every ProxyJump alias. D-126 made the
    layers ABOVE this durable (the `--user` forward, the ssh aliases) but never owned
    the base leg.

    **CLOSED for Office1 (D-126 amendment):** `scripts/site-baseleg.sh` +
    `tests/site-baseleg/` (24/24) -- a root `Type=oneshot` unit ordered `After=libvirtd`
    (MEASURED: monolithic `libvirtd` here; also orders after `virtnetworkd` for modular
    hosts) that re-adds the leg at boot. Bridge DISCOVERED from the stable net name
    (never a baked `virbrN`, PATTERN-1); `apply` idempotent; `check` = session-verify.
    INSTALLED + `enabled` on vcloud 2026-07-17; the `After=libvirtd`+poll BOOT-RACE stays
    UNPROVEN until an actual reboot self-heals `ssh voffice1`. See D-126 amendment +
    D-128; `docs/archive/changelogs/changelog-20260717-d126-baseleg-oneshot-d128-operating-model.md`.

    **DC generalization (so this gap does NOT reopen per-DC):** the tool is SITE-KEYED
    with a commented DC template row, and the harness FAILS on any un-commented DC
    supernet (hard rule 2). At each DC standup, MEASURE how the vcloud host reaches that
    DC's containment/service net: if it is an ISOLATED host-local net, add a MEASURED
    `site-baseleg.sh` leg row + `install` the unit as part of the standup definition-of-
    done; if the DC is reached over a routed/NAT net or by `qemu+ssh` to an address the
    host already holds (D-123 / R-5), no leg is needed. The skill carries this as
    standing discipline (routing row + operating-model invariant + the VR1 deploy loop).
    **Status: Office1 CLOSED; DC rows DEFERRED-until-measured (mechanism in place).**

---

## Where things stand today (2026-07-09) -- update this as stages close

| Layer | Status |
|---|---|
| Design (`dc-dc-buildout-design.md`) | done -- Section 10 now a ratified-ruling record; the doc's own header still reads "DESIGN / PLAN" (a status line, not a blocker -- its embedded decisions are ADOPTED per Stage 0 below) |
| Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED |
| DR mechanism seed | done (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 |
| Stage 0-7 runbooks | Stage 0 done (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. Stage 1 EXECUTED 2026-07-10. **Stage 2's runbook REWRITTEN 2026-07-13 to the D-114 model** (one `voffice1` site containment VM + MAAS-composed LXD VMs, replacing the three sibling service VMs and their Option A/B fork); it is PARTIALLY EXECUTED, and its MAAS/LXD command lines are marked PENDING VERIFICATION rather than guessed. Stages 3-7 NOT executed, and D-114 GATES them behind Office1 completing. See gaps #12-15 above for real design gaps the authoring pass surfaced. |
| **D-114 site containment VM (`voffice1`)** | **BUILT + running 2026-07-13.** `module "voffice1"` (`modules/cloudinit-vm` off `modules/base-image`, Ubuntu 24.04 noble) is instantiated and applied; the domain is up on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`. **Remaining:** MAAS-region + LXD (5.21 LTS track) ON it, the LXD-KVM-host registration back into MAAS, and the MAAS-COMPOSED service machines (NetBox/GitBucket/Tailscale). The first LXD VM that boots inside it is the **L3 nesting proof that gates DC1**. |
| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (tofu binary on the jumphost; pin authority `docs/CURRENT-STATE.md` section 7): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds the DC1 planes, pools, 3 mesh links, `office1-network`, the OPNsense edge, and (2026-07-13, D-114) the `ubuntu_noble_base` image + the **`voffice1` containment VM**. **`modules/cloudinit-vm` is now instantiated for real** and gained `expose_nested_virt`. **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. |
| OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + running 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. |
| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **superseded 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **done (reconciled 2026-07-16):** `templates/opnsense-config.xml.tmpl` is DELETED and the module's `config_seed`/cdrom + `config_iso_path` are RETIRED; edge config is the REST-API bootstrap. (Boot-measured `WAN_IF`/`LAN_IF` are still applied over the API at deploy time -- a deploy-time step, not an authoring gap.) |
| MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` |
| `tc netem` mechanism (`modules/netem-link`) | **WAS BROKEN, FIXED 2026-07-13 (DOCFIX-194)** -- its destroy provisioner referenced `var.*`, which OpenTofu rejects at INIT, so the module could not even initialize. Now validates. Real latency/jitter/loss/rate parameters are STILL an unruled D-100 item. Never applied. |
| `opentofu/` (DC2 planes) | not done -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` |
| Office1-local network (`modules/office1-network`) | BUILT + INSTANTIATED 2026-07-09 (DOCFIX-163, gap #12 CLOSED) -- see `opentofu/README.md` and `docs/archive/changelogs/changelog-20260709-office1-network-edge.md` |
| Office1 OPNsense edge ownership | **BUILT AND LIVE 2026-07-12/13.** `module "office1_opnsense"` is instantiated (not commented). **Gap #17 is CLOSED FOR OFFICE1**: `office1-wan` (virbr11, NAT, 172.30.1.0/24) is the dedicated per-site ISP uplink, and the edge's WAN sits on it -- the D-100-honouring answer (NOT a mesh leg). **DC1/DC2 still have NO uplink network** -- replicating the `office1-wan` shape per DC is now mechanical, not a decision. |
| Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` |
| `lib-net.sh` / `lib-hosts.sh` | `$DC` selector mechanism ADDED 2026-07-09 (DOCFIX-151, gap #1 CLOSED) -- `lib_net_select_dc()`/`lib_hosts_select_dc()`, backward compatible, 21/21 tests; Stage 4 and Stage 5's runbooks (both written 2026-07-09) mandate calling it, not yet executed for real |
| NetBox DC-DC pipeline (`netbox/dc-dc-prefixes-import.py`) | MECHANISM BUILT 2026-07-09 (DOCFIX-152, gap #3 tooling half CLOSED) -- 40/40 fake-NetBox tests (DOCFIX-160 fixed a dead-assertion test bug that had silently left 3 of these uncounted), UNVALIDATED live; real literals (ULA /48, GUA carve, DC2 supernet) still unassigned |
| Environment/skill docs (`environment.md`, README.md) | Still describe VR0 DC0/v1 single-DC scope wholesale -- flagged, not yet rewritten for VR1; not a blocker for Stage 0 but will need doing before/during Stage 1 |
