# DC-DC (VR1) deployment workflow overview

**Purpose.** A single followable checklist across the whole VR1 (two-DC virtual
regional) buildout, spanning many sessions. `docs/dc-dc-buildout-design.md` is the
WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT
sequence, cross-referenced to that plan, to D-100..D-110, and to what already
exists in this repo vs what still needs to be authored. STATUS AUTHORITY: stage and
gate status live ONLY in `docs/CURRENT-STATE.md` (GA-R1); this doc keeps stage
IDENTITY, links, and dated history -- it is no longer a status tracker.

**Governing constraint (unchanged throughout):** minimize delta to Roosevelt. Reuse
VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog
steps are the Section 9 shim register in the buildout design (OpenTofu node-VM
creation, `tc netem` WAN simulation, single-unit controllers/rbd-mirror).

---

## Stage 0 -- Decision ratification (CLEARED 2026-07-09)

**RATIFIED.** D-100..D-110 are ALL ADOPTED (`docs/design-decisions.md`;
D-102 MERGED into D-101 per the operator's own ruling, not deleted --
retained for numbering/history; D-109 confirmed standalone). All 6
redline items ruled (`docs/dc-dc-buildout-design.md` Section 10 is now a
ruling record, not an open list): Ceph size=3-by-default (size=2 only as an
explicit Phase-0 fallback); netem same-metro lean; D-102/D-101 merge +
D-109 standalone; metal-admin GAINS a ULA leg (a real reversal of the
stated lean, not a rubber-stamp); COS per-DC-only; mirror sync independent
per-DC. Verified via `bash scripts/ledger-scan.sh`: none of D-100..D-110
appear under "PROPOSED / OPEN decisions." Full ruling record:
`docs/archive/changelogs/changelog-20260709-stage0-ratification.md`.

**State:** cleared 2026-07-09 (history; status authority: docs/CURRENT-STATE.md).

---

## Stage 1 -- Phase 0: vcloud host preparation

| | |
|---|---|
| **Goal** | Turn the bare vcloud host into a substrate Office1 can deploy from. |
| **Build** | Nested KVM enabled; libvirt storage pools + thin-provisioning; dark-fiber + per-site ISP-edge virtual networks; the six per-DC plane segments as virtual networks; MTU plan set on the underlay. |
| **Gate** | Nested KVM verified; disk budget computed + Ceph size decision recorded; measured MTU recorded; virtual networks present and isolated as designed. |
| **Owns** | D-100 (fabric), D-101 (MTU -- folds in former D-102), buildout-design Section 3 (host/disk/MTU gates). |
| **Reuse vs new** | NEW. No VR0-DC0 analog -- that testcloud IS the single vcloud host, already prepared; this is preparing the host to carry TWO independent DC substrates plus Office1. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase0-vcloud-prep.md`** -- command-level steps (host identify -> measure CPU/RAM/disk/nested-KVM/MTU -> enable nested KVM -> prepare pool paths -> install/confirm OpenTofu -> tfvars -> init/validate/plan/apply -> post-apply gate verify), each MUTATION individually gated per this repo's discipline. Flags a real open structural question up front (the `provider "maas"` block in `main.tf` is unconditional even though this stage only touches libvirt resources -- may force `maas_api_url`/`maas_api_key` to be set at `plan` time with nothing yet using them; noted as a possible DOCFIX candidate, not silently worked around). NOT YET EXECUTED against real infrastructure -- this is the first runbook that will be, tomorrow morning, per the operator's own stated plan. `opentofu/` SCAFFOLD started 2026-07-09: `modules/dc-planes` (six per-DC planes) + `modules/mesh-link` (dark-fiber legs) + `modules/dc-storage-pool` wired for DC1 + Office1 -- see `opentofu/README.md` for what's built vs deliberately deferred (node-VM/domain resources, DC2 CIDRs, netem). UNVALIDATED: no `tofu` binary available to run `scripts/opentofu-validate.sh` yet -- this runbook's Step 8 is the first real run. |

**State:** executed 2026-07-10 (first DC-DC runbook run against real
infrastructure; vcloud host -- tofu pin authority: `docs/CURRENT-STATE.md` section 7). As-built: nested KVM already-on;
`underlay_mtu=9000` (jumbo, operator ruling -- the isolated planes/mesh are
host-internal virtio bridges, jumbo-capable regardless of the 1500 ISP uplink);
Ceph size=3 disk-budget plausibility PASS (4.60 TiB margin); 13 libvirt objects
applied (6 dc1 planes + dc1/office1/dc2 pools + office1-local + 3 mesh legs, all
MTU 9000). Prior VR0 topology torn down (`wan` kept as a gap-#17 model). DC2
PLANES deferred (Option B -- awaits NetBox supernet); DC2 storage pool wired.
Delivered DOCFIX-178 (prereq installers), DOCFIX-179 (scaffold: per-module
required_providers + fmt + maas-provider deferral). See those changelogs.

---

## Stage 2 -- Phase 1: Office1 SITE standup (RESHAPED 2026-07-13 by D-114)

> **!!! MODEL CHANGE (D-114, ADOPTED 2026-07-13). The three-sibling-service-VM model is
> SUPERSEDED.** Stage 2 no longer builds three peer VMs (MAAS-region, NetBox, GitBucket) on the
> vcloud host with Tailscale on the headend host. It builds **ONE Office1 site containment VM,
> `voffice1`** (Ubuntu 24.04, OpenTofu-created), which IS the facility: **MAAS-region runs ON it,
> LXD (pinned to the 5.21 LTS track) runs ON it**, that LXD is **registered BACK INTO MAAS as an
> LXD KVM host**, and the non-stack service machines (**NetBox, Tailscale** -- GitBucket removed per D-116) are VMs
> **COMPOSED BY MAAS** into it -- MAAS-visible: enlisted, commissioned, deployed, powered. This
> is VR0's own already-proven `lxd` + `tailscale` pattern, applied per site.
>
> **Explicitly NOT affected:** the Juju-deployed OpenStack services on the DC nodes. They stay in
> Juju-created LXD **CONTAINERS** (bundle `lxd:N` placements), invisible to MAAS, exactly as in
> VR0. D-114 grants MAAS a composition right scoped ONLY to the LXD VM host and the non-stack
> machines; `modules/node-vm` still pre-creates the OpenStack node VMs and MAAS merely DISCOVERS
> them (D-103 unchanged on that point).
>
> **Sequencing is an operator ruling:** Office1 FIRST, fully up on this model. **DC1 is GATED
> behind it** and is not started until Office1 is complete; DC2 follows DC1. Stages 3-7 below are
> therefore gated behind THIS stage, and their own content is otherwise unchanged by D-114.

| | |
|---|---|
| **Goal** | Stand up the Office1 SITE first, before any DC -- and prove the D-114 containment-VM model at its cheapest point. |
| **Build** | `voffice1` site containment VM (OpenTofu, `modules/cloudinit-vm` + `modules/base-image`, on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`); MAAS region controller ON `voffice1`; LXD ON `voffice1` (5.21 LTS track); that LXD registered into MAAS as an LXD KVM host; NetBox + Tailscale as MAAS-COMPOSED LXD VMs inside it (**GitBucket REMOVED from scope -- D-116**); Office1's own OPNsense simulated-ISP edge (gap #16, Stage 2 owns it). |
| **Gate** | Office1 OPNsense edge up (**MET** -- routing, NAT, egress, Kea DHCP; gap #17 closed for Office1 via `office1-wan`); `office1-local` network created (gap #12 CLOSED); OpenTofu reaches vcloud host libvirt (**MET**); **`voffice1` exists and runs (MET 2026-07-13, measured)**; `voffice1` has a reserved address + SSH + nested KVM exposed; MAAS region reachable ON `voffice1`; LXD on the 5.21 track with `lxdbr0` OFF `office1-local`; LXD registered as a MAAS KVM host; **the non-stack machines COMPOSED BY MAAS and deployed -- i.e. an LXD VM actually BOOTS at L3, which is D-114's DC1 entry gate**; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); Tailscale confirmed to Office1 only. **GitBucket is NO LONGER A GATE (D-116): no Office1-local GitBucket is built; `git.baldurkeep.com` remains the git service of record.** |
| **Owns** | **D-114** (site containment VM + MAAS-composed LXD VMs, incl. the LXD 5.21 pin), D-103 as AMENDED by D-114 (OpenTofu owns the containment VM and the site networks; MAAS composes INSIDE the site), D-107 (headend is not a core-service provider). |
| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but LESS new than it looks: the MAAS-composed-LXD-VM half is **exactly what VR0 runs today** (its `lxd` KVM host at LXD 5.21.4, with `tailscale` composed into it at 2 cores / 2 GiB / 25 GB) and carries **NO Roosevelt delta**. The containment VM itself IS a deliberate simulation-only delta -- it is what makes a "hard-down an entire facility" DR drill (`virsh destroy vvr1-dc0`) executable at all. NetBox-as-IPAM-apex discipline and the `netbox/*-import.py` pattern transfer unchanged. |
| **Authoring status** | **Runbook REWRITTEN 2026-07-13 to the D-114 model: `runbooks/dc-dc-phase1-office1-standup.md`.** The old Option A / Option B provisioning fork is **CLOSED** -- OpenTofu (Option A) is the path and `module "voffice1"` is a real, applied `cloudinit-vm` instantiation (real image source, real `user_data`/`meta_data`/`network_config`, plus the new `expose_nested_virt` variable). **PENDING VERIFICATION:** the MAAS/LXD install + LXD-KVM-host registration + VM-compose command lines are NOT yet confirmed against Canonical's current docs -- the runbook marks each unconfirmed flag/channel/subcommand rather than fabricating one, and a separate verification pass owns filling them in. |

### STAGE 2 WAS THE BRANCH MERGE POINT -- MERGED 2026-07-15 (PR #1, `e468d2e`)

Branch `dc-dc-stage1-phase0-first-exec` merged to `main` at Stage 2 CLOSE, once every close-out
gate bullet (C1..C5) was MET, the gauntlet was ALL GREEN (60) and repo-lint 0-fail. This was the
FIRST VR1 feature merge -- `main` had never taken one (its prior tip was ad-hoc uploads) -- so it
brought D-112..D-120, the Office1 headend, and the NetBox tooling onto trunk. **`main` is now
trunk-of-record for VR1 through Stage 2.** The merged branch was retired.

This established the STANDING **branch-per-stage + merge-at-close** discipline (see "Cross-cutting
discipline" above): every subsequent stage develops on its own branch and merges to `main` (merge
commit, operator-gated) as part of its definition-of-done. Stage 3 branches off the post-merge `main`.

---

**State (history; status authority: docs/CURRENT-STATE.md):** substrate + site both built by 2026-07-14; stage closed and merged to main (PR #1 `e468d2e`, 2026-07-15).

> The 2026-07-13 "not done" list that stood here is **SUPERSEDED and was already stale when
> written** -- MAAS-region, LXD, the LXD-KVM-host registration and both composed service
> machines all landed the same day. This doc's status table has now drifted three times; the
> correction below is measured, not assumed.

- **done (substrate):** `office1-local` + `office1-wan` networks; the **Office1 OPNsense edge is
  BUILT and LIVE** (routing, NAT, egress 0.0% loss, serial console, SSH-key managed, Kea DHCP on
  udp/67). Config is REST-API-managed per D-113(a2). OpenTofu reaches vcloud libvirt. **`voffice1`
  EXISTS and is running** (`module.voffice1` is in `opentofu/terraform.tfstate`).
- **done (site) -- D-114 IS PROVEN END TO END:** MAAS 3.7.2 region+rack ON `voffice1`; LXD 5.21.5
  ON `voffice1`; that LXD **registered back into MAAS as an LXD VM host**; and both service
  machines **COMPOSED BY MAAS** and deployed -- `office1-netbox` (10.10.1.10, NetBox 4.6.4) and
  `office1-tailscale` (10.10.1.11, subnet router for `10.10.0.0/22`, route now ENABLED on the
  self-hosted control server). GitBucket is out of scope (D-116), so Office1's composed machines
  are exactly TWO. Codified in `scripts/site-headend-install.sh` (harness 18/18) -- reusable for
  the DCs.
- **The L3 nesting proof is NO LONGER an unknown -- it is DEFINITIVE.** A guest booted, PXE'd and
  COMMISSIONED three levels deep. DC1 inherits the proof; D-114's model stands.
- **NetBox is populated in `office1-netbox` -- the VR1 IPAM APEX, a working draft (DOCFIX-195)** (D-115
  office carve + the six-plane roles + both DCs, 90 -> 134 prefixes). The apex literals that blocked it
  are now assigned: **D-117** (site naming) and **D-118** (`ORG_ULA_48 = fd50:840e:74e2::/48`).
  **Operator ruling 2026-07-15 (DOCFIX-195):** during VR1, `office1-netbox` (10.10.1.10) takes ALL
  NetBox writes and HOLDS THE IPAM APEX ROLE -- every consuming system (OpenTofu, MAAS, overlays, the
  Juju bundle) derives its NetBox values from it. It is a *working draft* (edited as the buildout
  proceeds), but it is the authoritative NetBox. `netbox.baldurkeep.com` is the production apex, held
  as a READ-ONLY v1 reference draft and NOT consulted by VR1 systems; the write-back to it is DEFERRED
  to end-of-deployment (may not happen) -- it is NOT a Stage 2 gate. So the "NetBox authoritative + populated" gate is met when `office1-netbox` holds
  the complete validated dataset, NOT by any upstream write. See C2 for what "complete" and "verified"
  require (the D-120 additions, plus a fidelity-check re-run -- the earlier "fidelity check clean"
  verdict predates the 2026-07-14 hardening and must be re-established).

**Stage 2 close-out checklist (what must be true before the merge):**

| # | Item | State |
|---|---|---|
| C1 | Office1 LAN IPv6 deployed (`2602:f3e2:f01:100::1/64` + RA on the edge LAN). WAN v6 leg DEFERRED: v6 does not egress the lab. | **done 2026-07-14** -- live and PROVEN ON THE WIRE: address on the kernel, radvd advertising the prefix (`mode=unmanaged`), `voffice1` autoconfigured `...:5054:ff:fe6a:87e5` by SLAAC, edge->voffice1 GUA ping 0.0% loss. NOTE the D-113 AMENDMENT: the ADDRESS half is **not** REST-API-doable (`scripts/opnsense-set-interface-v6.sh`); only the RA half is. `docs/archive/changelogs/changelog-20260714-office1-lan-ipv6-executed.md` |
| C2 | `office1-netbox` (10.10.1.10) is the VR1 IPAM apex -- a working draft (DOCFIX-195) -- and holds the complete validated dataset: D-115 carve + six-plane roles + both DCs' prefixes/aggregates/RIRs, PLUS the D-120 additions -- `10.10.1.0/24`'s child ranges (`ip-ranges`: static .2-.49 / dynamic .100-.200 / node .201-.254) and the two service IP assignments (`ip-addresses`: office1-netbox 10.10.1.10, office1-tailscale 10.10.1.11). `netbox.baldurkeep.com` stays a READ-ONLY v1 reference draft; the write-back to it is DEFERRED to end-of-deployment (DOCFIX-195, operator ruling 2026-07-15) -- NOT a Stage 2 gate. | **done 2026-07-15.** (i) The HARDENED fidelity check was re-run against office1-netbox -> exit 0 (faithful); it also surfaced + fixed a checker scope false-positive (`docs/archive/changelogs/changelog-20260715-fidelity-check-scope-correctness.md`). (ii) The tooling gap was CLOSED -- `prod-draft-dump.py` + `sandbox-fidelity-check.py` now cover `ip-ranges`/`ip-addresses` -- and the D-120 objects (3 bands + 2 service IPs) were loaded into office1-netbox via `netbox/d120-compose-bands.py --commit` and VERIFIED: re-dump + fidelity check -> exit 0, "EXACTLY the planned delta (D-115/D-117/D-118/D-120)." `docs/archive/changelogs/changelog-20260715-d120-load-and-iprange-tooling.md`. Gauntlet ALL GREEN (60). |
| C3 | The `$DC` shell-selector collision resolved. | **done 2026-07-14 (D-119)** -- selectors are REGION-QUALIFIED (`vr0-dc0`/`vr1-dc0`/`vr1-dc1`) across `lib-net.sh`, `lib-hosts.sh`, `opentofu/`, the importer and every runbook call site. Bare `dcN` is REJECTED. The importer's DC->site map is now an IDENTITY, so the off-by-one is structurally impossible. Gap #19 CLOSED. |
| C4 | The NetBox sandbox loop documented. | **done 2026-07-14** -- `docs/dc-dc-netbox-buildout-scope.md` section 8 (rationale + write-guard table + WAF trap) and `runbooks/dc-dc-phase1-office1-standup.md` Step 10b (operational sequence). |
| C5 | This doc's Stage 2 row reconciled against reality (it has drifted three times). | **done 2026-07-14** |

---

## Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

| | |
|---|---|
| **Goal** | From Office1, script each DC's "physical" layer into existence. |
| **Build** | Per DC, D-123 **Model B** (TWO OpenTofu roots + a bootstrap gate between): the OUTER root (vcloud) boots + sizes the containment VM `vvr1-dc0` (~416 GiB, `expose_nested_virt=true`) + the transit + the D-125 vcloud ISP uplink; a gated BOOTSTRAP (`site-headend-install.sh --host-nodes`) makes it a nested libvirt/KVM host AND the MAAS rack; the INNER root (qemu+ssh to `vvr1-dc0`, R-5) creates the 9 node-VM domains (3+2+4, R-3; SHIM, Section 9), the 6 plane networks, and the OPNsense edge INSIDE `vvr1-dc0`. `maas-vm-host` registers `vvr1-dc0`'s OWN (inner) virsh to the Office1 REGION. |
| **Gate** | `vvr1-dc0` up + sized + nested-KVM on (the rack IS `vvr1-dc0`, D-123); inner node VMs boot at depth-4 (D-114/D-123 boot gate); MAAS discovers them; edge routing + the D-125 per-DC ISP egress up (bridge-in isolation test passes); SEC-010 DC-LOCAL enforced; netem parameters applied and measured (D-100 gap #11, still unruled). |
| **Owns** | D-103 (seam, as AMENDED by D-123 Model B), D-100 (edges), D-121 (node layout, R-3 = 9/DC), D-122 (site/edge shape), D-123 (Model B site-down + two-root), D-124 (rack transit), D-125 (per-DC ISP egress). |
| **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** vr1-dc0-first (D-119 naming); vr1-dc1 explicitly hard-gated (HELD, gate G12: transit/rack addressing unruled -- its v4 supernet 10.12.64.0/19 IS assigned by D-115, R3-F11 fix) rather than run with placeholders. Originally walked a set of STOP-gate decisions; most are now RULED (see the rulings summary below): node sizing = D-121/R-3, edge/WAN shape = D-122, site-down/placement = D-123 Model B, rack transit = D-124, ISP egress = D-125, edge-config delivery = D-113(a2) REST (config.xml/ISO retired -- no more `config.xml` tokens). STILL open at execution: MAAS zone/pool + `power_address` (Stage-4 concerns), boot-measured `WAN_IF`/`LAN_IF` (real OPNsense boot), and netem params (D-100 gap #11). NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`; under Model B it targets `vvr1-dc0`'s inner virsh -> Office1 region, DOCFIX-179-deferred) all now exist AND are instantiated across the two Model B roots (D-123): the outer root (`opentofu/`) + the inner root (`opentofu/vr1-dc0-substrate/`), plus the new `modules/wan-bridge` (D-125). Both roots `tofu validate`. Remaining un-instantiated is deploy-time (MAAS zone/pool, `power_address`) not authoring. `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. |

**State:** CLOSED 2026-07-21 for the vr1-dc0 scope (operator-ruled; status authority:
docs/CURRENT-STATE.md sections 1 and 6; stage record
docs/archive/stage-records/vr1-stage3-record.md). vr1-dc1 remains the stage's designed HELD
remainder (gate G12) and returns when its addressing is ruled. Branch merged to `main` and
retired. NOTE the Gate line above predates two recorded corrections: `maas-vm-host`
registration was REFUTED for DCs (per-machine virsh power replaced it -- D-103/D-123
amendments 2026-07-20) and the netem target is the dc0<->dc1 leg (DOCFIX-197).

**Rulings landed (through 2026-07-16; supersedes the earlier "two blocking rulings landed 2026-07-15" note):**
- **D-121 (ADOPTED) + R-3** -- node layout = **Option C** = 3 control + 2 compute + **4 storage = 9
  nodes/DC** (R-3 added the 4th storage node; the earlier "3 storage / 16 VMs" is superseded). Whole-host
  fit re-validated under Model B: **FIT at 870/1024 GiB (154 GiB headroom)** via
  `scripts/dc-dc-whole-host-budget.py`. Vault-HA fork RULED (**R-2**: v-a MySQL-backed; Raft infeasible on
  the 1.8 charm, D-068/BUNDLEFIX-010).
- **D-122 (ADOPTED)** -- Office1-pattern edge (2048/2/nano, REST-API config), dedicated per-site ISP
  uplink, site nested in its own containment VM, per-site MAAS. Its WAN-uplink REALIZATION is superseded by
  **D-125** (bridge-in) and node PLACEMENT by **D-123**; D-122's intent stands.
- **D-123 (ADOPTED Model B)** -- DC nodes nest INSIDE `vvr1-dc0`; a single `virsh destroy vvr1-dc0` =
  site-down. Forces the TWO-ROOT split (a libvirt provider can't target a same-apply VM). Overturns the
  earlier inferred Model A (preserved as `docs/archive/model-a-fallback-plan.md` + git tag `model-a-fallback`).
- **D-124 (ADOPTED)** -- rack transit addressing (office1<->dc0 mesh /30, Scheme A); the entry's rack
  SIZING is VOID under Model B (`vvr1-dc0` now holds the fleet).
- **D-125 (ADOPTED)** -- per-DC ISP egress via bridge-in (resolves OBS-3's design gap; egress efficacy is a
  deploy-time gate).

Steps 4-5 are rewritten to the D-113(a2) REST-API bootstrap; naming is post-D-119. **First-run scope is
now the full outer -> bootstrap -> inner -> maas-register sequence** (the runbook's "D-123 MODEL B RESHAPE"
section). The per-DC MAAS rack source is NO LONGER a missing-module gap: `vvr1-dc0` IS the rack, stood up
by `site-headend-install.sh --host-nodes` (a bootstrap SCRIPT, not a tofu module) between the outer and
inner applies. The stale `vr1-dc1` "supernet not assigned" gate is corrected (D-115 assigned
`10.12.64.0/19`); vr1-dc1 stays out of scope by sequencing.

**Blocking to EXECUTE (all operator-gated):** the HELD NetBox addressing -- **D-124's rack transit/30 +
rack metal-admin IP** (via office1-netbox, apex token; D-125 needs NO new NetBox object -- its /24 is the
ruled `172.30.2.0/24`, OPNsense WAN stays `.2`); then the gated applies; then the deploy-time gates
(depth-4 nova boot D-114/D-123; D-125 foreign-MAC egress isolation
test; SEC-010 packet-drop).

---

## Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

| | |
|---|---|
| **Goal** | Bring the discovered node VMs to a deployed OS, ready for Juju. |
| **Build** | Per DC: MAAS commissions + deploys the node VMs; provider NIC left RAW (br-ex discipline, D-100); VLAN-103 metal-internal bridge stack pre-built by MAAS. |
| **Gate** | Nodes READY + carved + tagged (NOT "deployed" -- MAAS-deploy is SKIPPED per the 2026-07-23 READY-handoff ruling / DOCFIX-200; Juju provisions at Stage 5); six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC artifact source answering on its own address with an attested-current sync (dc0 = D-135 item-1 mirror, dc1 = the D-135-AMENDED apt caching proxy -- the two DCs differ BY RULING). Node-side reachability and the node time source are gate **G17**, split out 2026-07-27 (operator ruling, GA-R5) because powered-off nodes cannot be probed in this stage. `NTP from edge` was STRUCK -- superseded by D-129(iv) 2026-07-21, which keeps the MAAS hierarchy as time authority and gives the edge no NTP role (DOCFIX-204). |
| **Owns** | D-103, D-100 (br-ex), D-107 (mirror + NTP). |
| **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing now RULED (D-121/R-3 = 9/DC), so that sub-gap is closed; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md).

---

## Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

| | |
|---|---|
| **Goal** | A running independent OpenStack cloud per DC. |
| **Build** | Per DC: bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle; Vault init with a per-DC independent root, issuing v4 + IPv6-SAN certs; apply the family matrix (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider). |
| **Gate** | `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC; controller backup taken + stored; Ceph-over-v6 and geneve-over-v6 verified. |
| **Owns** | D-104 (controllers), D-109 (Vault), D-101 (families). |
| **Reuse vs new** | **This is the ground-truth/copy point** (`docs/dc-dc-buildout-design.md` line 12: seeded from `openstack-caracal-ipv4` at 8813efc). `bundle.yaml` and `runbooks/phase-01..08-*.md` are the per-DC template -- run essentially the existing VR0 DC0 deploy loop TWICE (once per DC), adapted for: independent Keystones (no cross-DC federation), the IPv6 family matrix (D-101/D-102), and per-DC Vault roots (D-109) instead of the single shared root. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`.** Does not duplicate `phase-01..08`'s content -- describes what changes to run that loop twice (once per DC): DC1 needs NO VIP/CIDR edits at all (D-101 inherits DC0 unchanged); DC2 is hard-blocked pending its NetBox literals. Requires every adapted `phase-01..08` command block that reads `lib-net.sh`/`lib-hosts.sh` to call `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` first (DOCFIX-151). Flags a real, unresolved design gap: the D-101 IPv6 family-matrix overlay (dual-stack/ULA-only per-plane charm config) does not exist yet -- the exact charm config option names for v6 addressing have not been confirmed against any charm's real schema, so no overlay file is invented; Step 6 names this as blocking work rather than gliding past it. Also flags D-109's IPv6-SAN cert-issuance support as an unverified charm-behavior assumption. NOT YET EXECUTED. |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md). **This is where the existing repo tooling (`preflight.sh`,
`cloud-assert.sh`, `scripts/lib-net.sh`/`lib-hosts.sh`, `run-logged.sh`) does the
most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they
apply to either DC's clone/session without modification.**

---

## Stage 6 -- Phase 5: DR wiring and failover drill

| | |
|---|---|
| **Goal** | Prove Tier-A recoverability, one-way first, then two-way. |
| **Build** | Replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones); rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency groups where needed. Run the failover/failback drill. |
| **Gate** | One-way drill clean (RTO/RPO measured); then two-way; controller restore drill (D-104) exercised; RTO recorded as a measured output, not an SLA. |
| **Owns** | D-108 (replication), D-104 (control-plane backup/restore in the drill). |
| **Reuse vs new** | Ground truth: `docs/archive/dc-dc-replication-DR-seed.md` (the original Tier-A/bidirectional/cinder-backup+rbd-mirror decision) -- D-108 corrected its carrier from the seed's v4 replication plane to IPv6-only ULA. The failover/failback drill skeleton is already drafted (buildout design Section 8): confirm DC1 down -> promote Glance -> restore Cinder -> rebuild instances -> record RTO/RPO; failback is split-brain-safe (no auto-resume, controlled window to flip primary back). |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase5-dr-failover-drill.md`.** Command-level implementation of Section 8's failover/failback skeleton, staged one-way-then-two-way per D-108's own requirement, plus the D-104 controller backup/restore drill. Found and flagged three real bundle gaps while writing it (grepped `bundle.yaml`): no `cinder-backup` charm deployed, no `ceph-rbd-mirror` charm deployed, and `ceph-mon`'s existing `rbd-mirror` endpoint binding is on `storage` instead of `replication` (inherited from the VR0/DC0 seed, predates D-108) -- all three become this runbook's own first gated mutation (Step 2), not silently routed around. Every realm/zone/pool/endpoint name is an explicit `<UPPER_SNAKE_TBD>` placeholder. Includes a negative-control sub-drill (sever only the replication link first, confirm the split-brain check correctly refuses to promote) before the real hard-down drill. NOT YET EXECUTED -- no live Ceph cluster this session (harness for gap #5's scripts is future work, logged, not built). |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md).

---

## Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

| | |
|---|---|
| **Goal** | Complete the service set in both DCs. |
| **Build** | Per DC: Designate (designate-bind) with per-DC zones + A/AAAA per the D-008 bootstrap order; COS observability; Magnum/CAPI with in-cloud single-homed capi-mgmt (`domain-setup` re-run per D-046). |
| **Gate** | Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); COS scraping; a CAPI workload cluster comes up per DC. |
| **Owns** | D-106 (Designate), D-105 (CAPI/COS). |
| **Reuse vs new** | HIGH REUSE for CAPI/Magnum: `runbooks/phase-06-incloud-mgmt-cluster.md` + `phase-07-conductor-graft.md` + `phase-08-workload-cluster-acceptance.md` are direct precedent, per-DC. Designate is a REVERSAL, not new build: VR0 DC0 deferred it (D-019, "DNS scope reduction"); D-106 reactivates it (supersedes D-019, reactivates the original D-008 bootstrap order) -- read D-008 and D-019 together before writing this stage's runbook, the static-`/etc/hosts` workaround they document goes away. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase6-designate-cos-magnum.md`.** Quotes `bundle.yaml`'s and `phase-01-bundle-deploy.md`'s actual current "NO designate (D-019)" text verbatim rather than paraphrasing, then the D-106 reversal against it. Flags that the reversal is bigger than "add designate" -- D-106's bootstrap order also reactivates `os-public-hostname` per API charm, reversing the bundle's separate B5 IP-only posture; proposes (not silently adopts) pushing per-DC hostname literals into new `overlays/vr1-dc0-hostnames.yaml`/`vr1-dc1-hostnames.yaml` files to keep `bundle.yaml` itself DC-agnostic, mirroring the existing octavia-pki overlay pattern (this cell said `dc1-hostnames.yaml`/`dc2-hostnames.yaml` until DOCFIX-205, 2026-07-30 -- the retired pre-D-117 spelling, which the runbook itself had already been corrected to the region-qualified form; under D-119 `${DC}` IS `vr1-dc0`, so `${DC}-hostnames.yaml` is now the correct interpolation). D-046's domain-setup trap quoted verbatim in a top callout and again inline. COS deployment mechanism explicitly left as an operator decision point (D-105 rules only the scope) rather than invented. Reuses `phase-06/07/08` by filename with concrete DC-parameterization notes (confirmed via tracing which scripts source `lib-net.sh`). NOT YET EXECUTED. |

**State:** runbook written, not yet executed (status authority: docs/CURRENT-STATE.md).

---

## Forward items -- DEFERRED BY RULING

Each item carries its OWN trigger. F1-F3 are deferred to the NEXT deployment; **F4 is due at
the END OF THIS ONE.** Check the trigger before assuming an item is out of scope for the
current work.

Named forward items live HERE, not as D-numbers. GA-R3 rule 1(b) as amended by A1 is
explicit: "Mentioning Roosevelt, or recording a Roosevelt-era preference, does not
qualify; changing what gets built or how it transfers does." An item below is a
preference for a FUTURE build with no present-tense half, so it is deliberately not in
the D-series. If a later deployment turns one into something that changes what gets
built -- a gate, a bootstrap order, a topology -- it earns a D-number on its own merits
AT THAT TIME. Status tokens do not appear here (GA-R1); this section records intent.

### F1. Regional NOC posture: Office1 -> DC0-as-NOC -> remote push to DC1

**The test this deployment was originally conceived to run**, described by the operator
2026-07-27 and found to exist in NO repo surface (`grep -rniE '\bNOC\b'` over `docs/`,
`runbooks/`, `scripts/`, `bundle.yaml` returned zero hits outside `docs/archive/`):

1. Office1 stands up first, simulating a real regional office.
2. Dark-fiber connectivity to DC0; the DC0 infrastructure is deployed REMOTELY (it may
   be built from vcloud where needed, since in the real world that hardware would have
   been racked and set up at DC0 already).
3. Once DC0 is online with a FULL set of OpenStack servers/services bootstrapped, **DC0
   is considered the NOC**.
4. The test proper: simulate a technician at their desk in Office1 connecting to the
   NOC and pushing a deployment to DC1.

**Why it is deferred.** The deployment steps for it were never in this roadmap, and by
the time that was found this deployment had progressed too far to re-sequence. Operator
ruling 2026-07-27 (GA-R5; utterance quoted in `docs/CURRENT-STATE.md`): step away from
the DC0-then-DC1 roadmap for THIS deployment, take the posture on the NEXT one after
this push, configuration and hardening complete. Both DCs deploy from voffice1 by the
same procedure instead, with per-DC overlays.

**What already supports it, so the next deployment does not start cold.** D-100 rules
the dark-fiber triangle and records that the Office1<->DC fiber carries management
traffic only (MAAS/Juju/operator) -- the exact path step 4 exercises. D-128 already
puts Plane-2 execution (MAAS / NetBox / inner tofu / tenant tests) ON voffice1, so
"a technician at Office1 pushes a deployment" is the operating model this deployment is
ALREADY using; what is missing is the DC0-first ORDER and a gate that proves the push
happened from there rather than from the jumphost.

**What it would need to become build-changing** (and therefore D-admissible): a named
executable check that the DC1 deployment was driven from Office1 against a DC0 NOC, and
a ruled ordering that makes DC0's full bootstrap a precondition of DC1's start.

RULING (GA-R5, 2026-07-27). Question as presented: does the deferred NOC posture get its
own D-number, or fold into D-100 as a sub-item -- noting GA-R3's A1 amendment appears to
exclude a Roosevelt-era preference from the D-series, and that the session ledger is the
wrong home because GA-R4 caps and rotates it. Operator answer, exact utterance: **"use
the workflow doc as a named forward item"**.

### F2. Render-pipeline AUTOMATION half (CI runner, event delivery, status-back)

**CARVE-OUT AGREED BY THE OPERATOR 2026-07-27**, when the render-pipeline tool buildout was
scoped. The renderer itself is built now; this half is DEFERRED. Pinned here because it is a
REQUIRED item later and would otherwise survive only in a session transcript.

**What is deferred:** the automated path around the renderer -- a CI job that renders on change,
delivers the event reliably, and writes validator verdicts back onto the merge request.

**Why (both reasons are measured gaps, not caution).** Per D-136 open question 5, TWO
requirements are **NOT YET VERIFIED against the running GitBucket instance**, and neither may be
promoted to a must-have until checked:

- **(a) Reliable event emission** -- webhook delivery with retry, a delivery log, and manual
  replay. A silently dropped webhook renders nothing and nobody notices, which is WORSE than no
  automation at all.
- **(b) A status-back API** -- so the runner can write the `provider-bundle-check` and
  `repo-lint` verdicts onto the merge request as commit status/checks. This is the requirement
  that PROTECTS the approve-on-rendered-diff gate: without it the gate cannot see the validator
  verdict, and a human approving a rendered diff is approving output no gate has judged.

**PLACEMENT IS ALSO OPEN** and needs an operator ruling when this is taken up: the runner is
**Jenkins**, which is operator-side and already running (v1 K8s + app deploys) -- nothing needs
standing up and no git-host/Actions-style runner is required. The candidate placement is a
Jenkins job on `voffice1` for VR1 (D-128 already routes Plane 2 there, and the render reads
Office1-side NetBox + git), with per-DC utility-node agents as the Roosevelt shape once D-132
ratifies -- BUT D-132 is PROPOSED and per-DC utility nodes are NOT built in VR1. Also unresolved:
whether an Office1 job emitting BOTH DCs' overlays brushes D-105's cross-DC roll-up prohibition.

**Interface to freeze now so this lands cleanly later:** the renderer is a deterministic
command-line tool whose inputs are the per-DC values file and whose outputs are committed,
reviewable artifacts. Nothing about automating it should require changing the renderer.

**Carry-forward trigger:** this is the revisit point for the deferred "Jenkins CI poll job" at
`docs/handoff-20260703-open-items.md:159`.

### F3. NetBox scope boundary for MACs and VLANs -- PERMANENT, not pending

**CARVE-OUT AGREED BY THE OPERATOR 2026-07-27.** Recorded as a forward item so a future session
does not read "the apex is the source of truth" and try to close a gap that is not one.

- **MAC pins:** the apex holds NO `dcim/devices` and NO `dcim/interfaces` (measured in
  `netbox/draft/vr1-office1-current-20260725.json`: the record carries only `ipam/*` plus
  `dcim/regions` and `dcim/sites`). There is therefore no NetBox front half to build for MACs.
  They stay sourced from `scripts/lib-hosts.sh` and become renderer OUTPUT into the per-DC
  overlay. See the D-136 correction of 2026-07-27 -- the earlier claim that MACs were
  "drift-free across substrate main.tf <-> lib-hosts.sh <-> bundle ovn-chassis <-> discovery"
  is FALSIFIED and the reason MACs are scoped out is availability, not drift-freedom.
- **VLANs / spaces / fabrics:** correctly absent and PERMANENTLY out of apex scope. VR1 is
  untagged-per-fabric on every plane in both DCs (D-133, superseding VR0's VID-103/104), so a
  VLAN object would document tagging that does not exist on the wire. The bundle binds by SPACE
  NAME -> MAAS subnet by CIDR, never by VLAN/fabric/subnet id. This layer is MAAS-sourced;
  NetBox is IPAM only.

### F4. IPv6 GUA host-part packing -- REVIEW AT THE END OF *THIS* DEPLOYMENT

**TRIGGER: end of THIS deployment, not the next.** Unlike F1-F3.

**Owed by ruling, 2026-07-27** (operator: "mirror the v4 octet for both legs but make a note
to review at the end of the project to review for adjustment"). The ruling is recorded under
D-134's R4 amendment in `docs/design-decisions.md`, which is the authority.

**What was ruled:** the v4 octet is mirrored as the IPv6 host part on BOTH leg types -- the
ULA metal legs AND the GUA provider leg. keystone at v4 `.50` is `::50` everywhere.

**What to review, and why it is worth a look:** the GUA provider leg has a DEDICATED VIP
`/64` per DC (`2602:f3e2:f02:11::/64` dc0, `f03:11::/64` dc1) with no node statics competing
for space. Mirroring the octet means that block is used from `::50` upward and `::1`-`::49`
sit unused. That was chosen deliberately -- ONE rule for every plane, so a VIP's host part is
predictable from its v4 octet and checkable by eye and by gate -- rather than carrying a
per-leg special case in the renderer. The review question is whether, with the cloud live and
v6 behaviour observed, the GUA leg should instead pack from `::1`.

**Cheap to change, by construction:** because the convention is uniform, adjusting it is a
renderer INPUT change plus a re-render, not a redesign. That is the main reason the uniform
rule was affordable to adopt first and review later.

**Related:** G18 (also project-close blocking) covers a different apex question -- whether the
charm-created Octavia `lb-mgmt-net` prefix is back-filled into NetBox. Do not conflate them.

---

## Cross-cutting discipline (applies at every stage, every session)

- **Session bootstrap, every session:** `git pull` -> `bash scripts/repo-lint.sh`
  (0 fail / 1 documented legacy WARN expected) -> `bash scripts/ledger-scan.sh`
  to reconcile open decisions/next-free numbers against `docs/session-ledger.md`.
- **Any session that will mutate the cloud:** `bash scripts/run-logged.sh <label>`
  first (as-executed log).
- **Operating model (D-128):** Claude runs ON the vcloud jumphost -- do not relocate
  the session to a workstation. **Plane 1** (substrate -- outer `tofu` `qemu:///system`
  + local `virsh`, building the DC *host* VMs) runs on vcloud (no Roosevelt analog);
  **Plane 2** (MAAS / NetBox / the inner `tofu` root `qemu+ssh`-from-Office1 / tenant
  tests) EXECUTES on `voffice1` via `ssh voffice1 '...'`, so runbooks read "from the
  Office1 headend" and transfer to Roosevelt. Durable jumphost->Office1 reach =
  `scripts/site-baseleg.sh` (D-126 amendment); run `check office1` if `ssh voffice1`
  is dead after a reboot. The workstation tailnet (D-107) is the human GUI path only.
- **Before any add-model-like mutation:** `bash scripts/preflight.sh` PASS.
- **After any deploy/restart/change, and as an incident-triage first step:**
  `bash scripts/cloud-assert.sh` (add `--capture` at real deploy-completion
  milestones for the committed BOM).
- **Before touching ANY built surface:** grep `docs/design-decisions.md` for the
  governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present
  options, do not implement.
- **Every deliverable (script, runbook, doc):** `repo-lint` clean, the touched
  script's harness green (write one if missing), a changelog entry with a
  per-item revert, and a `docs/session-ledger.md` update.
- **Branch-per-stage + MERGE AT STAGE CLOSE (standing, from Stage 2 on):** each
  stage is developed on its own branch (`dc-dc-stageN-...`). Merging that branch to
  `main` is PART OF the stage's definition-of-done, not an afterthought: when every
  close-out gate bullet is MET, the gauntlet is ALL GREEN and repo-lint is 0-fail,
  the branch is merged to `main` via a PR with a **merge commit** (NOT squash --
  preserve the per-item revertable history + changelogs), then the merged branch is
  RETIRED (local + remote). **The stage close ALSO SWEEPS THE OPERATING SKILL**
  (`.claude/skills/openstack-cloud-ops/`): fold in any new INVARIANT the stage
  established and reconcile any divergence from repo HEAD, in the same close (facts
  delegate to the repo and go stale; discipline/invariants live in the skill and must
  be kept current). The merge is an operator-gated action (present it, do not run it
  unasked). `main` is trunk-of-record only for stages that have merged; the next stage
  branches off the post-merge `main`. Precedent: Stage 2 -> PR #1 (`e468d2e`), 2026-07-15.
- **Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN:** `bash
  scripts/ledger-scan.sh` for next-free (it is the numbering authority, not
  repo-lint's informational count).
- **Hard rule reminder:** no inferred values -- every IP/ID/name/scope is
  measured this session or carried from confirmed as-built; verify before
  mutate, always; destructive/secret-handling steps are individually gated,
  never batched.

---

## Tooling gap register (audited 2026-07-09; update as items close)

> ### !!! STALE-CONTENT WARNING (2026-07-13) -- READ BEFORE ACTING ON ANY OPNSENSE ITEM BELOW !!!
>
> Everything in this document describing OPNsense edge config delivery via a rendered
> `config.xml` and/or a **config ISO** is **OBSOLETE and, against the live Office1 edge,
> DANGEROUS**:
>
> - **The config ISO is INERT (D-112).** The OPNsense Configuration Importer can NEVER fire on a
>   pre-installed nano image -- it probes for a read-only root, finds a writable one with a
>   factory `/conf/config.xml`, and exits without enumerating any device. Nothing ever reads it.
> - **Full-`config.xml` delivery is SUPERSEDED (D-113(a2), ruled + PROVEN 2026-07-13).** Edge
>   config is done over the **REST API** (`scripts/opnsense-api.sh`), proven end to end -- read
>   AND write -- against the live Office1 edge.
> - **Office1's DHCP is now API-MANAGED.** A full rendered `config.xml` push to `10.10.0.1` would
>   CLOBBER live state (it replaces `/conf/config.xml` wholesale, dropping ~667
>   migration-populated elements including the only 2 firewall pass rules).
>
> The as-built path that actually worked: prep image -> boot on factory defaults -> **D-112(c)**
> console bootstrap -> enable SSH + install key -> configure over SSH and the REST API. See
> `docs/archive/changelogs/changelog-20260713-opnsense-api-proven.md` and
> `docs/archive/changelogs/changelog-20260713-opnsense-api-write-proven.md`.
>
> **done (D-113(a2)) -- reconciled 2026-07-16; the repo won over this register's stale "OPEN".** The
> config-ISO path and the module's `config_seed`/cdrom wiring are RETIRED: `modules/opnsense-edge` no
> longer takes `config_iso_path` (verified -- no `config_seed`/`config_iso_path` anywhere in the module),
> `opentofu/templates/opnsense-config.xml.tmpl` is DELETED, and Stage 3's edge steps (phase-2 Steps 4-5)
> are REWRITTEN around the REST API (`scripts/opnsense-api.sh`, harnessed). So the OPNsense steps in the
> phase-1/phase-2 RUNBOOKS are the API path and are NOT reference-only. **Caveat:** THIS document's own
> prose below (and other gap items) still describe the old `config.xml`/ISO delivery in places -- treat
> **this workflow doc** as the stale surface and the runbooks + module as authoritative until a full
> gap-register reconciliation lands.

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not
inferred from the design doc alone. Ordered roughly by "you'll hit this first,"
not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;
#2 is the single largest build.

1. **`$DC` selector convention -- CLOSED 2026-07-09 (DOCFIX-151).**
   `scripts/lib-net.sh` gained `lib_net_select_dc()` (dc0/dc1 no-op, since
   D-101 rules DC1 inherits DC0's v4 layout unchanged; dc2 fails loud --
   NetBox literals not yet assigned, gap #3) and `scripts/lib-hosts.sh`
   gained `lib_hosts_select_dc()` (dc0 no-op; dc1 AND dc2 BOTH fail loud --
   no real per-DC host enrollment exists yet for either, an intentional
   asymmetry vs. the net selector, documented in both files' comments and
   `docs/archive/changelogs/changelog-20260709-dc-selector-convention.md`). Backward compatible
   by construction -- every existing caller of either library is unaffected.
   New harness `tests/dc-selector/run-tests.sh`: 21/21 PASS. The mechanism
   exists; Stage 5's runbook still needs to actually CALL
   `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` when it's
   authored -- that call-site wiring is part of writing that runbook, not
   this delivery.
2. **OpenTofu** -- **STATUS CORRECTED 2026-07-13. The "UNVALIDATED / no tofu
   binary" framing below is now FALSE and must not be relied on:**
   - **A `tofu` binary EXISTS on the jumphost** (version pin authority:
     `docs/CURRENT-STATE.md` section 7 ONLY). The tree can self-check. The whole "authored in a session with no binary" caveat that
     rides along with every OpenTofu instruction in this repo is now closed as
     a *capability* statement.
   - **The tree VALIDATES: 10/10 modules**, root + every module standalone
     (`bash scripts/opentofu-validate.sh`).
   - **It has also been APPLIED for real.** `opentofu/terraform.tfstate` holds
     11 resources: the DC1 planes, the pools, the three mesh links, the
     `office1-network`, and the Office1 OPNsense edge domain + volumes. This is
     no longer a paper tree.
   - **BUT two modules were BROKEN and had NEVER been parsed by any tool**
     (DOCFIX-194, 2026-07-13): `node-vm` (`libvirt_volume` with a bogus
     top-level `format` -- it nests under `target`) and `netem-link` (a
     destroy-time provisioner referencing `var.*`, which OpenTofu rejects at
     INIT -- the module could not even initialize). Root's `main.tf` does not
     call either one, and `tofu validate` on the root module never parses an
     uncalled module -- so the gate printed **PASS** over both. Both are fixed;
     the gate now validates EVERY module standalone (S3). **`node-vm` builds
     every DC node and `netem-link` is the Stage 6 failover-drill mechanism** --
     they would have detonated mid-deploy. See
     `docs/archive/changelogs/changelog-20260713-docfix194-opentofu-module-validation.md`.
   - **`modules/cloudinit-vm` IS NOW INSTANTIATED FOR REAL (2026-07-13, D-114).**
     The long-standing "the mechanism exists but no image source is chosen and no
     `user_data`/`meta_data`/`network_config` has been designed for it" caveat --
     which is what forced Stage 2's old Option A / Option B fork -- is **CLOSED
     for `voffice1`**: `module "voffice1"` in root `main.tf` carries a real image
     source (`modules/base-image` -> the official Ubuntu 24.04 noble cloud
     image), real cloud-init content, and a `network_config` that DHCPs off the
     Office1 edge's Kea (matching the NIC by glob, not by a guessed kernel name).
     The module also **gained an `expose_nested_virt` variable** -- load-bearing,
     because LXD virtual machines are qemu/KVM guests and D-114's composed
     service machines cannot boot without the host CPU feature passed through.
     Applied; the domain is running. **The three Office1 service VMs are NOT
     cloudinit-vm targets any more at all** -- under D-114 they are MAAS-COMPOSED
     LXD VMs, so no further `cloudinit-vm` design work is owed for them.
   - **Still true:** `tofu plan`/`apply` has NOT been exercised for the Stage 3
     DC substrate. Validation is not application.

   *Original 2026-07-09 text follows (kept for the module inventory; its status
   claims are superseded by the block above).*

   **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded
   2026-07-09 (was: zero files).** `opentofu/` now has: `modules/dc-planes`
   (six per-DC planes), `modules/mesh-link` (the D-100 dark-fiber triangle
   legs), `modules/dc-storage-pool`, `modules/node-vm` (blank disk, PXE-boot
   MAAS-managed node VM, the D-103 pattern for Stage 3), `modules/base-image`
   + `modules/cloudinit-vm` (the OTHER VM pattern -- pre-built image +
   cloud-init, for Office1's own service VMs), `modules/opnsense-edge` (see
   gap #4), `modules/maas-vm-host` (registers the vcloud host with MAAS as a
   virsh VM host -- via the official `canonical/maas` provider, confirmed
   against its real schema; deliberately NOT `maas_vm_host_machine`, which
   composes/creates new VMs itself and would fight `node-vm` for VM
   creation -- D-103 explicitly rules out MAAS composing), and
   `modules/netem-link` (`terraform_data` + SSH-wrapped `local-exec`
   provisioners applying `tc qdisc ... netem` to a mesh-link bridge --
   OpenTofu runs from Office1, not the vcloud host, so a bare local command
   can't reach the bridge). DC1 + Office1 pool/plane wiring is instantiated
   in root `main.tf`; everything else is NOT (node/VM sizing, MAAS zone/pool,
   netem parameters are all pending decisions -- neither invented here).
   Provider schema was confirmed against the actual current docs AND real
   example `.tf` files (not memory) -- it differs materially from older
   common examples: nested structures are attribute-style objects (`key =
   { ... }`), not classic HCL blocks. **A real bug from the first pass was
   found and fixed:** `dc-planes`/`mesh-link`/`dc-storage-pool` originally
   used block syntax for `libvirt_network`'s `domain`/`mtu` and
   `libvirt_pool`'s `target`; corrected once real provider examples surfaced
   the actual convention -- discovered precisely BECAUSE building the domain
   module forced fetching real example code instead of doc-summaries alone.
   Still missing, deliberately deferred (see `opentofu/README.md`):
   OPNsense's own bootability via `cloudinit-vm` is NOT confirmed (FreeBSD,
   not the same cloud-init/NoCloud guarantee as Linux images); the
   per-device `boot`-order attribute's exact internal shape (flagged
   UNVERIFIED in `node-vm/main.tf`, an inference by pattern not a confirmed
   field); and whether `devices.interfaces` list order reliably maps to
   LAN/WAN NIC enumeration (flagged in `opnsense-edge/main.tf`).
   **UNVALIDATED** -- no `tofu` binary was available to run
   `scripts/opentofu-validate.sh` this session; run it on a connected
   machine before trusting any of this.
3. **NetBox multi-DC/dual-stack import pipeline -- MECHANISM CLOSED
   2026-07-09 (DOCFIX-152), literals still pending.** `netbox/dc-dc-prefixes-
   import.py` extends the v1 pattern to VR1's two-DC dual-stack model:
   per-DC (`--dc dc1`/`dc2`) creates the six-plane IPv4 (DC1 inherited,
   hardcoded per D-101; DC2 via required `DC2_V4_SUPERNET`, no default) plus
   the D-101 family-matrix IPv6 legs (ULA/GUA), carved from two REQUIRED
   env vars (`ORG_ULA_48`, `DC_GUA_PREFIX`) with no defaults -- fails loud
   rather than inventing either. 40/40 tests (`tests/dc-dc-prefixes-
   import/`, fake-NetBox end-to-end). UNVALIDATED against a real NetBox (no
   instance reachable this session). The literals themselves (org ULA /48,
   per-DC GUA carve, DC2 supernet) still don't exist -- that operator/NetBox
   assignment step is unchanged by this delivery; this closes the TOOLING
   gap, not the DATA gap. Still blocks Stage 2/3 until those literals are
   assigned and this script is actually run against a live NetBox.
4. **OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) done
   2026-07-09, (d) is the only real gap left.** Each site (Office1, DC1, DC2)
   gets its own independent OPNsense edge with a simulated ISP uplink (D-100);
   OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge)
   and Stage 3 (per-DC edges) gates, which require netem parameters "applied
   and measured."
   - **(a) Network substrate -- done.** `opentofu/modules/mesh-link` creates
     the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an
     OPNsense VM's interfaces would eventually attach to.
   - **(b)+(c) The OPNsense VM/image AND its config -- BUILT
     (`opentofu/modules/opnsense-edge`), UNVALIDATED.** Cloud-init is
     confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum
     consensus, not just a hedge. The real native mechanism is the
     Configuration Importer (official docs): scans an attached volume for
     `/conf/config.xml` in a 2-3 second boot-time window; ISO9660 support was
     added to it specifically for VM/cloud automation (attach a config ISO as
     a secondary CD-ROM) -- mechanically IDENTICAL to `modules/cloudinit-vm`'s
     shape (a secondary volume attached as `cdrom`), just a different payload.
     Built as `opentofu/modules/opnsense-edge` + two prep scripts
     (`scripts/opnsense-prep-image.sh` for the nano image's decompress+
     convert, `scripts/opnsense-build-config-iso.sh` for the config.xml ISO).
     The real `config.xml` DESIGN is now also done:
     `opentofu/templates/opnsense-config.xml.tmpl` (a `{{TOKEN}}`-parameterized
     template built from OPNsense's own real shipped sample, covering
     interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall
     rules) + `scripts/opnsense-render-config.sh` (the renderer -- fully
     tested end-to-end, 8/8, unlike the other opnsense-* scripts, since it
     needs no external tool). UNVALIDATED regardless: no `tofu` binary, no
     ISO-building tool, and no real OPNsense boot were available to confirm
     the whole chain end-to-end. Several template tokens still can't be
     filled with real per-site values -- pending Stage 0 ratification
     (D-100/D-101/D-107) or only measurable on a real boot (which `vtnetN`
     device a network becomes). Full sourcing in `opentofu/README.md` and
     `opentofu/templates/README.md`'s token legend.
   - **(d) `tc netem` WAN-simulation -- mechanism BUILT (`modules/netem-link`),
     only the real parameters remain.** No Terraform/OpenTofu-native
     equivalent exists (confirmed against the `libvirt_network` schema, gap
     #2); built instead as a `terraform_data` + SSH-wrapped `local-exec`
     provisioner (OpenTofu runs from Office1, not the vcloud host where the
     bridge lives -- a bare local command can't reach it). UNVALIDATED, not
     instantiated. The exact parameters (latency/jitter/loss/rate) are
     themselves still an unruled D-100 open sub-item (gap #11) -- **a
     concrete PROPOSAL now exists** (`docs/dc-dc-netem-and-ula-gua-
     proposal.md`, DOCFIX-168, Part 1: 1ms/0.2ms jitter/0.01% loss/uncapped
     rate, with reasoning, presented for operator ratification, not
     silently applied) -- still needs an actual ruling before it can be
     applied for real.
5. **Ceph replication tooling (radosgw multisite + rbd-mirror) -- MECHANISM
   CLOSED 2026-07-09 (DOCFIX-165), data half still open.** `runbooks/dc-
   dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and 11.2-11.4's
   `radosgw-admin`/`rbd mirror` command sequences (verbatim upstream Ceph
   administration, not invented) are now callable, parameterized scripts:
   `scripts/dc-dc-radosgw-multisite.sh` (subcommands `master-init`/`join-
   readonly`/`enable-two-way` -- Step 4 one-way staging + Step 8's radosgw
   half), `scripts/dc-dc-rbd-mirror.sh` (subcommands `bootstrap-primary`/
   `bootstrap-secondary --direction rx-only|rx-tx` -- Step 5 one-way staging
   + Step 8's rbd-mirror half), and `scripts/dc-dc-dr-drill.sh` (subcommands
   `failover` -- Step 10.2 force-promote + a flagged, never-auto-executed
   Glance re-registration reminder -- and `failback` -- Steps 11.2-11.4, with
   the demote-CURRENT-PRIMARY-before-promote-RECOVERING safety ordering
   HARD-CODED as fixed call order, not left to documentation). Every
   realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument, no
   invented defaults; default mode is a `--dry-run` plan print, `--apply`
   executes one command at a time via `juju ssh`/`juju run`, aborting on
   first failure. New harnesses `tests/dc-dc-radosgw-multisite/`,
   `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23
   PASS) test ONLY these scripts' own argument parsing, guard clauses,
   dry-run output, and (critically) the failback step ORDER -- NOT real
   radosgw/rbd-mirror behavior, since no live Ceph cluster exists this
   session to test against (same UNVALIDATED posture as gap #3's NetBox
   pipeline). The MECHANISM exists; the DATA half (real DC1/DC2 Ceph
   cluster, real names/endpoints, the three `bundle.yaml` gaps in the Stage
   6 runbook's own "Known gaps" section) is unchanged and still blocks Stage
   6 for real. See `docs/archive/changelogs/changelog-20260709-ceph-replication-tooling.md`.
6. **Designate reactivation is a bundle change -- CLOSED 2026-07-10
   (DOCFIX-167).** `bundle.yaml` now carries `designate`/`designate-
   bind`/`designate-mysql-router`/`designate-hacluster` + 8 relations,
   researched against designate's real metadata.yaml (a real mistake
   caught during authoring: its provides-side endpoint is `dnsaas`, not
   `public` like the other API charms -- corrected). `os-public-hostname`
   re-activation (D-106's OTHER coupled change) deliberately NOT done in
   the shared bundle -- pushed to a proposed per-DC overlay instead, to
   keep `bundle.yaml` DC-agnostic. `python3 scripts/provider-bundle-
   check.py` PASS, all 6 existing invariants unaffected. See
   `docs/archive/changelogs/changelog-20260709-designate-cinderbackup-rbdmirror.md`. NOT
   applied to any live model -- repo-source change only.
7. **MTU/geneve budget + Ceph disk-budget calculators -- CLOSED 2026-07-09
   (DOCFIX-162).** D-102 and buildout-design Section 3 both describe
   measurement-then-computation gates (host L2 MTU -> geneve-over-v6
   overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback)
   that were manual arithmetic in prose, not a script, blocking the Stage 1
   Phase-0 gates. `scripts/dc-dc-mtu-geneve-budget.sh` takes the MEASURED
   underlay MTU via required `--underlay-mtu` (no default) and reproduces
   D-101's own worked example exactly (1500 - 56 = 1444), with a jumbo
   (>=9000) path leaving tenant MTU at 1500, plus an added IPv6-minimum-MTU
   (1280, RFC 8200) sanity floor clearly separated from D-101's own text.
   `scripts/dc-dc-ceph-disk-budget.sh` takes MEASURED total disk, per-DC
   OSD node counts and per-node footprints, and a required
   `--backup-overhead-fraction` (deliberately NO default -- not specified
   as a hard number anywhere in this repo, must come from real
   Ceph/radosgw-admin measurement once DC1 exists) and computes whether
   size=3 across both DCs plus overhead fits the measured total disk;
   if not, it names size=2 as the ONLY documented fallback (D-101 /
   Section 3) without silently applying it -- explicitly an operator-logged
   decision. New harnesses `tests/dc-dc-mtu-geneve-budget/run-tests.sh`
   (19/19 PASS) and `tests/dc-dc-ceph-disk-budget/run-tests.sh` (16/16
   PASS). Both scripts are read-only/mutate nothing and have not yet been
   run against the real vcloud host (prep-only session, no live
   infrastructure reachable) -- that first real run is part of executing
   `runbooks/dc-dc-phase0-vcloud-prep.md` Step 3. Changelog:
   `docs/archive/changelogs/changelog-20260709-mtu-ceph-budget-calculators.md`.
8. **The "developed IPv6 plan" D-101 cites does not exist in this repo.** D-101's
   context line references "the developed family-follows-reachability IPv6 plan
   (in-project, not yet committed)" as prior art -- grepped the whole repo, it is
   not present as a standalone doc anywhere. If the exact Phase A/B/C detail
   behind D-101/D-102 is needed by a future session, it currently is NOT
   retrievable from the repo alone. Not a blocker for Stage 0 ratification of
   what IS written, but a real content gap if finer detail is needed later.
9. **Runbook debt -- CLOSED 2026-07-09.** All seven Stage 1-7 runbooks are now
   written (`runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-
   designate-cos-magnum.md`), Stage 1 authored directly, Stages 2/3/4/6/7
   delegated to parallel subagents and Stage 5 authored directly, all
   reviewed and repo-lint clean. NONE have been executed against real
   infrastructure yet -- this was a prep-only session (the operator was away
   from the workstation, not the vcloud host). Each runbook is explicit about
   what it can and cannot close given today's real state (NetBox literals,
   node sizing, and several other decisions remain genuinely open -- see
   each runbook's own GATE section for the honest per-stage breakdown).
10. **Harness debt -- LARGELY CLOSED 2026-07-09/10.** This repo's own delivery
    rule -- no script change ships without its `tests/<name>/run-tests.sh`
    harness -- has been honored for every new script tonight: `dc-selector`
    (21/21), `dc-dc-prefixes-import` (40/40), `dc-dc-mtu-geneve-budget`
    (19/19), `dc-dc-ceph-disk-budget` (16/16), `dc-dc-radosgw-multisite`
    (19/19), `dc-dc-rbd-mirror` (19/19), `dc-dc-dr-drill` (23/23), plus new
    `$DC` test cases added to `carve-host-interfaces`/`phase-00-maas-
    standup` and a new `reenroll-hosts` harness. Gap #2 (OpenTofu modules)
    remains the one area still lacking a harness beyond `scripts/opentofu-
    validate.sh` itself (no `tofu` binary this session to validate against).
11. **Decision sub-items still genuinely open** (the big items are RULED per
    Stage 0; these specific literals/params are not -- but see
    `docs/archive/dc-dc-netem-and-ula-gua-proposal.md` (DOCFIX-168) for a concrete
    netem proposal and exact, safe ULA-generation guidance, drafted
    2026-07-10): exact netem parameters
    (D-100), exact NetBox literals for DC2/ULA/GUA (D-101).
12. **No Office1-local virtual network is modeled anywhere in `opentofu/` --
    CLOSED 2026-07-09.** `main.tf` previously wired only `office1_pool_path`
    (storage only) and the three inter-site mesh links (management-traffic-
    only per D-100's own sub-item ruling) -- nothing the three Office1
    service VMs (MAAS-region, NetBox, GitBucket), or the vcloud host's own
    management interface, could actually attach to as a shared local network.
    Resolved by a NEW `opentofu/modules/office1-network` module (a
    `dc-planes`-shaped isolated `libvirt_network`, sized for ONE network
    rather than six planes since Office1 is a headend, not an OpenStack DC),
    deliberately NOT a reused host bridge -- a hand-configured bridge would
    be the one network in this topology living outside OpenTofu's D-103-
    mandated inventory, silently breaking this repo's own `virsh net-list
    --all` completeness assumption. INSTANTIATED for real in root `main.tf`
    (`module "office1_network"`) -- unlike most of gap #2's other scaffolding,
    this needed no unmeasured value beyond `domain_suffix`/`underlay_mtu`,
    already real required inputs elsewhere in the same file. See
    `opentofu/README.md` and `docs/archive/changelogs/changelog-20260709-office1-network-edge.md`
    (DOCFIX-163) for the full design rationale. **Follow-on work also done
    2026-07-13 (D-114):** the "designing the service VMs' actual `cloudinit-vm`
    `network_config` against this network" tail of this gap is closed, but not
    the way it was written -- there are no longer three service VMs to attach.
    ONE VM attaches to `office1-local`: the `voffice1` containment VM, whose
    `network_config` DHCPs from the OPNsense edge's Kea. The service machines
    live INSIDE `voffice1`'s LXD (MAAS-composed) and never touch this network
    directly. **DHCP hazard to carry forward:** LXD's `lxdbr0` runs its OWN
    dnsmasq -- it must stay on LXD's internal NAT bridge and must NEVER be
    bridged onto `office1-local`, where Kea is authoritative.
13. **The D-101 IPv6 family-matrix overlay -- DRAFTED 2026-07-10, one real
    risk still open.** `overlays/dc-dc-ipv6-family-matrix.yaml` now exists,
    sourced against real charm config/source (`docs/dc-dc-ipv6-charm-
    research.md` has full citations): `prefer-ipv6` (confirmed real option,
    genuinely additive dual-stack HAProxy binding per `charm-nova-cloud-
    controller`'s actual commit, not an either/or switch) + appended v6/GUA
    VIPs for the API charms; `ceph-mon`'s own separate `prefer-ipv6` +
    `ceph-public-network`/`ceph-cluster-network` for the ULA-only planes;
    confirmed NO charm-config option exists for OVN (geneve family follows
    the bound interface automatically). D-109's IPv6-SAN cert support is
    CONFIRMED (read `charm-vault`'s actual `sort_sans()` code -- no v4/v6
    distinction). **RESOLVED 2026-07-27 by R8, superseding the
    text this sentence replaced:** Octavia's `lb-mgmt-net` is IPv6 ULA,
    charm-created and charm-owned. The two Launchpad numbers that argued
    otherwise were read in full and BOTH failed (#1913409 = kolla-ansible,
    Fix Released 2021; #1911788 = Incomplete, not an IPv6 defect, cause fixed
    in OVS 2.15 and these nodes run 2.17). The charm has no IPv4 code path for
    this subnet at all. Still owed: the LP #2018998 MTU check (now built into
    `scripts/phase-05-octavia-verify.sh`) and gate G18. Also unconfirmed:
    8 of the 9 dual-stack charms' `prefer-ipv6` presence is inferred by
    shared-framework pattern, not individually fetched. UNVALIDATED against
    any live model.
14. **Three real `bundle.yaml` gaps for D-108's DR mechanism -- CLOSED
    2026-07-10 (DOCFIX-167).** `cinder-backup` (a SUBORDINATE charm,
    confirmed via real metadata.yaml -- the first draft wrongly gave it
    principal-style `num_units`/`to`/`amqp`, corrected) and `ceph-rbd-
    mirror` (a principal with `ceph-local`/`ceph-remote` requires
    endpoints, confirmed via real metadata.yaml -- the first draft
    invented a wrong binding key, corrected) are now in `bundle.yaml`;
    `ceph-mon`'s `rbd-mirror` binding corrected from `storage` to
    `replication`. **Genuinely unresolved, documented not invented:**
    `ceph-remote` (the peer DC's ceph-mon) is a cross-Juju-controller
    relation D-104's per-DC-independent-controller design means cannot be
    a same-bundle relation line -- only this DC's own `ceph-local` leg is
    wired; the cross-DC peer relationship needs a `juju offer`/`consume`
    or the manual `rbd mirror pool peer bootstrap` CLI flow (Stage 6's
    runbook already documents the latter). `python3 scripts/provider-
    bundle-check.py` PASS. See `docs/changelog-20260709-designate-
    cinderbackup-rbdmirror.md`. NOT applied to any live model.
15. **Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the
    Stage 4 runbook** (full detail in that runbook's own "Known gaps"
    section). **Sub-item 2 (script `$DC`-parameterization) CLOSED
    2026-07-10 (DOCFIX-166)** -- `reenroll-hosts.sh`,
    `carve-host-interfaces.sh`, and `phase-00-maas-standup.sh` now accept an
    opt-in `$DC` env var and call `lib_net_select_dc`/`lib_hosts_select_dc`
    immediately after sourcing (unset `$DC` is byte-for-byte unchanged
    behavior; see `docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md`). This
    closes the CLI-invocation half of the gap only -- it does NOT create any
    new per-DC data (no dc1/dc2 hostnames, octets, boot MACs, or node-count
    figures were invented), and `phase-00-maas-standup.sh`'s own `PLANES`
    table remains hardcoded to D-052/D-053's literals rather than driven by
    `lib-net.sh`'s flat vars, so it is not truly dc2-topology-aware yet even
    though it now gates dc2 out cleanly. The other six sub-items remain
    OPEN: no OpenTofu module stands up a per-DC MAAS rack controller VM
    itself (only registration, `modules/maas-vm-host`, exists); per-DC node
    count/sizing is an unmade Phase-0 decision; DC2 is still blocked at the
    network-selector layer (gap #3, NetBox literals not assigned); the
    per-DC artifact mirror (D-107) has no identified owning stage/module;
    the MAAS status a VM-host-discovered (vs. PXE-enlisted) machine lands in
    is unconfirmed; `HOST_TAG`/boot-fabric-name literals are DC0-specific.
16. **No stage explicitly owns creating Office1's own OPNsense edge VM --
    CLOSED 2026-07-09.** The topology (buildout design Section 1) is
    explicit: "full dark-fiber triangle... plus a **per-site** OPNsense
    simulated ISP edge" -- three sites, three edges (DC1, DC2, AND Office1).
    Stage 3's Build list already covered DC1/DC2's edges
    (`modules/opnsense-edge` per DC); Stage 2's Build list (MAAS region,
    OpenTofu, NetBox, GitBucket, Tailscale) did not mention an Office1 edge
    at all, and neither runbook instantiated one. **Decision: YES, Office1
    gets its own `modules/opnsense-edge` call, a fourth site alongside
    DC1/DC2; Stage 2's runbook
    (`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it.**
    Rationale: Stage 2's entire scope is already "Office1 headend standup"
    (the OPNsense edge IS part of that headend, not DC substrate), and
    Stage 1 already creates the mesh-link networks this edge's neighbors
    depend on -- Stage 3 has no reason to reach back and create a piece of
    Office1's own infrastructure. Wired as a commented-out `module
    "office1_opnsense"` skeleton in root `main.tf` (mirroring the identical
    NOT-YET-REAL status of DC1/DC2's own edges -- neither is instantiated
    either) plus a new gated step in
    `runbooks/dc-dc-phase1-office1-standup.md` (Step 4b) and its own GATE
    bullet. Related to, but distinct from, gap #12 (Office1-local network,
    now also CLOSED) -- that gap was the service VMs' own attach point;
    this one was whether Office1 gets a simulated-ISP-edge boundary at all,
    and who builds it. The edge's LAN side now resolves cleanly to gap
    #12's new network; its WAN side does not -- see NEW gap #17
    immediately below, a genuinely deeper, cross-site gap surfaced while
    closing this one, not invented as a substitute for a real answer.
    **D-114 follow-on (2026-07-13):** the edge is now BUILT and LIVE, and it is
    a load-bearing PREREQUISITE of the rest of Stage 2, not just a gate bullet
    -- `voffice1` takes its DHCP lease from the edge's Kea and reaches the
    internet through it. Office1's edge is also the only one of the three that
    is live; DC1/DC2's edges are Stage 3's problem and are still blocked on gap
    #17's DC half.
17. **CLOSED -- Office1 2026-07-13, both DCs by 2026-07-22.** (Was carried as
    "PARTIALLY CLOSED ... DC1/DC2 still have no uplink network" until
    2026-07-29. That status was STALE-OPEN, and because the skill's VR1 deploy
    loop routes every session to read this register FIRST, it produced a FALSE
    STOP. Corrected against the tree and the live host, below.)
    - **Office1: CLOSED.** The answer was the one this gap proposed -- a
      dedicated per-site ISP-uplink `libvirt_network`, **not** a mesh leg.
      `office1-wan` exists and is live (measured 2026-07-13: `virbr11`, NAT
      forward, `172.30.1.0/24`). The Office1 edge's WAN sits on it at
      `172.30.1.2` with default route `172.30.1.1`, and egress to the internet
      is verified (0.0% loss). So the topology question is ANSWERED and the
      pattern is PROVEN in production, not merely designed.
    - **BOTH DCs: CLOSED.** The 2026-07-13 text here said "Neither has an
      ISP-uplink network ... there is no `dc1-wan`/`dc2-wan` at all" and that
      "Stage 3's `dc1_opnsense` still carries the unresolved
      `wan_network_name` placeholder". Every clause is now FALSE. Measured
      2026-07-29:
      - Outer root `opentofu/main.tf`: `module "vr1_dc0_uplink"`
        (`172.30.2.0/24`) and `module "vr1_dc1_uplink"` (`172.30.3.0/24`),
        both `modules/site-wan` NAT networks. (At `:379` and `:391` when
        measured; grep the module NAME, line numbers drift.)
      - Inner roots: `module "vr1_dc0_wan"` in
        `opentofu/vr1-dc0-substrate/main.tf` and `module "vr1_dc1_wan"` in
        `opentofu/vr1-dc1-substrate/main.tf`, each a `modules/wan-bridge`
        onto that containment VM's `br-vr1-dcN-wan`. Each DC edge's
        `wan_network_name` is wired to its own module's output -- no
        placeholder remains in either root.
      - LIVE on vcloud: `virsh net-list --all` shows `vr1-dc0-uplink` and
        `vr1-dc1-uplink` active/autostart; `ip -4 addr` shows `172.30.2.1/24`
        and `172.30.3.1/24` on their bridges.
      - Gate evidence: D-125 egress isolation PASS for dc0
        (`docs/audit/d125-egress-gate-20260720-matrix.txt`) and for dc1
        (`docs/audit/d125-egress-gate-20260722-dc1.txt`).
      - Naming: the searched-for `dc1-wan` does not exist because the DCs were
        renamed by D-117/D-119 -- the real names are `vr1-dc0-wan` /
        `vr1-dc1-wan`. A grep for the retired token returning nothing is not
        evidence of a missing network.
    - **How it was actually closed -- NOT by replicating `office1-wan` per
      DC.** This gap's own closing note used to say the fix was mechanical
      replication of the Office1 shape. It was not: D-125 rules **bridge-in**,
      with the single NAT at VCLOUD level (`vr1-dcN-uplink`) and the inner
      `vr1-dcN-wan` a BRIDGE onto the containment VM's netplan bridge, so the
      inner OPNsense WAN lands directly on that /24 and there is exactly ONE
      NAT. That was necessary because D-123 Model B nested the DC WAN inside a
      containment VM whose only routed leg is the East-West-only transit. The
      D-100 reasoning below (mesh legs are MANAGEMENT-TRAFFIC-ONLY and must
      NOT carry an edge's WAN side) stands unchanged and is what all three
      builds honoured.

    *Original 2026-07-09 text follows (the reasoning is still correct; the "for
    ANY site / nowhere" status claim is superseded by the block above).*

    **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in
    this repo, for ANY site** (found 2026-07-09 while closing gap #16 for
    Office1). Every `modules/opnsense-edge` caller needs a
    `wan_network_name` distinct from BOTH its LAN-side network (a
    `dc-planes` plane, or gap #12's new `office1-network`) AND the three
    D-100 mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC-
    ONLY by D-100's own sub-item ruling (Office1<->DC fiber carries MAAS/
    Juju/operator traffic only), not a simulated ISP uplink, so wiring an
    edge's WAN side to a mesh leg would misrepresent the topology rather
    than merely placeholder it. This is NOT an Office1-specific gap:
    `runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 5's own `dc1_opnsense`
    template carries the IDENTICAL unresolved placeholder
    (`wan_network_name = module.mesh_dc1_office1... # or a dedicated
    per-site ISP-uplink segment -- confirm which, do not assume`) --
    confirmed by reading that runbook directly, not assumed. Deliberately
    NOT invented here for Office1 alone (that would contradict D-100 and
    leave DC1/DC2 with the same unresolved gap under a different name).
    Needs a decision: a fourth network shape per site (one more
    `dc-planes`/`mesh-link`/`office1-network`-style dedicated
    `libvirt_network` standing in for that site's own "internet" uplink),
    or some other mechanism not yet considered. Blocks instantiating ANY of
    the three sites' `opnsense-edge` calls (`dc1_opnsense`, `dc2_opnsense`,
    `office1_opnsense`) end-to-end -- all three are otherwise ready
    (image/config mechanism per gap #4, LAN-side network per gap #12/this
    stage's plane modules) but for this one missing piece.

18. **OpenTofu state file had no documented protection -- CLOSED 2026-07-10
    (DOCFIX-175).** Found while reviewing what's missing from the buildout,
    before Stage 1's own first real `tofu apply` (`dc-dc-phase0-vcloud-
    prep.md` Step 10 -- confirmed by reading the runbook directly, not
    assumed from filenames; an earlier pass wrongly guessed Stage 3):
    `terraform.tfstate` stores `sensitive = true` variables like
    `var.maas_api_key` in plaintext (verified against HashiCorp's own docs,
    not assumed -- `sensitive` only suppresses CLI output, not state
    storage), and nothing gitignored it, documented backup/permission
    handling, or noted the local-backend's lack of locking. Fixed:
    `.gitignore` now excludes `opentofu/**/*.tfstate*` and related local
    artifacts, `opentofu/README.md` gained a full "State file handling"
    section, `variables.tf`'s `maas_api_key` description carries the caveat
    inline, and `dc-dc-phase0-vcloud-prep.md` Step 10 gained a "SECURE THE
    STATE FILE" callout (`chmod`, gitignore verification, out-of-band
    backup) as a standing discipline from the first real apply onward (a
    short pointer added at Stage 3 Step 8 too, the first step needing a REAL
    MAAS credential). NOT fixed (documented, not invented): a remote backend
    with real locking -- production-grade, Roosevelt-scope, not a rehearsal
    decision.

19. **(19a -- "gap #19, DOCFIX-176"; see the numbering note on 19b below.)
    No teardown/rollback runbook existed for the VR1 OpenTofu-provisioned
    infrastructure itself -- CLOSED 2026-07-10 (DOCFIX-176).**
    `runbooks/phase-00-teardown-*.sh` (D-061) covers tearing down the VR0
    juju/MAAS layer, a different pairing of tools; nothing documented how to
    `tofu destroy` DC1/DC2/Office1's networks, storage pools, node-VM
    domains, and OPNsense edges in safe dependency order, or what MAAS-side
    cleanup has to happen alongside it. Fixed:
    `runbooks/dc-dc-teardown-rollback.md` -- two paths (scoped single-site
    via `-target`, or full VR1 destroy), the D-061 principle carried over
    (MAAS-side cleanup before destroying the libvirt resource underneath),
    a rollback decision tree favoring fix-forward over destroy for the
    common partial-apply-failure case, and explicit non-scope for mesh-link
    legs (shared between two sites, only destroyed when both ends are
    going away). **NOT resolved (flagged, not invented):** the exact
    MAAS-side release/delete commands for machines enrolled under a VR1
    `vm_host`, since this repo has never reached Stage 4 for any VR1 DC and
    has no real machines to test against yet -- the runbook states the
    principle and points at MAAS's own docs rather than guessing flags.

19. **(19b) CLOSED 2026-07-14 by D-119 -- THE `dc1`/`dc0` NAME MEANT TWO
    DIFFERENT DATACENTERS.**

    **Numbering note (2026-07-29).** This register carried TWO items numbered
    19 -- this one and 19a above -- and this one still declared itself OPEN and
    a hard precondition of all Stage 3 OpenTofu work, five months after it was
    closed. Neither is renumbered, because both are cited by number from
    outside this file: "gap #19" in `docs/design-decisions.md:3746` ("Closes
    tooling-gap #19 and Stage 2 close-out item C3") and in
    `docs/archive/changelogs/changelog-20260714-d119-region-qualified-dc-namespace.md`
    means THIS item (19b); a citation naming DOCFIX-176 means 19a. Item 20
    keeps its number too -- the operating skill cites "register item 20" by
    number.

    **Status: CLOSED 2026-07-14 (D-119, ADOPTED -- `docs/design-decisions.md`
    D-119, which states in its own status block that it closes this gap and
    Stage 2 close-out item C3).** The remap was executed, not merely ruled --
    verified 2026-07-29: `scripts/lib-net.sh` and `scripts/lib-hosts.sh` take
    region-qualified selectors (`vr0-dc0` / `vr1-dc0` / `vr1-dc1`) and REFUSE a
    bare `dc0`/`dc1`/`dc2` with an explicit RETIRED message, and the OpenTofu
    module names are `vr1_dc0_*` / `vr1_dc1_*` / `mesh_vr1_dc0_vr1_dc1` with
    `moved` blocks in `opentofu/main.tf` recording the rename from the old
    `dc1_*`/`dc2_*` names. **Nothing below is a live blocker; it is kept as the
    record of WHY the namespace is region-qualified, which is still load-bearing
    when reading any pre-2026-07-14 surface.**

    *Original 2026-07-14 text follows (the reasoning is still correct; its OPEN
    status and its "BLOCKS Stage 3" claim are superseded by the block above).*

    **THE `dc1`/`dc0` NAME NOW MEANS TWO DIFFERENT DATACENTERS.
    This BLOCKS Stage 3 and is the same off-by-one D-117 just fixed, one layer down.**

    D-117 (ADOPTED, Option B) moved the repo onto the NetBox apex's 0-indexed
    convention: VR1's **first** DC is `vr1-dc0`, its **second** is `vr1-dc1`.
    That rename was executed on the IMPORT PATH ONLY (`netbox/dc-dc-prefixes-
    import.py` + the scope doc). It was **NOT** executed on the two other
    surfaces that speak `$DC`, and they now disagree with the apex:

    | Surface | says `dc1` means | says `dc0` means |
    |---|---|---|
    | NetBox apex (authority, D-117) | VR1's **SECOND** DC | VR1's **FIRST** DC |
    | `opentofu/main.tf` (`dc1_planes`, `dc2_storage`, `mesh_dc1_dc2`) | VR1's **FIRST** DC | -- (no `dc0`) |
    | `scripts/lib-net.sh` / `lib-hosts.sh` (DOCFIX-151 selectors) | VR1's first DC | **VR0's** DC0 -- a different REGION |

    So "the next DC we deploy" is, simultaneously: apex `vr1-dc0`, OpenTofu
    `dc1_*`, and a `lib-net.sh` `$DC` value that ALSO names VR0's live testcloud.
    Three names, one machine, and one of them collides with a **running cloud**.

    **Why a mechanical rename is a trap (D-117 TRAP 1/TRAP 2, now live here):**
    `sed s/dc1/dc0/` on `opentofu/main.tf` would hand VR1's first DC the string
    `dc0`, which `lib-net.sh` already resolves to **VR0's deployed literals**. The
    fix is a SEMANTIC remap through placeholders, and the shell selectors must be
    region-qualified (`vr0-dc0` / `vr1-dc0` / `vr1-dc1`), not merely renumbered.

    **Consequence if ignored:** `module "dc1_nodes"` gets authored against the
    wrong site's prefixes, and the first VR1 DC deploys with the second DC's
    addressing -- discovered, at best, at `juju deploy`.

    *(Its closing line read: "**Status: OPEN. It is Stage 2 close-out item C3
    and the FIRST step of any Stage 3 OpenTofu work.** Nothing in `opentofu/`
    should gain a new DC-keyed module until it is closed." Superseded -- see
    the CLOSED block at the head of 19b.)*

20. **NEW 2026-07-17 -- durable host->site (isolated-net) reachability had no owning
    tool -- CLOSED for Office1; DC path templated.**

    `office1-local` is an ISOLATED libvirt net (no `<ip>`/`<forward>`), so the vcloud
    host has no L3 presence on `10.10.0.0/24`: its base leg (`10.10.0.10/24` on the
    bridge) + compose route (`10.10.1.0/24 via 10.10.0.20`) were re-added BY HAND each
    session and dropped on every reboot. The vcloud OS-patch+reboot (2026-07-17) proved
    the cost -- it stranded `ssh voffice1` and every ProxyJump alias. D-126 made the
    layers ABOVE this durable (the `--user` forward, the ssh aliases) but never owned
    the base leg.

    **CLOSED for Office1 (D-126 amendment):** `scripts/site-baseleg.sh` +
    `tests/site-baseleg/` (24/24) -- a root `Type=oneshot` unit ordered `After=libvirtd`
    (MEASURED: monolithic `libvirtd` here; also orders after `virtnetworkd` for modular
    hosts) that re-adds the leg at boot. Bridge DISCOVERED from the stable net name
    (never a baked `virbrN`, PATTERN-1); `apply` idempotent; `check` = session-verify.
    INSTALLED + `enabled` on vcloud 2026-07-17; the `After=libvirtd`+poll BOOT-RACE stays
    UNPROVEN until an actual reboot self-heals `ssh voffice1`. See D-126 amendment +
    D-128; `docs/archive/changelogs/changelog-20260717-d126-baseleg-oneshot-d128-operating-model.md`.

    **DC generalization (so this gap does NOT reopen per-DC):** the tool is SITE-KEYED
    with a commented DC template row, and the harness FAILS on any un-commented DC
    supernet (hard rule 2). At each DC standup, MEASURE how the vcloud host reaches that
    DC's containment/service net: if it is an ISOLATED host-local net, add a MEASURED
    `site-baseleg.sh` leg row + `install` the unit as part of the standup definition-of-
    done; if the DC is reached over a routed/NAT net or by `qemu+ssh` to an address the
    host already holds (D-123 / R-5), no leg is needed. The skill carries this as
    standing discipline (routing row + operating-model invariant + the VR1 deploy loop).

    **PER-DC VERDICT, MEASURED 2026-07-29 on vcloud (read-only; this replaces the
    "DC rows DEFERRED-until-measured" status the entry carried until then -- which
    left "no leg needed" as an INFERENCE nobody had checked).**

    What was measured, and how:
    - Each DC's containment/service net is its office1<->dcN transit:
      `mesh-vr1-dc0-office1` (bridge `virbr7`) and `mesh-vr1-dc1-office1`
      (bridge `virbr3`) -- `virsh net-list --all`, `virsh domiflist vvr1-dc0`,
      `virsh domiflist vvr1-dc1`, `virsh domiflist voffice1`. Each containment VM
      has exactly two NICs: its transit and its D-125 uplink.
    - Both transits are ISOLATED: `virsh net-dumpxml` on each shows NO `<ip>` and
      NO `<forward>` element, so libvirt puts no address on `virbr7`/`virbr3`.
    - vcloud holds NO L3 presence on either: `ip -4 -o addr show` lists
      `virbr11`/`virbr0`/`virbr2`/`virbr4`/`virbr1` and NOT `virbr7`/`virbr3`.
    - vcloud has NO path to either rack transit address: `ip route get 172.31.0.2`
      and `ip route get 172.31.0.6` (the two `VIRSH_POWER_ADDRESS` values in
      `scripts/lib-hosts.sh`) both resolve `via 10.17.8.1 dev enp1s0` -- the
      corporate default route, i.e. off-site, not the DC.

    **VERDICT: NO `site-baseleg.sh` row is required for either DC -- but NOT for
    the reason the rule above anticipates, and the mismatch is recorded here
    deliberately rather than resolved silently.** The rule offers two branches and
    the measured state satisfies NEITHER: the net IS an isolated host-local net
    (branch 1, "add a leg"), AND vcloud does NOT hold the `qemu+ssh` address
    (branch 2, "no leg needed"). The tie is broken by D-128: **no
    vcloud-originated DC operation exists.** Plane 2 -- MAAS, the inner `tofu`
    roots, the per-DC `virsh -c qemu+ssh://...` power path -- EXECUTES on
    `voffice1`, which holds the region end of both transits (its NIC2/NIC3 sit on
    `mesh-vr1-dc0-office1` and `mesh-vr1-dc1-office1`, measured above). vcloud
    reaches the DCs only THROUGH voffice1, and voffice1 is reached over the
    office1 leg this item already closed. A DC leg would be adding host L3 to a
    path nothing originates on. **If a future change makes vcloud originate to a
    DC directly, this verdict expires and branch 1 applies.**

    **UNMEASURED, explicitly (needs an `ssh voffice1`, outside this session's
    read-only scope -- do not read this entry as "the path is fine"):**
    (a) that voffice1 currently holds `172.31.0.1/30` and `172.31.0.5/30` on
    those NICs, and (b) reboot-durability of that addressing -- the same class of
    failure that produced this item for Office1. **Resolves with:**
    `ssh voffice1 'ip -4 -o addr show; ip route show'` for (a), plus
    `virsh -c "$VIRSH_POWER_ADDRESS" version` per DC as the end-to-end proof;
    for (b), the determination is whether that addressing is netplan-persistent
    on voffice1 or was added by hand -- if by hand, it is a `site-baseleg`-class
    gap on voffice1, not on vcloud, and belongs in this item.

    **Status: Office1 CLOSED (leg installed). Both DC rows: MEASURED 2026-07-29,
    verdict NO LEG REQUIRED on vcloud, with the voffice1-side transit addressing
    UNMEASURED as above.**

21. **NEW 2026-07-31 -- OPERATOR-DIRECTED, QUEUED NOT BUILT: a per-DC Tailscale
    subnet router is a STANDING DC-STANDUP REQUIREMENT, dc0 first.**

    Operator direction, verbatim (2026-07-31): *"A closer real work analog would be to
    install tailscale in DC0, DC1, and any future DCs as they would be running their own
    tailscale on, or beside, the utility node for that particular DC. Queue up the
    tailscale installation in DC0 and add it as a requirement for DC1 and any future
    installations."*

    **>>> RULED + PULLED FORWARD 2026-08-07. <<<** All four sub-decisions (a)-(d) below are
    RULED (GA-R5, D-129(iii) AMENDMENT 2026-08-07 in `docs/design-decisions.md`): (a) dedicated
    VM at utility `.7`; (b) STAR (operator->DC only); (c) SINGLE router, HA scale-up PINNED;
    (d) SNAT ON now, source-IP preservation PINNED. Operator directive "Lets plan and push to
    both DC0 and DC1 in this step" -- so the build executes for BOTH DCs. Trigger to pull it
    forward: closing phase-03 Horizon "properly" needs the real operator access path (browser
    to the metal-admin dashboard VIP over the tailnet), not just the VIP serving TLS on-segment.
    Build in progress; the four-rulings-before-build precondition is DISCHARGED. The text below
    is retained as the requirement's origin + the vendor-research record.

    **QUEUED, NOT EXECUTED** (hard rule 1 -- the current step is the Stage-5 deploy). This
    row is the requirement's home until it is built.

    **Why it is a requirement and not an improvement.** Today operator reach into a DC is
    SSH ProxyJump only: workstation -> vcloud -> voffice1 -> the DC rack's transit /30 ->
    (on the rack) metal-admin. There is no tailnet presence in any DC. The Office1
    subnet router advertises `10.10.0.0/22` and NOTHING toward `10.12.x`, and it could not
    usefully advertise it either -- MEASURED 2026-07-31: voffice1 holds no route to
    `10.12.x` at all (`ip route get 10.12.8.1` falls through to the default via
    `10.10.0.1`; its only DC-facing addresses are the two transit /30s). A subnet router
    can only forward to nets its own host can reach, so the tailnet stops at Office1 by
    construction, not by configuration.

    **THE RULED SHAPE ALREADY EXISTS -- this row is its per-DC execution, not a new
    decision.** D-129 sub-decision (iii), RULED 2026-07-21 (GA-R5, operator: *"Dedicated
    node per site (Recommended)"*), makes a dedicated subnet-router node on each site's
    METAL-ADMIN plane the standing per-site pattern, with the edge excluded (no
    management leg, no `os-tailscale`). The operator's own clarification quoted inside
    that ruling is the standing context: *"Roosevelt deisgn, and this deployment, will
    have a site tailscale installation to access the metal-admin plane at each location."*
    [sic] D-129(iii) also states that per-site ACL scoping, key custody and a SEC row land
    AT IMPLEMENTATION -- i.e. here.

    **CITATION DEFECT, recorded so it is not propagated: D-129(iii) says "D-107 remains
    the governing decision for the Office1 installation". D-107 is titled "Airgap posture,
    per-DC artifact mirror, and NTP (VR1)" and rules NOTHING about Tailscale** -- read in
    full 2026-07-31. This is the SAME miscitation finding F7 raised against the skill's
    routing table on 2026-07-29, now found a second time inside a ruling. The Office1
    installation currently has no governing decision at all.

    **SEC-010 DOES NOT NEED RELAXING, and this is the load-bearing point.** Tailscale
    reaches each site's router as ordinary WireGuard UDP addressed to the router VM
    itself; the forwarding that must be permitted is ON THAT VM, between `tailscale0` and
    its metal-admin NIC. That is a different forwarding decision from the transit-leg
    DROP rule (`table inet sec010`), which is untouched. The isolation D-138 was ruled to
    work around stays intact -- the same application-layer-vs-kernel-forwarding
    distinction SEC-010's own justification draws, and the same one that makes the
    voffice1->rack MAAS tunnel legitimate.

    **WHAT MUST BE RULED BEFORE ANYTHING IS BUILT** (each its own GA-R5 exchange; none of
    these is engineering judgement):
    - **(a) The utility-band OCTET.** D-134's octet map is a STANDING CROSS-DC STANDARD
      and divergence between DCs at the same octet is a DEFECT, so this cannot be chosen
      per-DC at build time. Currently `.4` artifact service / `.5` Juju controller /
      `.6` MAAS region; `.7` is next free. Also to settle: "ON, or BESIDE, the utility
      node" -- a dedicated VM in the utility band (the D-129(iii) shape) versus
      co-locating the daemon on an existing utility VM.
    - **(b) THE ADMIN-REACHABILITY MODEL, and it is the real architecture question.** Does
      any DC need to reach ANOTHER DC's metal-admin plane, or is admin traffic strictly
      operator->DC? A star keeps a compromised DC's blast radius inside that DC and is
      what the existing SEC-010 / D-125 posture already encodes; a mesh creates lateral
      management paths and grows ACL surface as O(N^2), which does not survive the
      region-region-with-N-DCs shape the operator described on 2026-07-31. Cross-DC
      REPLICATION (Ceph rbd-mirror, RADOSGW multisite) runs on the replication plane, not
      metal-admin, so it is not an argument for a management mesh.
    - **(c) HA count per site** -- one router or two. Two is Tailscale's documented HA
      shape; one is a single point of failure for operator access.
    - **(d) SNAT on or off** (`--snat-subnet-routes=false`). Off preserves the operator's
      source IP in DC-side audit logs; it requires a return route for `100.64.0.0/10` on
      the served hosts.

    **ACCESS-MODEL CONTEXT (2026-08-06 operator discussion -- NOT a ruling; context that
    informs the four sub-decisions above when they are re-raised).** The operator extended
    the standing model: *operators reach each DC's metal-admin plane over the tailnet AND
    access ROUTED DASHBOARDS (Horizon) from there* -- i.e. a browser hitting the dashboard's
    metal-admin VIP (`10.12.8.58` on dc0) directly over the tailnet, not just SSH. This has
    consequences that reach OUTSIDE the Tailscale build and should be weighed with it:
    - **Reverse proxy (operator path) becomes unnecessary.** The external nginx proxy that
      fronts the PROVIDER VIP today (VR0's `10.12.4.7`, phase-03 Step 3.3) exists only
      because corporate clients have no route in. The tailnet gives operators a route to
      metal-admin, so the operator-facing proxy drops out; the TENANT path (provider VIP via
      the edge/FIP) is separate and unaffected. Clean split: operators -> metal-admin VIP via
      tailnet; tenants -> provider VIP via edge.
    - **The per-rebuild proxy accommodations SUNSET for operators.** D-044 (Secure-cookie
      override) and D-075 (root redirect) were artifacts of the PLAIN-HTTP proxy leg; over a
      real HTTPS tailnet leg they are unneeded. D-044 already names this the "Roosevelt
      sunset."
    - **Certs become load-bearing on metal-admin.** Direct operator HTTPS to the metal-admin
      VIP needs the dashboard cert to carry a SAN for `10.12.8.58` (and ultimately the FQDN
      once `os-public-hostname` + FQDN-SAN certs land at Stage 7, D-106/D-008). The current
      cert is minted for the provider VIP -- MEASURED apache warning `AH01909` on
      `10.12.4.58:443` is the preview of this gap.
    - **Binding placement -- reinforces metal-admin as the dashboard's serving plane.** This
      is the access-model argument for F-CV3 option A (apache terminates TLS on metal-admin)
      over option B (cluster->metal-internal): operators live on metal-admin, NOT the isolated
      metal-internal svc plane (D-125). metal-internal is deliberately NOT advertised to the
      tailnet, so a dashboard served only on metal-internal would be a plane operators cannot
      reach. The rest of the API split is unaffected (public->provider tenant; admin->metal-admin
      operators-via-tailnet; internal->metal-internal isolated).
    - **Relevance to sub-decision (b).** This is an OPERATOR->DC access statement, consistent
      with the star (not mesh) reading of (b): operators reach each DC's metal-admin; it is
      not an argument for DC-to-DC management paths.
    See F-CV3 in `docs/audit/queued-findings-20260806-phase03-coreverify.txt` for the live
    dashboard-TLS defect this context bears on.

    **VENDOR-DOCUMENTED REQUIREMENTS the build must honour** (researched 2026-07-31
    against tailscale.com/kb and headscale.net; full citations in the session record):
    - **Tagged identity, not user identity.** Tailscale KB 1068: "it's more common for
      subnet routers to use a tag-based identity", and applying a tag "removes any
      user-based authentication" and disables key expiry by default. The EXISTING Office1
      node is UNTAGGED (`AdvertiseTags: None`, MEASURED 2026-07-31) and therefore carries
      a user's 180-day key-expiry clock -- a standing outage waiting to happen and a
      defect to fix alongside this work.
    - **`autoApprovers` must be written BEFORE the router first advertises.** KB 1337:
      "Auto-approver policies only apply when Tailscale first receives a subnet route
      advertisement. Updating the tailnet policy file ... does not retroactively approve
      existing unapproved routes." Get the policy in first or approve by hand forever.
    - **Non-overlapping management CIDRs per DC, permanently.** Tailscale's remedy for
      overlapping subnets is 4via6 (KB 1201); the SELF-HOSTED Headscale control plane does
      not document 4via6 support and carries a closed issue reporting it fails. At
      region-region scale this makes "every DC's metal-admin gets a globally unique CIDR"
      a HARD ADDRESSING CONSTRAINT, cheap to hold now at three sites and expensive later.
      Currently satisfied: `10.10.0.0/22`, `10.12.8.0/22`, `10.12.68.0/22`.
    - **An unpoliced Headscale is allow-all.** Its policy reference states that by default
      "no policy is loaded which means that Headscale allows all traffic between nodes" --
      so decision (b) is not merely a design preference, it is the thing that stops DC0's
      router reaching DC1's management plane.
    - **Version gates, and the version is NOT YET MEASURED:** autoApprovers >= 0.17.0,
      subnet-router failover >= 0.18.0, Tailscale-model tags >= 0.28.0, HA health probing
      + `node.expiry` >= 0.29.0. **Establish the running Headscale version before relying
      on any of these.** Headscale documents NO failover time (10s/5s are sample-config
      probe defaults); do not carry Tailscale SaaS's documented 15 s across.
    - **Encryption terminates at the subnet router.** True by construction -- the served
      host runs no Tailscale and holds no key -- but Tailscale's own encryption KB pages
      do NOT state it (checked; only a blog post does). Treat last-hop cleartext on
      metal-admin as a design fact to reason about, not one to cite. Management protocols
      must carry their own encryption (SSH, TLS), which they do.

    **DEFINITION OF DONE, per DC** (so this does not reopen per-DC, the same shape as
    item 20's DC generalization): the router VM exists in the ruled utility octet on
    metal-admin; IP forwarding enabled ON THE VM only; joined with a TAGGED pre-auth key;
    route approved (ideally via `autoApprovers`); a NAMED executable check proving the
    route is live from outside the site; a SEC row for the node's key custody
    (D-129(iii)); and the addressing constraint above re-verified against the new DC's
    CIDR. **Roosevelt analog: this is the operator's out-of-band access path to every
    site, so it is a day-1 standup item, not a convenience added later.**

22. **NEW 2026-07-31 -- PINNED, NOT SETTLED: `ceph-rbd-mirror` is the last single-unit
    service in the control plane and must be brought into line with the HA posture.**

    Operator direction 2026-07-31: pin it with options and a recommendation "to make sure
    it is not missed as we need to bring it in line with the rest of the HA posture".

    **It is RULED, so this is not an oversight to fix -- it is a ruling to revisit.**
    D-108 (ADOPTED 2026-07-09): *"Glance images: `ceph-rbd-mirror` of the Glance pool,
    snapshot-based, two-way, with a re-register step on failover. Daemon topology: one
    rbd-mirror daemon per DC, peering over the replication plane; **daemon HA is
    single-unit for VR1 (matches the D-104 controller economy), noted for Roosevelt
    scale-up**."* D-121's sizing table repeats it: "Left single (NOT scaled):
    `ceph-rbd-mirror`=1 (**D-108** rules single-unit for VR1)".

    **WHY IT MATTERS.** After the 2026-07-31 memcached change, `ceph-rbd-mirror` is the
    only control-plane container left at one unit with no HA story (`designate-bind` and
    `glance-simplestreams-sync` are also 1, but D-121 classes them as a DNS backend and a
    periodic utility respectively -- neither is on a data-durability path).
    rbd-mirror IS on one: it is the mechanism carrying Glance images between DCs for the
    Tier-A DR posture. A single daemon means the cross-DC replication stream stops when
    that container stops -- silently, since nothing currently alarms on mirror lag.

    **WHY IT IS NOT A `num_units` EDIT.** Two things a bump does not address, and this is
    what makes it a D-108 amendment rather than an overlay line:
    - **Daemon identity and peering.** Each rbd-mirror daemon authenticates to both the
      local and remote Ceph clusters. Two daemons per DC is a topology choice
      (active/active with the pool split between them, or active/standby with one idle)
      and it changes the peer configuration on BOTH sides, not just this DC.
    - **It is symmetric by nature.** rbd-mirror peers DC-to-DC, so scaling one DC's
      daemon count without the other is not a half-step -- it is a different topology
      than the peer expects.

    **OPTIONS, for the ruling when it is taken:**
    - **(a) Leave at 1 for VR1, scale at Roosevelt -- what D-108 currently says.** No
      work now; the exposure is that a single container failure halts cross-DC Glance
      replication until noticed, and nothing watches mirror lag today.
    - **(b) Scale to 2 per DC, active/standby.** The cheap half-step: a second daemon
      that takes over if the first stops. Needs the D-108 amendment, a second client
      identity per DC, and symmetric application at BOTH DCs.
    - **(c) Scale to 2+ per DC, active/active with the pool split.** Real throughput and
      redundancy; the most work, and the least justified at VR1's rehearsal scale.
    - **(d) Leave at 1 but ADD the missing detection** -- a mirror-lag / daemon-liveness
      assertion in `cloud-assert.sh`, so a stopped daemon is loud rather than silent.

    **RECOMMENDATION: (d) NOW, then (b) AT ROOSEVELT.** The measured gap today is not
    redundancy, it is that a stopped mirror daemon is INVISIBLE -- `grep -i mtu`-style
    checks aside, nothing in `cloud-assert.sh` reads rbd-mirror state at all, so option
    (a)'s exposure is unbounded in time rather than merely present. (d) is small, needs no
    D-amendment (it adds an assertion, it does not change the ruled topology), and it
    converts a silent failure into a detected one. (b) is the right shape for Roosevelt
    where the DR posture is real, and D-108 already anticipates exactly that. (c) is
    over-engineering for a rehearsal.

    **DO NOT scale it (`ceph-rbd-mirror`) into HA without the D-108 amendment** -- if it
    is ever scaled, that edit goes in `bundle.yaml` (`dc-ha-scaleup.yaml` was retired
    2026-08-06, its HA chain folded into the base by BUNDLEFIX-053), never silently.

    **STATUS: PINNED, awaiting a GA-R5 ruling. NOT to be closed by silently scaling it.**

---

## Where things stand today (2026-07-09) -- update this as stages close

| Layer | Status |
|---|---|
| Design (`dc-dc-buildout-design.md`) | done -- Section 10 now a ratified-ruling record; the doc's own header still reads "DESIGN / PLAN" (a status line, not a blocker -- its embedded decisions are ADOPTED per Stage 0 below) |
| Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED |
| DR mechanism seed | done (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 |
| Stage 0-7 runbooks | Stage 0 done (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. Stage 1 EXECUTED 2026-07-10. **Stage 2's runbook REWRITTEN 2026-07-13 to the D-114 model** (one `voffice1` site containment VM + MAAS-composed LXD VMs, replacing the three sibling service VMs and their Option A/B fork); it is PARTIALLY EXECUTED, and its MAAS/LXD command lines are marked PENDING VERIFICATION rather than guessed. Stages 3-7 NOT executed, and D-114 GATES them behind Office1 completing. See gaps #12-15 above for real design gaps the authoring pass surfaced. |
| **D-114 site containment VM (`voffice1`)** | **BUILT + running 2026-07-13.** `module "voffice1"` (`modules/cloudinit-vm` off `modules/base-image`, Ubuntu 24.04 noble) is instantiated and applied; the domain is up on `office1-local`, DHCP from the edge's Kea, `expose_nested_virt = true`. **Remaining:** MAAS-region + LXD (5.21 LTS track) ON it, the LXD-KVM-host registration back into MAAS, and the MAAS-COMPOSED service machines (NetBox/GitBucket/Tailscale). The first LXD VM that boots inside it is the **L3 nesting proof that gates DC1**. |
| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | **VALIDATED 2026-07-13** (tofu binary on the jumphost; pin authority `docs/CURRENT-STATE.md` section 7): root + **10/10 modules** pass `scripts/opentofu-validate.sh`. **APPLIED for real** -- state holds the DC1 planes, pools, 3 mesh links, `office1-network`, the OPNsense edge, and (2026-07-13, D-114) the `ubuntu_noble_base` image + the **`voffice1` containment VM**. **`modules/cloudinit-vm` is now instantiated for real** and gained `expose_nested_virt`. **DOCFIX-194 fixed 2 modules that had NEVER been parsed** (`node-vm`, `netem-link`) because root does not call them and root-only validation skips uncalled modules. `tofu plan`/`apply` for the Stage 3 DC substrate is still UNEXERCISED. |
| OPNsense image+config mechanism (`modules/opnsense-edge`) | **BUILT + INSTANTIATED + running 2026-07-12/13.** `office1-opnsense` is UP: routing, NAT, egress 0.0% loss, serial console, SSH-key managed, **Kea DHCP serving on udp/67**. Four boot bugs closed (DOCFIX-188/190/191/192) + DHCP added (DOCFIX-193). The module's `config_seed`/cdrom wiring is now DEAD WEIGHT -- see the config.xml row. |
| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | **superseded 2026-07-13 by D-113(a2): edge config is done over the REST API** (`scripts/opnsense-api.sh`), proven end-to-end (read AND write) against the live edge. **DANGER: a full rendered config.xml push would CLOBBER the now-API-managed DHCP** -- do not run the old scp/install/reboot path. The config-ISO path is INERT (D-112: the Importer can never fire on a nano image). **done (reconciled 2026-07-16):** `templates/opnsense-config.xml.tmpl` is DELETED and the module's `config_seed`/cdrom + `config_iso_path` are RETIRED; edge config is the REST-API bootstrap. (Boot-measured `WAN_IF`/`LAN_IF` are still applied over the API at deploy time -- a deploy-time step, not an authoring gap.) |
| MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` |
| `tc netem` mechanism (`modules/netem-link`) | **WAS BROKEN, FIXED 2026-07-13 (DOCFIX-194)** -- its destroy provisioner referenced `var.*`, which OpenTofu rejects at INIT, so the module could not even initialize. Now validates. Real latency/jitter/loss/rate parameters are STILL an unruled D-100 item. Never applied. |
| `opentofu/` (DC2 planes) | not done -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` |
| Office1-local network (`modules/office1-network`) | BUILT + INSTANTIATED 2026-07-09 (DOCFIX-163, gap #12 CLOSED) -- see `opentofu/README.md` and `docs/archive/changelogs/changelog-20260709-office1-network-edge.md` |
| Office1 OPNsense edge ownership | **BUILT AND LIVE 2026-07-12/13.** `module "office1_opnsense"` is instantiated (not commented). **Gap #17 is CLOSED FOR OFFICE1**: `office1-wan` (virbr11, NAT, 172.30.1.0/24) is the dedicated per-site ISP uplink, and the edge's WAN sits on it -- the D-100-honouring answer (NOT a mesh leg). **DC1/DC2 still have NO uplink network** -- replicating the `office1-wan` shape per DC is now mechanical, not a decision. |
| Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` |
| `lib-net.sh` / `lib-hosts.sh` | `$DC` selector mechanism ADDED 2026-07-09 (DOCFIX-151, gap #1 CLOSED) -- `lib_net_select_dc()`/`lib_hosts_select_dc()`, backward compatible, 21/21 tests; Stage 4 and Stage 5's runbooks (both written 2026-07-09) mandate calling it, not yet executed for real |
| NetBox DC-DC pipeline (`netbox/dc-dc-prefixes-import.py`) | MECHANISM BUILT 2026-07-09 (DOCFIX-152, gap #3 tooling half CLOSED) -- 40/40 fake-NetBox tests (DOCFIX-160 fixed a dead-assertion test bug that had silently left 3 of these uncounted), UNVALIDATED live; real literals (ULA /48, GUA carve, DC2 supernet) still unassigned |
| Environment/skill docs (`environment.md`, README.md) | Still describe VR0 DC0/v1 single-DC scope wholesale -- flagged, not yet rewritten for VR1; not a blocker for Stage 0 but will need doing before/during Stage 1 |
