# DC-DC (VR1) deployment workflow overview

**Purpose.** A single followable checklist across the whole VR1 (two-DC virtual
regional) buildout, spanning many sessions. `docs/dc-dc-buildout-design.md` is the
WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT
sequence, cross-referenced to that plan, to D-100..D-110, and to what already
exists in this repo vs what still needs to be authored. Update the "state" column
as stages complete -- this doc is a tracker, not a one-time snapshot.

**Governing constraint (unchanged throughout):** minimize delta to Roosevelt. Reuse
VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog
steps are the Section 9 shim register in the buildout design (OpenTofu node-VM
creation, `tc netem` WAN simulation, single-unit controllers/rbd-mirror).

---

## Stage 0 -- Decision ratification (CLEARED 2026-07-09)

**RATIFIED.** D-100..D-110 are ALL ADOPTED (`docs/design-decisions.md`;
D-102 MERGED into D-101 per the operator's own ruling, not deleted --
retained for numbering/history; D-109 confirmed standalone). All 6
redline items ruled (`docs/dc-dc-buildout-design.md` Section 10 is now a
ruling record, not an open list): Ceph size=3-by-default (size=2 only as an
explicit Phase-0 fallback); netem same-metro lean; D-102/D-101 merge +
D-109 standalone; metal-admin GAINS a ULA leg (a real reversal of the
stated lean, not a rubber-stamp); COS per-DC-only; mirror sync independent
per-DC. Verified via `bash scripts/ledger-scan.sh`: none of D-100..D-110
appear under "PROPOSED / OPEN decisions." Full ruling record:
`docs/changelog-20260709-stage0-ratification.md`.

**State:** DONE.

---

## Stage 1 -- Phase 0: vcloud host preparation

| | |
|---|---|
| **Goal** | Turn the bare vcloud host into a substrate Office1 can deploy from. |
| **Build** | Nested KVM enabled; libvirt storage pools + thin-provisioning; dark-fiber + per-site ISP-edge virtual networks; the six per-DC plane segments as virtual networks; MTU plan set on the underlay. |
| **Gate** | Nested KVM verified; disk budget computed + Ceph size decision recorded; measured MTU recorded; virtual networks present and isolated as designed. |
| **Owns** | D-100 (fabric), D-101 (MTU -- folds in former D-102), buildout-design Section 3 (host/disk/MTU gates). |
| **Reuse vs new** | NEW. No VR0-DC0 analog -- that testcloud IS the single vcloud host, already prepared; this is preparing the host to carry TWO independent DC substrates plus Office1. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase0-vcloud-prep.md`** -- command-level steps (host identify -> measure CPU/RAM/disk/nested-KVM/MTU -> enable nested KVM -> prepare pool paths -> install/confirm OpenTofu -> tfvars -> init/validate/plan/apply -> post-apply gate verify), each MUTATION individually gated per this repo's discipline. Flags a real open structural question up front (the `provider "maas"` block in `main.tf` is unconditional even though this stage only touches libvirt resources -- may force `maas_api_url`/`maas_api_key` to be set at `plan` time with nothing yet using them; noted as a possible DOCFIX candidate, not silently worked around). NOT YET EXECUTED against real infrastructure -- this is the first runbook that will be, tomorrow morning, per the operator's own stated plan. `opentofu/` SCAFFOLD started 2026-07-09: `modules/dc-planes` (six per-DC planes) + `modules/mesh-link` (dark-fiber legs) + `modules/dc-storage-pool` wired for DC1 + Office1 -- see `opentofu/README.md` for what's built vs deliberately deferred (node-VM/domain resources, DC2 CIDRs, netem). UNVALIDATED: no `tofu` binary available to run `scripts/opentofu-validate.sh` yet -- this runbook's Step 8 is the first real run. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Stage 2 -- Phase 1: Office1 headend standup

| | |
|---|---|
| **Goal** | Stand up the operator's deployment headend first, before any DC. |
| **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only). |
| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only. |
| **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). |
| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. Flags a genuine, previously-unnoticed gap: no Office1-local virtual network is modeled anywhere in `opentofu/` for these VMs to attach to. NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned).

---

## Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

| | |
|---|---|
| **Goal** | From Office1, script each DC's "physical" layer into existence. |
| **Build** | Per DC: OpenTofu creates node-VM libvirt domains (SHIM, Section 9), the DC's plane networks, and its OPNsense edge; registers the DC libvirt host to that DC's MAAS rack controller as a virsh VM-host. |
| **Gate** | MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured. |
| **Owns** | D-103 (seam), D-100 (edges). |
| **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Walks every still-open decision (node sizing, MAAS zone/pool, `power_address`, config.xml tokens incl. boot-measured `WAN_IF`/`LAN_IF`, netem params) and stops at the corresponding step if unresolved rather than inventing values. States its own exit gate as "CONDITIONALLY MET AT BEST" pending Stage 2, node sizing, a real OPNsense boot measurement, and D-100's netem ruling. NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (2026-07-09, registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`, which composes new VMs and would fight `node-vm`) all now exist -- none instantiated (node sizing, OPNsense config.xml content, and a real MAAS zone/pool are all still pending decisions). `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED, hard-dependent on Stage 2 for its MAAS-registration half.

---

## Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

| | |
|---|---|
| **Goal** | Bring the discovered node VMs to a deployed OS, ready for Juju. |
| **Build** | Per DC: MAAS commissions + deploys the node VMs; provider NIC left RAW (br-ex discipline, D-100); VLAN-103 metal-internal bridge stack pre-built by MAAS. |
| **Gate** | Nodes deployed; six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC mirror reachable; NTP from edge working. |
| **Owns** | D-103, D-100 (br-ex), D-107 (mirror + NTP). |
| **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing undecided; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

| | |
|---|---|
| **Goal** | A running independent OpenStack cloud per DC. |
| **Build** | Per DC: bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle; Vault init with a per-DC independent root, issuing v4 + IPv6-SAN certs; apply the family matrix (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider). |
| **Gate** | `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC; controller backup taken + stored; Ceph-over-v6 and geneve-over-v6 verified. |
| **Owns** | D-104 (controllers), D-109 (Vault), D-101 (families). |
| **Reuse vs new** | **This is the ground-truth/copy point** (`docs/dc-dc-buildout-design.md` line 12: seeded from `openstack-caracal-ipv4` at 8813efc). `bundle.yaml` and `runbooks/phase-01..08-*.md` are the per-DC template -- run essentially the existing VR0 DC0 deploy loop TWICE (once per DC), adapted for: independent Keystones (no cross-DC federation), the IPv6 family matrix (D-101/D-102), and per-DC Vault roots (D-109) instead of the single shared root. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`.** Does not duplicate `phase-01..08`'s content -- describes what changes to run that loop twice (once per DC): DC1 needs NO VIP/CIDR edits at all (D-101 inherits DC0 unchanged); DC2 is hard-blocked pending its NetBox literals. Requires every adapted `phase-01..08` command block that reads `lib-net.sh`/`lib-hosts.sh` to call `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` first (DOCFIX-151). Flags a real, unresolved design gap: the D-101 IPv6 family-matrix overlay (dual-stack/ULA-only per-plane charm config) does not exist yet -- the exact charm config option names for v6 addressing have not been confirmed against any charm's real schema, so no overlay file is invented; Step 6 names this as blocking work rather than gliding past it. Also flags D-109's IPv6-SAN cert-issuance support as an unverified charm-behavior assumption. NOT YET EXECUTED. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. **This is where the existing repo tooling (`preflight.sh`,
`cloud-assert.sh`, `scripts/lib-net.sh`/`lib-hosts.sh`, `run-logged.sh`) does the
most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they
apply to either DC's clone/session without modification.**

---

## Stage 6 -- Phase 5: DR wiring and failover drill

| | |
|---|---|
| **Goal** | Prove Tier-A recoverability, one-way first, then two-way. |
| **Build** | Replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones); rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency groups where needed. Run the failover/failback drill. |
| **Gate** | One-way drill clean (RTO/RPO measured); then two-way; controller restore drill (D-104) exercised; RTO recorded as a measured output, not an SLA. |
| **Owns** | D-108 (replication), D-104 (control-plane backup/restore in the drill). |
| **Reuse vs new** | Ground truth: `docs/dc-dc-replication-DR-seed.md` (the original Tier-A/bidirectional/cinder-backup+rbd-mirror decision) -- D-108 corrected its carrier from the seed's v4 replication plane to IPv6-only ULA. The failover/failback drill skeleton is already drafted (buildout design Section 8): confirm DC1 down -> promote Glance -> restore Cinder -> rebuild instances -> record RTO/RPO; failback is split-brain-safe (no auto-resume, controlled window to flip primary back). |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase5-dr-failover-drill.md`.** Command-level implementation of Section 8's failover/failback skeleton, staged one-way-then-two-way per D-108's own requirement, plus the D-104 controller backup/restore drill. Found and flagged three real bundle gaps while writing it (grepped `bundle.yaml`): no `cinder-backup` charm deployed, no `ceph-rbd-mirror` charm deployed, and `ceph-mon`'s existing `rbd-mirror` endpoint binding is on `storage` instead of `replication` (inherited from the VR0/DC0 seed, predates D-108) -- all three become this runbook's own first gated mutation (Step 2), not silently routed around. Every realm/zone/pool/endpoint name is an explicit `<UPPER_SNAKE_TBD>` placeholder. Includes a negative-control sub-drill (sever only the replication link first, confirm the split-brain check correctly refuses to promote) before the real hard-down drill. NOT YET EXECUTED -- no live Ceph cluster this session (harness for gap #5's scripts is future work, logged, not built). |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

| | |
|---|---|
| **Goal** | Complete the service set in both DCs. |
| **Build** | Per DC: Designate (designate-bind) with per-DC zones + A/AAAA per the D-008 bootstrap order; COS observability; Magnum/CAPI with in-cloud single-homed capi-mgmt (`domain-setup` re-run per D-046). |
| **Gate** | Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); COS scraping; a CAPI workload cluster comes up per DC. |
| **Owns** | D-106 (Designate), D-105 (CAPI/COS). |
| **Reuse vs new** | HIGH REUSE for CAPI/Magnum: `runbooks/phase-06-incloud-mgmt-cluster.md` + `phase-07-conductor-graft.md` + `phase-08-workload-cluster-acceptance.md` are direct precedent, per-DC. Designate is a REVERSAL, not new build: VR0 DC0 deferred it (D-019, "DNS scope reduction"); D-106 reactivates it (supersedes D-019, reactivates the original D-008 bootstrap order) -- read D-008 and D-019 together before writing this stage's runbook, the static-`/etc/hosts` workaround they document goes away. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase6-designate-cos-magnum.md`.** Quotes `bundle.yaml`'s and `phase-01-bundle-deploy.md`'s actual current "NO designate (D-019)" text verbatim rather than paraphrasing, then the D-106 reversal against it. Flags that the reversal is bigger than "add designate" -- D-106's bootstrap order also reactivates `os-public-hostname` per API charm, reversing the bundle's separate B5 IP-only posture; proposes (not silently adopts) pushing per-DC hostname literals into new `overlays/dc1-hostnames.yaml`/`dc2-hostnames.yaml` files to keep `bundle.yaml` itself DC-agnostic, mirroring the existing octavia-pki overlay pattern. D-046's domain-setup trap quoted verbatim in a top callout and again inline. COS deployment mechanism explicitly left as an operator decision point (D-105 rules only the scope) rather than invented. Reuses `phase-06/07/08` by filename with concrete DC-parameterization notes (confirmed via tracing which scripts source `lib-net.sh`). NOT YET EXECUTED. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Cross-cutting discipline (applies at every stage, every session)

- **Session bootstrap, every session:** `git pull` -> `bash scripts/repo-lint.sh`
  (0 fail / 1 documented legacy WARN expected) -> `bash scripts/ledger-scan.sh`
  to reconcile open decisions/next-free numbers against `docs/session-ledger.md`.
- **Any session that will mutate the cloud:** `bash scripts/run-logged.sh <label>`
  first (as-executed log).
- **Before any add-model-like mutation:** `bash scripts/preflight.sh` PASS.
- **After any deploy/restart/change, and as an incident-triage first step:**
  `bash scripts/cloud-assert.sh` (add `--capture` at real deploy-completion
  milestones for the committed BOM).
- **Before touching ANY built surface:** grep `docs/design-decisions.md` for the
  governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present
  options, do not implement.
- **Every deliverable (script, runbook, doc):** `repo-lint` clean, the touched
  script's harness green (write one if missing), a changelog entry with a
  per-item revert, and a `docs/session-ledger.md` update.
- **Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN:** `bash
  scripts/ledger-scan.sh` for next-free (it is the numbering authority, not
  repo-lint's informational count).
- **Hard rule reminder:** no inferred values -- every IP/ID/name/scope is
  measured this session or carried from confirmed as-built; verify before
  mutate, always; destructive/secret-handling steps are individually gated,
  never batched.

---

## Tooling gap register (audited 2026-07-09; update as items close)

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not
inferred from the design doc alone. Ordered roughly by "you'll hit this first,"
not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;
#2 is the single largest build.

1. **`$DC` selector convention -- CLOSED 2026-07-09 (DOCFIX-151).**
   `scripts/lib-net.sh` gained `lib_net_select_dc()` (dc0/dc1 no-op, since
   D-101 rules DC1 inherits DC0's v4 layout unchanged; dc2 fails loud --
   NetBox literals not yet assigned, gap #3) and `scripts/lib-hosts.sh`
   gained `lib_hosts_select_dc()` (dc0 no-op; dc1 AND dc2 BOTH fail loud --
   no real per-DC host enrollment exists yet for either, an intentional
   asymmetry vs. the net selector, documented in both files' comments and
   `docs/changelog-20260709-dc-selector-convention.md`). Backward compatible
   by construction -- every existing caller of either library is unaffected.
   New harness `tests/dc-selector/run-tests.sh`: 21/21 PASS. The mechanism
   exists; Stage 5's runbook still needs to actually CALL
   `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` when it's
   authored -- that call-site wiring is part of writing that runbook, not
   this delivery.
2. **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded
   2026-07-09 (was: zero files).** `opentofu/` now has: `modules/dc-planes`
   (six per-DC planes), `modules/mesh-link` (the D-100 dark-fiber triangle
   legs), `modules/dc-storage-pool`, `modules/node-vm` (blank disk, PXE-boot
   MAAS-managed node VM, the D-103 pattern for Stage 3), `modules/base-image`
   + `modules/cloudinit-vm` (the OTHER VM pattern -- pre-built image +
   cloud-init, for Office1's own service VMs), `modules/opnsense-edge` (see
   gap #4), `modules/maas-vm-host` (registers the vcloud host with MAAS as a
   virsh VM host -- via the official `canonical/maas` provider, confirmed
   against its real schema; deliberately NOT `maas_vm_host_machine`, which
   composes/creates new VMs itself and would fight `node-vm` for VM
   creation -- D-103 explicitly rules out MAAS composing), and
   `modules/netem-link` (`terraform_data` + SSH-wrapped `local-exec`
   provisioners applying `tc qdisc ... netem` to a mesh-link bridge --
   OpenTofu runs from Office1, not the vcloud host, so a bare local command
   can't reach the bridge). DC1 + Office1 pool/plane wiring is instantiated
   in root `main.tf`; everything else is NOT (node/VM sizing, MAAS zone/pool,
   netem parameters are all pending decisions -- neither invented here).
   Provider schema was confirmed against the actual current docs AND real
   example `.tf` files (not memory) -- it differs materially from older
   common examples: nested structures are attribute-style objects (`key =
   { ... }`), not classic HCL blocks. **A real bug from the first pass was
   found and fixed:** `dc-planes`/`mesh-link`/`dc-storage-pool` originally
   used block syntax for `libvirt_network`'s `domain`/`mtu` and
   `libvirt_pool`'s `target`; corrected once real provider examples surfaced
   the actual convention -- discovered precisely BECAUSE building the domain
   module forced fetching real example code instead of doc-summaries alone.
   Still missing, deliberately deferred (see `opentofu/README.md`):
   OPNsense's own bootability via `cloudinit-vm` is NOT confirmed (FreeBSD,
   not the same cloud-init/NoCloud guarantee as Linux images); the
   per-device `boot`-order attribute's exact internal shape (flagged
   UNVERIFIED in `node-vm/main.tf`, an inference by pattern not a confirmed
   field); and whether `devices.interfaces` list order reliably maps to
   LAN/WAN NIC enumeration (flagged in `opnsense-edge/main.tf`).
   **UNVALIDATED** -- no `tofu` binary was available to run
   `scripts/opentofu-validate.sh` this session; run it on a connected
   machine before trusting any of this.
3. **NetBox multi-DC/dual-stack import pipeline -- MECHANISM CLOSED
   2026-07-09 (DOCFIX-152), literals still pending.** `netbox/dc-dc-prefixes-
   import.py` extends the v1 pattern to VR1's two-DC dual-stack model:
   per-DC (`--dc dc1`/`dc2`) creates the six-plane IPv4 (DC1 inherited,
   hardcoded per D-101; DC2 via required `DC2_V4_SUPERNET`, no default) plus
   the D-101 family-matrix IPv6 legs (ULA/GUA), carved from two REQUIRED
   env vars (`ORG_ULA_48`, `DC_GUA_PREFIX`) with no defaults -- fails loud
   rather than inventing either. 40/40 tests (`tests/dc-dc-prefixes-
   import/`, fake-NetBox end-to-end). UNVALIDATED against a real NetBox (no
   instance reachable this session). The literals themselves (org ULA /48,
   per-DC GUA carve, DC2 supernet) still don't exist -- that operator/NetBox
   assignment step is unchanged by this delivery; this closes the TOOLING
   gap, not the DATA gap. Still blocks Stage 2/3 until those literals are
   assigned and this script is actually run against a live NetBox.
4. **OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) DONE
   2026-07-09, (d) is the only real gap left.** Each site (Office1, DC1, DC2)
   gets its own independent OPNsense edge with a simulated ISP uplink (D-100);
   OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge)
   and Stage 3 (per-DC edges) gates, which require netem parameters "applied
   and measured."
   - **(a) Network substrate -- DONE.** `opentofu/modules/mesh-link` creates
     the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an
     OPNsense VM's interfaces would eventually attach to.
   - **(b)+(c) The OPNsense VM/image AND its config -- BUILT
     (`opentofu/modules/opnsense-edge`), UNVALIDATED.** Cloud-init is
     confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum
     consensus, not just a hedge. The real native mechanism is the
     Configuration Importer (official docs): scans an attached volume for
     `/conf/config.xml` in a 2-3 second boot-time window; ISO9660 support was
     added to it specifically for VM/cloud automation (attach a config ISO as
     a secondary CD-ROM) -- mechanically IDENTICAL to `modules/cloudinit-vm`'s
     shape (a secondary volume attached as `cdrom`), just a different payload.
     Built as `opentofu/modules/opnsense-edge` + two prep scripts
     (`scripts/opnsense-prep-image.sh` for the nano image's decompress+
     convert, `scripts/opnsense-build-config-iso.sh` for the config.xml ISO).
     The real `config.xml` DESIGN is now also done:
     `opentofu/templates/opnsense-config.xml.tmpl` (a `{{TOKEN}}`-parameterized
     template built from OPNsense's own real shipped sample, covering
     interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall
     rules) + `scripts/opnsense-render-config.sh` (the renderer -- fully
     tested end-to-end, 8/8, unlike the other opnsense-* scripts, since it
     needs no external tool). UNVALIDATED regardless: no `tofu` binary, no
     ISO-building tool, and no real OPNsense boot were available to confirm
     the whole chain end-to-end. Several template tokens still can't be
     filled with real per-site values -- pending Stage 0 ratification
     (D-100/D-101/D-107) or only measurable on a real boot (which `vtnetN`
     device a network becomes). Full sourcing in `opentofu/README.md` and
     `opentofu/templates/README.md`'s token legend.
   - **(d) `tc netem` WAN-simulation -- mechanism BUILT (`modules/netem-link`),
     only the real parameters remain.** No Terraform/OpenTofu-native
     equivalent exists (confirmed against the `libvirt_network` schema, gap
     #2); built instead as a `terraform_data` + SSH-wrapped `local-exec`
     provisioner (OpenTofu runs from Office1, not the vcloud host where the
     bridge lives -- a bare local command can't reach it). UNVALIDATED, not
     instantiated. The exact parameters (latency/jitter/loss/rate) are
     themselves still an unruled D-100 open sub-item (gap #11), so this
     needs a decision before it can be applied for real, not just scripting.
5. **Ceph replication tooling (radosgw multisite + rbd-mirror).** No script in
   the repo touches either (checked; `osd-blank-check.sh` is unrelated). D-108's
   entire mechanism -- zonegroup setup, rbd-mirror daemon peering, the
   failover/failback drill commands -- is net-new. Blocks Stage 6.
6. **Designate reactivation is a bundle change, not just a runbook.**
   `runbooks/phase-01-bundle-deploy.md` states the bundle explicitly ships
   "NO designate (D-019)" -- reactivating it per D-106 means editing
   `bundle.yaml` itself (a built-surface change, gated behind D-106 leaving
   PROPOSED), in addition to writing the reactivation runbook. Blocks Stage 7.
7. **MTU/geneve budget + Ceph disk-budget calculators.** D-102 and buildout-design
   Section 3 both describe measurement-then-computation gates (host L2 MTU ->
   geneve-over-v6 overhead; measured per-node OSD footprint -> size=3 vs size=2
   fallback) that are manual arithmetic in prose today, not a script. Blocks the
   Stage 1 Phase-0 gates.
8. **The "developed IPv6 plan" D-101 cites does not exist in this repo.** D-101's
   context line references "the developed family-follows-reachability IPv6 plan
   (in-project, not yet committed)" as prior art -- grepped the whole repo, it is
   not present as a standalone doc anywhere. If the exact Phase A/B/C detail
   behind D-101/D-102 is needed by a future session, it currently is NOT
   retrievable from the repo alone. Not a blocker for Stage 0 ratification of
   what IS written, but a real content gap if finer detail is needed later.
9. **Runbook debt -- CLOSED 2026-07-09.** All seven Stage 1-7 runbooks are now
   written (`runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-
   designate-cos-magnum.md`), Stage 1 authored directly, Stages 2/3/4/6/7
   delegated to parallel subagents and Stage 5 authored directly, all
   reviewed and repo-lint clean. NONE have been executed against real
   infrastructure yet -- this was a prep-only session (the operator was away
   from the workstation, not the vcloud host). Each runbook is explicit about
   what it can and cannot close given today's real state (NetBox literals,
   node sizing, and several other decisions remain genuinely open -- see
   each runbook's own GATE section for the honest per-stage breakdown).
10. **Harness debt.** This repo's own delivery rule -- no script change ships
    without its `tests/<name>/run-tests.sh` harness -- applies to everything in
    #2-7 above. None exist yet because none of those scripts exist yet; expect
    roughly 5-6 new test suites alongside whatever gets built.
11. **Decision sub-items still genuinely open** (the big items are RULED per
    Stage 0; these specific literals/params are not): exact netem parameters
    (D-100), exact NetBox literals for DC2/ULA/GUA (D-101).
12. **No Office1-local virtual network is modeled anywhere in `opentofu/`**
    (found 2026-07-09 while drafting the Stage 2 runbook). `main.tf` wires
    `office1_pool_path` (storage only) and the three inter-site mesh links
    (management-traffic-only per D-100's own sub-item ruling) -- nothing the
    three Office1 service VMs (MAAS-region, NetBox, GitBucket), or the vcloud
    host's own management interface, could actually attach to as a shared
    local network. Blocks designing real `cloudinit-vm` `network_config`
    content for any of the three VMs. Needs either a new `modules/dc-planes`-
    shaped Office1-local network, or a deliberate, documented reuse of an
    existing host bridge -- not decided yet.
13. **The D-101 IPv6 family-matrix overlay does not exist** (found 2026-07-09
    while drafting the Stage 5 runbook). Which planes need ULA-only vs.
    dual-stack vs. GUA addressing is fully specified by D-101 (Section 2 of
    the buildout design), but the actual Juju charm config option names
    needed to configure that per application (`ceph-mon`, `ovn-central`,
    `octavia`, the HA-cluster/VIP charms, etc.) have not been confirmed
    against any charm's real current schema. Blocks Stage 5's "Ceph-over-v6
    and geneve-over-v6 verified" gate criterion. D-109's IPv6-SAN cert-
    issuance charm support is a related, separately-unverified assumption.
14. **Three real `bundle.yaml` gaps for D-108's DR mechanism** (found
    2026-07-09 while drafting the Stage 6 runbook, by grepping the actual
    bundle): no `cinder-backup` charm deployed; no `ceph-rbd-mirror` charm
    deployed; `ceph-mon`'s existing `rbd-mirror` endpoint binding is on
    `storage`, not `replication` (inherited from the VR0/DC0 seed, predates
    D-108's ULA-carrier requirement). All three are Stage 6's own first
    gated mutation (its Step 2), not fixed by this delivery -- no live Ceph
    cluster exists this session to test the fix against.
15. **Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the
    Stage 4 runbook** (full detail in that runbook's own "Known gaps"
    section): no OpenTofu module stands up a per-DC MAAS rack controller VM
    itself (only registration, `modules/maas-vm-host`, exists); `reenroll-
    hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not
    `$DC`-parameterized (DOCFIX-151 only touched the two library files);
    per-DC node count/sizing is an unmade Phase-0 decision; the per-DC
    artifact mirror (D-107) has no identified owning stage/module; the MAAS
    status a VM-host-discovered (vs. PXE-enlisted) machine lands in is
    unconfirmed; `HOST_TAG`/boot-fabric-name literals are DC0-specific.

---

## Where things stand today (2026-07-09) -- update this as stages close

| Layer | Status |
|---|---|
| Design (`dc-dc-buildout-design.md`) | DONE, PROPOSED |
| Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED |
| DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 |
| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. NONE executed against real infrastructure (prep-only session). See gaps #12-15 above for real design gaps the authoring pass surfaced. |
| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` |
| OPNsense image+config mechanism (`modules/opnsense-edge`) | BUILT 2026-07-09, UNVALIDATED (no `tofu`/ISO-tool/real boot to confirm), see `opentofu/README.md` |
| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | BUILT 2026-07-09, renderer fully tested (8/8) -- several tokens still pending Stage 0 ratification or a real boot, see `opentofu/templates/README.md` |
| MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` |
| `tc netem` mechanism (`modules/netem-link`) | BUILT 2026-07-09, UNVALIDATED -- real latency/jitter/loss/rate parameters still an unruled D-100 item, see `opentofu/README.md` |
| `opentofu/` (DC2 planes) | NOT DONE -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` |
| Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` |
| `lib-net.sh` / `lib-hosts.sh` | `$DC` selector mechanism ADDED 2026-07-09 (DOCFIX-151, gap #1 CLOSED) -- `lib_net_select_dc()`/`lib_hosts_select_dc()`, backward compatible, 21/21 tests; Stage 5 runbook still needs to call it |
| NetBox DC-DC pipeline (`netbox/dc-dc-prefixes-import.py`) | MECHANISM BUILT 2026-07-09 (DOCFIX-152, gap #3 tooling half CLOSED) -- 40/40 fake-NetBox tests (DOCFIX-160 fixed a dead-assertion test bug that had silently left 3 of these uncounted), UNVALIDATED live; real literals (ULA /48, GUA carve, DC2 supernet) still unassigned |
| Environment/skill docs (`environment.md`, README.md) | Still describe VR0 DC0/v1 single-DC scope wholesale -- flagged, not yet rewritten for VR1; not a blocker for Stage 0 but will need doing before/during Stage 1 |