# DC-DC (VR1) deployment workflow overview

**Purpose.** A single followable checklist across the whole VR1 (two-DC virtual
regional) buildout, spanning many sessions. `docs/dc-dc-buildout-design.md` is the
WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT
sequence, cross-referenced to that plan, to D-100..D-110, and to what already
exists in this repo vs what still needs to be authored. Update the "state" column
as stages complete -- this doc is a tracker, not a one-time snapshot.

**Governing constraint (unchanged throughout):** minimize delta to Roosevelt. Reuse
VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog
steps are the Section 9 shim register in the buildout design (OpenTofu node-VM
creation, `tc netem` WAN simulation, single-unit controllers/rbd-mirror).

---

## Stage 0 -- Decision ratification (CLEARED 2026-07-09)

**RATIFIED.** D-100..D-110 are ALL ADOPTED (`docs/design-decisions.md`;
D-102 MERGED into D-101 per the operator's own ruling, not deleted --
retained for numbering/history; D-109 confirmed standalone). All 6
redline items ruled (`docs/dc-dc-buildout-design.md` Section 10 is now a
ruling record, not an open list): Ceph size=3-by-default (size=2 only as an
explicit Phase-0 fallback); netem same-metro lean; D-102/D-101 merge +
D-109 standalone; metal-admin GAINS a ULA leg (a real reversal of the
stated lean, not a rubber-stamp); COS per-DC-only; mirror sync independent
per-DC. Verified via `bash scripts/ledger-scan.sh`: none of D-100..D-110
appear under "PROPOSED / OPEN decisions." Full ruling record:
`docs/changelog-20260709-stage0-ratification.md`.

**State:** DONE.

---

## Stage 1 -- Phase 0: vcloud host preparation

| | |
|---|---|
| **Goal** | Turn the bare vcloud host into a substrate Office1 can deploy from. |
| **Build** | Nested KVM enabled; libvirt storage pools + thin-provisioning; dark-fiber + per-site ISP-edge virtual networks; the six per-DC plane segments as virtual networks; MTU plan set on the underlay. |
| **Gate** | Nested KVM verified; disk budget computed + Ceph size decision recorded; measured MTU recorded; virtual networks present and isolated as designed. |
| **Owns** | D-100 (fabric), D-101 (MTU -- folds in former D-102), buildout-design Section 3 (host/disk/MTU gates). |
| **Reuse vs new** | NEW. No VR0-DC0 analog -- that testcloud IS the single vcloud host, already prepared; this is preparing the host to carry TWO independent DC substrates plus Office1. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase0-vcloud-prep.md`** -- command-level steps (host identify -> measure CPU/RAM/disk/nested-KVM/MTU -> enable nested KVM -> prepare pool paths -> install/confirm OpenTofu -> tfvars -> init/validate/plan/apply -> post-apply gate verify), each MUTATION individually gated per this repo's discipline. Flags a real open structural question up front (the `provider "maas"` block in `main.tf` is unconditional even though this stage only touches libvirt resources -- may force `maas_api_url`/`maas_api_key` to be set at `plan` time with nothing yet using them; noted as a possible DOCFIX candidate, not silently worked around). NOT YET EXECUTED against real infrastructure -- this is the first runbook that will be, tomorrow morning, per the operator's own stated plan. `opentofu/` SCAFFOLD started 2026-07-09: `modules/dc-planes` (six per-DC planes) + `modules/mesh-link` (dark-fiber legs) + `modules/dc-storage-pool` wired for DC1 + Office1 -- see `opentofu/README.md` for what's built vs deliberately deferred (node-VM/domain resources, DC2 CIDRs, netem). UNVALIDATED: no `tofu` binary available to run `scripts/opentofu-validate.sh` yet -- this runbook's Step 8 is the first real run. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Stage 2 -- Phase 1: Office1 headend standup

| | |
|---|---|
| **Goal** | Stand up the operator's deployment headend first, before any DC. |
| **Build** | MAAS region controller; OpenTofu; NetBox (vcloud-local, importer extended for multi-DC + dual-stack); GitBucket (vcloud-local mirror); Tailscale front door (Office1 only); Office1's own OPNsense simulated-ISP edge (gap #16, DECIDED 2026-07-09: Stage 2 owns it). |
| **Gate** | MAAS region reachable; NetBox authoritative + populated (planes, per-DC v4, ULA/GUA carve); GitBucket serving; OpenTofu reaches vcloud host libvirt; Tailscale confirmed to Office1 only; Office1-local network created (gap #12 CLOSED); Office1 OPNsense edge module wired (instantiation still blocked on real specs + gap #17's WAN-side network). |
| **Owns** | D-103 (Office1 VMs are OpenTofu-created), D-107 (headend is not a core-service provider). |
| **Reuse vs new** | NEW infra (Office1 didn't exist for VR0 DC0), but the *practices* transfer directly: NetBox-as-IPAM-apex discipline, the `netbox/*-import.py` pattern, and this repo's own git/GitBucket hosting model are all precedent, just extended. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09, UPDATED 2026-07-09: `runbooks/dc-dc-phase1-office1-standup.md`.** Gives each of the three service VMs (MAAS-region, NetBox, GitBucket) an explicit Option A (OpenTofu `cloudinit-vm`, BLOCKED pending `user_data`/`meta_data`/`network_config` design) vs. Option B (manual `virt-install`, logged as debt) fork rather than silently picking one. The Office1-local-network gap it originally flagged (gap #12) is now CLOSED (`opentofu/modules/office1-network`, instantiated for real), and a new Step 4b now creates Office1's own OPNsense edge (gap #16, CLOSED -- ownership decided). NOT YET EXECUTED. `opentofu/modules/base-image` + `cloudinit-vm` (2026-07-09) give the VM-creation mechanism -- neither instantiated yet: no image source chosen, and the real `user_data`/`meta_data`/`network_config` content for any of the three VMs hasn't been designed. See `opentofu/README.md`. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned).

---

## Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

| | |
|---|---|
| **Goal** | From Office1, script each DC's "physical" layer into existence. |
| **Build** | Per DC: OpenTofu creates node-VM libvirt domains (SHIM, Section 9), the DC's plane networks, and its OPNsense edge; registers the DC libvirt host to that DC's MAAS rack controller as a virsh VM-host. |
| **Gate** | MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured. |
| **Owns** | D-103 (seam), D-100 (edges). |
| **Reuse vs new** | NEW (no OpenTofu/multi-rack precedent in VR0 DC0). Explicitly tagged shim: node-VM creation has no Roosevelt analog (physical racking + BMC enlistment replaces it there) -- do not treat as reusable production IaC. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase2-tofu-dc-substrate.md`.** DC1-first; DC2 explicitly hard-gated (D-101 supernet unassigned) rather than run with placeholders. Walks every still-open decision (node sizing, MAAS zone/pool, `power_address`, config.xml tokens incl. boot-measured `WAN_IF`/`LAN_IF`, netem params) and stops at the corresponding step if unresolved rather than inventing values. States its own exit gate as "CONDITIONALLY MET AT BEST" pending Stage 2, node sizing, a real OPNsense boot measurement, and D-100's netem ruling. NOT YET EXECUTED. The plane/link/pool layer of `opentofu/` (see Stage 1) is reusable here too. `modules/node-vm` (blank disk, PXE-boot), `modules/opnsense-edge`, and `modules/maas-vm-host` (2026-07-09, registers the virsh host with MAAS via the official `canonical/maas` provider -- deliberately NOT `maas_vm_host_machine`, which composes new VMs and would fight `node-vm`) all now exist -- none instantiated (node sizing, OPNsense config.xml content, and a real MAAS zone/pool are all still pending decisions). `modules/netem-link` gives the netem mechanism for this stage's edge gate too. See `opentofu/README.md`. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED, hard-dependent on Stage 2 for its MAAS-registration half.

---

## Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

| | |
|---|---|
| **Goal** | Bring the discovered node VMs to a deployed OS, ready for Juju. |
| **Build** | Per DC: MAAS commissions + deploys the node VMs; provider NIC left RAW (br-ex discipline, D-100); VLAN-103 metal-internal bridge stack pre-built by MAAS. |
| **Gate** | Nodes deployed; six planes present per node with correct fabrics/VLANs; provider NIC raw; PXE (v4) working; per-DC mirror reachable; NTP from edge working. |
| **Owns** | D-103, D-100 (br-ex), D-107 (mirror + NTP). |
| **Reuse vs new** | PARTIAL reuse. `runbooks/phase-00-teardown-maas-reset.md` carries the MAAS commission/deploy + br-ex discipline this repo already validated for a single rack; the per-DC multi-rack registration is new, the deploy mechanics are precedent. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase3-maas-enlist-deploy.md`.** Adapts `phase-00-teardown-maas-reset.md`'s validated single-rack commission/deploy/carve mechanics per DC, run once per `$DC` (dc1/dc2). Requires `lib_net_select_dc "$DC"` first (DOCFIX-151); explicitly documents that `lib_hosts_select_dc "$DC"` is EXPECTED to fail for both DCs until this stage's own real host enrollment populates `lib-hosts.sh` -- a deliberate chicken-and-egg, not a bug. Surfaces 7 real gaps in a named section (no OpenTofu module stands up a per-DC MAAS rack controller VM itself; `reenroll-hosts.sh`/`carve-host-interfaces.sh`/`phase-00-maas-standup.sh` are not `$DC`-parameterized; node count/sizing undecided; DC2 additionally blocked at the selector layer; no identified owner for the per-DC artifact mirror; MAAS status for VM-host-discovered domains unconfirmed; `HOST_TAG`/fabric-name literals are DC0-specific). NOT YET EXECUTED. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

| | |
|---|---|
| **Goal** | A running independent OpenStack cloud per DC. |
| **Build** | Per DC: bootstrap the single-unit Juju controller in-DC; deploy the Caracal bundle; Vault init with a per-DC independent root, issuing v4 + IPv6-SAN certs; apply the family matrix (ULA on data/storage/replication, dual-stack metal-internal API, GUA provider). |
| **Gate** | `preflight.sh` PASS before add-model; post-deploy `cloud-assert.sh --capture` per DC; controller backup taken + stored; Ceph-over-v6 and geneve-over-v6 verified. |
| **Owns** | D-104 (controllers), D-109 (Vault), D-101 (families). |
| **Reuse vs new** | **This is the ground-truth/copy point** (`docs/dc-dc-buildout-design.md` line 12: seeded from `openstack-caracal-ipv4` at 8813efc). `bundle.yaml` and `runbooks/phase-01..08-*.md` are the per-DC template -- run essentially the existing VR0 DC0 deploy loop TWICE (once per DC), adapted for: independent Keystones (no cross-DC federation), the IPv6 family matrix (D-101/D-102), and per-DC Vault roots (D-109) instead of the single shared root. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase4-juju-bundle-per-dc.md`.** Does not duplicate `phase-01..08`'s content -- describes what changes to run that loop twice (once per DC): DC1 needs NO VIP/CIDR edits at all (D-101 inherits DC0 unchanged); DC2 is hard-blocked pending its NetBox literals. Requires every adapted `phase-01..08` command block that reads `lib-net.sh`/`lib-hosts.sh` to call `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` first (DOCFIX-151). Flags a real, unresolved design gap: the D-101 IPv6 family-matrix overlay (dual-stack/ULA-only per-plane charm config) does not exist yet -- the exact charm config option names for v6 addressing have not been confirmed against any charm's real schema, so no overlay file is invented; Step 6 names this as blocking work rather than gliding past it. Also flags D-109's IPv6-SAN cert-issuance support as an unverified charm-behavior assumption. NOT YET EXECUTED. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED. **This is where the existing repo tooling (`preflight.sh`,
`cloud-assert.sh`, `scripts/lib-net.sh`/`lib-hosts.sh`, `run-logged.sh`) does the
most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they
apply to either DC's clone/session without modification.**

---

## Stage 6 -- Phase 5: DR wiring and failover drill

| | |
|---|---|
| **Goal** | Prove Tier-A recoverability, one-way first, then two-way. |
| **Build** | Replication-plane peering (ULA); radosgw multisite zonegroup (two DCs as zones); rbd-mirror for the Glance pool; cinder-backup to the replicated object store; consistency groups where needed. Run the failover/failback drill. |
| **Gate** | One-way drill clean (RTO/RPO measured); then two-way; controller restore drill (D-104) exercised; RTO recorded as a measured output, not an SLA. |
| **Owns** | D-108 (replication), D-104 (control-plane backup/restore in the drill). |
| **Reuse vs new** | Ground truth: `docs/dc-dc-replication-DR-seed.md` (the original Tier-A/bidirectional/cinder-backup+rbd-mirror decision) -- D-108 corrected its carrier from the seed's v4 replication plane to IPv6-only ULA. The failover/failback drill skeleton is already drafted (buildout design Section 8): confirm DC1 down -> promote Glance -> restore Cinder -> rebuild instances -> record RTO/RPO; failback is split-brain-safe (no auto-resume, controlled window to flip primary back). |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase5-dr-failover-drill.md`.** Command-level implementation of Section 8's failover/failback skeleton, staged one-way-then-two-way per D-108's own requirement, plus the D-104 controller backup/restore drill. Found and flagged three real bundle gaps while writing it (grepped `bundle.yaml`): no `cinder-backup` charm deployed, no `ceph-rbd-mirror` charm deployed, and `ceph-mon`'s existing `rbd-mirror` endpoint binding is on `storage` instead of `replication` (inherited from the VR0/DC0 seed, predates D-108) -- all three become this runbook's own first gated mutation (Step 2), not silently routed around. Every realm/zone/pool/endpoint name is an explicit `<UPPER_SNAKE_TBD>` placeholder. Includes a negative-control sub-drill (sever only the replication link first, confirm the split-brain check correctly refuses to promote) before the real hard-down drill. NOT YET EXECUTED -- no live Ceph cluster this session (harness for gap #5's scripts is future work, logged, not built). |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

| | |
|---|---|
| **Goal** | Complete the service set in both DCs. |
| **Build** | Per DC: Designate (designate-bind) with per-DC zones + A/AAAA per the D-008 bootstrap order; COS observability; Magnum/CAPI with in-cloud single-homed capi-mgmt (`domain-setup` re-run per D-046). |
| **Gate** | Designate resolving A + AAAA; FQDN-SAN certs valid (D-019/D-021 root cause closed); COS scraping; a CAPI workload cluster comes up per DC. |
| **Owns** | D-106 (Designate), D-105 (CAPI/COS). |
| **Reuse vs new** | HIGH REUSE for CAPI/Magnum: `runbooks/phase-06-incloud-mgmt-cluster.md` + `phase-07-conductor-graft.md` + `phase-08-workload-cluster-acceptance.md` are direct precedent, per-DC. Designate is a REVERSAL, not new build: VR0 DC0 deferred it (D-019, "DNS scope reduction"); D-106 reactivates it (supersedes D-019, reactivates the original D-008 bootstrap order) -- read D-008 and D-019 together before writing this stage's runbook, the static-`/etc/hosts` workaround they document goes away. |
| **Authoring status** | **Runbook WRITTEN 2026-07-09: `runbooks/dc-dc-phase6-designate-cos-magnum.md`.** Quotes `bundle.yaml`'s and `phase-01-bundle-deploy.md`'s actual current "NO designate (D-019)" text verbatim rather than paraphrasing, then the D-106 reversal against it. Flags that the reversal is bigger than "add designate" -- D-106's bootstrap order also reactivates `os-public-hostname` per API charm, reversing the bundle's separate B5 IP-only posture; proposes (not silently adopts) pushing per-DC hostname literals into new `overlays/dc1-hostnames.yaml`/`dc2-hostnames.yaml` files to keep `bundle.yaml` itself DC-agnostic, mirroring the existing octavia-pki overlay pattern. D-046's domain-setup trap quoted verbatim in a top callout and again inline. COS deployment mechanism explicitly left as an operator decision point (D-105 rules only the scope) rather than invented. Reuses `phase-06/07/08` by filename with concrete DC-parameterization notes (confirmed via tracing which scripts source `lib-net.sh`). NOT YET EXECUTED. |

**State:** RUNBOOK WRITTEN, NOT YET EXECUTED.

---

## Cross-cutting discipline (applies at every stage, every session)

- **Session bootstrap, every session:** `git pull` -> `bash scripts/repo-lint.sh`
  (0 fail / 1 documented legacy WARN expected) -> `bash scripts/ledger-scan.sh`
  to reconcile open decisions/next-free numbers against `docs/session-ledger.md`.
- **Any session that will mutate the cloud:** `bash scripts/run-logged.sh <label>`
  first (as-executed log).
- **Before any add-model-like mutation:** `bash scripts/preflight.sh` PASS.
- **After any deploy/restart/change, and as an incident-triage first step:**
  `bash scripts/cloud-assert.sh` (add `--capture` at real deploy-completion
  milestones for the committed BOM).
- **Before touching ANY built surface:** grep `docs/design-decisions.md` for the
  governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present
  options, do not implement.
- **Every deliverable (script, runbook, doc):** `repo-lint` clean, the touched
  script's harness green (write one if missing), a changelog entry with a
  per-item revert, and a `docs/session-ledger.md` update.
- **Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN:** `bash
  scripts/ledger-scan.sh` for next-free (it is the numbering authority, not
  repo-lint's informational count).
- **Hard rule reminder:** no inferred values -- every IP/ID/name/scope is
  measured this session or carried from confirmed as-built; verify before
  mutate, always; destructive/secret-handling steps are individually gated,
  never batched.

---

## Tooling gap register (audited 2026-07-09; update as items close)

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not
inferred from the design doc alone. Ordered roughly by "you'll hit this first,"
not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;
#2 is the single largest build.

1. **`$DC` selector convention -- CLOSED 2026-07-09 (DOCFIX-151).**
   `scripts/lib-net.sh` gained `lib_net_select_dc()` (dc0/dc1 no-op, since
   D-101 rules DC1 inherits DC0's v4 layout unchanged; dc2 fails loud --
   NetBox literals not yet assigned, gap #3) and `scripts/lib-hosts.sh`
   gained `lib_hosts_select_dc()` (dc0 no-op; dc1 AND dc2 BOTH fail loud --
   no real per-DC host enrollment exists yet for either, an intentional
   asymmetry vs. the net selector, documented in both files' comments and
   `docs/changelog-20260709-dc-selector-convention.md`). Backward compatible
   by construction -- every existing caller of either library is unaffected.
   New harness `tests/dc-selector/run-tests.sh`: 21/21 PASS. The mechanism
   exists; Stage 5's runbook still needs to actually CALL
   `lib_net_select_dc "$DC"` / `lib_hosts_select_dc "$DC"` when it's
   authored -- that call-site wiring is part of writing that runbook, not
   this delivery.
2. **OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded
   2026-07-09 (was: zero files).** `opentofu/` now has: `modules/dc-planes`
   (six per-DC planes), `modules/mesh-link` (the D-100 dark-fiber triangle
   legs), `modules/dc-storage-pool`, `modules/node-vm` (blank disk, PXE-boot
   MAAS-managed node VM, the D-103 pattern for Stage 3), `modules/base-image`
   + `modules/cloudinit-vm` (the OTHER VM pattern -- pre-built image +
   cloud-init, for Office1's own service VMs), `modules/opnsense-edge` (see
   gap #4), `modules/maas-vm-host` (registers the vcloud host with MAAS as a
   virsh VM host -- via the official `canonical/maas` provider, confirmed
   against its real schema; deliberately NOT `maas_vm_host_machine`, which
   composes/creates new VMs itself and would fight `node-vm` for VM
   creation -- D-103 explicitly rules out MAAS composing), and
   `modules/netem-link` (`terraform_data` + SSH-wrapped `local-exec`
   provisioners applying `tc qdisc ... netem` to a mesh-link bridge --
   OpenTofu runs from Office1, not the vcloud host, so a bare local command
   can't reach the bridge). DC1 + Office1 pool/plane wiring is instantiated
   in root `main.tf`; everything else is NOT (node/VM sizing, MAAS zone/pool,
   netem parameters are all pending decisions -- neither invented here).
   Provider schema was confirmed against the actual current docs AND real
   example `.tf` files (not memory) -- it differs materially from older
   common examples: nested structures are attribute-style objects (`key =
   { ... }`), not classic HCL blocks. **A real bug from the first pass was
   found and fixed:** `dc-planes`/`mesh-link`/`dc-storage-pool` originally
   used block syntax for `libvirt_network`'s `domain`/`mtu` and
   `libvirt_pool`'s `target`; corrected once real provider examples surfaced
   the actual convention -- discovered precisely BECAUSE building the domain
   module forced fetching real example code instead of doc-summaries alone.
   Still missing, deliberately deferred (see `opentofu/README.md`):
   OPNsense's own bootability via `cloudinit-vm` is NOT confirmed (FreeBSD,
   not the same cloud-init/NoCloud guarantee as Linux images); the
   per-device `boot`-order attribute's exact internal shape (flagged
   UNVERIFIED in `node-vm/main.tf`, an inference by pattern not a confirmed
   field); and whether `devices.interfaces` list order reliably maps to
   LAN/WAN NIC enumeration (flagged in `opnsense-edge/main.tf`).
   **UNVALIDATED** -- no `tofu` binary was available to run
   `scripts/opentofu-validate.sh` this session; run it on a connected
   machine before trusting any of this.
3. **NetBox multi-DC/dual-stack import pipeline -- MECHANISM CLOSED
   2026-07-09 (DOCFIX-152), literals still pending.** `netbox/dc-dc-prefixes-
   import.py` extends the v1 pattern to VR1's two-DC dual-stack model:
   per-DC (`--dc dc1`/`dc2`) creates the six-plane IPv4 (DC1 inherited,
   hardcoded per D-101; DC2 via required `DC2_V4_SUPERNET`, no default) plus
   the D-101 family-matrix IPv6 legs (ULA/GUA), carved from two REQUIRED
   env vars (`ORG_ULA_48`, `DC_GUA_PREFIX`) with no defaults -- fails loud
   rather than inventing either. 40/40 tests (`tests/dc-dc-prefixes-
   import/`, fake-NetBox end-to-end). UNVALIDATED against a real NetBox (no
   instance reachable this session). The literals themselves (org ULA /48,
   per-DC GUA carve, DC2 supernet) still don't exist -- that operator/NetBox
   assignment step is unchanged by this delivery; this closes the TOOLING
   gap, not the DATA gap. Still blocks Stage 2/3 until those literals are
   assigned and this script is actually run against a live NetBox.
4. **OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) DONE
   2026-07-09, (d) is the only real gap left.** Each site (Office1, DC1, DC2)
   gets its own independent OPNsense edge with a simulated ISP uplink (D-100);
   OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge)
   and Stage 3 (per-DC edges) gates, which require netem parameters "applied
   and measured."
   - **(a) Network substrate -- DONE.** `opentofu/modules/mesh-link` creates
     the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an
     OPNsense VM's interfaces would eventually attach to.
   - **(b)+(c) The OPNsense VM/image AND its config -- BUILT
     (`opentofu/modules/opnsense-edge`), UNVALIDATED.** Cloud-init is
     confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum
     consensus, not just a hedge. The real native mechanism is the
     Configuration Importer (official docs): scans an attached volume for
     `/conf/config.xml` in a 2-3 second boot-time window; ISO9660 support was
     added to it specifically for VM/cloud automation (attach a config ISO as
     a secondary CD-ROM) -- mechanically IDENTICAL to `modules/cloudinit-vm`'s
     shape (a secondary volume attached as `cdrom`), just a different payload.
     Built as `opentofu/modules/opnsense-edge` + two prep scripts
     (`scripts/opnsense-prep-image.sh` for the nano image's decompress+
     convert, `scripts/opnsense-build-config-iso.sh` for the config.xml ISO).
     The real `config.xml` DESIGN is now also done:
     `opentofu/templates/opnsense-config.xml.tmpl` (a `{{TOKEN}}`-parameterized
     template built from OPNsense's own real shipped sample, covering
     interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall
     rules) + `scripts/opnsense-render-config.sh` (the renderer -- fully
     tested end-to-end, 8/8, unlike the other opnsense-* scripts, since it
     needs no external tool). UNVALIDATED regardless: no `tofu` binary, no
     ISO-building tool, and no real OPNsense boot were available to confirm
     the whole chain end-to-end. Several template tokens still can't be
     filled with real per-site values -- pending Stage 0 ratification
     (D-100/D-101/D-107) or only measurable on a real boot (which `vtnetN`
     device a network becomes). Full sourcing in `opentofu/README.md` and
     `opentofu/templates/README.md`'s token legend.
   - **(d) `tc netem` WAN-simulation -- mechanism BUILT (`modules/netem-link`),
     only the real parameters remain.** No Terraform/OpenTofu-native
     equivalent exists (confirmed against the `libvirt_network` schema, gap
     #2); built instead as a `terraform_data` + SSH-wrapped `local-exec`
     provisioner (OpenTofu runs from Office1, not the vcloud host where the
     bridge lives -- a bare local command can't reach it). UNVALIDATED, not
     instantiated. The exact parameters (latency/jitter/loss/rate) are
     themselves still an unruled D-100 open sub-item (gap #11) -- **a
     concrete PROPOSAL now exists** (`docs/dc-dc-netem-and-ula-gua-
     proposal.md`, DOCFIX-168, Part 1: 1ms/0.2ms jitter/0.01% loss/uncapped
     rate, with reasoning, presented for operator ratification, not
     silently applied) -- still needs an actual ruling before it can be
     applied for real.
5. **Ceph replication tooling (radosgw multisite + rbd-mirror) -- MECHANISM
   CLOSED 2026-07-09 (DOCFIX-165), data half still open.** `runbooks/dc-
   dc-phase5-dr-failover-drill.md` Steps 4, 5, 8, 10.2, and 11.2-11.4's
   `radosgw-admin`/`rbd mirror` command sequences (verbatim upstream Ceph
   administration, not invented) are now callable, parameterized scripts:
   `scripts/dc-dc-radosgw-multisite.sh` (subcommands `master-init`/`join-
   readonly`/`enable-two-way` -- Step 4 one-way staging + Step 8's radosgw
   half), `scripts/dc-dc-rbd-mirror.sh` (subcommands `bootstrap-primary`/
   `bootstrap-secondary --direction rx-only|rx-tx` -- Step 5 one-way staging
   + Step 8's rbd-mirror half), and `scripts/dc-dc-dr-drill.sh` (subcommands
   `failover` -- Step 10.2 force-promote + a flagged, never-auto-executed
   Glance re-registration reminder -- and `failback` -- Steps 11.2-11.4, with
   the demote-CURRENT-PRIMARY-before-promote-RECOVERING safety ordering
   HARD-CODED as fixed call order, not left to documentation). Every
   realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument, no
   invented defaults; default mode is a `--dry-run` plan print, `--apply`
   executes one command at a time via `juju ssh`/`juju run`, aborting on
   first failure. New harnesses `tests/dc-dc-radosgw-multisite/`,
   `tests/dc-dc-rbd-mirror/`, `tests/dc-dc-dr-drill/` (19/19, 19/19, 23/23
   PASS) test ONLY these scripts' own argument parsing, guard clauses,
   dry-run output, and (critically) the failback step ORDER -- NOT real
   radosgw/rbd-mirror behavior, since no live Ceph cluster exists this
   session to test against (same UNVALIDATED posture as gap #3's NetBox
   pipeline). The MECHANISM exists; the DATA half (real DC1/DC2 Ceph
   cluster, real names/endpoints, the three `bundle.yaml` gaps in the Stage
   6 runbook's own "Known gaps" section) is unchanged and still blocks Stage
   6 for real. See `docs/changelog-20260709-ceph-replication-tooling.md`.
6. **Designate reactivation is a bundle change -- CLOSED 2026-07-10
   (DOCFIX-167).** `bundle.yaml` now carries `designate`/`designate-
   bind`/`designate-mysql-router`/`designate-hacluster` + 8 relations,
   researched against designate's real metadata.yaml (a real mistake
   caught during authoring: its provides-side endpoint is `dnsaas`, not
   `public` like the other API charms -- corrected). `os-public-hostname`
   re-activation (D-106's OTHER coupled change) deliberately NOT done in
   the shared bundle -- pushed to a proposed per-DC overlay instead, to
   keep `bundle.yaml` DC-agnostic. `python3 scripts/provider-bundle-
   check.py` PASS, all 6 existing invariants unaffected. See
   `docs/changelog-20260709-designate-cinderbackup-rbdmirror.md`. NOT
   applied to any live model -- repo-source change only.
7. **MTU/geneve budget + Ceph disk-budget calculators -- CLOSED 2026-07-09
   (DOCFIX-162).** D-102 and buildout-design Section 3 both describe
   measurement-then-computation gates (host L2 MTU -> geneve-over-v6
   overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback)
   that were manual arithmetic in prose, not a script, blocking the Stage 1
   Phase-0 gates. `scripts/dc-dc-mtu-geneve-budget.sh` takes the MEASURED
   underlay MTU via required `--underlay-mtu` (no default) and reproduces
   D-101's own worked example exactly (1500 - 56 = 1444), with a jumbo
   (>=9000) path leaving tenant MTU at 1500, plus an added IPv6-minimum-MTU
   (1280, RFC 8200) sanity floor clearly separated from D-101's own text.
   `scripts/dc-dc-ceph-disk-budget.sh` takes MEASURED total disk, per-DC
   OSD node counts and per-node footprints, and a required
   `--backup-overhead-fraction` (deliberately NO default -- not specified
   as a hard number anywhere in this repo, must come from real
   Ceph/radosgw-admin measurement once DC1 exists) and computes whether
   size=3 across both DCs plus overhead fits the measured total disk;
   if not, it names size=2 as the ONLY documented fallback (D-101 /
   Section 3) without silently applying it -- explicitly an operator-logged
   decision. New harnesses `tests/dc-dc-mtu-geneve-budget/run-tests.sh`
   (19/19 PASS) and `tests/dc-dc-ceph-disk-budget/run-tests.sh` (16/16
   PASS). Both scripts are read-only/mutate nothing and have not yet been
   run against the real vcloud host (prep-only session, no live
   infrastructure reachable) -- that first real run is part of executing
   `runbooks/dc-dc-phase0-vcloud-prep.md` Step 3. Changelog:
   `docs/changelog-20260709-mtu-ceph-budget-calculators.md`.
8. **The "developed IPv6 plan" D-101 cites does not exist in this repo.** D-101's
   context line references "the developed family-follows-reachability IPv6 plan
   (in-project, not yet committed)" as prior art -- grepped the whole repo, it is
   not present as a standalone doc anywhere. If the exact Phase A/B/C detail
   behind D-101/D-102 is needed by a future session, it currently is NOT
   retrievable from the repo alone. Not a blocker for Stage 0 ratification of
   what IS written, but a real content gap if finer detail is needed later.
9. **Runbook debt -- CLOSED 2026-07-09.** All seven Stage 1-7 runbooks are now
   written (`runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-
   designate-cos-magnum.md`), Stage 1 authored directly, Stages 2/3/4/6/7
   delegated to parallel subagents and Stage 5 authored directly, all
   reviewed and repo-lint clean. NONE have been executed against real
   infrastructure yet -- this was a prep-only session (the operator was away
   from the workstation, not the vcloud host). Each runbook is explicit about
   what it can and cannot close given today's real state (NetBox literals,
   node sizing, and several other decisions remain genuinely open -- see
   each runbook's own GATE section for the honest per-stage breakdown).
10. **Harness debt -- LARGELY CLOSED 2026-07-09/10.** This repo's own delivery
    rule -- no script change ships without its `tests/<name>/run-tests.sh`
    harness -- has been honored for every new script tonight: `dc-selector`
    (21/21), `dc-dc-prefixes-import` (40/40), `dc-dc-mtu-geneve-budget`
    (19/19), `dc-dc-ceph-disk-budget` (16/16), `dc-dc-radosgw-multisite`
    (19/19), `dc-dc-rbd-mirror` (19/19), `dc-dc-dr-drill` (23/23), plus new
    `$DC` test cases added to `carve-host-interfaces`/`phase-00-maas-
    standup` and a new `reenroll-hosts` harness. Gap #2 (OpenTofu modules)
    remains the one area still lacking a harness beyond `scripts/opentofu-
    validate.sh` itself (no `tofu` binary this session to validate against).
11. **Decision sub-items still genuinely open** (the big items are RULED per
    Stage 0; these specific literals/params are not -- but see
    `docs/dc-dc-netem-and-ula-gua-proposal.md` (DOCFIX-168) for a concrete
    netem proposal and exact, safe ULA-generation guidance, drafted
    2026-07-10): exact netem parameters
    (D-100), exact NetBox literals for DC2/ULA/GUA (D-101).
12. **No Office1-local virtual network is modeled anywhere in `opentofu/` --
    CLOSED 2026-07-09.** `main.tf` previously wired only `office1_pool_path`
    (storage only) and the three inter-site mesh links (management-traffic-
    only per D-100's own sub-item ruling) -- nothing the three Office1
    service VMs (MAAS-region, NetBox, GitBucket), or the vcloud host's own
    management interface, could actually attach to as a shared local network.
    Resolved by a NEW `opentofu/modules/office1-network` module (a
    `dc-planes`-shaped isolated `libvirt_network`, sized for ONE network
    rather than six planes since Office1 is a headend, not an OpenStack DC),
    deliberately NOT a reused host bridge -- a hand-configured bridge would
    be the one network in this topology living outside OpenTofu's D-103-
    mandated inventory, silently breaking this repo's own `virsh net-list
    --all` completeness assumption. INSTANTIATED for real in root `main.tf`
    (`module "office1_network"`) -- unlike most of gap #2's other scaffolding,
    this needed no unmeasured value beyond `domain_suffix`/`underlay_mtu`,
    already real required inputs elsewhere in the same file. See
    `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md`
    (DOCFIX-163) for the full design rationale. Designing the three
    service VMs' actual `cloudinit-vm` `network_config` content against this
    network remains separate, un-actioned work.
13. **The D-101 IPv6 family-matrix overlay -- DRAFTED 2026-07-10, one real
    risk still open.** `overlays/dc-dc-ipv6-family-matrix.yaml` now exists,
    sourced against real charm config/source (`docs/dc-dc-ipv6-charm-
    research.md` has full citations): `prefer-ipv6` (confirmed real option,
    genuinely additive dual-stack HAProxy binding per `charm-nova-cloud-
    controller`'s actual commit, not an either/or switch) + appended v6/GUA
    VIPs for the API charms; `ceph-mon`'s own separate `prefer-ipv6` +
    `ceph-public-network`/`ceph-cluster-network` for the ULA-only planes;
    confirmed NO charm-config option exists for OVN (geneve family follows
    the bound interface automatically). D-109's IPv6-SAN cert support is
    CONFIRMED (read `charm-vault`'s actual `sort_sans()` code -- no v4/v6
    distinction). **Genuinely unresolved:** Octavia's `lb-mgmt-net` IPv6
    support is excluded from the overlay on purpose -- two real, still-
    referenced upstream Launchpad bugs (#1911788, #1913409) document IPv6
    lb-mgmt-net failures; needs an explicit operator decision (accept the
    risk and test for real, or keep Octavia's lb-mgmt-net as a deliberate
    v4-only D-101 exception), not silently resolved here. Also unconfirmed:
    8 of the 9 dual-stack charms' `prefer-ipv6` presence is inferred by
    shared-framework pattern, not individually fetched. UNVALIDATED against
    any live model.
14. **Three real `bundle.yaml` gaps for D-108's DR mechanism -- CLOSED
    2026-07-10 (DOCFIX-167).** `cinder-backup` (a SUBORDINATE charm,
    confirmed via real metadata.yaml -- the first draft wrongly gave it
    principal-style `num_units`/`to`/`amqp`, corrected) and `ceph-rbd-
    mirror` (a principal with `ceph-local`/`ceph-remote` requires
    endpoints, confirmed via real metadata.yaml -- the first draft
    invented a wrong binding key, corrected) are now in `bundle.yaml`;
    `ceph-mon`'s `rbd-mirror` binding corrected from `storage` to
    `replication`. **Genuinely unresolved, documented not invented:**
    `ceph-remote` (the peer DC's ceph-mon) is a cross-Juju-controller
    relation D-104's per-DC-independent-controller design means cannot be
    a same-bundle relation line -- only this DC's own `ceph-local` leg is
    wired; the cross-DC peer relationship needs a `juju offer`/`consume`
    or the manual `rbd mirror pool peer bootstrap` CLI flow (Stage 6's
    runbook already documents the latter). `python3 scripts/provider-
    bundle-check.py` PASS. See `docs/changelog-20260709-designate-
    cinderbackup-rbdmirror.md`. NOT applied to any live model.
15. **Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the
    Stage 4 runbook** (full detail in that runbook's own "Known gaps"
    section). **Sub-item 2 (script `$DC`-parameterization) CLOSED
    2026-07-10 (DOCFIX-166)** -- `reenroll-hosts.sh`,
    `carve-host-interfaces.sh`, and `phase-00-maas-standup.sh` now accept an
    opt-in `$DC` env var and call `lib_net_select_dc`/`lib_hosts_select_dc`
    immediately after sourcing (unset `$DC` is byte-for-byte unchanged
    behavior; see `docs/changelog-20260709-maas-scripts-dc-param.md`). This
    closes the CLI-invocation half of the gap only -- it does NOT create any
    new per-DC data (no dc1/dc2 hostnames, octets, boot MACs, or node-count
    figures were invented), and `phase-00-maas-standup.sh`'s own `PLANES`
    table remains hardcoded to D-052/D-053's literals rather than driven by
    `lib-net.sh`'s flat vars, so it is not truly dc2-topology-aware yet even
    though it now gates dc2 out cleanly. The other six sub-items remain
    OPEN: no OpenTofu module stands up a per-DC MAAS rack controller VM
    itself (only registration, `modules/maas-vm-host`, exists); per-DC node
    count/sizing is an unmade Phase-0 decision; DC2 is still blocked at the
    network-selector layer (gap #3, NetBox literals not assigned); the
    per-DC artifact mirror (D-107) has no identified owning stage/module;
    the MAAS status a VM-host-discovered (vs. PXE-enlisted) machine lands in
    is unconfirmed; `HOST_TAG`/boot-fabric-name literals are DC0-specific.
16. **No stage explicitly owns creating Office1's own OPNsense edge VM --
    CLOSED 2026-07-09.** The topology (buildout design Section 1) is
    explicit: "full dark-fiber triangle... plus a **per-site** OPNsense
    simulated ISP edge" -- three sites, three edges (DC1, DC2, AND Office1).
    Stage 3's Build list already covered DC1/DC2's edges
    (`modules/opnsense-edge` per DC); Stage 2's Build list (MAAS region,
    OpenTofu, NetBox, GitBucket, Tailscale) did not mention an Office1 edge
    at all, and neither runbook instantiated one. **Decision: YES, Office1
    gets its own `modules/opnsense-edge` call, a fourth site alongside
    DC1/DC2; Stage 2's runbook
    (`runbooks/dc-dc-phase1-office1-standup.md`) owns creating it.**
    Rationale: Stage 2's entire scope is already "Office1 headend standup"
    (the OPNsense edge IS part of that headend, not DC substrate), and
    Stage 1 already creates the mesh-link networks this edge's neighbors
    depend on -- Stage 3 has no reason to reach back and create a piece of
    Office1's own infrastructure. Wired as a commented-out `module
    "office1_opnsense"` skeleton in root `main.tf` (mirroring the identical
    NOT-YET-REAL status of DC1/DC2's own edges -- neither is instantiated
    either) plus a new gated step in
    `runbooks/dc-dc-phase1-office1-standup.md` (Step 4b) and its own GATE
    bullet. Related to, but distinct from, gap #12 (Office1-local network,
    now also CLOSED) -- that gap was the service VMs' own attach point;
    this one was whether Office1 gets a simulated-ISP-edge boundary at all,
    and who builds it. The edge's LAN side now resolves cleanly to gap
    #12's new network; its WAN side does not -- see NEW gap #17
    immediately below, a genuinely deeper, cross-site gap surfaced while
    closing this one, not invented as a substitute for a real answer.
17. **No dedicated per-site ISP-uplink/WAN-side network exists anywhere in
    this repo, for ANY site** (found 2026-07-09 while closing gap #16 for
    Office1). Every `modules/opnsense-edge` caller needs a
    `wan_network_name` distinct from BOTH its LAN-side network (a
    `dc-planes` plane, or gap #12's new `office1-network`) AND the three
    D-100 mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC-
    ONLY by D-100's own sub-item ruling (Office1<->DC fiber carries MAAS/
    Juju/operator traffic only), not a simulated ISP uplink, so wiring an
    edge's WAN side to a mesh leg would misrepresent the topology rather
    than merely placeholder it. This is NOT an Office1-specific gap:
    `runbooks/dc-dc-phase2-tofu-dc-substrate.md` Step 5's own `dc1_opnsense`
    template carries the IDENTICAL unresolved placeholder
    (`wan_network_name = module.mesh_dc1_office1... # or a dedicated
    per-site ISP-uplink segment -- confirm which, do not assume`) --
    confirmed by reading that runbook directly, not assumed. Deliberately
    NOT invented here for Office1 alone (that would contradict D-100 and
    leave DC1/DC2 with the same unresolved gap under a different name).
    Needs a decision: a fourth network shape per site (one more
    `dc-planes`/`mesh-link`/`office1-network`-style dedicated
    `libvirt_network` standing in for that site's own "internet" uplink),
    or some other mechanism not yet considered. Blocks instantiating ANY of
    the three sites' `opnsense-edge` calls (`dc1_opnsense`, `dc2_opnsense`,
    `office1_opnsense`) end-to-end -- all three are otherwise ready
    (image/config mechanism per gap #4, LAN-side network per gap #12/this
    stage's plane modules) but for this one missing piece.

---

## Where things stand today (2026-07-09) -- update this as stages close

| Layer | Status |
|---|---|
| Design (`dc-dc-buildout-design.md`) | DONE -- Section 10 now a ratified-ruling record; the doc's own header still reads "DESIGN / PLAN" (a status line, not a blocker -- its embedded decisions are ADOPTED per Stage 0 below) |
| Decisions (D-100..D-110) | **ADOPTED 2026-07-09** (D-102 merged into D-101; D-109 standalone) -- Stage 0 CLEARED |
| DR mechanism seed | DONE (`dc-dc-replication-DR-seed.md`), superseded-carrier corrected into D-108 |
| Stage 0-7 runbooks | Stage 0 DONE (ratification). **Stages 1-7 ALL WRITTEN 2026-07-09** (gap #9 CLOSED) -- `runbooks/dc-dc-phase0-vcloud-prep.md` through `dc-dc-phase6-designate-cos-magnum.md`. NONE executed against real infrastructure (prep-only session). See gaps #12-15 above for real design gaps the authoring pass surfaced. |
| `opentofu/` (networks/pools/node-VM PXE/cloud-init-VM patterns) | SCAFFOLDED 2026-07-09, UNVALIDATED (no `tofu` binary available this session) -- see gap #2 and `opentofu/README.md` |
| OPNsense image+config mechanism (`modules/opnsense-edge`) | BUILT 2026-07-09, UNVALIDATED (no `tofu`/ISO-tool/real boot to confirm), see `opentofu/README.md` |
| OPNsense config.xml design (`templates/opnsense-config.xml.tmpl`) | BUILT 2026-07-09, renderer fully tested (8/8) -- several tokens still pending Stage 0 ratification or a real boot, see `opentofu/templates/README.md` |
| MAAS VM-host registration (`modules/maas-vm-host`) | BUILT 2026-07-09, UNVALIDATED -- needs a real MAAS zone/pool + vcloud power_address, see `opentofu/README.md` |
| `tc netem` mechanism (`modules/netem-link`) | BUILT 2026-07-09, UNVALIDATED -- real latency/jitter/loss/rate parameters still an unruled D-100 item, see `opentofu/README.md` |
| `opentofu/` (DC2 planes) | NOT DONE -- deliberately deferred pending NetBox CIDR assignment, see `opentofu/README.md` |
| Office1-local network (`modules/office1-network`) | BUILT + INSTANTIATED 2026-07-09 (DOCFIX-163, gap #12 CLOSED) -- see `opentofu/README.md` and `docs/changelog-20260709-office1-network-edge.md` |
| Office1 OPNsense edge ownership | DECIDED 2026-07-09 (DOCFIX-163, gap #16 CLOSED) -- Stage 2 owns it; `module "office1_opnsense"` skeleton in `main.tf` (commented, same as DC1/DC2's own not-yet-instantiated edges); WAN-side network still blocked on NEW gap #17 (no per-site ISP-uplink network exists anywhere yet, DC1/DC2 included) |
| Reusable as-is, repo-agnostic (2026-07-09 sweep) | `bundle.yaml`, `phase-01..08` VR0 DC0 runbooks (Stage 5/7 template, once DC-parameterized -- gap #1), `preflight.sh`, `cloud-assert.sh`, `repo-lint.sh`, `run-logged.sh`, `ledger-scan.sh` |
| `lib-net.sh` / `lib-hosts.sh` | `$DC` selector mechanism ADDED 2026-07-09 (DOCFIX-151, gap #1 CLOSED) -- `lib_net_select_dc()`/`lib_hosts_select_dc()`, backward compatible, 21/21 tests; Stage 4 and Stage 5's runbooks (both written 2026-07-09) mandate calling it, not yet executed for real |
| NetBox DC-DC pipeline (`netbox/dc-dc-prefixes-import.py`) | MECHANISM BUILT 2026-07-09 (DOCFIX-152, gap #3 tooling half CLOSED) -- 40/40 fake-NetBox tests (DOCFIX-160 fixed a dead-assertion test bug that had silently left 3 of these uncounted), UNVALIDATED live; real literals (ULA /48, GUA carve, DC2 supernet) still unassigned |
| Environment/skill docs (`environment.md`, README.md) | Still describe VR0 DC0/v1 single-DC scope wholesale -- flagged, not yet rewritten for VR1; not a blocker for Stage 0 but will need doing before/during Stage 1 |