DC-DC (VR1) deployment workflow overview

Purpose. A single followable checklist across the whole VR1 (two-DC virtual regional) buildout, spanning many sessions. docs/dc-dc-buildout-design.md is the WHY/WHAT (goals, gates, D-refs per phase); this doc is the HOW-TO-FOLLOW-IT sequence, cross-referenced to that plan, to D-100..D-110, and to what already exists in this repo vs what still needs to be authored. Update the "state" column as stages complete -- this doc is a tracker, not a one-time snapshot.

Governing constraint (unchanged throughout): minimize delta to Roosevelt. Reuse VR0 DC0 material wherever it transfers; the only deliberate no-Roosevelt-analog steps are the Section 9 shim register in the buildout design (OpenTofu node-VM creation, tc netem WAN simulation, single-unit controllers/rbd-mirror).

Stage 0 -- Decision ratification (CLEARED 2026-07-09)

RATIFIED. D-100..D-110 are ALL ADOPTED (docs/design-decisions.md; D-102 MERGED into D-101 per the operator's own ruling, not deleted -- retained for numbering/history; D-109 confirmed standalone). All 6 redline items ruled (docs/dc-dc-buildout-design.md Section 10 is now a ruling record, not an open list): Ceph size=3-by-default (size=2 only as an explicit Phase-0 fallback); netem same-metro lean; D-102/D-101 merge + D-109 standalone; metal-admin GAINS a ULA leg (a real reversal of the stated lean, not a rubber-stamp); COS per-DC-only; mirror sync independent per-DC. Verified via bash scripts/ledger-scan.sh: none of D-100..D-110 appear under "PROPOSED / OPEN decisions." Full ruling record: docs/changelog-20260709-stage0-ratification.md.

State: DONE.

Stage 1 -- Phase 0: vcloud host preparation

State: DONE -- executed 2026-07-10 (first DC-DC runbook run against real infrastructure; vcloud host, OpenTofu v1.12.3). As-built: nested KVM already-on; underlay_mtu=9000 (jumbo, operator ruling -- the isolated planes/mesh are host-internal virtio bridges, jumbo-capable regardless of the 1500 ISP uplink); Ceph size=3 disk-budget plausibility PASS (4.60 TiB margin); 13 libvirt objects applied (6 dc1 planes + dc1/office1/dc2 pools + office1-local + 3 mesh legs, all MTU 9000). Prior VR0 topology torn down (wan kept as a gap-#17 model). DC2 PLANES deferred (Option B -- awaits NetBox supernet); DC2 storage pool wired. Delivered DOCFIX-178 (prereq installers), DOCFIX-179 (scaffold: per-module required_providers + fmt + maas-provider deferral). See those changelogs.

Stage 2 -- Phase 1: Office1 headend standup

State: RUNBOOK WRITTEN, NOT YET EXECUTED. Honest exit state per the runbook's own GATE section: MAAS/GitBucket/Tailscale/OpenTofu-reach bullets closeable in one run; NetBox's "authoritative and populated" bullet only PARTIAL (mechanism done via DOCFIX-152, literals still unassigned).

Stage 3 -- Phase 2: OpenTofu builds each DC substrate (from Office1)

State: RUNBOOK WRITTEN, NOT YET EXECUTED, hard-dependent on Stage 2 for its MAAS-registration half.

Stage 4 -- Phase 3: MAAS enlist / commission / deploy (per DC)

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Stage 5 -- Phase 4: Juju controller + OpenStack bundle (per DC) -- HIGH REUSE

State: RUNBOOK WRITTEN, NOT YET EXECUTED. This is where the existing repo tooling (preflight.sh, cloud-assert.sh, scripts/lib-net.sh/lib-hosts.sh, run-logged.sh) does the most direct work -- confirmed repo-agnostic as of the 2026-07-09 sweep, so they apply to either DC's clone/session without modification.

Stage 6 -- Phase 5: DR wiring and failover drill

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Stage 7 -- Phase 6: Designate, COS, Magnum/CAPI (per DC)

State: RUNBOOK WRITTEN, NOT YET EXECUTED.

Cross-cutting discipline (applies at every stage, every session)

• Session bootstrap, every session: git pull -> bash scripts/repo-lint.sh (0 fail / 1 documented legacy WARN expected) -> bash scripts/ledger-scan.sh to reconcile open decisions/next-free numbers against docs/session-ledger.md.
• Any session that will mutate the cloud: bash scripts/run-logged.sh <label> first (as-executed log).
• Before any add-model-like mutation: bash scripts/preflight.sh PASS.
• After any deploy/restart/change, and as an incident-triage first step: bash scripts/cloud-assert.sh (add --capture at real deploy-completion milestones for the committed BOM).
• Before touching ANY built surface: grep docs/design-decisions.md for the governing D-NNN. PROPOSED/OPEN means the operator has not ruled -- present options, do not implement.
• Every deliverable (script, runbook, doc): repo-lint clean, the touched script's harness green (write one if missing), a changelog entry with a per-item revert, and a docs/session-ledger.md update.
• Before assigning a new D-/DOCFIX-/BUNDLEFIX-NNN: bash scripts/ledger-scan.sh for next-free (it is the numbering authority, not repo-lint's informational count).
• Hard rule reminder: no inferred values -- every IP/ID/name/scope is measured this session or carried from confirmed as-built; verify before mutate, always; destructive/secret-handling steps are individually gated, never batched.

Tooling gap register (audited 2026-07-09; update as items close)

Confirmed by grepping the actual repo (scripts/, tests/, netbox/, runbooks/), not inferred from the design doc alone. Ordered roughly by "you'll hit this first," not by size -- #1 is cheap and unblocks a reuse claim made elsewhere in this doc;

#2 is the single largest build.

1. $DC selector convention -- CLOSED 2026-07-09 (DOCFIX-151). scripts/lib-net.sh gained lib_net_select_dc() (dc0/dc1 no-op, since D-101 rules DC1 inherits DC0's v4 layout unchanged; dc2 fails loud -- NetBox literals not yet assigned, gap #3) and scripts/lib-hosts.sh gained lib_hosts_select_dc() (dc0 no-op; dc1 AND dc2 BOTH fail loud -- no real per-DC host enrollment exists yet for either, an intentional asymmetry vs. the net selector, documented in both files' comments and docs/changelog-20260709-dc-selector-convention.md). Backward compatible by construction -- every existing caller of either library is unaffected. New harness tests/dc-selector/run-tests.sh: 21/21 PASS. The mechanism exists; Stage 5's runbook still needs to actually CALL lib_net_select_dc "$DC" / lib_hosts_select_dc "$DC" when it's authored -- that call-site wiring is part of writing that runbook, not this delivery.
2. OpenTofu -- network/pool/node-VM/cloud-init/MAAS/netem layer scaffolded 2026-07-09 (was: zero files). opentofu/ now has: modules/dc-planes (six per-DC planes), modules/mesh-link (the D-100 dark-fiber triangle legs), modules/dc-storage-pool, modules/node-vm (blank disk, PXE-boot MAAS-managed node VM, the D-103 pattern for Stage 3), modules/base-image
   • modules/cloudinit-vm (the OTHER VM pattern -- pre-built image + cloud-init, for Office1's own service VMs), modules/opnsense-edge (see gap #4), modules/maas-vm-host (registers the vcloud host with MAAS as a virsh VM host -- via the official canonical/maas provider, confirmed against its real schema; deliberately NOT maas_vm_host_machine, which composes/creates new VMs itself and would fight node-vm for VM creation -- D-103 explicitly rules out MAAS composing), and modules/netem-link (terraform_data + SSH-wrapped local-exec provisioners applying tc qdisc ... netem to a mesh-link bridge -- OpenTofu runs from Office1, not the vcloud host, so a bare local command can't reach the bridge). DC1 + Office1 pool/plane wiring is instantiated in root main.tf; everything else is NOT (node/VM sizing, MAAS zone/pool, netem parameters are all pending decisions -- neither invented here). Provider schema was confirmed against the actual current docs AND real example .tf files (not memory) -- it differs materially from older common examples: nested structures are attribute-style objects (key = { ... }), not classic HCL blocks. A real bug from the first pass was found and fixed: dc-planes/mesh-link/dc-storage-pool originally used block syntax for libvirt_network's domain/mtu and libvirt_pool's target; corrected once real provider examples surfaced the actual convention -- discovered precisely BECAUSE building the domain module forced fetching real example code instead of doc-summaries alone. Still missing, deliberately deferred (see opentofu/README.md): OPNsense's own bootability via cloudinit-vm is NOT confirmed (FreeBSD, not the same cloud-init/NoCloud guarantee as Linux images); the per-device boot-order attribute's exact internal shape (flagged UNVERIFIED in node-vm/main.tf, an inference by pattern not a confirmed field); and whether devices.interfaces list order reliably maps to LAN/WAN NIC enumeration (flagged in opnsense-edge/main.tf). UNVALIDATED -- no tofu binary was available to run scripts/opentofu-validate.sh this session; run it on a connected machine before trusting any of this.
3. NetBox multi-DC/dual-stack import pipeline -- MECHANISM CLOSED 2026-07-09 (DOCFIX-152), literals still pending. netbox/dc-dc-prefixes- import.py extends the v1 pattern to VR1's two-DC dual-stack model: per-DC (--dc dc1/dc2) creates the six-plane IPv4 (DC1 inherited, hardcoded per D-101; DC2 via required DC2_V4_SUPERNET, no default) plus the D-101 family-matrix IPv6 legs (ULA/GUA), carved from two REQUIRED env vars (ORG_ULA_48, DC_GUA_PREFIX) with no defaults -- fails loud rather than inventing either. 40/40 tests (tests/dc-dc-prefixes- import/, fake-NetBox end-to-end). UNVALIDATED against a real NetBox (no instance reachable this session). The literals themselves (org ULA /48, per-DC GUA carve, DC2 supernet) still don't exist -- that operator/NetBox assignment step is unchanged by this delivery; this closes the TOOLING gap, not the DATA gap. Still blocks Stage 2/3 until those literals are assigned and this script is actually run against a live NetBox.
4. OPNsense edge deployment/config, and netem WAN-simulation -- (a)-(c) DONE 2026-07-09, (d) is the only real gap left. Each site (Office1, DC1, DC2) gets its own independent OPNsense edge with a simulated ISP uplink (D-100); OpenTofu owns creating the edge VMs (D-103). Blocks Stage 1 (Office1 edge) and Stage 3 (per-DC edges) gates, which require netem parameters "applied and measured."
   • (a) Network substrate -- DONE. opentofu/modules/mesh-link creates the inter-site L2 segments (DC1<->DC2, DC1<->Office1, DC2<->Office1) an OPNsense VM's interfaces would eventually attach to.
   • (b)+(c) The OPNsense VM/image AND its config -- BUILT (opentofu/modules/opnsense-edge), UNVALIDATED. Cloud-init is confirmed unreliable on OPNsense (FreeBSD) -- OPNsense's own forum consensus, not just a hedge. The real native mechanism is the Configuration Importer (official docs): scans an attached volume for /conf/config.xml in a 2-3 second boot-time window; ISO9660 support was added to it specifically for VM/cloud automation (attach a config ISO as a secondary CD-ROM) -- mechanically IDENTICAL to modules/cloudinit-vm's shape (a secondary volume attached as cdrom), just a different payload. Built as opentofu/modules/opnsense-edge + two prep scripts (scripts/opnsense-prep-image.sh for the nano image's decompress+ convert, scripts/opnsense-build-config-iso.sh for the config.xml ISO). The real config.xml DESIGN is now also done: opentofu/templates/opnsense-config.xml.tmpl (a {{TOKEN}}-parameterized template built from OPNsense's own real shipped sample, covering interfaces/gateway/staticroutes/NAT/DNS/NTP and D-107-shaped firewall rules) + scripts/opnsense-render-config.sh (the renderer -- fully tested end-to-end, 8/8, unlike the other opnsense-* scripts, since it needs no external tool). UNVALIDATED regardless: no tofu binary, no ISO-building tool, and no real OPNsense boot were available to confirm the whole chain end-to-end. Several template tokens still can't be filled with real per-site values -- pending Stage 0 ratification (D-100/D-101/D-107) or only measurable on a real boot (which vtnetN device a network becomes). Full sourcing in opentofu/README.md and opentofu/templates/README.md's token legend.
   • (d) tc netem WAN-simulation -- mechanism BUILT (modules/netem-link), only the real parameters remain. No Terraform/OpenTofu-native equivalent exists (confirmed against the libvirt_network schema, gap #2); built instead as a terraform_data + SSH-wrapped local-exec provisioner (OpenTofu runs from Office1, not the vcloud host where the bridge lives -- a bare local command can't reach it). UNVALIDATED, not instantiated. The exact parameters (latency/jitter/loss/rate) are themselves still an unruled D-100 open sub-item (gap #11) -- a concrete PROPOSAL now exists (docs/dc-dc-netem-and-ula-gua- proposal.md, DOCFIX-168, Part 1: 1ms/0.2ms jitter/0.01% loss/uncapped rate, with reasoning, presented for operator ratification, not silently applied) -- still needs an actual ruling before it can be applied for real.
5. Ceph replication tooling (radosgw multisite + rbd-mirror) -- MECHANISM CLOSED 2026-07-09 (DOCFIX-165), data half still open. runbooks/dc- dc-phase5-dr-failover-drill.md Steps 4, 5, 8, 10.2, and 11.2-11.4's radosgw-admin/rbd mirror command sequences (verbatim upstream Ceph administration, not invented) are now callable, parameterized scripts: scripts/dc-dc-radosgw-multisite.sh (subcommands master-init/join- readonly/enable-two-way -- Step 4 one-way staging + Step 8's radosgw half), scripts/dc-dc-rbd-mirror.sh (subcommands bootstrap-primary/ bootstrap-secondary --direction rx-only|rx-tx -- Step 5 one-way staging
   • Step 8's rbd-mirror half), and scripts/dc-dc-dr-drill.sh (subcommands failover -- Step 10.2 force-promote + a flagged, never-auto-executed Glance re-registration reminder -- and failback -- Steps 11.2-11.4, with the demote-CURRENT-PRIMARY-before-promote-RECOVERING safety ordering HARD-CODED as fixed call order, not left to documentation). Every realm/zonegroup/zone/pool/endpoint/unit name is a REQUIRED argument, no invented defaults; default mode is a --dry-run plan print, --apply executes one command at a time via juju ssh/juju run, aborting on first failure. New harnesses tests/dc-dc-radosgw-multisite/, tests/dc-dc-rbd-mirror/, tests/dc-dc-dr-drill/ (19/19, 19/19, 23/23 PASS) test ONLY these scripts' own argument parsing, guard clauses, dry-run output, and (critically) the failback step ORDER -- NOT real radosgw/rbd-mirror behavior, since no live Ceph cluster exists this session to test against (same UNVALIDATED posture as gap #3's NetBox pipeline). The MECHANISM exists; the DATA half (real DC1/DC2 Ceph cluster, real names/endpoints, the three bundle.yaml gaps in the Stage 6 runbook's own "Known gaps" section) is unchanged and still blocks Stage 6 for real. See docs/changelog-20260709-ceph-replication-tooling.md.
6. Designate reactivation is a bundle change -- CLOSED 2026-07-10 (DOCFIX-167). bundle.yaml now carries designate/designate- bind/designate-mysql-router/designate-hacluster + 8 relations, researched against designate's real metadata.yaml (a real mistake caught during authoring: its provides-side endpoint is dnsaas, not public like the other API charms -- corrected). os-public-hostname re-activation (D-106's OTHER coupled change) deliberately NOT done in the shared bundle -- pushed to a proposed per-DC overlay instead, to keep bundle.yaml DC-agnostic. python3 scripts/provider-bundle- check.py PASS, all 6 existing invariants unaffected. See docs/changelog-20260709-designate-cinderbackup-rbdmirror.md. NOT applied to any live model -- repo-source change only.
7. MTU/geneve budget + Ceph disk-budget calculators -- CLOSED 2026-07-09 (DOCFIX-162). D-102 and buildout-design Section 3 both describe measurement-then-computation gates (host L2 MTU -> geneve-over-v6 overhead; measured per-node OSD footprint -> size=3 vs size=2 fallback) that were manual arithmetic in prose, not a script, blocking the Stage 1 Phase-0 gates. scripts/dc-dc-mtu-geneve-budget.sh takes the MEASURED underlay MTU via required --underlay-mtu (no default) and reproduces D-101's own worked example exactly (1500 - 56 = 1444), with a jumbo (>=9000) path leaving tenant MTU at 1500, plus an added IPv6-minimum-MTU (1280, RFC 8200) sanity floor clearly separated from D-101's own text. scripts/dc-dc-ceph-disk-budget.sh takes MEASURED total disk, per-DC OSD node counts and per-node footprints, and a required --backup-overhead-fraction (deliberately NO default -- not specified as a hard number anywhere in this repo, must come from real Ceph/radosgw-admin measurement once DC1 exists) and computes whether size=3 across both DCs plus overhead fits the measured total disk; if not, it names size=2 as the ONLY documented fallback (D-101 / Section 3) without silently applying it -- explicitly an operator-logged decision. New harnesses tests/dc-dc-mtu-geneve-budget/run-tests.sh (19/19 PASS) and tests/dc-dc-ceph-disk-budget/run-tests.sh (16/16 PASS). Both scripts are read-only/mutate nothing and have not yet been run against the real vcloud host (prep-only session, no live infrastructure reachable) -- that first real run is part of executing runbooks/dc-dc-phase0-vcloud-prep.md Step 3. Changelog: docs/changelog-20260709-mtu-ceph-budget-calculators.md.
8. The "developed IPv6 plan" D-101 cites does not exist in this repo. D-101's context line references "the developed family-follows-reachability IPv6 plan (in-project, not yet committed)" as prior art -- grepped the whole repo, it is not present as a standalone doc anywhere. If the exact Phase A/B/C detail behind D-101/D-102 is needed by a future session, it currently is NOT retrievable from the repo alone. Not a blocker for Stage 0 ratification of what IS written, but a real content gap if finer detail is needed later.
9. Runbook debt -- CLOSED 2026-07-09. All seven Stage 1-7 runbooks are now written (runbooks/dc-dc-phase0-vcloud-prep.md through dc-dc-phase6- designate-cos-magnum.md), Stage 1 authored directly, Stages 2/3/4/6/7 delegated to parallel subagents and Stage 5 authored directly, all reviewed and repo-lint clean. NONE have been executed against real infrastructure yet -- this was a prep-only session (the operator was away from the workstation, not the vcloud host). Each runbook is explicit about what it can and cannot close given today's real state (NetBox literals, node sizing, and several other decisions remain genuinely open -- see each runbook's own GATE section for the honest per-stage breakdown).
10. Harness debt -- LARGELY CLOSED 2026-07-09/10. This repo's own delivery rule -- no script change ships without its tests/<name>/run-tests.sh harness -- has been honored for every new script tonight: dc-selector (21/21), dc-dc-prefixes-import (40/40), dc-dc-mtu-geneve-budget (19/19), dc-dc-ceph-disk-budget (16/16), dc-dc-radosgw-multisite (19/19), dc-dc-rbd-mirror (19/19), dc-dc-dr-drill (23/23), plus new $DC test cases added to carve-host-interfaces/phase-00-maas- standup and a new reenroll-hosts harness. Gap #2 (OpenTofu modules) remains the one area still lacking a harness beyond scripts/opentofu- validate.sh itself (no tofu binary this session to validate against).
11. Decision sub-items still genuinely open (the big items are RULED per Stage 0; these specific literals/params are not -- but see docs/dc-dc-netem-and-ula-gua-proposal.md (DOCFIX-168) for a concrete netem proposal and exact, safe ULA-generation guidance, drafted 2026-07-10): exact netem parameters (D-100), exact NetBox literals for DC2/ULA/GUA (D-101).
12. No Office1-local virtual network is modeled anywhere in opentofu/ -- CLOSED 2026-07-09. main.tf previously wired only office1_pool_path (storage only) and the three inter-site mesh links (management-traffic- only per D-100's own sub-item ruling) -- nothing the three Office1 service VMs (MAAS-region, NetBox, GitBucket), or the vcloud host's own management interface, could actually attach to as a shared local network. Resolved by a NEW opentofu/modules/office1-network module (a dc-planes-shaped isolated libvirt_network, sized for ONE network rather than six planes since Office1 is a headend, not an OpenStack DC), deliberately NOT a reused host bridge -- a hand-configured bridge would be the one network in this topology living outside OpenTofu's D-103- mandated inventory, silently breaking this repo's own virsh net-list --all completeness assumption. INSTANTIATED for real in root main.tf (module "office1_network") -- unlike most of gap #2's other scaffolding, this needed no unmeasured value beyond domain_suffix/underlay_mtu, already real required inputs elsewhere in the same file. See opentofu/README.md and docs/changelog-20260709-office1-network-edge.md (DOCFIX-163) for the full design rationale. Designing the three service VMs' actual cloudinit-vm network_config content against this network remains separate, un-actioned work.
13. The D-101 IPv6 family-matrix overlay -- DRAFTED 2026-07-10, one real risk still open. overlays/dc-dc-ipv6-family-matrix.yaml now exists, sourced against real charm config/source (docs/dc-dc-ipv6-charm- research.md has full citations): prefer-ipv6 (confirmed real option, genuinely additive dual-stack HAProxy binding per charm-nova-cloud- controller's actual commit, not an either/or switch) + appended v6/GUA VIPs for the API charms; ceph-mon's own separate prefer-ipv6 + ceph-public-network/ceph-cluster-network for the ULA-only planes; confirmed NO charm-config option exists for OVN (geneve family follows the bound interface automatically). D-109's IPv6-SAN cert support is CONFIRMED (read charm-vault's actual sort_sans() code -- no v4/v6 distinction). Genuinely unresolved: Octavia's lb-mgmt-net IPv6 support is excluded from the overlay on purpose -- two real, still- referenced upstream Launchpad bugs (#1911788, #1913409) document IPv6 lb-mgmt-net failures; needs an explicit operator decision (accept the risk and test for real, or keep Octavia's lb-mgmt-net as a deliberate v4-only D-101 exception), not silently resolved here. Also unconfirmed: 8 of the 9 dual-stack charms' prefer-ipv6 presence is inferred by shared-framework pattern, not individually fetched. UNVALIDATED against any live model.
14. Three real bundle.yaml gaps for D-108's DR mechanism -- CLOSED 2026-07-10 (DOCFIX-167). cinder-backup (a SUBORDINATE charm, confirmed via real metadata.yaml -- the first draft wrongly gave it principal-style num_units/to/amqp, corrected) and ceph-rbd- mirror (a principal with ceph-local/ceph-remote requires endpoints, confirmed via real metadata.yaml -- the first draft invented a wrong binding key, corrected) are now in bundle.yaml; ceph-mon's rbd-mirror binding corrected from storage to replication. Genuinely unresolved, documented not invented: ceph-remote (the peer DC's ceph-mon) is a cross-Juju-controller relation D-104's per-DC-independent-controller design means cannot be a same-bundle relation line -- only this DC's own ceph-local leg is wired; the cross-DC peer relationship needs a juju offer/consume or the manual rbd mirror pool peer bootstrap CLI flow (Stage 6's runbook already documents the latter). python3 scripts/provider- bundle-check.py PASS. See docs/changelog-20260709-designate- cinderbackup-rbdmirror.md. NOT applied to any live model.
15. Seven real MAAS/multi-rack gaps found 2026-07-09 while drafting the Stage 4 runbook (full detail in that runbook's own "Known gaps" section). Sub-item 2 (script $DC-parameterization) CLOSED 2026-07-10 (DOCFIX-166) -- reenroll-hosts.sh, carve-host-interfaces.sh, and phase-00-maas-standup.sh now accept an opt-in $DC env var and call lib_net_select_dc/lib_hosts_select_dc immediately after sourcing (unset $DC is byte-for-byte unchanged behavior; see docs/changelog-20260709-maas-scripts-dc-param.md). This closes the CLI-invocation half of the gap only -- it does NOT create any new per-DC data (no dc1/dc2 hostnames, octets, boot MACs, or node-count figures were invented), and phase-00-maas-standup.sh's own PLANES table remains hardcoded to D-052/D-053's literals rather than driven by lib-net.sh's flat vars, so it is not truly dc2-topology-aware yet even though it now gates dc2 out cleanly. The other six sub-items remain OPEN: no OpenTofu module stands up a per-DC MAAS rack controller VM itself (only registration, modules/maas-vm-host, exists); per-DC node count/sizing is an unmade Phase-0 decision; DC2 is still blocked at the network-selector layer (gap #3, NetBox literals not assigned); the per-DC artifact mirror (D-107) has no identified owning stage/module; the MAAS status a VM-host-discovered (vs. PXE-enlisted) machine lands in is unconfirmed; HOST_TAG/boot-fabric-name literals are DC0-specific.
16. No stage explicitly owns creating Office1's own OPNsense edge VM -- CLOSED 2026-07-09. The topology (buildout design Section 1) is explicit: "full dark-fiber triangle... plus a per-site OPNsense simulated ISP edge" -- three sites, three edges (DC1, DC2, AND Office1). Stage 3's Build list already covered DC1/DC2's edges (modules/opnsense-edge per DC); Stage 2's Build list (MAAS region, OpenTofu, NetBox, GitBucket, Tailscale) did not mention an Office1 edge at all, and neither runbook instantiated one. Decision: YES, Office1 gets its own modules/opnsense-edge call, a fourth site alongside DC1/DC2; Stage 2's runbook (runbooks/dc-dc-phase1-office1-standup.md) owns creating it. Rationale: Stage 2's entire scope is already "Office1 headend standup" (the OPNsense edge IS part of that headend, not DC substrate), and Stage 1 already creates the mesh-link networks this edge's neighbors depend on -- Stage 3 has no reason to reach back and create a piece of Office1's own infrastructure. Wired as a commented-out module "office1_opnsense" skeleton in root main.tf (mirroring the identical NOT-YET-REAL status of DC1/DC2's own edges -- neither is instantiated either) plus a new gated step in runbooks/dc-dc-phase1-office1-standup.md (Step 4b) and its own GATE bullet. Related to, but distinct from, gap #12 (Office1-local network, now also CLOSED) -- that gap was the service VMs' own attach point; this one was whether Office1 gets a simulated-ISP-edge boundary at all, and who builds it. The edge's LAN side now resolves cleanly to gap #12's new network; its WAN side does not -- see NEW gap #17 immediately below, a genuinely deeper, cross-site gap surfaced while closing this one, not invented as a substitute for a real answer.

17. No dedicated per-site ISP-uplink/WAN-side network exists anywhere in this repo, for ANY site (found 2026-07-09 while closing gap #16 for Office1). Every modules/opnsense-edge caller needs a wan_network_name distinct from BOTH its LAN-side network (a dc-planes plane, or gap #12's new office1-network) AND the three D-100 mesh-link legs -- the mesh legs are confirmed MANAGEMENT-TRAFFIC- ONLY by D-100's own sub-item ruling (Office1<->DC fiber carries MAAS/ Juju/operator traffic only), not a simulated ISP uplink, so wiring an edge's WAN side to a mesh leg would misrepresent the topology rather than merely placeholder it. This is NOT an Office1-specific gap: runbooks/dc-dc-phase2-tofu-dc-substrate.md Step 5's own dc1_opnsense template carries the IDENTICAL unresolved placeholder (wan_network_name = module.mesh_dc1_office1... # or a dedicated per-site ISP-uplink segment -- confirm which, do not assume) -- confirmed by reading that runbook directly, not assumed. Deliberately NOT invented here for Office1 alone (that would contradict D-100 and leave DC1/DC2 with the same unresolved gap under a different name). Needs a decision: a fourth network shape per site (one more dc-planes/mesh-link/office1-network-style dedicated libvirt_network standing in for that site's own "internet" uplink), or some other mechanism not yet considered. Blocks instantiating ANY of the three sites' opnsense-edge calls (dc1_opnsense, dc2_opnsense, office1_opnsense) end-to-end -- all three are otherwise ready (image/config mechanism per gap #4, LAN-side network per gap #12/this stage's plane modules) but for this one missing piece.

18. OpenTofu state file had no documented protection -- CLOSED 2026-07-10 (DOCFIX-175). Found while reviewing what's missing from the buildout, before Stage 1's own first real tofu apply (dc-dc-phase0-vcloud- prep.md Step 10 -- confirmed by reading the runbook directly, not assumed from filenames; an earlier pass wrongly guessed Stage 3): terraform.tfstate stores sensitive = true variables like var.maas_api_key in plaintext (verified against HashiCorp's own docs, not assumed -- sensitive only suppresses CLI output, not state storage), and nothing gitignored it, documented backup/permission handling, or noted the local-backend's lack of locking. Fixed: .gitignore now excludes opentofu/**/*.tfstate* and related local artifacts, opentofu/README.md gained a full "State file handling" section, variables.tf's maas_api_key description carries the caveat inline, and dc-dc-phase0-vcloud-prep.md Step 10 gained a "SECURE THE STATE FILE" callout (chmod, gitignore verification, out-of-band backup) as a standing discipline from the first real apply onward (a short pointer added at Stage 3 Step 8 too, the first step needing a REAL MAAS credential). NOT fixed (documented, not invented): a remote backend with real locking -- production-grade, Roosevelt-scope, not a rehearsal decision.

19. No teardown/rollback runbook existed for the VR1 OpenTofu-provisioned infrastructure itself -- CLOSED 2026-07-10 (DOCFIX-176). runbooks/phase-00-teardown-*.sh (D-061) covers tearing down the VR0 juju/MAAS layer, a different pairing of tools; nothing documented how to tofu destroy DC1/DC2/Office1's networks, storage pools, node-VM domains, and OPNsense edges in safe dependency order, or what MAAS-side cleanup has to happen alongside it. Fixed: runbooks/dc-dc-teardown-rollback.md -- two paths (scoped single-site via -target, or full VR1 destroy), the D-061 principle carried over (MAAS-side cleanup before destroying the libvirt resource underneath), a rollback decision tree favoring fix-forward over destroy for the common partial-apply-failure case, and explicit non-scope for mesh-link legs (shared between two sites, only destroyed when both ends are going away). NOT resolved (flagged, not invented): the exact MAAS-side release/delete commands for machines enrolled under a VR1 vm_host, since this repo has never reached Stage 4 for any VR1 DC and has no real machines to test against yet -- the runbook states the principle and points at MAAS's own docs rather than guessing flags.

Where things stand today (2026-07-09) -- update this as stages close