# DC-DC Phase 3 -- MAAS enlist / commission / deploy (per DC) (Stage 4) Bring the per-DC node VMs that Stage 3 (`docs/dc-dc-deployment-workflow.md` -- "Stage 3 -- Phase 2: OpenTofu builds each DC substrate") caused MAAS to DISCOVER through to a deployed OS, ready for Juju (Stage 5 / Phase 4). Run this runbook ONCE PER DC: set `DC=dc1` or `DC=dc2` for the whole session and do not mix the two DCs' commands in one shell. This is the FIRST DC-DC runbook whose subject is MAAS itself in the multi-rack VR1 shape -- everything before it (Stage 0-3) either ratified decisions or built the substrate MAAS now sits on top of. **Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "### Phase 3" (goal/build/gate, quoted in the GATE section below) and Section 5 (OpenTofu/MAAS/Juju boundary: "MAAS: enlist (discover) / commission / deploy / power / release of node VMs. Does not compose. One region (Office1) + one rack per DC (virsh VM-host)."); `docs/dc-dc-deployment-workflow.md` Stage 4 row ("Reuse vs new: PARTIAL reuse... the per-DC multi-rack registration is new, the deploy mechanics are precedent"); `docs/design-decisions.md` **D-103** (lifecycle seam: MAAS owns commission/deploy/power/release, does NOT compose), **D-100** (br-ex / provider-NIC raw discipline, Pattern A), **D-107** (per-DC airgap mirror + edge NTP). `runbooks/phase-00-teardown-maas-reset.md` Steps 5-7 are the validated single-rack precedent this runbook adapts -- read it first if you have not; this doc assumes you have and does not re-derive what it already established. Decisions this runbook owns: **D-103**, **D-100** (br-ex), **D-107** (mirror + NTP). --- ## Precondition -- Stage 3's gate must already be green (READ-ONLY) Stage 3's own runbook (`runbooks/dc-dc-phase2-tofu-dc-substrate.md`) is written but NOT YET EXECUTED as of this writing -- confirm it has actually been RUN (not merely authored) for `$DC` before proceeding, by checking its own EXIT GATE against `docs/dc-dc-buildout-design.md` Section 4 Phase 2 gate text: "MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured." If that gate is not green for `$DC`, STOP here -- do not attempt to stand up a rack controller, register a VM host, or create node VMs from inside THIS runbook. That is Stage 3's job (D-103: OpenTofu creates the node VMs and registers the libvirt host as a MAAS VM host); this runbook only picks up from "MAAS already sees them." **CHECK (read-only) -- confirm the rack controller MAAS thinks it has for this DC** ```bash maas "${MAAS_PROFILE:-admin}" rack-controllers read | jq -r '.[] | "\(.hostname)\t\(.system_id)"' ``` Identify, by eye, which entry (if any) is `$DC`'s rack controller -- there is no established naming convention yet to grep for (Stage 3 has not run; do not assume a name like `dc1-rack`). If nothing plausible appears, Stage 3's gate is not met -- stop and go finish Stage 3. --- ## Known gaps flagged before you start (not fixed by this runbook) These are real, current repo gaps surfaced while adapting phase-00's single-rack sequence to the multi-rack VR1 shape. Each is a FINDING to log (changelog / D-NNN / DOCFIX candidate per this repo's discipline) at the point it is actually hit, not something to route around silently here. 1. **No OpenTofu module stands up a per-DC MAAS rack controller VM itself.** `opentofu/README.md`'s built-module list has `maas-vm-host` (registers an existing libvirt/virsh connection as a VM host WITH a MAAS rack), but nothing that creates the rack-controller service VM the buildout design's Section 5 says each DC needs ("one region (Office1) + one rack per DC"). If Stage 3's precondition check above finds no rack controller for `$DC`, that absence is this gap manifesting, not a Stage 4 problem to solve here. 2. **CLOSED 2026-07-10 (DOCFIX-166), CLI half only.** `scripts/reenroll-hosts.sh`, `scripts/carve-host-interfaces.sh`, and `scripts/phase-00-maas-standup.sh` now accept an opt-in `$DC` env var and call `lib_net_select_dc`/`lib_hosts_select_dc` immediately after sourcing (unset `$DC` is byte-for-byte unchanged VR0/DC0 behavior). See `docs/changelog-20260709-maas-scripts-dc-param.md`. This means `DC=dc1 bash scripts/reenroll-hosts.sh` (etc.) is now a REAL invocation, not hypothetical -- but it does not by itself unblock anything: calling any of the three with `DC=dc1` or `DC=dc2` today fails loud immediately at `lib_hosts_select_dc` (reenroll-hosts.sh, carve-host-interfaces.sh -- no per-DC host inventory exists yet) or, for dc2, at `lib_net_select_dc` first (gap #4 below). The one case genuinely simplified is `phase-00-maas-standup.sh` against `dc1`: since D-101 makes dc1 inherit dc0's plane literals unchanged, `DC=dc1 bash scripts/phase-00-maas-standup.sh` is now a real, correct, gated call -- see the updated Step 8. Parameterizing these scripts to accept REAL per-DC data (once it exists) required no further script change; the remaining blocker is populating `lib-hosts.sh` with a real `$DC` block (delivery checklist item below) and, separately, `phase-00-maas-standup.sh`'s own `PLANES` table is still a D-052/D-053 hardcoded literal, not derived from `lib-net.sh`'s flat vars, so it is not dc2-topology-aware even once dc2's CIDRs exist. 3. **Node count and sizing per DC is an unmade Phase-0 decision.** `opentofu/README.md`'s `node-vm` module note: "node count/memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision... that hasn't been made -- call the module once real values exist." Do NOT assume DC0's four hosts (`openstack0-3`) carry over to `$DC` -- discover however many nodes Stage 3 actually created, live, in Step 2 below. 4. **DC2 is additionally blocked at the network-selector layer.** `lib_net_select_dc dc2` FAILS LOUD today (D-101's NetBox-literals open item, tooling gap register #3 -- no CIDRs assigned yet for DC2). This runbook cannot proceed past Step 1 for `DC=dc2` until that closes. `dc1` is unblocked at this layer (D-101: DC1 inherits DC0's v4 layout unchanged, so `lib_net_select_dc dc1` is a documented no-op). 5. **The per-DC artifact mirror (D-107) has no identified owning stage/module in this repo as surveyed for this runbook.** Stage 2's build list (Office1 headend) mentions GitBucket (the REPO mirror) and NetBox -- neither is the node-package/apt/snap/charmhub/image mirror D-107 describes. Before Step 7's mirror-reachability check, CONFIRM the mirror actually exists and where; if it does not, that is a gap to log against Stage 2 or a new stage, not something to invent a reachability check for. 6. **The MAAS status a VM-host-discovered (not PXE-enlisted) pre-existing domain lands in is not confirmed anywhere in this repo.** Phase-00's `reenroll-hosts.sh` header states PXE-enlisted hosts "auto-commission on create" -- but Stage 3's node VMs are discovered via `maas_vm_host` pod scan of already-existing libvirt domains, a different MAAS code path. Step 2 below measures the real status live rather than assuming either "New" or "auto-Ready." 7. **`lib-hosts.sh`'s `HOST_TAG`/boot-fabric-name conventions (`openstack`, `2_metal`) are DC0-specific literals.** Whether `$DC`'s bundle placement tag and PXE fabric carry the same names is not established -- confirm live per DC rather than assuming reuse of DC0's literal strings. --- ## Sequence ``` 0. Precondition confirm Stage 3 gate (above) (read-only) 1. DC selector lib_net_select_dc "$DC"; note lib_hosts_select_dc fails (read-only) 2. Discover find $DC's node VMs live in MAAS (read-only) 3. Commission per discovered node [MUTATION: gated] 4. Deploy per discovered node [MUTATION: gated] 5. Pattern A carve (adapted) per node, RAW provider NIC + VLAN-103 stack [MUTATION: gated] 6. PXE / boot-fabric verify confirm v4 PXE + fabric landing (read-only) 7. Mirror + NTP verify per-DC mirror reachable; edge NTP (read-only) 8. Topology check (adapted) phase-00-maas-standup.sh mechanics, per DC (read-only) -> EXIT GATE -> Stage 5 (Juju controller + bundle, per DC) ``` --- ## Step 1 -- DC selector (READ-ONLY) Every session/script that sources `scripts/lib-net.sh` for this stage calls the selector ONCE, immediately after sourcing, before using any flat plane variable: ```bash SCRIPT_DIR="$(pwd)/scripts" . "$SCRIPT_DIR/lib-net.sh" . "$SCRIPT_DIR/lib-hosts.sh" lib_net_select_dc "$DC" # dc1: documented no-op (D-101 inherits DC0 v4). # dc2: FAILS LOUD today -- gap #3, stop here for dc2. ``` Then, honestly, also call the host selector and expect it to fail for BOTH DCs right now: ```bash lib_hosts_select_dc "$DC" # EXPECTED TO FAIL for dc1 AND dc2 until THIS # stage's own completion populates real per-DC # host data into lib-hosts.sh (Step 5 below). ``` This is a real, expected chicken-and-egg, not a bug to work around: the selector exists specifically to PREVENT a `$DC` script from silently reusing `HOSTS`/`HOST_OCTET`/`HOST_BOOT_MAC`/`VIRSH_POWER_ADDRESS` from DC0's real, already-enrolled hosts for a different DC's different VMs. Until this stage's enrollment produces `$DC`'s own real hostnames, MACs, and system_ids, there is nothing correct for `lib_hosts_select_dc "$DC"` to select -- so it fails, on purpose, for both `dc1` and `dc2`, and will keep failing until Step 5's data goes into `lib-hosts.sh` for real. Do not comment out the call, do not pre-populate placeholder host data to make it pass -- populating it with REAL data once Step 5 has measured it is this stage's own delivery item (see the checklist). Because `lib_hosts_select_dc` cannot yet be relied on, every step below resolves `$DC`'s host identity LIVE from MAAS (by whatever the discovery in Step 2 actually finds), never from `lib-hosts.sh`'s DC0 maps. --- ## Step 2 -- Discover `$DC`'s node VMs live in MAAS (READ-ONLY) There is no fixed hostname list for `$DC` (gap #3 above) and no established zone/pool/tag naming convention yet (Stage 3 has not run to record one). Confirm, from what Stage 3 actually did for `$DC` (its as-executed log / session notes), which of these MAAS groupings it used, then discover against THAT -- do not guess a name like `"dc1"`: **CHECK -- by zone (if Stage 3 recorded a zone name for `$DC`)** ```bash maas "${MAAS_PROFILE:-admin}" zones read | jq -r '.[] | "\(.name)"' # then, once you have confirmed the real zone name for $DC (call it Z below): maas "${MAAS_PROFILE:-admin}" machines read \ | jq -r --arg z "Z" '.[] | select(.zone.name==$z) | "\(.hostname)\t\(.system_id)\t\(.status_name)"' ``` **CHECK -- by resource pool (if Stage 3 used a pool instead)** ```bash maas "${MAAS_PROFILE:-admin}" resource-pools read | jq -r '.[] | "\(.name)"' maas "${MAAS_PROFILE:-admin}" machines read \ | jq -r --arg p "P" '.[] | select(.pool.name==$p) | "\(.hostname)\t\(.system_id)\t\(.status_name)"' ``` **Cross-check -- by fabric/VLAN** (independent confirmation: D-100 makes each DC's metal-admin/PXE plane its own isolated segment, so `$DC`'s machines' boot interfaces should all land on the SAME fabric, and that fabric should be DIFFERENT from DC0's `2_metal` and from the other DC's): ```bash maas "${MAAS_PROFILE:-admin}" interfaces read \ | jq -r '.[] | select(.type=="physical") | "\(.mac_address)\t\(.vlan.fabric)"' ``` Run this per candidate system_id from the zone/pool query above; every node claimed for `$DC` should share one fabric name, and that name should not be `2_metal` (DC0's) nor match the other DC's. If the grouping mechanism disagrees with the fabric cross-check (e.g. a machine tagged into `$DC`'s pool but sitting on a different DC's fabric), STOP -- that is a real enrollment inconsistency, not something to paper over by picking whichever signal you prefer. **GATE (informal, before mutating):** you have a concrete, confirmed list of system_ids that both (a) belong to `$DC` by whatever grouping Stage 3 used and (b) share one boot fabric distinct from DC0's and the peer DC's. Record that list -- Steps 3-5 iterate over it. Record the count too; it is whatever Stage 3 actually created (gap #3), not an assumed 4. --- ## Step 3 -- Commission each discovered node [MUTATION: gated, per node] Phase-00 precedent (`reenroll-hosts.sh`): freshly PXE-enlisted hosts auto-commission. Stage 3's nodes arrived via `maas_vm_host` pod discovery of pre-existing libvirt domains, a different code path whose landing status this repo has not previously observed (gap #6) -- Step 2's `status_name` column tells you what you actually got. If a node already reads `Ready`, skip it (idempotent). If it reads `New`, commission it explicitly: **CHECK (read-only) -- current status, one more time, immediately before mutating** ```bash maas "${MAAS_PROFILE:-admin}" machine read | jq -r '.status_name' ``` > CAUTION: commissioning boots the machine and runs MAAS's commissioning > scripts against it -- gated, one node at a time, re-read status after each. **RUN -- per node that reads `New`** ```bash maas "${MAAS_PROFILE:-admin}" machine commission ``` **VERIFY -- poll to `Ready`** ```bash maas "${MAAS_PROFILE:-admin}" machine read | jq -r '.status_name' ``` Expect `Ready` within a reasonable commissioning window (phase-00 uses a ~20 min poll budget as a reference, not a hard rule). If a node lands anywhere other than `Ready`/`Failed commissioning` after a full poll cycle, treat the unexpected status itself as gap #6 manifesting -- log the actual status observed rather than assuming it will resolve. --- ## Step 4 -- Deploy each node [MUTATION: gated, per node] Do not pass a hardcoded `distro_series`/`osystem` -- no script in this repo sets one (checked: `reenroll-hosts.sh`, `phase-00-maas-standup.sh`, and the rest of `scripts/` are all silent on OS release, meaning DC0's own hosts deploy off MAAS's region-level default). Confirm `$DC`'s rack/region default live before deploying, rather than assuming it matches DC0's: **CHECK -- MAAS's configured default OS release** ```bash maas "${MAAS_PROFILE:-admin}" maas get-config name=default_distro_series ``` If `$DC` needs a different release than the default (a real design decision, not assumed here), that override is itself a value to record and justify, not silently pass. > CAUTION: deploy installs the OS and reboots the machine -- gated, one node > at a time. **RUN -- per Ready node** ```bash maas "${MAAS_PROFILE:-admin}" machine deploy ``` **VERIFY** ```bash maas "${MAAS_PROFILE:-admin}" machine read | jq -r '.status_name' ``` Expect `Deployed`. **Apply the deploy-placement tag** (phase-00 precedent: `reenroll-hosts.sh`'s "Apply MAAS tag ... (deploy placement prereq)" step -- without a tag the bundle cannot bind a unit to the machine via `constraint tags=...`). `lib-hosts.sh`'s `HOST_TAG="openstack"` is DC0's literal (gap #7) -- confirm whether `$DC`'s Stage-5 bundle will constrain on the same tag name or a DC-specific one before tagging; do not assume reuse without checking whatever Stage 5's authoring records. ```bash maas "${MAAS_PROFILE:-admin}" tags read | jq -r '.[].name' # create/confirm the confirmed tag name, then: maas "${MAAS_PROFILE:-admin}" tag update-nodes "" add= ``` --- ## Step 5 -- Pattern A interface carve, adapted per node [MUTATION: gated, per node] `scripts/carve-host-interfaces.sh` is the validated precedent (D-060 Pattern A revert, D-100 br-ex/raw-NIC discipline) -- read its header tree again: ``` enp1s0 --> br-ex (OVS bridge) + STATIC provider-public (Pattern A; MAAS builds the OVS bridge; ovn-chassis consumes it by MAC, does not build its own -- D-100) enp7s0 --> br-metal (bridge) + STATIC metal-admin br-metal.103 (VLAN, VID 103) --> br-internal (bridge) + STATIC metal-internal enp8/9/10s0 raw + STATIC data-tenant / storage / replication enp11s0 idle (ex-lbaas; no VR1 analog needed unless re-added deliberately) ``` The VLAN-103 metal-internal stack is a REAL, already-established repo constant, safe to cite verbatim (`scripts/lib-net.sh`): `METAL_INTERNAL_VID="103"`, `METAL_INTERNAL_IFACE="br-internal"`, `METAL_INTERNAL_CIDR="10.12.12.0/22"` (the last one is DC0/DC1's value only under `lib_net_select_dc dc1`'s no-op -- confirm the same call resolves the right CIDR for whichever `$DC` you are running against; it fails loud for `dc2` per gap #4). **The script now accepts a `$DC` env var (gap #2's CLI half CLOSED 2026-07-10, DOCFIX-166)** -- `DC=$DC bash scripts/carve-host-interfaces.sh --apply` calls `lib_net_select_dc`/`lib_hosts_select_dc` immediately after sourcing the libraries, before any interface logic runs. This closes the earlier worry about a silent empty-octet carve: for `dc1` and `dc2` today, `lib_hosts_select_dc` FAILS LOUD before the script ever reaches `HOST_OCTET[]`, because no per-DC host inventory has been captured into `lib-hosts.sh` yet (true for every DC1/DC2 node right now) -- it does not fall through to an empty lookup. That data gap, not the CLI wiring, is what still blocks a direct invocation for `$DC != dc0`. Before carving each node: 1. Assign it a real octet within whatever per-DC octet band Stage 1/3 planning actually recorded (not invented here -- if no band was recorded, that is itself a finding to log before proceeding). 2. Resolve its system_id and boot MAC live (`maas machine read`, `maas interfaces read` -- same pattern as `host_sysid()` and `HOST_BOOT_MAC` in `lib-hosts.sh`, just not yet captured there for `$DC`). 3. Record hostname -> octet -> boot MAC -> system_id as you measure each, per node, THIS SESSION -- this record is exactly what Step 5's completion feeds back into `lib-hosts.sh` as a real `$DC` block (delivery checklist item below), not a placeholder. 4. Only then either (a) temporarily add that mapping to a local, uncommitted copy of `lib-hosts.sh` so `DC=$DC bash scripts/carve-host-interfaces.sh --apply` resolves correctly and runs for real (the `$DC` plumbing itself is no longer the blocker, per gap #2's closure -- only the missing data was), or (b) run the equivalent `maas` CLI calls the script itself issues (`link-subnet`, bridge/VLAN create) by hand with the same measured values. Either way, re-read the resulting interface tree per node before moving to the next (per the script's own "apply ONE host at a time" discipline). > CAUTION: mutates MAAS interface definitions on each node -- requires the > node in `Ready`... but Step 4 already deployed it. **Sequencing note:** > phase-00's precedent carves BEFORE deploy (Ready-only; deploy applies the > carved netplan on boot). If Stage 3's discovery+commission flow leaves > `$DC`'s nodes at `Ready` before you reach this step, carve THEN deploy > (matching phase-00's order) rather than the Step-3/4/5 ordering written > above -- reorder Steps 4 and 5 for your actual live sequence and note which > order you used in the as-executed log; do not silently follow this doc's > numbering if the real machine states say otherwise. **GATE (per node):** `br-ex` (OVS) STATIC on provider-public; `br-metal` STATIC on metal-admin; `br-internal` (VID 103) STATIC on metal-internal; data/storage/replication planes raw + STATIC. Provider NIC (`enp1s0`) carries NO other configuration -- confirm it stayed raw/untagged per D-100 (MAAS must not bridge the provider NIC itself outside the OVS build ovn- chassis expects). --- ## Step 6 -- PXE / boot-fabric verify (READ-ONLY) **CHECK -- boot NIC landed on `$DC`'s own metal/PXE fabric** ```bash maas "${MAAS_PROFILE:-admin}" interfaces read \ | jq -r --arg m "" '.[]|select(.mac_address==$m)|.vlan.fabric' ``` Expect the SAME fabric name every `$DC` node cross-checked against in Step 2 -- not `2_metal` (DC0's), not the peer DC's fabric. **CHECK -- PXE (v4) actually working** Confirm via the commission/deploy transcripts already captured in Steps 3-4 (a machine that successfully commissioned and deployed necessarily PXE'd successfully) rather than re-deriving a separate synthetic PXE test -- per D-101, PXE is v4-first regardless of the metal-admin dual-stack amendment, so no v6 PXE path exists to additionally check. --- ## Step 7 -- Per-DC mirror + edge NTP verify (READ-ONLY) Per gap #5: CONFIRM the per-DC artifact mirror this D-107 gate item requires actually exists and where, before running a reachability check against it. If it does not exist yet, STOP this sub-check, log the gap (it blocks the "airgap nodes pull only from an in-DC mirror" posture D-107 requires), and do not fabricate a mirror endpoint to check against. **CHECK (once the real mirror address is known) -- reachable from a deployed node** ```bash # from the deployed node, or via its MAAS-reported address: curl -sI http:/// | head -1 ``` **CHECK -- NTP from `$DC`'s own OPNsense edge (D-107: nodes get chrony from their DC edge, which syncs upstream; Office1 is NOT in this path)** ```bash chronyc sources # or, if chrony is not yet configured on a freshly-deployed node: timedatectl show -p NTP -p NTPSynchronized ``` Confirm the source is `$DC`'s own edge address (measured, not assumed) -- not Office1, not a public pool directly (D-107: the edge is the only component with controlled internet egress; nodes are airgapped at the node boundary). --- ## Step 8 -- Topology consistency check, adapted per DC (READ-ONLY) `scripts/phase-00-maas-standup.sh` is the validated precedent for "does MAAS's fabric/VLAN/subnet/space topology match the target plane scheme." **Gap #2's CLI half is CLOSED (2026-07-10, DOCFIX-166):** the script now accepts a `$DC` env var and calls `lib_net_select_dc "$DC"` immediately after sourcing `lib-net.sh`. This genuinely simplifies the `dc1` case: since D-101 makes dc1 inherit dc0's plane literals unchanged, the script's hardcoded `PLANES` table is ALSO dc1's real target, so ```bash DC=dc1 bash scripts/phase-00-maas-standup.sh ``` is now a real, correct, gated call for `dc1` -- run it directly instead of the manual space-read-and-compare below. For `dc2`, run it the same way and expect it to fail loud immediately at the selector (gap #4, NetBox literals not assigned) -- a clean stop, not a false DRIFT. **What is NOT closed:** once dc2's CIDRs exist, this script's `PLANES` table itself is still a D-052/D-053 hardcoded literal (not derived from `lib-net.sh`'s flat vars), so it will need its own follow-up parameterization before it is genuinely dc2-topology-aware -- until then, for a real dc2 buildout, fall back to the manual cross-check below instead of trusting the script's literals: ```bash maas "${MAAS_PROFILE:-admin}" spaces read \ | jq -r '.[] | "\(.name)\t\([.subnets[]?.cidr] | join(", "))"' | sort ``` Cross-check the six spaces/CIDRs against whatever `lib_net_select_dc "$DC"` resolved in Step 1 -- confirm no DRIFT (a subnet present but bound to the wrong plane or VID) before treating `$DC` as ready for Stage 5's Juju controller bootstrap. **Explicitly NOT this stage's job:** `scripts/provider-bundle-check.py` (the bundle-invariant gate phase-00 runs alongside the topology check) validates `bundle.yaml`'s DC0-specific chassis MACs, VIP triples, and unit counts. Stage 5's own authoring status already flags that `bundle.yaml` needs a per-DC parameterization pass before reuse -- do not run `provider-bundle-check.py` against `$DC` from this stage; that check belongs at the START of Stage 5, once that parameterization exists. --- ## GATE (Stage 4 exit condition, per DC -- restated honestly from the buildout design) Per `docs/dc-dc-buildout-design.md` Section 4 Phase 3: - Nodes deployed (Steps 3-4: all discovered `$DC` nodes read `Deployed`). - The six planes present per node with correct fabrics/VLANs (Step 5's per-node GATE, Step 8's topology cross-check). - Provider NIC raw (Step 5 GATE: `enp1s0`/equivalent carries no config beyond what MAAS's own br-ex build applies). - PXE (v4) working (Step 6). - Per-DC mirror reachable from nodes (Step 7 -- CONDITIONAL on gap #5 being resolved; if the mirror does not exist yet, this GATE bullet is honestly NOT MET and Stage 4 is not complete for `$DC`, full stop, regardless of how clean the MAAS-side steps came out). - NTP from the DC's own OPNsense edge working (Step 7). All six true, for every node Step 2 discovered for `$DC` -> Stage 4 complete for that DC. Run the whole runbook again with the other `$DC` value for the second DC; the two DCs' completions are independent (D-100: no shared control plane), so one can be done before the other. --- ## Next Stage 5 (`docs/dc-dc-deployment-workflow.md` -- Phase 4: Juju controller + OpenStack bundle, per DC): bootstrap the per-DC Juju controller onto these now-deployed machines, parameterize `bundle.yaml` for `$DC` (D-101 family matrix, D-109 Vault root), and run `provider-bundle-check.py` for the first time against that parameterized bundle before `preflight.sh`. --- ## Delivery checklist (this repo's standard discipline) - [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing anything touched while executing this runbook. - [ ] **Populate `scripts/lib-hosts.sh`'s per-DC host data from this stage's real enrollment, as its own follow-up delivery** -- the real hostnames, boot MACs, octets, and a note on how to resolve each node's system_id live, captured in Steps 2-5 above, go into a new `$DC`-scoped block in `lib-hosts.sh` (mirroring the existing DC0 arrays' shape, not overwriting them) so that `lib_hosts_select_dc "$DC"` can go from FAIL to a real, populated selection. Do not commit this data before it has actually been measured against a live enrollment. - [ ] Log gap #1 (no OpenTofu module creates a per-DC MAAS rack controller VM) against Stage 3 if it was hit as a real blocker, not just noted. - [x] Gap #2's CLI half CLOSED 2026-07-10 (DOCFIX-166): `reenroll-hosts.sh` / `carve-host-interfaces.sh` / `phase-00-maas-standup.sh` now accept `$DC` and call the selectors -- see `docs/changelog-20260709-maas-scripts-dc-param.md`. Remaining: once `lib-hosts.sh` gets a real `$DC` block (checklist item above), `reenroll-hosts.sh`/`carve-host-interfaces.sh` become directly callable for that DC with no further script change; separately, `phase-00-maas-standup.sh`'s hardcoded `PLANES` table still needs its own parameterization pass before it is genuinely dc2-topology-aware. - [ ] Log gap #5 (no identified owning stage/module for the per-DC artifact mirror) against Stage 2 (or a new stage) if the mirror was found not to exist when Step 7 was reached -- this GATE bullet cannot be met until that closes, so it blocks Stage 4 completion honestly rather than being paperable-over. - [ ] Changelog entry for this runbook's first real execution per DC (next DOCFIX number via `bash scripts/ledger-scan.sh`), noting actual measured values (node count, hostnames, octets, fabric names, tag name used) -- these are as-built facts this repo commits, not secrets. - [ ] `docs/session-ledger.md` updated with the outcome, per DC. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 4 row and tracker table updated from NOT STARTED to the honest actual state (may be PARTIAL -- e.g. DC1 done, DC2 still blocked on gap #4's NetBox literals).