DC-DC Phase 3 -- MAAS enlist / commission / deploy (per DC) (Stage 4)

Bring the per-DC node VMs that Stage 3 (docs/dc-dc-deployment-workflow.md -- "Stage 3 -- Phase 2: OpenTofu builds each DC substrate") caused MAAS to DISCOVER through to a deployed OS, ready for Juju (Stage 5 / Phase 4). Run this runbook ONCE PER DC: set DC=dc1 or DC=dc2 for the whole session and do not mix the two DCs' commands in one shell. This is the FIRST DC-DC runbook whose subject is MAAS itself in the multi-rack VR1 shape -- everything before it (Stage 0-3) either ratified decisions or built the substrate MAAS now sits on top of.

Governing docs: docs/dc-dc-buildout-design.md Section 4 "### Phase 3" (goal/build/gate, quoted in the GATE section below) and Section 5 (OpenTofu/MAAS/Juju boundary: "MAAS: enlist (discover) / commission / deploy / power / release of node VMs. Does not compose. One region (Office1) + one rack per DC (virsh VM-host)."); docs/dc-dc-deployment-workflow.md Stage 4 row ("Reuse vs new: PARTIAL reuse... the per-DC multi-rack registration is new, the deploy mechanics are precedent"); docs/design-decisions.md D-103 (lifecycle seam: MAAS owns commission/deploy/power/release, does NOT compose), D-100 (br-ex / provider-NIC raw discipline, Pattern A), D-107 (per-DC airgap mirror + edge NTP). runbooks/phase-00-teardown-maas-reset.md Steps 5-7 are the validated single-rack precedent this runbook adapts -- read it first if you have not; this doc assumes you have and does not re-derive what it already established.

Decisions this runbook owns: D-103, D-100 (br-ex), D-107 (mirror

NTP).

Precondition -- Stage 3's gate must already be green (READ-ONLY)

Stage 3's own runbook (runbooks/dc-dc-phase2-tofu-dc-substrate.md) is written but NOT YET EXECUTED as of this writing -- confirm it has actually been RUN (not merely authored) for $DC before proceeding, by checking its own EXIT GATE against docs/dc-dc-buildout-design.md Section 4 Phase 2 gate text: "MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured." If that gate is not green for $DC, STOP here -- do not attempt to stand up a rack controller, register a VM host, or create node VMs from inside THIS runbook. That is Stage 3's job (D-103: OpenTofu creates the node VMs and registers the libvirt host as a MAAS VM host); this runbook only picks up from "MAAS already sees them."

CHECK (read-only) -- confirm the rack controller MAAS thinks it has for this DC

maas "${MAAS_PROFILE:-admin}" rack-controllers read | jq -r '.[] | "\(.hostname)\t\(.system_id)"'

Identify, by eye, which entry (if any) is $DC's rack controller -- there is no established naming convention yet to grep for (Stage 3 has not run; do not assume a name like dc1-rack). If nothing plausible appears, Stage 3's gate is not met -- stop and go finish Stage 3.

Known gaps flagged before you start (not fixed by this runbook)

These are real, current repo gaps surfaced while adapting phase-00's single-rack sequence to the multi-rack VR1 shape. Each is a FINDING to log (changelog / D-NNN / DOCFIX candidate per this repo's discipline) at the point it is actually hit, not something to route around silently here.

No OpenTofu module stands up a per-DC MAAS rack controller VM itself. opentofu/README.md's built-module list has maas-vm-host (registers an existing libvirt/virsh connection as a VM host WITH a MAAS rack), but nothing that creates the rack-controller service VM the buildout design's Section 5 says each DC needs ("one region (Office1) + one rack per DC"). If Stage 3's precondition check above finds no rack controller for $DC, that absence is this gap manifesting, not a Stage 4 problem to solve here.
scripts/reenroll-hosts.sh, scripts/carve-host-interfaces.sh, and scripts/phase-00-maas-standup.sh do not accept a $DC argument and do not call lib_net_select_dc/lib_hosts_select_dc. Only lib-net.sh and lib-hosts.sh themselves gained the selector functions (DOCFIX-151, 2026-07-09). This runbook therefore describes ADAPTING each script's MECHANICS by hand per DC (the CLI calls it makes, in the order it makes them) rather than literally invoking e.g. bash scripts/reenroll-hosts.sh dc1 -- that invocation does not exist. Parameterizing these three scripts for $DC directly is a real follow-up (log it), not done here.
Node count and sizing per DC is an unmade Phase-0 decision. opentofu/README.md's node-vm module note: "node count/memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision... that hasn't been made -- call the module once real values exist." Do NOT assume DC0's four hosts (openstack0-3) carry over to $DC -- discover however many nodes Stage 3 actually created, live, in Step 2 below.
DC2 is additionally blocked at the network-selector layer. lib_net_select_dc dc2 FAILS LOUD today (D-101's NetBox-literals open item, tooling gap register #3 -- no CIDRs assigned yet for DC2). This runbook cannot proceed past Step 1 for DC=dc2 until that closes. dc1 is unblocked at this layer (D-101: DC1 inherits DC0's v4 layout unchanged, so lib_net_select_dc dc1 is a documented no-op).
The per-DC artifact mirror (D-107) has no identified owning stage/module in this repo as surveyed for this runbook. Stage 2's build list (Office1 headend) mentions GitBucket (the REPO mirror) and NetBox -- neither is the node-package/apt/snap/charmhub/image mirror D-107 describes. Before Step 7's mirror-reachability check, CONFIRM the mirror actually exists and where; if it does not, that is a gap to log against Stage 2 or a new stage, not something to invent a reachability check for.
The MAAS status a VM-host-discovered (not PXE-enlisted) pre-existing domain lands in is not confirmed anywhere in this repo. Phase-00's reenroll-hosts.sh header states PXE-enlisted hosts "auto-commission on create" -- but Stage 3's node VMs are discovered via maas_vm_host pod scan of already-existing libvirt domains, a different MAAS code path. Step 2 below measures the real status live rather than assuming either "New" or "auto-Ready."
lib-hosts.sh's HOST_TAG/boot-fabric-name conventions (openstack, 2_metal) are DC0-specific literals. Whether $DC's bundle placement tag and PXE fabric carry the same names is not established -- confirm live per DC rather than assuming reuse of DC0's literal strings.

Sequence

0.  Precondition            confirm Stage 3 gate (above)                (read-only)
1.  DC selector              lib_net_select_dc "$DC"; note lib_hosts_select_dc fails  (read-only)
2.  Discover                 find $DC's node VMs live in MAAS            (read-only)
3.  Commission                per discovered node                        [MUTATION: gated]
4.  Deploy                    per discovered node                        [MUTATION: gated]
5.  Pattern A carve (adapted) per node, RAW provider NIC + VLAN-103 stack [MUTATION: gated]
6.  PXE / boot-fabric verify  confirm v4 PXE + fabric landing             (read-only)
7.  Mirror + NTP verify       per-DC mirror reachable; edge NTP           (read-only)
8.  Topology check (adapted)  phase-00-maas-standup.sh mechanics, per DC  (read-only)
    -> EXIT GATE -> Stage 5 (Juju controller + bundle, per DC)

Step 1 -- DC selector (READ-ONLY)

Every session/script that sources scripts/lib-net.sh for this stage calls the selector ONCE, immediately after sourcing, before using any flat plane variable:

SCRIPT_DIR="$(pwd)/scripts"
. "$SCRIPT_DIR/lib-net.sh"
. "$SCRIPT_DIR/lib-hosts.sh"
lib_net_select_dc "$DC"     # dc1: documented no-op (D-101 inherits DC0 v4).
                            # dc2: FAILS LOUD today -- gap #3, stop here for dc2.

Then, honestly, also call the host selector and expect it to fail for BOTH DCs right now:

lib_hosts_select_dc "$DC"   # EXPECTED TO FAIL for dc1 AND dc2 until THIS
                             # stage's own completion populates real per-DC
                             # host data into lib-hosts.sh (Step 5 below).

This is a real, expected chicken-and-egg, not a bug to work around: the selector exists specifically to PREVENT a $DC script from silently reusing HOSTS/HOST_OCTET/HOST_BOOT_MAC/VIRSH_POWER_ADDRESS from DC0's real, already-enrolled hosts for a different DC's different VMs. Until this stage's enrollment produces $DC's own real hostnames, MACs, and system_ids, there is nothing correct for lib_hosts_select_dc "$DC" to select -- so it fails, on purpose, for both dc1 and dc2, and will keep failing until Step 5's data goes into lib-hosts.sh for real. Do not comment out the call, do not pre-populate placeholder host data to make it pass -- populating it with REAL data once Step 5 has measured it is this stage's own delivery item (see the checklist).

Because lib_hosts_select_dc cannot yet be relied on, every step below resolves $DC's host identity LIVE from MAAS (by whatever the discovery in Step 2 actually finds), never from lib-hosts.sh's DC0 maps.

Step 2 -- Discover `$DC`'s node VMs live in MAAS (READ-ONLY)

There is no fixed hostname list for $DC (gap #3 above) and no established zone/pool/tag naming convention yet (Stage 3 has not run to record one). Confirm, from what Stage 3 actually did for $DC (its as-executed log / session notes), which of these MAAS groupings it used, then discover against THAT -- do not guess a name like "dc1":

CHECK -- by zone (if Stage 3 recorded a zone name for $DC)

maas "${MAAS_PROFILE:-admin}" zones read | jq -r '.[] | "\(.name)"'
# then, once you have confirmed the real zone name for $DC (call it Z below):
maas "${MAAS_PROFILE:-admin}" machines read \
  | jq -r --arg z "Z" '.[] | select(.zone.name==$z) | "\(.hostname)\t\(.system_id)\t\(.status_name)"'

CHECK -- by resource pool (if Stage 3 used a pool instead)

maas "${MAAS_PROFILE:-admin}" resource-pools read | jq -r '.[] | "\(.name)"'
maas "${MAAS_PROFILE:-admin}" machines read \
  | jq -r --arg p "P" '.[] | select(.pool.name==$p) | "\(.hostname)\t\(.system_id)\t\(.status_name)"'

Cross-check -- by fabric/VLAN (independent confirmation: D-100 makes each DC's metal-admin/PXE plane its own isolated segment, so $DC's machines' boot interfaces should all land on the SAME fabric, and that fabric should be DIFFERENT from DC0's 2_metal and from the other DC's):

maas "${MAAS_PROFILE:-admin}" interfaces read <system_id> \
  | jq -r '.[] | select(.type=="physical") | "\(.mac_address)\t\(.vlan.fabric)"'

Run this per candidate system_id from the zone/pool query above; every node claimed for $DC should share one fabric name, and that name should not be 2_metal (DC0's) nor match the other DC's. If the grouping mechanism disagrees with the fabric cross-check (e.g. a machine tagged into $DC's pool but sitting on a different DC's fabric), STOP -- that is a real enrollment inconsistency, not something to paper over by picking whichever signal you prefer.

GATE (informal, before mutating): you have a concrete, confirmed list of system_ids that both (a) belong to $DC by whatever grouping Stage 3 used and (b) share one boot fabric distinct from DC0's and the peer DC's. Record that list -- Steps 3-5 iterate over it. Record the count too; it is whatever Stage 3 actually created (gap #3), not an assumed 4.

Step 3 -- Commission each discovered node [MUTATION: gated, per node]

Phase-00 precedent (reenroll-hosts.sh): freshly PXE-enlisted hosts auto-commission. Stage 3's nodes arrived via maas_vm_host pod discovery of pre-existing libvirt domains, a different code path whose landing status this repo has not previously observed (gap #6) -- Step 2's status_name column tells you what you actually got. If a node already reads Ready, skip it (idempotent). If it reads New, commission it explicitly:

CHECK (read-only) -- current status, one more time, immediately before mutating

maas "${MAAS_PROFILE:-admin}" machine read <system_id> | jq -r '.status_name'

CAUTION: commissioning boots the machine and runs MAAS's commissioning scripts against it -- gated, one node at a time, re-read status after each.

RUN -- per node that reads New

maas "${MAAS_PROFILE:-admin}" machine commission <system_id>

VERIFY -- poll to Ready

maas "${MAAS_PROFILE:-admin}" machine read <system_id> | jq -r '.status_name'

Expect Ready within a reasonable commissioning window (phase-00 uses a ~20 min poll budget as a reference, not a hard rule). If a node lands anywhere other than Ready/Failed commissioning after a full poll cycle, treat the unexpected status itself as gap #6 manifesting -- log the actual status observed rather than assuming it will resolve.

Step 4 -- Deploy each node [MUTATION: gated, per node]

Do not pass a hardcoded distro_series/osystem -- no script in this repo sets one (checked: reenroll-hosts.sh, phase-00-maas-standup.sh, and the rest of scripts/ are all silent on OS release, meaning DC0's own hosts deploy off MAAS's region-level default). Confirm $DC's rack/region default live before deploying, rather than assuming it matches DC0's:

CHECK -- MAAS's configured default OS release

maas "${MAAS_PROFILE:-admin}" maas get-config name=default_distro_series

If $DC needs a different release than the default (a real design decision, not assumed here), that override is itself a value to record and justify, not silently pass.

CAUTION: deploy installs the OS and reboots the machine -- gated, one node at a time.

RUN -- per Ready node

maas "${MAAS_PROFILE:-admin}" machine deploy <system_id>

VERIFY

maas "${MAAS_PROFILE:-admin}" machine read <system_id> | jq -r '.status_name'

Expect Deployed.

Apply the deploy-placement tag (phase-00 precedent: reenroll-hosts.sh's "Apply MAAS tag ... (deploy placement prereq)" step -- without a tag the bundle cannot bind a unit to the machine via constraint tags=...). lib-hosts.sh's HOST_TAG="openstack" is DC0's literal (gap #7) -- confirm whether $DC's Stage-5 bundle will constrain on the same tag name or a DC-specific one before tagging; do not assume reuse without checking whatever Stage 5's authoring records.

maas "${MAAS_PROFILE:-admin}" tags read | jq -r '.[].name'
# create/confirm the confirmed tag name, then:
maas "${MAAS_PROFILE:-admin}" tag update-nodes "<confirmed-tag>" add=<system_id>

Step 5 -- Pattern A interface carve, adapted per node [MUTATION: gated, per node]

scripts/carve-host-interfaces.sh is the validated precedent (D-060 Pattern A revert, D-100 br-ex/raw-NIC discipline) -- read its header tree again:

enp1s0  --> br-ex (OVS bridge) + STATIC provider-public   (Pattern A; MAAS builds
                                                            the OVS bridge; ovn-chassis
                                                            consumes it by MAC, does not
                                                            build its own -- D-100)
enp7s0  --> br-metal (bridge) + STATIC metal-admin
            br-metal.103 (VLAN, VID 103)
              --> br-internal (bridge) + STATIC metal-internal
enp8/9/10s0  raw + STATIC data-tenant / storage / replication
enp11s0      idle (ex-lbaas; no VR1 analog needed unless re-added deliberately)

The VLAN-103 metal-internal stack is a REAL, already-established repo constant, safe to cite verbatim (scripts/lib-net.sh): METAL_INTERNAL_VID="103", METAL_INTERNAL_IFACE="br-internal", METAL_INTERNAL_CIDR="10.12.12.0/22" (the last one is DC0/DC1's value only under lib_net_select_dc dc1's no-op -- confirm the same call resolves the right CIDR for whichever $DC you are running against; it fails loud for dc2 per gap #4).

The script itself is not yet DC-parameterized (gap #2). It resolves a per-host static octet from lib-hosts.sh's HOST_OCTET[<hostname>], keyed by hostname -- for a $DC node whose hostname is not yet in that map (true for every DC1/DC2 node right now), do NOT run bash scripts/carve-host-interfaces.sh <new-hostname> --apply as-is: the octet lookup would return empty and the script would either fail cleanly (preferred) or, worse, carve an interface with no static address. Before carving each node:

Assign it a real octet within whatever per-DC octet band Stage 1/3 planning actually recorded (not invented here -- if no band was recorded, that is itself a finding to log before proceeding).
Resolve its system_id and boot MAC live (maas machine read, maas interfaces read -- same pattern as host_sysid() and HOST_BOOT_MAC in lib-hosts.sh, just not yet captured there for $DC).
Record hostname -> octet -> boot MAC -> system_id as you measure each, per node, THIS SESSION -- this record is exactly what Step 5's completion feeds back into lib-hosts.sh as a real $DC block (delivery checklist item below), not a placeholder.
Only then either (a) temporarily add that mapping to a local, uncommitted copy of lib-hosts.sh so carve-host-interfaces.sh <hostname> --apply resolves correctly, or (b) run the equivalent maas CLI calls the script itself issues (link-subnet, bridge/VLAN create) by hand with the same measured values. Either way, re-read the resulting interface tree per node before moving to the next (per the script's own "apply ONE host at a time" discipline).

CAUTION: mutates MAAS interface definitions on each node -- requires the node in Ready... but Step 4 already deployed it. Sequencing note: phase-00's precedent carves BEFORE deploy (Ready-only; deploy applies the carved netplan on boot). If Stage 3's discovery+commission flow leaves $DC's nodes at Ready before you reach this step, carve THEN deploy (matching phase-00's order) rather than the Step-3/4/5 ordering written above -- reorder Steps 4 and 5 for your actual live sequence and note which order you used in the as-executed log; do not silently follow this doc's numbering if the real machine states say otherwise.

GATE (per node): br-ex (OVS) STATIC on provider-public; br-metal STATIC on metal-admin; br-internal (VID 103) STATIC on metal-internal; data/storage/replication planes raw + STATIC. Provider NIC (enp1s0) carries NO other configuration -- confirm it stayed raw/untagged per D-100 (MAAS must not bridge the provider NIC itself outside the OVS build ovn- chassis expects).

Step 6 -- PXE / boot-fabric verify (READ-ONLY)

CHECK -- boot NIC landed on $DC's own metal/PXE fabric

maas "${MAAS_PROFILE:-admin}" interfaces read <system_id> \
  | jq -r --arg m "<boot-mac-measured-in-step-2-or-5>" '.[]|select(.mac_address==$m)|.vlan.fabric'

Expect the SAME fabric name every $DC node cross-checked against in Step 2 -- not 2_metal (DC0's), not the peer DC's fabric.

CHECK -- PXE (v4) actually working Confirm via the commission/deploy transcripts already captured in Steps 3-4 (a machine that successfully commissioned and deployed necessarily PXE'd successfully) rather than re-deriving a separate synthetic PXE test -- per D-101, PXE is v4-first regardless of the metal-admin dual-stack amendment, so no v6 PXE path exists to additionally check.

Step 7 -- Per-DC mirror + edge NTP verify (READ-ONLY)

Per gap #5: CONFIRM the per-DC artifact mirror this D-107 gate item requires actually exists and where, before running a reachability check against it. If it does not exist yet, STOP this sub-check, log the gap (it blocks the "airgap nodes pull only from an in-DC mirror" posture D-107 requires), and do not fabricate a mirror endpoint to check against.

CHECK (once the real mirror address is known) -- reachable from a deployed node

# from the deployed node, or via its MAAS-reported address:
curl -sI http://<measured-mirror-address>/ | head -1

CHECK -- NTP from $DC's own OPNsense edge (D-107: nodes get chrony from their DC edge, which syncs upstream; Office1 is NOT in this path)

chronyc sources
# or, if chrony is not yet configured on a freshly-deployed node:
timedatectl show -p NTP -p NTPSynchronized

Confirm the source is $DC's own edge address (measured, not assumed) -- not Office1, not a public pool directly (D-107: the edge is the only component with controlled internet egress; nodes are airgapped at the node boundary).

Step 8 -- Topology consistency check, adapted per DC (READ-ONLY)

scripts/phase-00-maas-standup.sh is the validated precedent for "does MAAS's fabric/VLAN/subnet/space topology match the target plane scheme" -- but it is hardcoded to DC0/D-052/D-053's literal space names and is not yet $DC-aware (gap #2). Adapt its CHECKS (not its literals) per DC:

maas "${MAAS_PROFILE:-admin}" spaces read \
  | jq -r '.[] | "\(.name)\t\([.subnets[]?.cidr] | join(", "))"' | sort

Cross-check the six spaces/CIDRs against whatever lib_net_select_dc "$DC" resolved in Step 1 (a no-op equal to DC0's values for dc1; blocked for dc2) -- confirm no DRIFT (a subnet present but bound to the wrong plane or VID) before treating $DC as ready for Stage 5's Juju controller bootstrap. Do NOT run phase-00-maas-standup.sh itself unmodified against $DC -- its target-state literals are DC0's; running it as-is against $DC would either no-op uselessly (dc1, since the CIDRs happen to match) or produce false DRIFT findings (dc2, once its CIDRs exist and differ from DC0's).

Explicitly NOT this stage's job: scripts/provider-bundle-check.py (the bundle-invariant gate phase-00 runs alongside the topology check) validates bundle.yaml's DC0-specific chassis MACs, VIP triples, and unit counts. Stage 5's own authoring status already flags that bundle.yaml needs a per-DC parameterization pass before reuse -- do not run provider-bundle-check.py against $DC from this stage; that check belongs at the START of Stage 5, once that parameterization exists.

GATE (Stage 4 exit condition, per DC -- restated honestly from the buildout design)

Per docs/dc-dc-buildout-design.md Section 4 Phase 3:

Nodes deployed (Steps 3-4: all discovered $DC nodes read Deployed).
The six planes present per node with correct fabrics/VLANs (Step 5's per-node GATE, Step 8's topology cross-check).
Provider NIC raw (Step 5 GATE: enp1s0/equivalent carries no config beyond what MAAS's own br-ex build applies).
PXE (v4) working (Step 6).
Per-DC mirror reachable from nodes (Step 7 -- CONDITIONAL on gap #5 being resolved; if the mirror does not exist yet, this GATE bullet is honestly NOT MET and Stage 4 is not complete for $DC, full stop, regardless of how clean the MAAS-side steps came out).
NTP from the DC's own OPNsense edge working (Step 7).

All six true, for every node Step 2 discovered for $DC -> Stage 4 complete for that DC. Run the whole runbook again with the other $DC value for the second DC; the two DCs' completions are independent (D-100: no shared control plane), so one can be done before the other.

Stage 5 (docs/dc-dc-deployment-workflow.md -- Phase 4: Juju controller + OpenStack bundle, per DC): bootstrap the per-DC Juju controller onto these now-deployed machines, parameterize bundle.yaml for $DC (D-101 family matrix, D-109 Vault root), and run provider-bundle-check.py for the first time against that parameterized bundle before preflight.sh.

Delivery checklist (this repo's standard discipline)

bash scripts/repo-lint.sh clean (0 fail) before committing anything
```
touched while executing this runbook.
```

**Populate scripts/lib-hosts.sh's per-DC host data from this stage's

real enrollment, as its own follow-up delivery** -- the real hostnames,
boot MACs, octets, and a note on how to resolve each node's system_id
live, captured in Steps 2-5 above, go into a new `$DC`-scoped block in
`lib-hosts.sh` (mirroring the existing DC0 arrays' shape, not
overwriting them) so that `lib_hosts_select_dc "$DC"` can go from
FAIL to a real, populated selection. Do not commit this data before it
has actually been measured against a live enrollment.

Log gap #1 (no OpenTofu module creates a per-DC MAAS rack controller

VM) against Stage 3 if it was hit as a real blocker, not just noted.

Log gap #2 (reenroll-hosts.sh / carve-host-interfaces.sh /

phase-00-maas-standup.sh are not `$DC`-parameterized) as a DOCFIX
candidate once the manual adaptation in Steps 3/5/8 has been run at
least once for real, so the next DC's run (or a redeploy of this one)
has a real script to call instead of hand-adapted CLI.

Log gap #5 (no identified owning stage/module for the per-DC artifact

mirror) against Stage 2 (or a new stage) if the mirror was found not
to exist when Step 7 was reached -- this GATE bullet cannot be met
until that closes, so it blocks Stage 4 completion honestly rather
than being paperable-over.

Changelog entry for this runbook's first real execution per DC (next

DOCFIX number via `bash scripts/ledger-scan.sh`), noting actual
measured values (node count, hostnames, octets, fabric names, tag
name used) -- these are as-built facts this repo commits, not secrets.

docs/session-ledger.md updated with the outcome, per DC.

docs/dc-dc-deployment-workflow.md Stage 4 row and tracker table

updated from NOT STARTED to the honest actual state (may be PARTIAL
-- e.g. DC1 done, DC2 still blocked on gap #4's NetBox literals).