Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / runbooks / dc-dc-phase3-maas-enlist-deploy.md
@JANeumatrix JANeumatrix 11 hours ago 33 KB G17 opened by operator ruling; Stage 4 DoD bullets 5-6 repaired (DOCFIX-204)
Raw Blame History

DC-DC Phase 3 -- MAAS enlist / commission / deploy (per DC) (Stage 4)

Bring the per-DC node VMs that Stage 3 (docs/dc-dc-deployment-workflow.md -- "Stage 3 -- Phase 2: OpenTofu builds each DC substrate") caused MAAS to DISCOVER through to a deployed OS, ready for Juju (Stage 5 / Phase 4). Run this runbook ONCE PER DC: set DC=vr1-dc0 or DC=vr1-dc1 for the whole session and do not mix the two DCs' commands in one shell. This is the FIRST DC-DC runbook whose subject is MAAS itself in the multi-rack VR1 shape -- everything before it (Stage 0-3) either ratified decisions or built the substrate MAAS now sits on top of.

Governing docs: docs/dc-dc-buildout-design.md Section 4 "### Phase 3" (goal/build/gate, quoted in the GATE section below) and Section 5 (OpenTofu/MAAS/Juju boundary: "MAAS: enlist (discover) / commission / deploy / power / release of node VMs. Does not compose. One region (Office1) + one rack per DC (virsh VM-host)."); docs/dc-dc-deployment-workflow.md Stage 4 row ("Reuse vs new: PARTIAL reuse... the per-DC multi-rack registration is new, the deploy mechanics are precedent"); docs/design-decisions.md D-103 (lifecycle seam: MAAS owns commission/deploy/power/release, does NOT compose), D-100 (br-ex / provider-NIC raw discipline, Pattern A), D-107 (per-DC airgap mirror + edge NTP). runbooks/phase-00-teardown-maas-reset.md Steps 5-7 are the validated single-rack precedent this runbook adapts -- read it first if you have not; this doc assumes you have and does not re-derive what it already established.

Decisions this runbook owns: D-103, D-100 (br-ex), D-107 (mirror

  • NTP).

Precondition -- Stage 3's gate must already be green (READ-ONLY)

Stage 3's own runbook (runbooks/dc-dc-phase2-tofu-dc-substrate.md) is written but NOT YET EXECUTED as of this writing -- confirm it has actually been RUN (not merely authored) for $DC before proceeding, by checking its own EXIT GATE against docs/dc-dc-buildout-design.md Section 4 Phase 2 gate text: "MAAS rack controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge routing + simulated ISP uplink up; netem parameters applied and measured." If that gate is not green for $DC, STOP here -- do not attempt to stand up a rack controller, register a VM host, or create node VMs from inside THIS runbook. That is Stage 3's job (D-103: OpenTofu creates the node VMs and registers the libvirt host as a MAAS VM host); this runbook only picks up from "MAAS already sees them."

CHECK (read-only) -- confirm the rack controller MAAS thinks it has for this DC

maas "${MAAS_PROFILE:-admin}" rack-controllers read | jq -r '.[] | "\(.hostname)\t\(.system_id)"'

Identify, by eye, which entry (if any) is $DC's rack controller -- there is no established naming convention yet to grep for (Stage 3 has not run; do not assume a name like dc1-rack). If nothing plausible appears, Stage 3's gate is not met -- stop and go finish Stage 3.

Known gaps flagged before you start (not fixed by this runbook)

These are real, current repo gaps surfaced while adapting phase-00's single-rack sequence to the multi-rack VR1 shape. Each is a FINDING to log (changelog / D-NNN / DOCFIX candidate per this repo's discipline) at the point it is actually hit, not something to route around silently here.

  1. No OpenTofu module stands up a per-DC MAAS rack controller VM itself. opentofu/README.md's built-module list has maas-vm-host (registers an existing libvirt/virsh connection as a VM host WITH a MAAS rack), but nothing that creates the rack-controller service VM the buildout design's Section 5 says each DC needs ("one region (Office1) + one rack per DC"). If Stage 3's precondition check above finds no rack controller for $DC, that absence is this gap manifesting, not a Stage 4 problem to solve here.
  2. CLOSED 2026-07-10 (DOCFIX-166), CLI half only. scripts/reenroll-hosts.sh, scripts/carve-host-interfaces.sh, and scripts/phase-00-maas-standup.sh now accept an opt-in $DC env var and call lib_net_select_dc/lib_hosts_select_dc immediately after sourcing (unset $DC is byte-for-byte unchanged VR0/DC0 behavior). See docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md. This means DC=vr1-dc0 bash scripts/reenroll-hosts.sh (etc.) is now a REAL invocation, not hypothetical -- but it does not by itself unblock anything: calling any of the three with DC=vr1-dc0 or DC=vr1-dc1 today fails loud immediately at lib_hosts_select_dc (reenroll-hosts.sh, carve-host-interfaces.sh -- no per-DC host inventory exists yet) or, for dc2, at lib_net_select_dc first (gap #4 below). The one case genuinely simplified is phase-00-maas-standup.sh against dc1: since D-101 makes dc1 inherit dc0's plane literals unchanged, DC=vr1-dc0 bash scripts/phase-00-maas-standup.sh is now a real, correct, gated call -- see the updated Step 8. Parameterizing these scripts to accept REAL per-DC data (once it exists) required no further script change; the remaining blocker is populating lib-hosts.sh with a real $DC block (delivery checklist item below) and, separately, phase-00-maas-standup.sh's own PLANES table is still a D-052/D-053 hardcoded literal, not derived from lib-net.sh's flat vars, so it is not dc2-topology-aware even once dc2's CIDRs exist.
  3. Node count and sizing per DC is an unmade Phase-0 decision. opentofu/README.md's node-vm module note: "node count/memory/vcpu/disk sizing is a Phase-0 host/disk-budget decision... that hasn't been made -- call the module once real values exist." Do NOT assume DC0's four hosts (openstack0-3) carry over to $DC -- discover however many nodes Stage 3 actually created, live, in Step 2 below.
  4. DC2 is additionally blocked at the network-selector layer. lib_net_select_dc vr1-dc1 FAILS LOUD today (D-101's NetBox-literals open item, tooling gap register #3 -- no CIDRs assigned yet for DC2). This runbook cannot proceed past Step 1 for DC=vr1-dc1 until that closes. dc1 is unblocked at this layer (D-101: DC1 inherits DC0's v4 layout unchanged, so lib_net_select_dc vr1-dc0 is a documented no-op).
  5. The per-DC artifact mirror (D-107) has no identified owning stage/module in this repo as surveyed for this runbook. Stage 2's build list (Office1 headend) mentions GitBucket (the REPO mirror) and NetBox -- neither is the node-package/apt/snap/charmhub/image mirror D-107 describes. Before Step 7's mirror-reachability check, CONFIRM the mirror actually exists and where; if it does not, that is a gap to log against Stage 2 or a new stage, not something to invent a reachability check for.
  6. The MAAS status a VM-host-discovered (not PXE-enlisted) pre-existing domain lands in is not confirmed anywhere in this repo. Phase-00's reenroll-hosts.sh header states PXE-enlisted hosts "auto-commission on create" -- but Stage 3's node VMs are discovered via maas_vm_host pod scan of already-existing libvirt domains, a different MAAS code path. Step 2 below measures the real status live rather than assuming either "New" or "auto-Ready."
  7. lib-hosts.sh's HOST_TAG/boot-fabric-name conventions (openstack, 2_metal) are DC0-specific literals. Whether $DC's bundle placement tag and PXE fabric carry the same names is not established -- confirm live per DC rather than assuming reuse of DC0's literal strings.

Sequence

0.  Precondition            confirm Stage 3 gate (above)                (read-only)
1.  DC selector              lib_net_select_dc "$DC"; note lib_hosts_select_dc fails  (read-only)
2.  Discover                 find $DC's node VMs live in MAAS            (read-only)
3.  Commission                per discovered node                        [MUTATION: gated]
4.  Tag + base images         nodes STAY Ready (DOCFIX-200)              [MUTATION: gated]
5.  Pattern A carve (adapted) per node, RAW provider NIC + VLAN-103 stack [MUTATION: gated]
6.  PXE / boot-fabric verify  confirm v4 PXE + fabric landing             (read-only)
7.  Mirror + NTP verify       per-DC mirror reachable; edge NTP           (read-only)
8.  Topology check (adapted)  phase-00-maas-standup.sh mechanics, per DC  (read-only)
    -> EXIT GATE -> Stage 5 (Juju controller + bundle, per DC)

Step 1 -- DC selector (READ-ONLY)

Every session/script that sources scripts/lib-net.sh for this stage calls the selector ONCE, immediately after sourcing, before using any flat plane variable:

SCRIPT_DIR="$(pwd)/scripts"
. "$SCRIPT_DIR/lib-net.sh"
. "$SCRIPT_DIR/lib-hosts.sh"
lib_net_select_dc "$DC"     # dc1: documented no-op (D-101 inherits DC0 v4).
                            # dc2: FAILS LOUD today -- gap #3, stop here for dc2.

Then, honestly, also call the host selector and expect it to fail for BOTH DCs right now:

lib_hosts_select_dc "$DC"   # EXPECTED TO FAIL for dc1 AND dc2 until THIS
                             # stage's own completion populates real per-DC
                             # host data into lib-hosts.sh (Step 5 below).

This is a real, expected chicken-and-egg, not a bug to work around: the selector exists specifically to PREVENT a $DC script from silently reusing HOSTS/HOST_OCTET/HOST_BOOT_MAC/VIRSH_POWER_ADDRESS from DC0's real, already-enrolled hosts for a different DC's different VMs. Until this stage's enrollment produces $DC's own real hostnames, MACs, and system_ids, there is nothing correct for lib_hosts_select_dc "$DC" to select -- so it fails, on purpose, for both dc1 and dc2, and will keep failing until Step 5's data goes into lib-hosts.sh for real. Do not comment out the call, do not pre-populate placeholder host data to make it pass -- populating it with REAL data once Step 5 has measured it is this stage's own delivery item (see the checklist).

Because lib_hosts_select_dc cannot yet be relied on, every step below resolves $DC's host identity LIVE from MAAS (by whatever the discovery in Step 2 actually finds), never from lib-hosts.sh's DC0 maps.

Step 2 -- Discover $DC's node VMs live in MAAS (READ-ONLY)

There is no fixed hostname list for $DC (gap #3 above) and no established zone/pool/tag naming convention yet (Stage 3 has not run to record one). Confirm, from what Stage 3 actually did for $DC (its as-executed log / session notes), which of these MAAS groupings it used, then discover against THAT -- do not guess a name like "dc1":

CHECK -- by zone (if Stage 3 recorded a zone name for $DC)

maas "${MAAS_PROFILE:-admin}" zones read | jq -r '.[] | "\(.name)"'
# then, once you have confirmed the real zone name for $DC (call it Z below):
maas "${MAAS_PROFILE:-admin}" machines read \
  | jq -r --arg z "Z" '.[] | select(.zone.name==$z) | "\(.hostname)\t\(.system_id)\t\(.status_name)"'

CHECK -- by resource pool (if Stage 3 used a pool instead)

maas "${MAAS_PROFILE:-admin}" resource-pools read | jq -r '.[] | "\(.name)"'
maas "${MAAS_PROFILE:-admin}" machines read \
  | jq -r --arg p "P" '.[] | select(.pool.name==$p) | "\(.hostname)\t\(.system_id)\t\(.status_name)"'

Cross-check -- by fabric/VLAN (independent confirmation: D-100 makes each DC's metal-admin/PXE plane its own isolated segment, so $DC's machines' boot interfaces should all land on the SAME fabric, and that fabric should be DIFFERENT from DC0's 2_metal and from the other DC's):

maas "${MAAS_PROFILE:-admin}" interfaces read <system_id> \
  | jq -r '.[] | select(.type=="physical") | "\(.mac_address)\t\(.vlan.fabric)"'

Run this per candidate system_id from the zone/pool query above; every node claimed for $DC should share one fabric name, and that name should not be 2_metal (DC0's) nor match the other DC's. If the grouping mechanism disagrees with the fabric cross-check (e.g. a machine tagged into $DC's pool but sitting on a different DC's fabric), STOP -- that is a real enrollment inconsistency, not something to paper over by picking whichever signal you prefer.

GATE (informal, before mutating): you have a concrete, confirmed list of system_ids that both (a) belong to $DC by whatever grouping Stage 3 used and (b) share one boot fabric distinct from DC0's and the peer DC's. Record that list -- Steps 3-5 iterate over it. Record the count too; it is whatever Stage 3 actually created (gap #3), not an assumed 4.

Step 3 -- Commission each discovered node [MUTATION: gated, per node]

Phase-00 precedent (reenroll-hosts.sh): freshly PXE-enlisted hosts auto-commission. Stage 3's nodes arrived via maas_vm_host pod discovery of pre-existing libvirt domains, a different code path whose landing status this repo has not previously observed (gap #6) -- Step 2's status_name column tells you what you actually got. If a node already reads Ready, skip it (idempotent). If it reads New, commission it explicitly:

CHECK (read-only) -- current status, one more time, immediately before mutating

maas "${MAAS_PROFILE:-admin}" machine read <system_id> | jq -r '.status_name'

CAUTION: commissioning boots the machine and runs MAAS's commissioning scripts against it -- gated, one node at a time, re-read status after each.

RUN -- per node that reads New

maas "${MAAS_PROFILE:-admin}" machine commission <system_id>

VERIFY -- poll to Ready

maas "${MAAS_PROFILE:-admin}" machine read <system_id> | jq -r '.status_name'

Expect Ready within a reasonable commissioning window (phase-00 uses a ~20 min poll budget as a reference, not a hard rule). If a node lands anywhere other than Ready/Failed commissioning after a full poll cycle, treat the unexpected status itself as gap #6 manifesting -- log the actual status observed rather than assuming it will resolve.

Step 4 -- Tag each node; nodes STAY Ready (DOCFIX-200; NO MAAS deploy)

DOCFIX-200 (2026-07-23, operator-ruled at first execution). This step originally instructed maas machine deploy per node. That CONTRADICTED the validated VR0 flow and would have BROKEN Stage 5: phase-01-bundle-deploy.md's own precondition is machines "Ready/power=off; MAAS carve applied", and juju deploy bundle.yaml then provisions them itself -- Juju allocates READY MAAS machines and requests its own base (default-base: ubuntu@22.04); it cannot allocate an already-Deployed machine. Ruling (question: "skip MAAS-deploy ... and sync jammy images into the region now as the enabling step?"; operator answer: "Ready + jammy images (Recommended)"): Stage 4 ends with nodes READY + carved + tagged; the enabling mutation is syncing the bundle's base images into the region instead.

RUN (once per region, not per node) -- ensure the bundle's base images are synced. The bundle's default-base (read it from bundle.yaml, do not assume) must exist as a Synced boot resource before Stage 5:

maas "${MAAS_PROFILE:-admin}" boot-source-selections create 1 os=ubuntu release=jammy arches=amd64 subarches='*' labels='*'
maas "${MAAS_PROFILE:-admin}" boot-resources import
# poll until the release shows Synced:
maas "${MAAS_PROFILE:-admin}" boot-resources read | jq -r '.[] | select(.type=="Synced") | .name' | sort -u

Apply the deploy-placement tag (phase-00 precedent: reenroll-hosts.sh's "Apply MAAS tag ... (deploy placement prereq)" step -- without a tag the bundle cannot bind a unit to the machine via constraint tags=...). lib-hosts.sh's HOST_TAG="openstack" is DC0's literal (gap #7) -- confirm whether $DC's Stage-5 bundle will constrain on the same tag name or a DC-specific one before tagging; do not assume reuse without checking whatever Stage 5's authoring records.

maas "${MAAS_PROFILE:-admin}" tags read | jq -r '.[].name'
# create/confirm the confirmed tag name, then:
maas "${MAAS_PROFILE:-admin}" tag update-nodes "<confirmed-tag>" add=<system_id>

Step 5 -- Pattern A interface carve, adapted per node [MUTATION: gated, per node]

scripts/carve-host-interfaces.sh is the validated precedent (D-060 Pattern A revert, D-100 br-ex/raw-NIC discipline) -- read its header tree again:

enp1s0  --> br-ex (OVS bridge) + STATIC provider-public   (Pattern A; MAAS builds
                                                            the OVS bridge; ovn-chassis
                                                            consumes it by MAC, does not
                                                            build its own -- D-100)
enp7s0  --> br-metal (bridge) + STATIC metal-admin
            br-metal.103 (VLAN, VID 103)
              --> br-internal (bridge) + STATIC metal-internal
enp8/9/10s0  raw + STATIC data-tenant / storage / replication
enp11s0      idle (ex-lbaas; no VR1 analog needed unless re-added deliberately)

The VLAN-103 metal-internal stack is a REAL, already-established repo constant, safe to cite verbatim (scripts/lib-net.sh): METAL_INTERNAL_VID="103", METAL_INTERNAL_IFACE="br-internal", METAL_INTERNAL_CIDR="10.12.12.0/22" (the last one is DC0/DC1's value only under lib_net_select_dc vr1-dc0's no-op -- confirm the same call resolves the right CIDR for whichever $DC you are running against; it fails loud for dc2 per gap #4).

The script now accepts a $DC env var (gap #2's CLI half CLOSED 2026-07-10, DOCFIX-166) -- `DC=$DC bash scripts/carve-host-interfaces.sh

--applycallslib_net_select_dc/lib_hosts_select_dcimmediately after sourcing the libraries, before any interface logic runs. This closes the earlier worry about a silent empty-octet carve: fordc1anddc2today,lib_hosts_select_dcFAILS LOUD before the script ever reachesHOST_OCTET[<hostname>], because no per-DC host inventory has been captured intolib-hosts.shyet (true for every DC1/DC2 node right now) -- it does not fall through to an empty lookup. That data gap, not the CLI wiring, is what still blocks a direct invocation for$DC != dc0`. Before carving each node:

  1. Assign it a real octet within whatever per-DC octet band Stage 1/3 planning actually recorded (not invented here -- if no band was recorded, that is itself a finding to log before proceeding).
  2. Resolve its system_id and boot MAC live (maas machine read, maas interfaces read -- same pattern as host_sysid() and HOST_BOOT_MAC in lib-hosts.sh, just not yet captured there for $DC).
  3. Record hostname -> octet -> boot MAC -> system_id as you measure each, per node, THIS SESSION -- this record is exactly what Step 5's completion feeds back into lib-hosts.sh as a real $DC block (delivery checklist item below), not a placeholder.
  4. Only then either (a) temporarily add that mapping to a local, uncommitted copy of lib-hosts.sh so DC=$DC bash scripts/carve-host-interfaces.sh <hostname> --apply resolves correctly and runs for real (the $DC plumbing itself is no longer the blocker, per gap #2's closure -- only the missing data was), or (b) run the equivalent maas CLI calls the script itself issues (link-subnet, bridge/VLAN create) by hand with the same measured values. Either way, re-read the resulting interface tree per node before moving to the next (per the script's own "apply ONE host at a time" discipline).

CAUTION: mutates MAAS interface definitions on each node -- requires the node in Ready... but Step 4 already deployed it. Sequencing note: phase-00's precedent carves BEFORE deploy (Ready-only; deploy applies the carved netplan on boot). If Stage 3's discovery+commission flow leaves $DC's nodes at Ready before you reach this step, carve THEN deploy (matching phase-00's order) rather than the Step-3/4/5 ordering written above -- reorder Steps 4 and 5 for your actual live sequence and note which order you used in the as-executed log; do not silently follow this doc's numbering if the real machine states say otherwise.

GATE (per node): br-ex (OVS) STATIC on provider-public; br-metal STATIC on metal-admin; br-internal (VID 103) STATIC on metal-internal; data/storage/replication planes raw + STATIC. Provider NIC (enp1s0) carries NO other configuration -- confirm it stayed raw/untagged per D-100 (MAAS must not bridge the provider NIC itself outside the OVS build ovn- chassis expects).

Step 6 -- PXE / boot-fabric verify (READ-ONLY)

CHECK -- boot NIC landed on $DC's own metal/PXE fabric

maas "${MAAS_PROFILE:-admin}" interfaces read <system_id> \
  | jq -r --arg m "<boot-mac-measured-in-step-2-or-5>" '.[]|select(.mac_address==$m)|.vlan.fabric'

Expect the SAME fabric name every $DC node cross-checked against in Step 2 -- not 2_metal (DC0's), not the peer DC's fabric.

CHECK -- PXE (v4) actually working Confirm via the commission/deploy transcripts already captured in Steps 3-4 (a machine that successfully commissioned and deployed necessarily PXE'd successfully) rather than re-deriving a separate synthetic PXE test -- per D-101, PXE is v4-first regardless of the metal-admin dual-stack amendment, so no v6 PXE path exists to additionally check.

Step 7 -- Per-DC artifact source + time authority verify (READ-ONLY)

REWRITTEN 2026-07-27 (DOCFIX-204). As originally written this step was unrunnable and, in one half, contradicted a later ruling. Both defects are recorded here rather than quietly patched, because the step's old text is the kind that gets followed:

  1. Gap #5 (no owner for the per-DC artifact mirror) is RESOLVED -- D-135 built it, so the "if it does not exist yet, STOP" branch is history. But the two DCs deliberately have DIFFERENT artifact paths (D-135 AMENDMENT 2026-07-24), and a single mirror check applied to both fails on dc1 by design.
  2. The edge-NTP half is SUPERSEDED by D-129(iv) (RULED 2026-07-21, "Keep MAAS hierarchy (Recommended)" -- no NTP role on the edge). It asked you to confirm the edge is the time source, which the ruling explicitly refused.
  3. Both checks said "from a deployed node". Under the READY-handoff ruling (DOCFIX-200) nodes are never deployed in this stage -- they stay powered off in Ready and Juju provisions them at Stage 5. The node-side half is therefore tracked at gate G17, not here.

CHECK -- the artifact source answers on its own address, with an ATTESTED-CURRENT sync. Run the site's own checker; do not hand-roll a curl. Both were fixed on 2026-07-27 to assert on sync status instead of merely printing it -- a PASS from before that date does not mean what it appears to.

# dc0 -- D-135 item 1: a full debmirror served by nginx on the D-134 utility .4
ssh -i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@<dc0-rack-transit-ip> \
    'sudo bash -s' -- check dc0 < scripts/dc-mirror.sh

# dc1 -- D-135 AMENDMENT: the ruled path is an apt CACHING PROXY, not a mirror.
# dc1's debmirror is PAUSED and dormant as the fallback; checking it as a mirror
# is expected to FAIL and is NOT this gate.
ssh -i ~/vr1-dc1-creds/vr1-dc1_svc_ed25519 -J voffice1 jessea123@<dc1-rack-transit-ip> \
    'sudo bash -s' -- check dc1 < scripts/dc-cache-proxy.sh

Take the rack transit IP from scripts/lib-hosts.sh (VIRSH_POWER_ADDRESS), never from memory -- hard rule 2.

CHECK -- time authority is the MAAS hierarchy (D-129(iv), NOT the edge). MAAS serves NTP to the nodes it manages; the edge carries no NTP role. This is a node-side observation, so it runs when a node has actually booted an OS -- i.e. at Stage 5 first boot, alongside G17, not in this stage:

# on a booted node (Stage 5), confirm the source is the MAAS hierarchy:
chronyc sources
timedatectl show -p NTP -p NTPSynchronized

Confirm the source is the MAAS-served address (measured, not assumed) -- NOT the DC edge, which D-129(iv) removed from this path.

Step 8 -- Topology consistency check, adapted per DC (READ-ONLY)

scripts/phase-00-maas-standup.sh is the validated precedent for "does MAAS's fabric/VLAN/subnet/space topology match the target plane scheme." Gap #2's CLI half is CLOSED (2026-07-10, DOCFIX-166): the script now accepts a $DC env var and calls lib_net_select_dc "$DC" immediately after sourcing lib-net.sh. This genuinely simplifies the dc1 case: since D-101 makes dc1 inherit dc0's plane literals unchanged, the script's hardcoded PLANES table is ALSO dc1's real target, so

DC=vr1-dc0 bash scripts/phase-00-maas-standup.sh

is now a real, correct, gated call for dc1 -- run it directly instead of the manual space-read-and-compare below. For dc2, run it the same way and expect it to fail loud immediately at the selector (gap #4, NetBox literals not assigned) -- a clean stop, not a false DRIFT. What is NOT closed: once dc2's CIDRs exist, this script's PLANES table itself is still a D-052/D-053 hardcoded literal (not derived from lib-net.sh's flat vars), so it will need its own follow-up parameterization before it is genuinely dc2-topology-aware -- until then, for a real dc2 buildout, fall back to the manual cross-check below instead of trusting the script's literals:

maas "${MAAS_PROFILE:-admin}" spaces read \
  | jq -r '.[] | "\(.name)\t\([.subnets[]?.cidr] | join(", "))"' | sort

Cross-check the six spaces/CIDRs against whatever lib_net_select_dc "$DC" resolved in Step 1 -- confirm no DRIFT (a subnet present but bound to the wrong plane or VID) before treating $DC as ready for Stage 5's Juju controller bootstrap.

Explicitly NOT this stage's job: scripts/provider-bundle-check.py (the bundle-invariant gate phase-00 runs alongside the topology check) validates bundle.yaml's DC0-specific chassis MACs, VIP triples, and unit counts. Stage 5's own authoring status already flags that bundle.yaml needs a per-DC parameterization pass before reuse -- do not run provider-bundle-check.py against $DC from this stage; that check belongs at the START of Stage 5, once that parameterization exists.

GATE (Stage 4 exit condition, per DC -- restated honestly from the buildout design)

Per docs/dc-dc-buildout-design.md Section 4 Phase 3 (its "Nodes deployed" wording corrected by DOCFIX-200 -- deployment is Stage 5's juju deploy, per the phase-01 precondition this stage hands off to):

  • Nodes READY + carved + tagged; the bundle's base images Synced in the region (Steps 3-4 as corrected).
  • The six planes present per node with correct fabrics/VLANs (Step 5's per-node GATE, Step 8's topology cross-check).
  • Provider NIC raw (Step 5 GATE: enp1s0/equivalent carries no config beyond what MAAS's own br-ex build applies).
  • PXE (v4) working (Step 6).
  • Per-DC artifact source reachable ON ITS OWN ADDRESS, with an ATTESTED-CURRENT sync (Step 7). SPLIT 2026-07-27 (operator ruling, GA-R5 -- recorded in the G17 gate row of docs/CURRENT-STATE.md, which is the authority). Gap #5 is resolved -- D-135 built the artifact source per DC -- but the original bullet said "reachable FROM NODES", and the READY-handoff ruling (DOCFIX-200) leaves every node powered off in Ready, so a node-side probe cannot run in this stage at all. GA-R6 E3 forbids a conditional close, so the node-side half moved to its own gate row G17 (trigger: Stage 5 first boot). What this bullet now requires, per DC:
    • dc0 -- dc-mirror.sh check dc0 PASS, which as of 2026-07-27 ASSERTS on last-sync.status rather than merely printing it. It previously passed on a FAIL status; do not trust any pre-2026-07-27 PASS for this bullet.
    • dc1 -- dc-cache-proxy.sh check dc1 PASS. dc1's ruled artifact path is the apt CACHING PROXY, not a mirror (D-135 AMENDMENT 2026-07-24); its debmirror is paused and dormant as the fallback. Checking dc1 as a mirror fails by design and is not this bullet.
  • NTP from the DC's own OPNsense edge working (Step 7).STRUCK -- SUPERSEDED by D-129(iv), RULED 2026-07-21: "Keep MAAS hierarchy (Recommended)", no NTP role on the edge. The replacement check is that the node's time source is the MAAS hierarchy (see Step 7). This bullet was unsatisfiable as written: it required the edge to be the time source, which the ruling explicitly refused.

All bullets true, for every node Step 2 discovered for $DC -> Stage 4 complete for that DC (with the node-side reachability half tracked separately at G17). Run the whole runbook again with the other $DC value for the second DC; the two DCs' completions are independent (D-100: no shared control plane), so one can be done before the other.

CREDS-CONSOLIDATION CLOSE-OUT (SEC-009 enforcement -- non-negotiable). Standing up a DC MINTS secrets on its VMs and on the jumphost -- MAAS API key, per-DC NetBox token (if any), node/edge SSH keys, OPNsense root creds, tailscale authkey, Juju/Vault material. The moment each is minted, consolidate it into ~/$DC-creds/ (0700; files 0600), add a row to creds-matrix.tsv, and -- if it introduces a new place secrets live -- add a row to creds-manifests/vm-secret-locations. UPDATED 2026-07-26 (D-137): do NOT hand-edit creds-manifests/$DC.manifest. Manifests are now DERIVED from the matrix (sub-ruling 2), and an undeclared LOCATION is never audited at all (sub-ruling 3) -- which is how SEC-022 happened. Policy text: docs/design-decisions.md D-137.

Do NOT defer this -- "later" is when secrets get lost (and after a compaction they are gone). This is the exact class that bit us: office1's sandbox NetBox token, minted on the VM at /root/netbox-secrets/, went un-consolidated until it was needed (SEC-009 addendum). Before declaring this DC done, BOTH must hold: bash scripts/creds-audit.sh $DC reports CLEAN (folder perms, every manifest entry present, no undeclared files, no home-root sprawl), AND python3 scripts/creds-matrix.py shows no NEW finding class for this DC -- it is the blocking preflight gate P5, so a DC standup that adds an unregistered credential will stop the next deploy rather than surface later.

ALSO AT DC-STANDUP CLOSE (operator-run, D-137 tier 2 remote + tier 3, ruled 2026-07-26 -- these are NOT deploy gates, because a host outage or a snap refresh must not turn the deploy red for reasons that are not about credentials):

  • python3 scripts/creds-matrix.py --tier2 --tier3 --remote --privileged -- the remote existence sweep plus cross-copy sha256 provenance. Capture to docs/audit/ and cite it in the close.
  • bash scripts/creds-probe.sh --list, then --run once you accept the blast radius. --run AUTHENTICATES and writes an audit/login record on every target; that is the cost of knowing a credential still works.

Next

Stage 5 (docs/dc-dc-deployment-workflow.md -- Phase 4: Juju controller + OpenStack bundle, per DC): bootstrap the per-DC Juju controller onto these now-deployed machines, parameterize bundle.yaml for $DC (D-101 family matrix, D-109 Vault root), and run provider-bundle-check.py for the first time against that parameterized bundle before preflight.sh.

Delivery checklist (this repo's standard discipline)

  • bash scripts/repo-lint.sh clean (0 fail) before committing anything 
    touched while executing this runbook.
  • **Populate scripts/lib-hosts.sh's per-DC host data from this stage's 
    real enrollment, as its own follow-up delivery** -- the real hostnames,
boot MACs, octets, and a note on how to resolve each node's system_id
live, captured in Steps 2-5 above, go into a new `$DC`-scoped block in
`lib-hosts.sh` (mirroring the existing DC0 arrays' shape, not
overwriting them) so that `lib_hosts_select_dc "$DC"` can go from
FAIL to a real, populated selection. Do not commit this data before it
has actually been measured against a live enrollment.
  • Log gap #1 (no OpenTofu module creates a per-DC MAAS rack controller 
    VM) against Stage 3 if it was hit as a real blocker, not just noted.
  • Gap #2's CLI half CLOSED 2026-07-10 (DOCFIX-166): 
    `reenroll-hosts.sh` / `carve-host-interfaces.sh` /
`phase-00-maas-standup.sh` now accept `$DC` and call the selectors --
see `docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md`. Remaining:
once `lib-hosts.sh` gets a real `$DC` block (checklist item above),
`reenroll-hosts.sh`/`carve-host-interfaces.sh` become directly
callable for that DC with no further script change; separately,
`phase-00-maas-standup.sh`'s hardcoded `PLANES` table still needs its
own parameterization pass before it is genuinely dc2-topology-aware.
  • Log gap #5 (no identified owning stage/module for the per-DC artifact 
    mirror) against Stage 2 (or a new stage) if the mirror was found not
to exist when Step 7 was reached -- this GATE bullet cannot be met
until that closes, so it blocks Stage 4 completion honestly rather
than being paperable-over.
  • Changelog entry for this runbook's first real execution per DC (next 
    DOCFIX number via `bash scripts/ledger-scan.sh`), noting actual
measured values (node count, hostnames, octets, fabric names, tag
name used) -- these are as-built facts this repo commits, not secrets.
  • docs/session-ledger.md updated with the outcome, per DC.
  • docs/dc-dc-deployment-workflow.md Stage 4 row and tracker table 
    updated from NOT STARTED to the honest actual state (may be PARTIAL
-- e.g. DC1 done, DC2 still blocked on gap #4's NetBox literals).