# DC-DC Phase 3 -- MAAS enlist / commission / deploy (per DC) (Stage 4)

Bring the per-DC node VMs that Stage 3 (`docs/dc-dc-deployment-workflow.md` --
"Stage 3 -- Phase 2: OpenTofu builds each DC substrate") caused MAAS to
DISCOVER through to a deployed OS, ready for Juju (Stage 5 / Phase 4). Run
this runbook ONCE PER DC: set `DC=vr1-dc0` or `DC=vr1-dc1` for the whole session and
do not mix the two DCs' commands in one shell. This is the FIRST DC-DC
runbook whose subject is MAAS itself in the multi-rack VR1 shape -- everything
before it (Stage 0-3) either ratified decisions or built the substrate MAAS
now sits on top of.

**Governing docs:** `docs/dc-dc-buildout-design.md` Section 4 "### Phase 3"
(goal/build/gate, quoted in the GATE section below) and Section 5
(OpenTofu/MAAS/Juju boundary: "MAAS: enlist (discover) / commission / deploy
/ power / release of node VMs. Does not compose. One region (Office1) + one
rack per DC (virsh VM-host)."); `docs/dc-dc-deployment-workflow.md` Stage 4
row ("Reuse vs new: PARTIAL reuse... the per-DC multi-rack registration is
new, the deploy mechanics are precedent"); `docs/design-decisions.md`
**D-103** (lifecycle seam: MAAS owns
commission/deploy/power/release, does NOT compose), **D-100** (br-ex /
provider-NIC raw discipline, Pattern A), **D-107** (per-DC airgap mirror +
edge NTP). `runbooks/phase-00-teardown-maas-reset.md` Steps 5-7 are the
validated single-rack precedent this runbook adapts -- read it first if you
have not; this doc assumes you have and does not re-derive what it already
established.

Decisions this runbook owns: **D-103**, **D-100** (br-ex), **D-107** (mirror
+ NTP).

---

## Precondition -- Stage 3's gate must already be green (READ-ONLY)

Stage 3's own runbook (`runbooks/dc-dc-phase2-tofu-dc-substrate.md`) is
written but NOT YET EXECUTED as of this writing -- confirm it has actually
been RUN (not merely authored) for `$DC` before proceeding, by checking its
own EXIT GATE against
`docs/dc-dc-buildout-design.md` Section 4 Phase 2 gate text: "MAAS rack
controller per DC up; MAAS discovers the OpenTofu-created node VMs; edge
routing + simulated ISP uplink up; netem parameters applied and measured."
If that gate is not green for `$DC`, STOP here -- do not attempt to
stand up a rack controller, register a VM host, or create node VMs from
inside THIS runbook. That is Stage 3's job (D-103: OpenTofu creates the node
VMs and registers the libvirt host as a MAAS VM host); this runbook only
picks up from "MAAS already sees them."

**CHECK (read-only) -- confirm the rack controller MAAS thinks it has for this DC**
```bash
maas "${MAAS_PROFILE:-admin}" rack-controllers read | jq -r '.[] | "\(.hostname)\t\(.system_id)"'
```
Identify, by eye, which entry (if any) is `$DC`'s rack controller -- there is
no established naming convention yet to grep for (Stage 3 has not run; do
not assume a name like `dc1-rack`). If nothing plausible appears, Stage 3's
gate is not met -- stop and go finish Stage 3.

---

## Known gaps flagged before you start (not fixed by this runbook)

These are real, current repo gaps surfaced while adapting phase-00's
single-rack sequence to the multi-rack VR1 shape. Each is a FINDING to log
(changelog / D-NNN / DOCFIX candidate per this repo's discipline) at the
point it is actually hit, not something to route around silently here.

1. **No OpenTofu module stands up a per-DC MAAS rack controller VM itself.**
   `opentofu/README.md`'s built-module list has `maas-vm-host` (registers an
   existing libvirt/virsh connection as a VM host WITH a MAAS rack), but
   nothing that creates the rack-controller service VM the buildout design's
   Section 5 says each DC needs ("one region (Office1) + one rack per DC").
   If Stage 3's precondition check above finds no rack controller for `$DC`,
   that absence is this gap manifesting, not a Stage 4 problem to solve here.
2. **CLOSED 2026-07-10 (DOCFIX-166), CLI half only.**
   `scripts/reenroll-hosts.sh`, `scripts/carve-host-interfaces.sh`, and
   `scripts/phase-00-maas-standup.sh` now accept an opt-in `$DC` env var and
   call `lib_net_select_dc`/`lib_hosts_select_dc` immediately after sourcing
   (unset `$DC` is byte-for-byte unchanged VR0/DC0 behavior). See
   `docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md`. This means `DC=vr1-dc0
   bash scripts/reenroll-hosts.sh` (etc.) is now a REAL invocation, not
   hypothetical -- but it does not by itself unblock anything: calling any
   of the three with `DC=vr1-dc0` or `DC=vr1-dc1` today fails loud immediately at
   `lib_hosts_select_dc` (reenroll-hosts.sh, carve-host-interfaces.sh -- no
   per-DC host inventory exists yet) or, for dc2, at `lib_net_select_dc`
   first (gap #4 below). The one case genuinely simplified is
   `phase-00-maas-standup.sh` against `dc1`: since D-101 makes dc1 inherit
   dc0's plane literals unchanged, `DC=vr1-dc0 bash
   scripts/phase-00-maas-standup.sh` is now a real, correct, gated call --
   see the updated Step 8. Parameterizing these scripts to accept REAL
   per-DC data (once it exists) required no further script change; the
   remaining blocker is populating `lib-hosts.sh` with a real `$DC` block
   (delivery checklist item below) and, separately,
   `phase-00-maas-standup.sh`'s own `PLANES` table is still a D-052/D-053
   hardcoded literal, not derived from `lib-net.sh`'s flat vars, so it is
   not dc2-topology-aware even once dc2's CIDRs exist.
3. **Node count and sizing per DC is an unmade Phase-0 decision.**
   `opentofu/README.md`'s `node-vm` module note: "node count/memory/vcpu/disk
   sizing is a Phase-0 host/disk-budget decision... that hasn't been made --
   call the module once real values exist." Do NOT assume DC0's four hosts
   (`openstack0-3`) carry over to `$DC` -- discover however many nodes Stage
   3 actually created, live, in Step 2 below.
4. **DC2 is additionally blocked at the network-selector layer.**
   `lib_net_select_dc vr1-dc1` FAILS LOUD today (D-101's NetBox-literals open
   item, tooling gap register #3 -- no CIDRs assigned yet for DC2). This
   runbook cannot proceed past Step 1 for `DC=vr1-dc1` until that closes. `dc1`
   is unblocked at this layer (D-101: DC1 inherits DC0's v4 layout
   unchanged, so `lib_net_select_dc vr1-dc0` is a documented no-op).
5. **The per-DC artifact mirror (D-107) has no identified owning
   stage/module in this repo as surveyed for this runbook.** Stage 2's build
   list (Office1 headend) mentions GitBucket (the REPO mirror) and NetBox --
   neither is the node-package/apt/snap/charmhub/image mirror D-107
   describes. Before Step 7's mirror-reachability check, CONFIRM the mirror
   actually exists and where; if it does not, that is a gap to log against
   Stage 2 or a new stage, not something to invent a reachability check for.
6. **The MAAS status a VM-host-discovered (not PXE-enlisted) pre-existing
   domain lands in is not confirmed anywhere in this repo.** Phase-00's
   `reenroll-hosts.sh` header states PXE-enlisted hosts "auto-commission on
   create" -- but Stage 3's node VMs are discovered via `maas_vm_host` pod
   scan of already-existing libvirt domains, a different MAAS code path.
   Step 2 below measures the real status live rather than assuming either
   "New" or "auto-Ready."
7. **`lib-hosts.sh`'s `HOST_TAG`/boot-fabric-name conventions (`openstack`,
   `2_metal`) are DC0-specific literals.** Whether `$DC`'s bundle placement
   tag and PXE fabric carry the same names is not established -- confirm
   live per DC rather than assuming reuse of DC0's literal strings.

---

## Sequence
```
0.  Precondition            confirm Stage 3 gate (above)                (read-only)
1.  DC selector              lib_net_select_dc "$DC"; note lib_hosts_select_dc fails  (read-only)
2.  Discover                 find $DC's node VMs live in MAAS            (read-only)
3.  Commission                per discovered node                        [MUTATION: gated]
4.  Tag + base images         nodes STAY Ready (DOCFIX-200)              [MUTATION: gated]
5.  Pattern A carve (adapted) per node, RAW provider NIC + VLAN-103 stack [MUTATION: gated]
6.  PXE / boot-fabric verify  confirm v4 PXE + fabric landing             (read-only)
7.  Mirror + NTP verify       per-DC mirror reachable; edge NTP           (read-only)
8.  Topology check (adapted)  phase-00-maas-standup.sh mechanics, per DC  (read-only)
    -> EXIT GATE -> Stage 5 (Juju controller + bundle, per DC)
```

---

## Step 1 -- DC selector (READ-ONLY)

Every session/script that sources `scripts/lib-net.sh` for this stage calls
the selector ONCE, immediately after sourcing, before using any flat plane
variable:

```bash
SCRIPT_DIR="$(pwd)/scripts"
. "$SCRIPT_DIR/lib-net.sh"
. "$SCRIPT_DIR/lib-hosts.sh"
lib_net_select_dc "$DC"     # dc1: documented no-op (D-101 inherits DC0 v4).
                            # dc2: FAILS LOUD today -- gap #3, stop here for dc2.
```

Then, honestly, also call the host selector and expect it to fail for
BOTH DCs right now:

```bash
lib_hosts_select_dc "$DC"   # EXPECTED TO FAIL for dc1 AND dc2 until THIS
                             # stage's own completion populates real per-DC
                             # host data into lib-hosts.sh (Step 5 below).
```

This is a real, expected chicken-and-egg, not a bug to work around: the
selector exists specifically to PREVENT a `$DC` script from silently reusing
`HOSTS`/`HOST_OCTET`/`HOST_BOOT_MAC`/`VIRSH_POWER_ADDRESS` from DC0's real,
already-enrolled hosts for a different DC's different VMs. Until this
stage's enrollment produces `$DC`'s own real hostnames, MACs, and
system_ids, there is nothing correct for `lib_hosts_select_dc "$DC"` to
select -- so it fails, on purpose, for both `dc1` and `dc2`, and will keep
failing until Step 5's data goes into `lib-hosts.sh` for real. Do not
comment out the call, do not pre-populate placeholder host data to make it
pass -- populating it with REAL data once Step 5 has measured it is this
stage's own delivery item (see the checklist).

Because `lib_hosts_select_dc` cannot yet be relied on, every step below
resolves `$DC`'s host identity LIVE from MAAS (by whatever the discovery in
Step 2 actually finds), never from `lib-hosts.sh`'s DC0 maps.

---

## Step 2 -- Discover `$DC`'s node VMs live in MAAS (READ-ONLY)

There is no fixed hostname list for `$DC` (gap #3 above) and no established
zone/pool/tag naming convention yet (Stage 3 has not run to record one).
Confirm, from what Stage 3 actually did for `$DC` (its as-executed log /
session notes), which of these MAAS groupings it used, then discover
against THAT -- do not guess a name like `"dc1"`:

**CHECK -- by zone (if Stage 3 recorded a zone name for `$DC`)**
```bash
maas "${MAAS_PROFILE:-admin}" zones read | jq -r '.[] | "\(.name)"'
# then, once you have confirmed the real zone name for $DC (call it Z below):
maas "${MAAS_PROFILE:-admin}" machines read \
  | jq -r --arg z "Z" '.[] | select(.zone.name==$z) | "\(.hostname)\t\(.system_id)\t\(.status_name)"'
```

**CHECK -- by resource pool (if Stage 3 used a pool instead)**
```bash
maas "${MAAS_PROFILE:-admin}" resource-pools read | jq -r '.[] | "\(.name)"'
maas "${MAAS_PROFILE:-admin}" machines read \
  | jq -r --arg p "P" '.[] | select(.pool.name==$p) | "\(.hostname)\t\(.system_id)\t\(.status_name)"'
```

**Cross-check -- by fabric/VLAN** (independent confirmation: D-100 makes
each DC's metal-admin/PXE plane its own isolated segment, so `$DC`'s
machines' boot interfaces should all land on the SAME fabric, and that
fabric should be DIFFERENT from DC0's `2_metal` and from the other DC's):
```bash
maas "${MAAS_PROFILE:-admin}" interfaces read <system_id> \
  | jq -r '.[] | select(.type=="physical") | "\(.mac_address)\t\(.vlan.fabric)"'
```
Run this per candidate system_id from the zone/pool query above; every node
claimed for `$DC` should share one fabric name, and that name should not be
`2_metal` (DC0's) nor match the other DC's. If the grouping mechanism
disagrees with the fabric cross-check (e.g. a machine tagged into `$DC`'s
pool but sitting on a different DC's fabric), STOP -- that is a real
enrollment inconsistency, not something to paper over by picking whichever
signal you prefer.

**GATE (informal, before mutating):** you have a concrete, confirmed list of
system_ids that both (a) belong to `$DC` by whatever grouping Stage 3 used
and (b) share one boot fabric distinct from DC0's and the peer DC's. Record
that list -- Steps 3-5 iterate over it. Record the count too; it is
whatever Stage 3 actually created (gap #3), not an assumed 4.

---

## Step 3 -- Commission each discovered node [MUTATION: gated, per node]

Phase-00 precedent (`reenroll-hosts.sh`): freshly PXE-enlisted hosts
auto-commission. Stage 3's nodes arrived via `maas_vm_host` pod discovery of
pre-existing libvirt domains, a different code path whose landing status
this repo has not previously observed (gap #6) -- Step 2's `status_name`
column tells you what you actually got. If a node already reads `Ready`,
skip it (idempotent). If it reads `New`, commission it explicitly:

**CHECK (read-only) -- current status, one more time, immediately before mutating**
```bash
maas "${MAAS_PROFILE:-admin}" machine read <system_id> | jq -r '.status_name'
```

> CAUTION: commissioning boots the machine and runs MAAS's commissioning
> scripts against it -- gated, one node at a time, re-read status after each.

**RUN -- per node that reads `New`**
```bash
maas "${MAAS_PROFILE:-admin}" machine commission <system_id>
```

**VERIFY -- poll to `Ready`**
```bash
maas "${MAAS_PROFILE:-admin}" machine read <system_id> | jq -r '.status_name'
```
Expect `Ready` within a reasonable commissioning window (phase-00 uses a
~20 min poll budget as a reference, not a hard rule). If a node lands
anywhere other than `Ready`/`Failed commissioning` after a full poll cycle,
treat the unexpected status itself as gap #6 manifesting -- log the actual
status observed rather than assuming it will resolve.

---

## Step 4 -- Tag each node; nodes STAY Ready (DOCFIX-200; NO MAAS deploy)

> **DOCFIX-200 (2026-07-23, operator-ruled at first execution).** This step
> originally instructed `maas machine deploy` per node. That CONTRADICTED the
> validated VR0 flow and would have BROKEN Stage 5: `phase-01-bundle-deploy.md`'s
> own precondition is machines "Ready/power=off; MAAS carve applied", and
> `juju deploy bundle.yaml` then provisions them itself -- Juju allocates READY
> MAAS machines and requests its own base (`default-base: ubuntu@22.04`); it
> cannot allocate an already-Deployed machine. Ruling (question: "skip
> MAAS-deploy ... and sync jammy images into the region now as the enabling
> step?"; operator answer: "Ready + jammy images (Recommended)"): Stage 4 ends
> with nodes READY + carved + tagged; the enabling mutation is syncing the
> bundle's base images into the region instead.

**RUN (once per region, not per node) -- ensure the bundle's base images are
synced.** The bundle's `default-base` (read it from `bundle.yaml`, do not
assume) must exist as a Synced boot resource before Stage 5:
```bash
maas "${MAAS_PROFILE:-admin}" boot-source-selections create 1 os=ubuntu release=jammy arches=amd64 subarches='*' labels='*'
maas "${MAAS_PROFILE:-admin}" boot-resources import
# poll until the release shows Synced:
maas "${MAAS_PROFILE:-admin}" boot-resources read | jq -r '.[] | select(.type=="Synced") | .name' | sort -u
```

**Apply the deploy-placement tag** (phase-00 precedent: `reenroll-hosts.sh`'s
"Apply MAAS tag ... (deploy placement prereq)" step -- without a tag the
bundle cannot bind a unit to the machine via `constraint tags=...`).
`lib-hosts.sh`'s `HOST_TAG="openstack"` is DC0's literal (gap #7) -- confirm
whether `$DC`'s Stage-5 bundle will constrain on the same tag name or a
DC-specific one before tagging; do not assume reuse without checking
whatever Stage 5's authoring records.
```bash
maas "${MAAS_PROFILE:-admin}" tags read | jq -r '.[].name'
# create/confirm the confirmed tag name, then:
maas "${MAAS_PROFILE:-admin}" tag update-nodes "<confirmed-tag>" add=<system_id>
```

---

## Step 5 -- Pattern A interface carve, adapted per node [MUTATION: gated, per node]

`scripts/carve-host-interfaces.sh` is the validated precedent (D-060 Pattern
A revert, D-100 br-ex/raw-NIC discipline) -- read its header tree again:

```
enp1s0  --> br-ex (OVS bridge) + STATIC provider-public   (Pattern A; MAAS builds
                                                            the OVS bridge; ovn-chassis
                                                            consumes it by MAC, does not
                                                            build its own -- D-100)
enp7s0  --> br-metal (bridge) + STATIC metal-admin
            br-metal.103 (VLAN, VID 103)
              --> br-internal (bridge) + STATIC metal-internal
enp8/9/10s0  raw + STATIC data-tenant / storage / replication
enp11s0      idle (ex-lbaas; no VR1 analog needed unless re-added deliberately)
```

The VLAN-103 metal-internal stack is a REAL, already-established repo
constant, safe to cite verbatim (`scripts/lib-net.sh`): `METAL_INTERNAL_VID="103"`,
`METAL_INTERNAL_IFACE="br-internal"`, `METAL_INTERNAL_CIDR="10.12.12.0/22"` (the
last one is DC0/DC1's value only under `lib_net_select_dc vr1-dc0`'s no-op --
confirm the same call resolves the right CIDR for whichever `$DC` you are
running against; it fails loud for `dc2` per gap #4).

**The script now accepts a `$DC` env var (gap #2's CLI half CLOSED
2026-07-10, DOCFIX-166)** -- `DC=$DC bash scripts/carve-host-interfaces.sh
<hostname> --apply` calls `lib_net_select_dc`/`lib_hosts_select_dc`
immediately after sourcing the libraries, before any interface logic runs.
This closes the earlier worry about a silent empty-octet carve: for `dc1`
and `dc2` today, `lib_hosts_select_dc` FAILS LOUD before the script ever
reaches `HOST_OCTET[<hostname>]`, because no per-DC host inventory has been
captured into `lib-hosts.sh` yet (true for every DC1/DC2 node right now) --
it does not fall through to an empty lookup. That data gap, not the CLI
wiring, is what still blocks a direct invocation for `$DC != dc0`. Before
carving each node:

1. Assign it a real octet within whatever per-DC octet band Stage 1/3
   planning actually recorded (not invented here -- if no band was recorded,
   that is itself a finding to log before proceeding).
2. Resolve its system_id and boot MAC live (`maas machine read`,
   `maas interfaces read` -- same pattern as `host_sysid()` and
   `HOST_BOOT_MAC` in `lib-hosts.sh`, just not yet captured there for `$DC`).
3. Record hostname -> octet -> boot MAC -> system_id as you measure each,
   per node, THIS SESSION -- this record is exactly what Step 5's completion
   feeds back into `lib-hosts.sh` as a real `$DC` block (delivery checklist
   item below), not a placeholder.
4. Only then either (a) temporarily add that mapping to a local, uncommitted
   copy of `lib-hosts.sh` so `DC=$DC bash scripts/carve-host-interfaces.sh
   <hostname> --apply` resolves correctly and runs for real (the `$DC`
   plumbing itself is no longer the blocker, per gap #2's closure -- only
   the missing data was), or (b) run the equivalent `maas` CLI calls the
   script itself issues (`link-subnet`, bridge/VLAN create) by hand with the
   same measured values. Either way, re-read the resulting interface tree
   per node before moving to the next (per the script's own "apply ONE host
   at a time" discipline).

> CAUTION: mutates MAAS interface definitions on each node -- requires the
> node in `Ready`... but Step 4 already deployed it. **Sequencing note:**
> phase-00's precedent carves BEFORE deploy (Ready-only; deploy applies the
> carved netplan on boot). If Stage 3's discovery+commission flow leaves
> `$DC`'s nodes at `Ready` before you reach this step, carve THEN deploy
> (matching phase-00's order) rather than the Step-3/4/5 ordering written
> above -- reorder Steps 4 and 5 for your actual live sequence and note which
> order you used in the as-executed log; do not silently follow this doc's
> numbering if the real machine states say otherwise.

**GATE (per node):** `br-ex` (OVS) STATIC on provider-public; `br-metal`
STATIC on metal-admin; `br-internal` (VID 103) STATIC on metal-internal;
data/storage/replication planes raw + STATIC. Provider NIC (`enp1s0`)
carries NO other configuration -- confirm it stayed raw/untagged per D-100
(MAAS must not bridge the provider NIC itself outside the OVS build ovn-
chassis expects).

---

## Step 6 -- PXE / boot-fabric verify (READ-ONLY)

**CHECK -- boot NIC landed on `$DC`'s own metal/PXE fabric**
```bash
maas "${MAAS_PROFILE:-admin}" interfaces read <system_id> \
  | jq -r --arg m "<boot-mac-measured-in-step-2-or-5>" '.[]|select(.mac_address==$m)|.vlan.fabric'
```
Expect the SAME fabric name every `$DC` node cross-checked against in Step
2 -- not `2_metal` (DC0's), not the peer DC's fabric.

**CHECK -- PXE (v4) actually working**
Confirm via the commission/deploy transcripts already captured in Steps 3-4
(a machine that successfully commissioned and deployed necessarily PXE'd
successfully) rather than re-deriving a separate synthetic PXE test --
per D-101, PXE is v4-first regardless of the metal-admin dual-stack
amendment, so no v6 PXE path exists to additionally check.

---

## Step 7 -- Per-DC artifact source + time authority verify (READ-ONLY)

**REWRITTEN 2026-07-27 (DOCFIX-204).** As originally written this step was
unrunnable and, in one half, contradicted a later ruling. Both defects are
recorded here rather than quietly patched, because the step's old text is the
kind that gets followed:

1. Gap #5 (no owner for the per-DC artifact mirror) is **RESOLVED** -- D-135
   built it, so the "if it does not exist yet, STOP" branch is history. But the
   two DCs deliberately have **DIFFERENT** artifact paths (D-135 AMENDMENT
   2026-07-24), and a single mirror check applied to both fails on dc1 by design.
2. The edge-NTP half is **SUPERSEDED by D-129(iv)** (RULED 2026-07-21, "Keep MAAS
   hierarchy (Recommended)" -- no NTP role on the edge). It asked you to confirm
   the edge is the time source, which the ruling explicitly refused.
3. Both checks said "from a deployed node". Under the READY-handoff ruling
   (DOCFIX-200) nodes are never deployed in this stage -- they stay powered off
   in `Ready` and Juju provisions them at Stage 5. The node-side half is
   therefore tracked at gate **G17**, not here.

**CHECK -- the artifact source answers on its own address, with an ATTESTED-CURRENT
sync.** Run the site's own checker; do not hand-roll a curl. Both were fixed on
2026-07-27 to assert on sync status instead of merely printing it -- a PASS from
before that date does not mean what it appears to.

```bash
# dc0 -- D-135 item 1: a full debmirror served by nginx on the D-134 utility .4
ssh -i ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 -J voffice1 jessea123@<dc0-rack-transit-ip> \
    'sudo bash -s' -- check dc0 < scripts/dc-mirror.sh

# dc1 -- D-135 AMENDMENT: the ruled path is an apt CACHING PROXY, not a mirror.
# dc1's debmirror is PAUSED and dormant as the fallback; checking it as a mirror
# is expected to FAIL and is NOT this gate.
ssh -i ~/vr1-dc1-creds/vr1-dc1_svc_ed25519 -J voffice1 jessea123@<dc1-rack-transit-ip> \
    'sudo bash -s' -- check dc1 < scripts/dc-cache-proxy.sh
```
Take the rack transit IP from `scripts/lib-hosts.sh` (`VIRSH_POWER_ADDRESS`), never
from memory -- hard rule 2.

**CHECK -- time authority is the MAAS hierarchy (D-129(iv), NOT the edge).** MAAS
serves NTP to the nodes it manages; the edge carries no NTP role. This is a
node-side observation, so it runs when a node has actually booted an OS -- i.e.
at Stage 5 first boot, alongside G17, not in this stage:
```bash
# on a booted node (Stage 5), confirm the source is the MAAS hierarchy:
chronyc sources
timedatectl show -p NTP -p NTPSynchronized
```
Confirm the source is the MAAS-served address (measured, not assumed) -- NOT the
DC edge, which D-129(iv) removed from this path.

---

## Step 8 -- Topology consistency check, adapted per DC (READ-ONLY)

`scripts/phase-00-maas-standup.sh` is the validated precedent for "does
MAAS's fabric/VLAN/subnet/space topology match the target plane scheme."
**Gap #2's CLI half is CLOSED (2026-07-10, DOCFIX-166):** the script now
accepts a `$DC` env var and calls `lib_net_select_dc "$DC"` immediately
after sourcing `lib-net.sh`. This genuinely simplifies the `dc1` case: since
D-101 makes dc1 inherit dc0's plane literals unchanged, the script's
hardcoded `PLANES` table is ALSO dc1's real target, so

```bash
DC=vr1-dc0 bash scripts/phase-00-maas-standup.sh
```
is now a real, correct, gated call for `dc1` -- run it directly instead of
the manual space-read-and-compare below. For `dc2`, run it the same way and
expect it to fail loud immediately at the selector (gap #4, NetBox literals
not assigned) -- a clean stop, not a false DRIFT. **What is NOT closed:**
once dc2's CIDRs exist, this script's `PLANES` table itself is still a
D-052/D-053 hardcoded literal (not derived from `lib-net.sh`'s flat vars),
so it will need its own follow-up parameterization before it is genuinely
dc2-topology-aware -- until then, for a real dc2 buildout, fall back to the
manual cross-check below instead of trusting the script's literals:

```bash
maas "${MAAS_PROFILE:-admin}" spaces read \
  | jq -r '.[] | "\(.name)\t\([.subnets[]?.cidr] | join(", "))"' | sort
```
Cross-check the six spaces/CIDRs against whatever `lib_net_select_dc "$DC"`
resolved in Step 1 -- confirm no DRIFT (a subnet present but bound to the
wrong plane or VID) before treating `$DC` as ready for Stage 5's Juju
controller bootstrap.

**Explicitly NOT this stage's job:** `scripts/provider-bundle-check.py` (the
bundle-invariant gate phase-00 runs alongside the topology check) validates
`bundle.yaml`'s DC0-specific chassis MACs, VIP triples, and unit counts.
Stage 5's own authoring status already flags that `bundle.yaml` needs a
per-DC parameterization pass before reuse -- do not run
`provider-bundle-check.py` against `$DC` from this stage; that check belongs
at the START of Stage 5, once that parameterization exists.

---

## GATE (Stage 4 exit condition, per DC -- restated honestly from the buildout design)

Per `docs/dc-dc-buildout-design.md` Section 4 Phase 3 (its "Nodes deployed"
wording corrected by DOCFIX-200 -- deployment is Stage 5's `juju deploy`, per
the phase-01 precondition this stage hands off to):
- Nodes READY + carved + tagged; the bundle's base images Synced in the
  region (Steps 3-4 as corrected).
- The six planes present per node with correct fabrics/VLANs (Step 5's
  per-node GATE, Step 8's topology cross-check).
- Provider NIC raw (Step 5 GATE: `enp1s0`/equivalent carries no config
  beyond what MAAS's own br-ex build applies).
- PXE (v4) working (Step 6).
- Per-DC artifact source reachable ON ITS OWN ADDRESS, with an ATTESTED-CURRENT
  sync (Step 7). **SPLIT 2026-07-27 (operator ruling, GA-R5 -- recorded in the
  G17 gate row of `docs/CURRENT-STATE.md`, which is the authority).** Gap #5 is
  resolved -- D-135 built the artifact source per DC -- but the original bullet
  said "reachable FROM NODES", and the READY-handoff ruling (DOCFIX-200) leaves
  every node powered off in `Ready`, so a node-side probe cannot run in this
  stage at all. GA-R6 E3 forbids a conditional close, so the node-side half
  moved to its own gate row **G17** (trigger: Stage 5 first boot). What this
  bullet now requires, per DC:
  - dc0 -- `dc-mirror.sh check dc0` PASS, which as of 2026-07-27 ASSERTS on
    `last-sync.status` rather than merely printing it. It previously passed on a
    `FAIL` status; do not trust any pre-2026-07-27 PASS for this bullet.
  - dc1 -- `dc-cache-proxy.sh check dc1` PASS. dc1's ruled artifact path is the
    apt CACHING PROXY, not a mirror (D-135 AMENDMENT 2026-07-24); its debmirror
    is paused and dormant as the fallback. Checking dc1 as a mirror fails by
    design and is not this bullet.
- ~~NTP from the DC's own OPNsense edge working (Step 7).~~ **STRUCK -- SUPERSEDED
  by D-129(iv), RULED 2026-07-21: "Keep MAAS hierarchy (Recommended)", no NTP
  role on the edge.** The replacement check is that the node's time source is the
  MAAS hierarchy (see Step 7). This bullet was unsatisfiable as written: it
  required the edge to be the time source, which the ruling explicitly refused.

All bullets true, for every node Step 2 discovered for `$DC` -> Stage 4 complete
for that DC (with the node-side reachability half tracked separately at G17).
Run the whole runbook again with the other `$DC` value for the second DC; the two
DCs' completions are independent (D-100: no shared control plane), so one can be
done before the other.

> **CREDS-CONSOLIDATION CLOSE-OUT (SEC-009 enforcement -- non-negotiable).** Standing up a DC MINTS
> secrets on its VMs and on the jumphost -- MAAS API key, per-DC NetBox token (if any), node/edge SSH
> keys, OPNsense root creds, tailscale authkey, Juju/Vault material. **The moment each is minted,
> consolidate it into `~/$DC-creds/` (0700; files 0600), add a row to `creds-matrix.tsv`, and --
> if it introduces a new place secrets live -- add a row to `creds-manifests/vm-secret-locations`.**
> UPDATED 2026-07-26 (D-137): do NOT hand-edit `creds-manifests/$DC.manifest`. Manifests are now
> DERIVED from the matrix (sub-ruling 2), and an undeclared LOCATION is never audited at all
> (sub-ruling 3) -- which is how SEC-022 happened. Policy text: `docs/design-decisions.md` D-137.
>
> Do NOT defer this -- "later" is when secrets get lost (and after a compaction they are gone). This is the exact class that bit us: office1's sandbox
> NetBox token, minted on the VM at `/root/netbox-secrets/`, went un-consolidated until it was needed
> (SEC-009 addendum). Before declaring this DC done, BOTH must hold:
> `bash scripts/creds-audit.sh $DC` reports **CLEAN** (folder perms, every manifest entry
> present, no undeclared files, no home-root sprawl), AND
> `python3 scripts/creds-matrix.py` shows no NEW finding class for this DC -- it is the
> blocking preflight gate P5, so a DC standup that adds an unregistered credential will stop
> the next deploy rather than surface later.
>
> ALSO AT DC-STANDUP CLOSE (operator-run, D-137 tier 2 remote + tier 3, ruled 2026-07-26 --
> these are NOT deploy gates, because a host outage or a snap refresh must not turn the
> deploy red for reasons that are not about credentials):
> - `python3 scripts/creds-matrix.py --tier2 --tier3 --remote --privileged`
>   -- the remote existence sweep plus cross-copy sha256 provenance. Capture to
>   `docs/audit/` and cite it in the close.
> - `bash scripts/creds-probe.sh --list`, then `--run` once you accept the blast radius.
>   **`--run` AUTHENTICATES** and writes an audit/login record on every target; that is the
>   cost of knowing a credential still works.

---

## Next

Stage 5 (`docs/dc-dc-deployment-workflow.md` -- Phase 4: Juju controller +
OpenStack bundle, per DC): bootstrap the per-DC Juju controller onto these
now-deployed machines, parameterize `bundle.yaml` for `$DC` (D-101 family
matrix, D-109 Vault root), and run `provider-bundle-check.py` for the first
time against that parameterized bundle before `preflight.sh`.

---

## Delivery checklist (this repo's standard discipline)

- [ ] `bash scripts/repo-lint.sh` clean (0 fail) before committing anything
      touched while executing this runbook.
- [ ] **Populate `scripts/lib-hosts.sh`'s per-DC host data from this stage's
      real enrollment, as its own follow-up delivery** -- the real hostnames,
      boot MACs, octets, and a note on how to resolve each node's system_id
      live, captured in Steps 2-5 above, go into a new `$DC`-scoped block in
      `lib-hosts.sh` (mirroring the existing DC0 arrays' shape, not
      overwriting them) so that `lib_hosts_select_dc "$DC"` can go from
      FAIL to a real, populated selection. Do not commit this data before it
      has actually been measured against a live enrollment.
- [ ] Log gap #1 (no OpenTofu module creates a per-DC MAAS rack controller
      VM) against Stage 3 if it was hit as a real blocker, not just noted.
- [x] Gap #2's CLI half CLOSED 2026-07-10 (DOCFIX-166):
      `reenroll-hosts.sh` / `carve-host-interfaces.sh` /
      `phase-00-maas-standup.sh` now accept `$DC` and call the selectors --
      see `docs/archive/changelogs/changelog-20260709-maas-scripts-dc-param.md`. Remaining:
      once `lib-hosts.sh` gets a real `$DC` block (checklist item above),
      `reenroll-hosts.sh`/`carve-host-interfaces.sh` become directly
      callable for that DC with no further script change; separately,
      `phase-00-maas-standup.sh`'s hardcoded `PLANES` table still needs its
      own parameterization pass before it is genuinely dc2-topology-aware.
- [ ] Log gap #5 (no identified owning stage/module for the per-DC artifact
      mirror) against Stage 2 (or a new stage) if the mirror was found not
      to exist when Step 7 was reached -- this GATE bullet cannot be met
      until that closes, so it blocks Stage 4 completion honestly rather
      than being paperable-over.
- [ ] Changelog entry for this runbook's first real execution per DC (next
      DOCFIX number via `bash scripts/ledger-scan.sh`), noting actual
      measured values (node count, hostnames, octets, fabric names, tag
      name used) -- these are as-built facts this repo commits, not secrets.
- [ ] `docs/session-ledger.md` updated with the outcome, per DC.
- [ ] `docs/dc-dc-deployment-workflow.md` Stage 4 row and tracker table
      updated from NOT STARTED to the honest actual state (may be PARTIAL
      -- e.g. DC1 done, DC2 still blocked on gap #4's NetBox literals).
