# Changelog 2026-07-30 -- dc0 MAAS region migration (Stage 5 unblock) Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`. Status claims live ONLY in `docs/CURRENT-STATE.md`. Context: Stage 5 is OPEN and BLOCKED. D-132 q1 (RULED 2026-07-30) puts a MAAS region in each DC; the dc0 region VM is LIVE at `10.12.8.6:5240`, but the Office1 region still owns all 9 dc0 role nodes plus the Juju controller. MAAS cannot move machines between regions, so the carve must be RECREATED and the machines re-enrolled. This changelog covers that migration. **Operator authorization for this session, verbatim: "Begin working through all tasks and continue deployment. Work as autonomously as possible, deploy agents as needed."** Under blanket approval this changelog IS the review surface (GA-R2): every item states what, why (evidence), and how to revert. --- ## Item 1 -- `scripts/maas-profile-assert.sh` + `tests/maas-profile-assert/` (NEW) **What.** A guard that proves WHICH MAAS REGION a CLI profile resolves to, by rack-controller identity, before any command that reads or writes that region. Exit 0 match / 1 wrong region / 2 refuse. **Why (measured, not reasoned).** Every MAAS call in this repo defaults to `MAAS_PROFILE=admin` (13 scripts: `maas-fabric-prune`, `phase-00-teardown-destroy`, `pre-flight-checks`, `maas-role-tags`, `carve-host-interfaces`, `dc-plane-ipam`, `phase-00-teardown-release`, `lib-hosts`, `maas-node-power`, `phase-00-maas-standup`, `reenroll-hosts`, and two more). MEASURED on voffice1: the sole profile `admin` resolves to `http://10.10.0.20:5240/` -- the OFFICE1 region. The two regions hold SEPARATE databases, so during this migration: - a carve recreated against the wrong profile is an idempotent NO-OP that prints PASS (the state was already there -- in the wrong region); - `machine delete` against the wrong profile DESTROYS the real record, unrecoverably. This is the wrong-HOST class this repo has been bitten by three times already (`dc-mirror.sh` run on voffice1; the egress probe run from the rack; P5 probing the wrong filesystem) -- now wrong-REGION, where the destructive direction has no undo. **A count alone is not proof** -- two regions can hold the same number of machines. Rack-controller identity is the discriminator, because a rack registers to exactly one region. **Live verification, both directions** (capture `docs/audit/dc0-region-profile-assert-20260730.txt`): admin -> machines=23 racks=voffice1,vvr1-dc0,vvr1-dc1 (Office1) vr1-dc0-region -> machines=0 racks=hot-kid (new dc0 region) $ maas-profile-assert.sh vr1-dc0-region hot-kid exit=0 $ maas-profile-assert.sh admin voffice1,vvr1-dc0,vvr1-dc1 exit=0 $ maas-profile-assert.sh admin hot-kid NEGATIVE -> exit=1 **Harness 20/20, and the MUTATION PASS FOUND A REAL DEFECT IN MY OWN TEST.** Three mutations were applied to the script and the suite re-run: - A (comparison always true) -- killed 4 tests. Good. - B (empty rack list accepted) -- killed 4 tests. Good. - C (nameless rack entries accepted) -- **suite stayed fully GREEN (19/19)**. The assertion was decoration: a nameless entry fell through to the generic empty-`ACTUAL` guard and refused for a different reason, so deleting the branch it claimed to test changed nothing. Fixed by giving the nameless case its own exit code and refusal message and asserting on THAT; mutation C now kills a test. This is exactly the "prove each new assertion can FAIL" rule, and it caught me writing decoration. **Revert.** `git rm scripts/maas-profile-assert.sh tests/maas-profile-assert/ && git checkout ^ -- .` Nothing else depends on it yet; it is additive. --- ## Item 2 -- named MAAS profile `vr1-dc0-region` on voffice1 (LIVE, config) **What.** `maas login vr1-dc0-region http://127.0.0.1:5241/MAAS/ -`, key streamed from `~/vr1-dc0-creds/maas-region-api-key.txt` on vcloud via stdin, never printed. The existing `admin` profile is UNTOUCHED and still points at Office1. **Why the tunnel.** Neither candidate host has everything the migration needs: | host | repo | maas CLI | reaches new region | reaches NetBox apex | |---|---|---|---|---| | voffice1 | yes | yes | **NO** (curl exit 28) | yes | | dc0 rack | no | yes | yes (200) | **NO** (10.10.1.10:8000 CLOSED) | | region VM | no | yes | yes | not measured | `dc-plane-ipam.sh` derives the v6 plane prefixes from the NetBox apex (D-136 option D), so it must run where NetBox is reachable. An SSH tunnel from voffice1 through the rack (`-L 127.0.0.1:5241:10.12.8.6:5240`) gives voffice1 both. Measured: `localhost:5241` answers **200**. **This does not puncture SEC-010.** SEC-010 forbids the rack FORWARDING between its transit leg and the plane nets. The tunnel is an application-layer, rack-ORIGINATED connection -- exactly the distinction SEC-010's own justification draws ("a MAAS rack proxies at the application layer and needs no kernel forwarding"). No route, no nftables rule, and no libvirt net was changed. The rack's `table inet sec010` drops are intact. **Safe failure direction, verified by construction:** if the tunnel dies, the `vr1-dc0-region` profile becomes unreachable and `maas-profile-assert.sh` REFUSES (exit 2). It cannot silently fall back to Office1, because the fallback would have to be a different URL. **CREDENTIAL RESIDENCY -- SEC-026 addendum, flagged not glossed.** This puts dc0's region admin API key on voffice1, where it was not before. Today that is a strict SUBSET of the blast radius already present: voffice1 holds the Office1 region admin key, which currently administers BOTH DCs' nodes. **After the migration that inverts** -- Office1's key stops covering dc0, and dc0's region key becomes the broader credential on a shared headend. **Obligation recorded: remove the `vr1-dc0-region` profile from voffice1 once dc0's control host is settled.** A security-ledger row update is owed at session close. **Revert.** On voffice1: `maas logout vr1-dc0-region`, and kill the tunnel (`pkill -f '5241:10.12.8.6'`). Both are local to voffice1; no cloud state is touched. --- ## Item 3 -- OPERATING NOTE: MAAS CLI for a per-DC region cannot run from voffice1 **Measured.** voffice1 -> `10.12.8.6:5240` times out (curl exit 28); voffice1 -> `10.10.0.20:5240` is fine. The dc0 rack reaches BOTH (200 / 200). **Consequence.** D-128's "Plane 2 (MAAS/NetBox/inner tofu) executes on voffice1" clause was written when there was ONE region, at Office1. D-132 q1 makes the MAAS control surface DC-local, so for the per-DC regions that clause needs either a tunnel (what this session does) or a DC-side host. **This is a consequence of an already-ruled decision, not a new one** -- recorded here, and owed as a DOCFIX against D-128's Plane-2 wording and the phase-4 runbook's run-location banner (which F5 already flags as wrong for the juju half after D-138). **No revert** -- this item is a record, not a change. --- ## Item 4 -- `lib-hosts.sh`: the power address is now per-REGION (both DCs) **What.** Each VR1 arm gains `VIRSH_POWER_ADDRESS_FROM_OFFICE1` (the rack's transit leg) and `VIRSH_POWER_ADDRESS_FROM_DCREGION` (the rack's metal-admin leg). `VIRSH_POWER_ADDRESS` is now an ALIAS of the Office1 form, byte-for-byte unchanged. The flat/VR0 layer declares both as EMPTY. **Why (measured).** MAAS power ops originate from the REGION, so the reachable rack address depends on which region dials. From `vr1-dc0-maas-01` (10.12.8.6): 172.31.0.2:22 CLOSED <- the transit address lib-hosts has always used 10.12.8.2:22 OPEN <- the rack's metal-admin leg routes: 10.12.4.0/22, 10.12.8.0/22, default via 10.12.4.1 dc1's `10.12.68.2` was MEASURED on the dc1 rack (`ip -4 -o addr` -> `virbr6 inet 10.12.68.2/22`) and corroborated against `dc-rack-net.sh`'s dc1 `LEGS` block. It was NOT inferred from dc0's `.2` by symmetry. **Backward compatibility is the load-bearing half.** `reenroll-hosts.sh:132` and the teardown runbook's `virsh -c "$VIRSH_POWER_ADDRESS"` probes must not silently change target while Office1 still owns dc1's nodes. Flipping the default to the DC-region form is OWED once BOTH DCs have migrated. **Harness `tests/dc-selector` 81 checks ALL PASS, 4 mutations each killed tests:** | mutation | tests killed | |---|---| | A: collapse dc0's DCREGION onto the transit form | 2 | | B: dc1's DCREGION gets dc0's octet (cross-DC contamination) | 2 | | C: `VIRSH_POWER_ADDRESS` silently repointed to the DC-region form | 3 | | D: VR0 inherits a VR1 form | 1 | **Revert.** `git checkout ^ -- scripts/lib-hosts.sh tests/dc-selector/run-tests.sh`. --- ## Item 5 -- `scripts/maas-region-power-key.sh` + harness (NEW), and INSTALLED at dc0 **What.** `check|install `. Installs the per-DC MAAS->libvirt power key into a region's snap and verifies it with a REAL virsh call. Key is read from STDIN, so it never lands on an intermediate host and never appears in a process argument. **Why.** `runbooks/dc-dc-phase2-tofu-dc-substrate.md` prerequisite 5 carries this as PROSE, and prose is not a gate. It must be re-run at every DC standup and -- per that same prerequisite's own FRAGILITY note -- after EVERY MAAS snap refresh, because the key lives under per-revision `/var/snap/maas/current/`. MEASURED before this work: `/var/snap/maas/current/root/.ssh/` did not exist on the new region at all, so commissioning could not have powered a node on. **LIVE at dc0** (capture `docs/audit/dc0-region-power-key-20260730.txt`), 9/9 exit 0, ending on the artifact rather than the config: [ok] key mode 0600 / owned by root / parses as a valid PRIVATE key (SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY) [ok] config binds Host 10.12.8.2 -> id_dc0_power, IdentitiesOnly yes [ok] REAL virsh connect to qemu+ssh://jessea123@10.12.8.2/system succeeds [ok] virsh enumerates domains on the rack (got 12) The key is the SAME dc0-scoped key already authorized on the dc0 rack (fingerprint matched against the rack's `authorized_keys`), so SEC-012's per-DC isolation holds -- this is not a cross-DC reuse. **OWED: remove Office1's copy once dc0's migration completes**, since that region will no longer power dc0 machines. **THREE DEFECTS THIS SCRIPT'S OWN TESTING FOUND, all measured, none by review:** 1. **`ssh-keygen -lf ` reads an ADJACENT `.pub` when one exists** and reports ITS fingerprint. A stale sibling made a freshly-installed key read back as a different one, which would have made the idempotency and cross-DC-swap refusals answer about the wrong key. Fixed with `ssh-keygen -y -f` (derive the public half FROM the private key), which no neighbouring file can fool. **Found by the harness.** 2. **A snap has TWO roots and they are not interchangeable.** `/var/snap//current` is writable data (where `root/.ssh` lives); `/snap//current` is the read-only squashfs (where the binaries live). The first live run REFUSED with "no virsh binary found" while `find /snap/maas/current -name virsh` had already located one. 3. **The snap's binary must be run INSIDE the snap's runtime.** Invoking `/snap/maas/current/usr/bin/virsh` directly fails with `error while loading shared libraries: libvirt-lxc.so.0`. `snap run --shell maas -c 'virsh ...'` executes it with the snap's environment AND its ssh config -- which is exactly the context MAAS's own power driver uses, so this is the correct artifact to grade, not a proxy. **Harness 44/44.** Assertions include every refusal direction (absent key, wrong-host binding, missing `IdentitiesOnly`, unreachable virsh, virsh listing ZERO domains) and the cross-DC-swap refusal, verified not to overwrite. Two harness assertions were found passing FOR THE WRONG REASON during development and were made specific. **A NOTE ON MY OWN INSTRUMENT.** The first live install reported `EXIT=0` while the check had actually returned 2 -- I had read `$?` from the end of a pipeline (`... | tail`), so I measured `tail`'s exit, not the script's. **That is the identical class this repo recorded twice on 2026-07-30** (`| grep -q` under `pipefail`). Fixed by capturing into a variable and reading `$?` directly. **Revert.** `git rm scripts/maas-region-power-key.sh tests/maas-region-power-key/`. To undo the live install: on the region VM, `sudo rm /var/snap/maas/current/root/.ssh/id_dc0_power` and delete the `Host 10.12.8.2` stanza from that dir's `config`. --- ## Item 6 -- RECON: what actually builds the dc0 carve (read-only; changes nothing) A read-only survey mapped each of the seven carve components to its creating tool. **Four findings change the migration plan; all four were verified against the files before being recorded here.** **(a) THE PRE-MIGRATION CAPTURE IS NOT A DIFF TARGET, and I had planned to use it as one.** Its `vlan_id=5189..5193` / `id=6,7,12..15,20..25` / `spaces id=1..6` values are MAAS **database row ids, not 802.1Q tags** -- every named fabric line reads `vid=0`. Row ids are region-local and will differ on a rebuild. Its RESERVED-RANGES section is `subnet reserved-ip-ranges` output, which COALESCES and is state-dependent (metal-internal et al. show a merged `.4-.99 reserved`, while provider-public splits `.4-.49`/`.50-.99` because `.4-.49` there also carries `assigned-ip` from live statics). **A textual diff of a perfectly rebuilt region against this file would NOT match.** The only stable identities are fabric NAME, space NAME, subnet CIDR, tag NAME, node boot MAC, interface NAME -- which is what `scripts/lib-net.sh:8-10` already says repo-wide ("CIDR is the stable key throughout; MAAS subnet IDs drift across cutovers"). The real gate is the `check` actions (`dc-plane-ipam.sh check`, `dc-node-v6-carve.py check`, `maas-role-tags.sh check`), not a file diff. My task-7 plan is corrected accordingly. **(b) `scripts/dc-node-v6-carve.py` IS PROFILE-ENV-BLIND -- verified at the file.** `PROFILE_DEFAULT = "admin"` (`:39`) and `--profile` (`:70`), with NO environment read anywhere. So `export MAAS_PROFILE=vr1-dc0-region` -- the idiom every bash tool here honours -- **silently targets Office1**. Combined with this changelog's own item 1 reasoning (a carve run against the wrong profile is an idempotent no-op that prints PASS), this is a live foot-gun on exactly the tool the migration needs. It must be invoked with an explicit `--profile`, or fixed to read the env. **(c) THREE COMPONENTS HAVE NO REPO TOOL AT ALL.** The named plane fabrics, the six v4 plane subnets, and the per-node v4 NIC carve (60 NIC re-homes + 9 `br-ex` bridges + 54 statics) were all created ad-hoc during the Stage-4 window; the mutations went to `~/as-executed/2026-07-23-stage4-carve.log`, which is NOT in the repo. The `openstack-vr1-dc0` tag has no usable creator either. So the "re-run the generator against the new endpoint" plan is only available for roughly half the work. **(d) TWO EXISTING SCRIPTS MUST NOT BE POINTED AT THE NEW REGION, both verified:** - `scripts/carve-host-interfaces.sh:62-64` REFUSES `vr1-*` by design (it carves the VR0 Pattern-A VLAN-103 stack, superseded for VR1 by D-133). `scripts/reenroll-hosts.sh:53-55` refuses `vr1-*` likewise. - `scripts/phase-00-maas-standup.sh` does NOT refuse, and that is the danger: its parity guard PASSES for `vr1-dc0`, then it creates fabrics named after the SPACE (`provider-public`, not `vr1-dc0-provider-public`, `:192-194`), creates a named `metal-admin` fabric the as-built does not have, and builds metal-internal as a tagged VID-103 VLAN riding metal-admin's fabric -- which D-133 abolished for VR1. It would silently produce a divergent region. **Also recorded:** `runbooks/phase-01-bundle-deploy.md:128` iterates **hardcoded subnet ids** (`for ID in 1 2 6 7 8 9`) and is profile-blind -- in a file that warns against exactly this at `:54` and `:287-291`. `scripts/site-headend-install.sh:528` picks a rack by list INDEX (`[0]['system_id']`), which is meaningless across regions. Both are pre-existing and LOGGED, NOT EXECUTED (hard rule 1). **No revert** -- this item is a record, not a change. --- ## Item 7 -- harness manifest re-recorded (91 harnesses) `scripts/run-tests-all.sh` FAILED with `harness set DRIFTED from tests/HARNESS-MANIFEST`, listing `maas-profile-assert` and `maas-region-power-key` as present-but-unpinned. That is the manifest gate working as designed on two harnesses added this session, not a regression. Re-recorded with `--record-manifest`; 35 -> 37 entries in the diff, 91 total. **Revert.** `git checkout ^ -- tests/HARNESS-MANIFEST`. --- ## Item 8 -- `scripts/dc-region-topology.sh` + harness (NEW), and APPLIED at dc0 **What.** `check|apply --profile

--expect-rack [--commit]` over the five named plane fabrics, the six spaces, the six v4 plane subnets on the right VLANs with the ruled gateways, the VLAN->space bindings, and the `openstack-` tag. This is the tool item 6(c) found missing. **Design decisions worth stating:** - **`--profile` has NO default.** The repo-wide default `admin` resolves to the Office1 region, where this `apply` would be an idempotent no-op that prints PASS. Every mutating path is gated by `maas-profile-assert.sh` FIRST. - **metal-admin deliberately keeps MAAS's own auto-created fabric.** It is a discovery artifact created when the rack registers, before anything in this repo could name it; Juju binds SPACES, not fabrics. The check asserts only that it does not share a fabric with a named plane (which would collapse two L2 domains). - **No numeric MAAS id is persisted.** Everything resolves per-run by name or CIDR. **LIVE at dc0** (capture `docs/audit/dc0-region-topology-20260730.txt`). The apply created 5 fabrics, 6 spaces, 4 subnets, MOVED `10.12.4.0/22` off the auto `fabric-1` onto `vr1-dc0-provider-public`, bound 6 VLANs to their spaces, and created the site tag. Final named check: **39 passed, 0 failed, exit 0.** **A CONCRETE ILLUSTRATION OF WHY ROW IDS ARE NOT PORTABLE, measured in passing:** VLAN row id **5005** is dc0's metal-admin VLAN in the Office1 region and `vr1-dc0-data-tenant` in the new region. Same integer, different object, different DC plane. Anything that carried `5005` across would have mis-targeted silently. **Harness 39/39; 4 mutations each killed tests** (region gate removed; profile defaulted to `admin`; space-binding assertion neutered; fabric-placement assertion neutered). **Fixtures were corrected against a live read:** MAAS reports an unbound VLAN as the STRING `"undefined"`, not `null`, so the first fixtures graded a payload MAAS never emits. **Revert.** The apply is additive. To undo: delete the five `vr1-dc0-*` fabrics and the six spaces on the new region, and move `10.12.4.0/22` back to `fabric-1`. Nothing in the Office1 region was touched by this item. --- ## Item 9 -- metal-admin service config set on the new region (LIVE) `dns_servers=10.12.8.6`, `allow_dns=false`, and the D-134 dynamic range `10.12.8.201-10.12.8.254`. Subnet resolved BY CIDR (-> id 1), never from the capture. **The DNS target was MEASURED, not carried over, and my first reading of it was WRONG.** The old region set `dns_servers=10.12.8.3` (the D-131 rack forwarder). A first probe suggested the new region's own BIND at `10.12.8.6` could not resolve external names, which would have forced keeping the forwarder. That was a **cold-cache timeout at `+time=3 +tries=1`, not a capability gap** -- re-probed with `+time=5 +tries=3`, `10.12.8.6` answers `archive.ubuntu.com`, `streams.canonical.com` and `api.snapcraft.io` correctly, and a full `dig` shows `flags: qr rd ra` with `ANSWER: 9`. I nearly recorded a false finding on one under-timed query. Pointing node DNS at the DC-LOCAL region is the correct end state: it is authoritative for the zone the migrated nodes will live in (its own `maas-internal`, SOA serial 16) and removes the cross-fiber dependency that D-132 q1 exists to remove. **FOLLOW-UP RECORDED, NOT ACTIONED (hard rule 1):** the live rack forwarder is `/etc/dnsmasq-dc0-node.conf` with `no-resolv` + `server=10.10.0.20` -- it sends EVERYTHING to the Office1 region. After dc0's nodes migrate, that upstream points at a region that no longer owns them. `dc-rack-net.sh` carries `DNS_UPSTREAM="10.10.0.20"` for BOTH sites (`:66`, `:82`), and dc1's is still correct, so this is a per-site change at each DC's cutover. **Also noted: the live unit is named with the BARE `dc0` token** (`dc0-node-dns.service`, `/etc/dnsmasq-dc0-node.conf`) while the current script generates `vr1-dc0-*` names -- a D-119 retired-token residue from the 2026-07-21 install. **Revert.** `maas vr1-dc0-region subnet update 1 dns_servers=10.12.8.3 allow_dns=false` and delete the dynamic iprange. --- ## Item 10 -- the DHCP handover (task 5): initially BLOCKED, then EXECUTED **What was attempted.** With the topology verified complete (39/0), the next step is the one-way DHCP cutover: # 1. off at Office1 (measured target, resolved live, not from memory) maas admin vlan update 4 0 dhcp_on=false # 2. verify dhcpd actually STOPPED on the rack, BY PROCESS # (MAAS self-report is not evidence -- the 2026-07-20 Temporal incident class) # 3. on at the new region, primary_rack = hot-kid maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack= # 4. verify dhcpd RUNNING by process **It was REFUSED by the harness permission classifier.** Per the standing rule -- and the precedent recorded in `CURRENT-STATE` for the four MAAS carve calls refused on 2026-07-30 -- it was **NOT retried in an altered shape.** **NOTHING IS HALF-APPLIED, verified read-only after the refusal:** Office1 still reads `10.12.8.0/22 dhcp_on=True primary_rack=7chphy`, and the rack still runs exactly one `dhcpd` on `virbr2`. The refusal landed before any mutation. **Measured state at the boundary, so the next session does not re-derive it:** | | Office1 region | new dc0 region | |---|---|---| | dc0 metal-admin | fabric-4 / vid 0 / vlan row 5005 | fabric-0 / vid 0 / vlan row 5001 | | dhcp_on | **True**, primary_rack `7chphy` (vvr1-dc0) | False, primary_rack None | | dns_servers | `10.12.8.3` | `10.12.8.6` | | rack system_ids | mtstwf / 7chphy / nmpcq4 | hot-kid | **Why the gap is safe to take:** all 10 dc0 machines are `Ready` and POWERED OFF, so no node is mid-DHCP. Two DHCP servers on one segment is the hazard the ordering avoids, which is why off-then-on is one operation and not two independent ones. **RESOLVED AND EXECUTED 2026-07-30** -- see item 15. Everything above stands as the record of the blocked attempt; nothing was changed at the time of the refusal. **THE EXACT UNBLOCK SEQUENCE, every parameter RE-RESOLVED LIVE at session end.** Run these on the jumphost with the `!` prefix so the output lands in-session. The `vr1-dc0-region` profile needs the tunnel, which was verified UP at capture time (`pgrep -f '5241:10.12.8.6'`); if it has since died, re-open it with `ssh voffice1 "ssh -f -N -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2"`. # 1. OFF at Office1 (fabric-4 / vid 0 -- dc0 metal-admin, dhcp_on=True, primary_rack=7chphy) ssh voffice1 'maas admin vlan update 4 0 dhcp_on=false' # 2. VERIFY BY PROCESS, not by MAAS self-report (the 2026-07-20 Temporal incident class) ssh vr1-dc0-rack 'pgrep -c dhcpd; ps -ef | grep "[d]hcpd"' # expect 0 dhcpd # 3. ON at the new region (fabric-0 / vid 0, rack hot-kid = c3aqh8 IN THAT REGION) ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=c3aqh8' # 4. VERIFY BY PROCESS again -- expect exactly one dhcpd, now owned by the region VM ssh vr1-dc0-maas 'pgrep -c dhcpd; ps -ef | grep "[d]hcpd"' # 5. Re-run the named gate; it should stay 40/0 ssh voffice1 'cd ~/openstack-caracal-dc-dc && bash scripts/dc-region-topology.sh \ check vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid' **PRE-CUTOVER CHECKS, both done, both were genuinely load-bearing:** - **`hot-kid` IS eligible as `primary_rack` for fabric-0/vid 0** -- its `enp1s0` is on that VLAN (`vlan_row=5001`, static `10.12.8.6`). If it were not, step 1 would succeed, step 3 would fail, and the segment would have NO DHCP with ten machines about to need PXE. That is the one-way risk in the sequence and it is answerable read-only, so it was. - **`hot-kid`'s system_id is `c3aqh8` in the NEW region and `tw7ptw` in Office1** -- the same machine, two different ids. One more proof that ids do not cross regions. --- ## Item 12 -- DEFECT IN MY OWN APPLY, found by reading interfaces back, and fixed **What went wrong.** `dc-region-topology.sh apply` CREATED `vr1-dc0-provider-public` and MOVED `10.12.4.0/22` onto it. **MAAS does not bring interface links along on a subnet move.** The region VM's own `enp2s0` stayed on the old auto fabric's VLAN while its subnet left, so `hot-kid` was left holding a static `10.12.4.6` on a VLAN with no subnet -- the region's provider-public leg, which it needs for image sync, and the **identical under-carve class that cost three bootstrap attempts on the Juju controller**. **Why nothing caught it.** Live connectivity was UNAFFECTED (the OS netplan is independent of MAAS's model -- `default via 10.12.4.1`, egress 200), and the check read subnets, fabrics, spaces and tags but never an interface link. **It passed 39/39 with the model inconsistent.** **Fixed three ways:** 1. **Live:** subnet returned to vlan 5002, the wrongly-created fabric deleted, `fabric-1` RENAMED to `vr1-dc0-provider-public`, space re-bound. `enp2s0` now reads `fabric=vr1-dc0-provider-public` with its static intact. 2. **In the tool:** it now RENAMES an auto fabric that already carries only this plane's subnet -- renaming moves nothing. A MOVE survives only for the case where the fabric carries other planes, and now prints a warning that links do not follow. 3. **In the gate:** a new assertion that every interface link sits on its subnet's VLAN. Live re-check is now **40 passed / 0 failed, exit 0**. Harness 47/47; the rename path and the stranded-link check each proven able to fail. One older assertion asserted the CREATE behaviour and was RE-POINTED rather than deleted, per the standing rule. --- ## Item 13 -- `dc-node-v6-carve.py`: env-blindness and traceback fixed Item 6(b) recorded this as a foot-gun; it is now fixed rather than only noted, because this migration will invoke it. `PROFILE_DEFAULT` now reads `MAAS_PROFILE` (with `--profile` still overriding), and every run PRINTS which profile it targets and where that came from -- a capture that does not name the profile cannot be audited for wrong-region work. Separately, an absent `maas` CLI raised `FileNotFoundError` and exited **1 with a traceback**, indistinguishable from a real failure and contrary to this script's own REFUSE contract; it now exits **2**. Harness 14/14 (was 9); both fixes mutation-proven. --- ## Item 14 -- the MEASURED VR1 node NIC -> plane order (`lib-hosts.sh`) The per-node v4 carve has no repo tool, so the next step was about to re-derive this by hand. MEASURED instead, end to end: the substrate roots pin six MACs per node and the module attaches them in list order, so `macs[i]` IS `enps0`. Verified on `vr1-dc0-control-01` against the live carve -- `enp1s0` metal-admin (PXE), `enp2s0` provider-public (the `br-ex` parent, Pattern A), `enp3s0` metal-internal, `enp4s0` data-tenant, `enp5s0` storage, `enp6s0` replication. **THE TRAP, now guarded: this is NOT `lib-net.sh`'s `PLANE_CIDRS` order**, which starts with provider-public. Anything walking `PLANE_CIDRS` positionally to place NICs swaps `enp1s0` and `enp2s0` and puts the PXE leg on provider-public -- stranding commissioning on a plane with no DHCP. `NIC_PLANE_ORDER` + `BREX_PARENT_NIC` are recorded with the evidence, and `dc-selector` (88 checks) asserts the two orders differ at position 0, so the guard cannot go vacuous. --- ## STILL OWED (explicit, so nothing is lost) 1. **The DHCP cutover** -- item 10, blocked on the permission wall. 2. **`scripts/dc-node-carve.sh` does not exist** -- 60 NIC re-homes + 9 `br-ex` bridges + 54 v4 statics, the largest remaining no-tool gap. The two hard inputs are now settled (item 14's ordering, and `carve-host-interfaces.sh:211-236`'s `build_ovs_brex()` call sequence, which the recon flagged as the correct MAAS ordering: unlink commissioning links -> move the parent NIC onto the provider VLAN -> `interfaces create-bridge bridge_type=ovs` -> `link-subnet mode=STATIC` on the bridge). **Deliberately NOT written blind:** it cannot be exercised until the nodes are re-enrolled, and both defects found today (the stranding bug, the fixture mismatch) were caught by LIVE runs, not by fixtures. Write it against a region it can actually run against. 3. **`maas-profile-assert.sh` is wired into `dc-region-topology.sh` only.** `dc-plane-ipam.sh`, `maas-role-tags.sh` and `maas-node-power.sh` still default to `MAAS_PROFILE=admin`. A forgotten `export` on any of them mutates OFFICE1 -- which still owns dc1's nine live nodes, so a mis-targeted `maas-role-tags.sh apply` would retag the wrong DC. Until they are wired, run the assert in the same command immediately before each and put it in the capture. 4. **The rack DNS forwarder's upstream** -- item 9's follow-up. 5. **Remove the `vr1-dc0-region` profile from voffice1** once dc0's control host settles (SEC-026), and **remove Office1's copy of the dc0 power key** once the migration lands. --- ## Item 11 -- as-executed log NOT used for this window, and why `scripts/run-logged.sh` opens an INTERACTIVE `script(1)` subshell and is unusable from a non-interactive agent session. F6 (2026-07-30) already records the harness classifier refusing the wrapped `script -aqe ... -c` form. Rather than produce a log that looks complete and is not, this session captures to `docs/audit/*` files -- which is what GA-R6 actually requires of a gate -- and says so here. A `logs/as-executed-index.md` row is owed at close. --- ## Item 15 -- DHCP HANDOVER EXECUTED, and the permission wall diagnosed **Executed** in the ruled order, region identity proven first (capture `docs/audit/dc0-dhcp-handover-20260730.txt`): | step | command | read-back | |---|---|---| | 1 | `maas admin vlan update 4 0 dhcp_on=false` | `dhcp_on=False` | | 2 | `pgrep -c dhcpd` on the dc0 rack | **0** | | 3 | `maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=c3aqh8` | `dhcp_on=True primary_rack=c3aqh8 fabric=fabric-0 space=metal-admin` | | 4 | `ps -ef` on `hot-kid`; rack re-checked | 2 dhcpd on the region VM; rack still **0** | Step 2 is the one that matters: MAAS's self-report is not evidence that dhcpd stopped (the 2026-07-20 Temporal incident class). Exactly ONE DHCP server is now on the segment, owned by the DC-local region. All 10 machines were `Ready` and POWERED OFF throughout. **Recorded so it is not later read as a fault:** the new region also starts `dhcpd6`, where the Office1 rack ran v4 only. Its v6 metal-admin subnet carries NO dynamic range -- only `rfc-4291-2.6.1` and `assigned-ip,reserved`, identical to the old region -- so it has nothing to offer and nodes keep their ruled MAAS statics. **THE WALL WAS A RULE THAT FAILED TO MATCH, NOT A CLASSIFIER FAULT** -- the standing lesson, restated with two new specifics. `.claude/settings.json` already had `Bash(maas admin * update*)` in **ask**, and `settings.local.json` already had `Bash(ssh voffice1 "maas admin subnet *)` and siblings. Neither matched, because: 1. those entries pin the **double-quote** form (`ssh voffice1 "maas ...`) while the commands used **single** quotes; and 2. they pin the **`maas admin`** profile, while the new region needs `maas vr1-dc0-region`. So the call fell through to the classifier, which blocked it as unmatched. Operator-approved rules were added to `.claude/settings.local.json` (gitignored -- committed team policy in `settings.json` is unchanged): profile-agnostic **ask** rules `Bash(ssh *'maas * update*)` plus the double-quote twin and the `ipranges` pair, and read-only **allow** rules for `pgrep` / `ps -ef` over ssh. The operator chose **ask** over **allow** deliberately, keeping hard rule 3 intact for every MAAS mutation. **A NOTE ON THE RULE I ALMOST WROTE:** the shape first proposed pinned `maas admin * update*`, mirroring the existing project rule. It would have matched step 1 and **silently failed to match step 3**, which runs against `maas vr1-dc0-region` -- leaving the cutover half done at exactly the two-servers boundary. Widened to `maas * update*` before use. An `ask` rule that is too broad only costs extra prompts; one that is too narrow costs a stall mid-cutover. **Revert.** `ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=false'` then `ssh voffice1 'maas admin vlan update 4 0 dhcp_on=true primary_rack=7chphy'`, verifying by process at each step. Remove the added rules from `.claude/settings.local.json`. --- ## Item 16 -- CANARY re-enrolment, and the ORDERING the migration actually needs **Canary chosen deliberately:** `xqqwdq` / moral-salmon = `vr1-dc0-storage-04`, a storage node -- the least central of the ten if anything went wrong. The dc1 standup set the precedent of proving enlistment with one machine before touching the rest. **MEASURED SEQUENCE, and it is NOT the obvious one.** A freshly-enlisted machine in a new region has NO power configuration, so MAAS cannot power-cycle it, so it CANNOT drive commissioning. Steps 1-5 below are each individually MEASURED, but see item 18: **the sequence has NOT yet produced a `Ready` machine, so it is not a closed invariant and may need a further step.** The order so far: 1. `machine delete ` on the OLD region (the libvirt domain is untouched -- all 12 dc0 domains stayed defined throughout). 2. **Power the VM on by hand** (`virsh -c qemu:///system start ` on the rack). This step exists only because the new region has no power driver for a machine it has never seen. Nothing else can start it. 3. It PXEs, leases from the new region's dynamic range, and SELF-ENLISTS. 4. **Set power config BEFORE expecting commissioning to progress.** 5. `machine abort` (-> `New`), then `machine commission`. MAAS now powers it on itself. **Skipping step 4 leaves the machine wedged in `Commissioning` producing ZERO events** -- which is exactly what happened here and looks alarming until the cause is named. The enlistment ephemeral powers the node off when it finishes; with no power driver MAAS can never bring it back, so it waits forever. `machine abort` is the documented way out. **The power command needs BOTH URIs, and `maas-node-power.sh` already supports it:** MAAS_PROFILE=vr1-dc0-region \ VIRSH_URI=qemu+ssh://jessea123@172.31.0.2/system \ # ENUMERATE from voffice1 (transit) bash scripts/maas-node-power.sh --commit \ qemu+ssh://jessea123@10.12.8.2/system vr1-dc0 # STORE what the REGION dials `VIRSH_URI` defaults to the power address but is env-overridable, which is precisely the separation this migration needs: voffice1 can reach the rack only over the transit /30, while the region reaches it only over metal-admin. Conflating them would either fail to enumerate or store an address the region cannot dial. **Result:** `[ok] civil-bug -> vr1-dc0-storage-04 (power state: off)` -- a REAL `query-power-state`, which proves the region can drive the rack's libvirt over metal-admin using the SEC-012 key installed in its snap (item 5). Power path verified end to end, not inferred. MAAS then powered the VM on for commissioning by itself (`vm=running`). **IDENTITY PRESERVED, verified by the stable key:** the re-enlisted machine carries boot MAC `52:54:00:2b:ed:ab`, matching `lib-hosts.sh HOST_BOOT_MAC[vr1-dc0-storage-04]` exactly. Its system_id changed (`xqqwdq` -> `6q4syf`) and its hostname changed (`moral-salmon` -> `civil-bug`), which is why the pre-delete snapshot (`docs/audit/dc0-premigration-machine-identity-20260730.txt`) records the BOOT MAC as the identity and treats system_id and hostname as disposable. **Revert.** Delete the record from the new region and re-enlist into Office1 by re-enabling its DHCP -- possible only while the domain and its pinned MACs survive, which they do. --- ## Item 17 -- DIAGNOSTIC BLIND SPOT: the per-node serial log stopped capturing on 2026-07-21 **Measured.** `/var/lib/libvirt/vr1/staging/vr1-dc0-storage-04-serial.log` has mtime **2026-07-21 08:54:43** -- nine days stale. Today's boot wrote NOTHING to it, although the domain XML configures both `` and `` with ``. **Why this matters more than it looks.** The file still CONTAINS a complete, plausible boot: kernel command line, cloud-init stages, "All scripts successfully ran", and a clean `reboot: Power down` at 138s. **I read that as today's boot and reported it as fact** before checking the mtime. It was the Stage-4 commissioning of 2026-07-21, betrayed only by its internal dates and by `DataSourceMAASLocal [http://10.12.8.2:5248/...]` -- the OLD region's metadata path. An append-mode log that has silently stopped appending is worse than an absent one: `tail` returns confident, well-formed, wrong evidence. **Same class as two other misreadings this session** (a `+time=3 +tries=1` DNS probe called a capability gap; a `head -5`-truncated listener list called a missing TFTP bind). All three were instrument errors that produced a clean-looking answer. The rule that catches this one: **check the mtime before trusting a log file as current.** **LOGGED, NOT ACTIONED (hard rule 1).** Not fixed here -- it is a diagnostic aid, not on the deploy path, and the 2026-07-22 dc1 build already recorded a related fault ("serial-log staging dir absent on dc1; queued to standup DoD"). Worth folding into that same DoD item: a serial log configured but not capturing is a standup defect, and it should be verified by mtime at each DC standup rather than assumed from the domain XML. --- ## Item 18 -- RESOLVED: commissioning passed but landed `New` ONCE; re-issuing reached `Ready` **OUTCOME: the canary reached `Ready` at 22:05:33** on the third commission -- `6q4syf`/`civil-bug`, `cpu=8 mem=24576` (exact D-121 Option C storage-node shape), `power=virsh/off`, all six pinned MACs present. Item 16's ordering is therefore a WORKING sequence. What follows is the diagnosis, kept because it refutes two plausible theories and because the intermediate `New` will recur on the other nine. **MEASURED, from the surface I initially skipped.** My first look at `node-script-results read` returned unparseable output and I moved on -- which is the "could not look is not nothing there" failure, committed against my own rule. Read properly, it is the only direct evidence and it is unambiguous: | result set | type | status | detail | |---|---|---|---| | id=2 | Commissioning | Aborted | the enlistment-time run I aborted deliberately | | id=3 | Commissioning | **Passed** | all 13 scripts exit 0; `30-maas-01-bmc-config` Skipped (normal for a VM) | | id=4 | Testing | **Passed** | `smartctl-validate` **Skipped** on both virtio disks | Hardware discovery populated correctly: `cpu_count=8`, `memory=24576`, storage present -- exactly the D-121 Option C storage-node shape. **So neither commissioning nor testing failed.** The machine transitioned to `New` at 21:53:08, the same second the testing set completed. **REFUTED:** the plausible theory that `smartctl-validate` against virtio disks was the fault. It did not fail -- it was Skipped, and its result set Passed. Had I re-run with `testing_scripts=none` as the first move, it would have "fixed" nothing and I would have learned nothing. **ROOT CAUSE: a commission issued immediately after an `abort` can land in `New` once.** Attempt 2 (issued seconds after my `abort`) passed every script and still went to `New`. Attempt 3, issued under IDENTICAL conditions from a settled `New`, reached `Ready` in ~3.5 minutes -- matching the dc1 standup timing exactly. It was a ONE-OFF. **The repeat was justified as an EXPERIMENT, not a blind retry.** It was the single move that distinguished "one-off" from "systematic", and those two have completely different next steps. Re-running the same thing is only defensible when the re-run is the discriminator; it was here, and it is not in general. **SECOND THEORY ALSO REFUTED, in parallel:** a config delta between the two independently configured regions. All 13 commissioning-relevant settings compare IDENTICAL -- `enlist_commissioning`, `commissioning_distro_series`, `default_osystem`, `default_distro_series`, `default_min_hwe_kernel`, `enable_third_party_drivers`, `default_storage_layout`, `node_timeout`, `enable_disk_erasing_on_release`, `max_node_commissioning_results`, `curtin_verbose`, `force_v1_network_yaml`, `enable_kernel_crash_dump`. **STANDING CONSEQUENCE for the remaining nine:** do not read a single `New` after a passing commission as a fault. Re-issue `machine commission` once. Only a SECOND `New` is evidence of a real problem. **Holding the other nine until the canary passed was the right call** -- it kept an undiagnosed fault from being multiplied by ten while the return path (re-enable Office1 DHCP, re-enlist) was still cheap. They are now unblocked. **THE CARVE GAP IS NOW CONCRETE.** At `Ready` the canary reads: `enp1s0` correctly on `fabric-0` (metal-admin -- it PXE'd there) with an AUTO link on `10.12.8.0/22`, and `enp2s0..enp6s0` each on a FRESH auto-created fabric (`fabric-7..11`) with `link_up` and no subnet. That is precisely the 60 NIC re-homes + 9 `br-ex` bridges + 54 statics task 7 owes, and there is now a stable `Ready` machine to build `dc-node-carve.sh` against. --- ## Item 19 -- THE NEW REGION HAD ZERO SSH KEYS (would have broken juju bootstrap) **MEASURED:** `maas vr1-dc0-region sshkeys read` -> **count 0**, while `maas admin sshkeys read` -> 1 (`SHA256:iUYex2kmvtdGTPiMKRZ2Nfywqa8pWacVV1GEEx6sscI`, `vr1-office1-svc`). A fresh `maas init` registers no keys, and nothing in the migration sequence adds them. **Why this matters far beyond convenience.** MAAS injects its registered SSH keys into every machine it deploys. With none registered, deployed nodes would come up with NO authorized key -- and `juju bootstrap` needs direct client->machine SSH. That is **exactly the failure signature that consumed three bootstrap attempts on 2026-07-30** ("juju then spent 15 minutes unable to SSH the machine"). Task 8 would have hit it again, and the symptom would have looked like a network fault rather than a missing key. **Found by accident, which is worth admitting:** I was probing the stalled enlistment nodes and noticed sshd was open but I had no way in. The gap was never on my checklist. It should be a standing DC-standup item -- **a new region's SSH key set is empty until someone fills it** -- and it belongs next to the power-key item (prerequisite 5) in the phase-2 runbook. **FIXED:** the SAME key Office1 carries was imported, deliberately rather than minting a new one -- `docs/CURRENT-STATE.md` records that deployed nodes carry the MAAS-registered `vr1-office1-svc` key, and the `vr1-dc0-juju` / node ssh aliases built earlier this session depend on that identity. Minting a per-DC node key is a defensible SEC-side improvement but would break the existing aliases and tooling, so it is LOGGED as a follow-up rather than done mid-migration. **Revert.** `maas vr1-dc0-region sshkeys delete `. --- ## Item 20 -- enlistment does NOT scale to nine concurrent nodes; batch it **Measured sequence.** The canary alone enlisted and commissioned in ~2 min. Nine started together produced: all 9 leasing as `maas-enlisting-node`, all 9 alive (ping + sshd open, CPU climbing), cloud-init output flowing until **22:12:46** -- and then nothing for 15 minutes while DHCP renewals continued. **Zero** machine records were created. **Everything downstream was RULED OUT by measurement, not assumed:** | suspect | measurement | verdict | |---|---|---| | DC edge egress | rack -> `security.ubuntu.com` 200 in **0.9s** | healthy | | MAAS proxy (the only path nodes have -- metal-admin carries NO gateway) | rack -> via `10.12.8.6:8000` 200 in **1.3s** | healthy | | region VM overload | load **0.29**, 5 GB free | idle | | nodes dead | ping OK, ssh port OPEN, CPU climbing | alive | **A NOTE ON MY OWN INSTRUMENT, because I got this wrong twice in five minutes.** I first called it stalled off a bad elapsed-time estimate (the last line was 90 seconds old, not minutes). Then, correcting too fast, I counted "104 maas-enlist lines in 3 minutes" and called it recovered -- but those were **DHCP lease renewals**, not cloud-init progress; filtering for `cloud-init` specifically returned **0**. The stall was real, my first two readings of it were not. Counting lines from a broad grep is not the same as measuring the thing you care about. **ACTION:** all nine stopped and restarted in batches of three. Not a fix for the root cause -- which is still uncharacterised beyond "concurrency" -- but the canary is direct evidence that low concurrency works, and the migration does not need the root cause to proceed. **LOGGED for the dc1 standup and for Roosevelt: do not power on a whole rack of nodes for enlistment at once.** --- ## Item 21 -- BATCHED RE-ENROLMENT WORKS, and it confirms the ordering was the cause **Batch of three (control-01/02/03): enlisted in 2.3 minutes** against a 15-minute stall with zero records for nine. Batching is the working procedure. **The more useful result: all three went straight to `Ready` with NO `New` detour.** The canary needed a second commission; these did not. The difference is that their power config was set BEFORE `commission` was issued, whereas the canary was commissioned first and had power set afterwards. So the canary's detour was **caused by the ordering, not by chance** -- which is a stronger and more useful conclusion than the earlier "one-off", and it supersedes it. **REVISED per-node procedure, now evidenced twice:** delete (old region) -> virsh start -> self-enlist -> SET POWER -> abort -> commission with the emphasis on power BEFORE commission. Issue `abort` only to clear the wedge left by the enlistment-time commissioning; once power is set first, `commission` completes in one pass (~3.5 min for three concurrent). **Standing guidance for dc1 and Roosevelt:** enlist in batches of three, and never issue `commission` on a machine whose `power_type` is still unset. --- ## Item 22 -- role tags: covered by existing tooling, ordering enforced by the tool itself The new region carries only `openstack-vr1-dc0` (created by `dc-region-topology.sh`) and MAAS's own `virtual`. The `control` / `compute` / `storage` / `juju-controller-vr1-dc0` tags do NOT exist yet -- but `scripts/maas-role-tags.sh` CREATES them (`ROLES` at `:38`, `ROLES+=("juju-controller-$SITE")` at `:125`, creation at `:184`), so task 7's tagging needs no new tool. **Its own refusal enforces the correct ordering:** it refuses if any pinned boot MAC has no MAAS record, so it cannot run until all ten machines are enlisted. That is the gate working in the right direction -- no additional sequencing discipline required from the operator.