# Changelog 2026-07-30 -- dc0 MAAS region migration (Stage 5 unblock) Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`. Status claims live ONLY in `docs/CURRENT-STATE.md`. Context: Stage 5 is OPEN and BLOCKED. D-132 q1 (RULED 2026-07-30) puts a MAAS region in each DC; the dc0 region VM is LIVE at `10.12.8.6:5240`, but the Office1 region still owns all 9 dc0 role nodes plus the Juju controller. MAAS cannot move machines between regions, so the carve must be RECREATED and the machines re-enrolled. This changelog covers that migration. **Operator authorization for this session, verbatim: "Begin working through all tasks and continue deployment. Work as autonomously as possible, deploy agents as needed."** Under blanket approval this changelog IS the review surface (GA-R2): every item states what, why (evidence), and how to revert. --- ## Item 1 -- `scripts/maas-profile-assert.sh` + `tests/maas-profile-assert/` (NEW) **What.** A guard that proves WHICH MAAS REGION a CLI profile resolves to, by rack-controller identity, before any command that reads or writes that region. Exit 0 match / 1 wrong region / 2 refuse. **Why (measured, not reasoned).** Every MAAS call in this repo defaults to `MAAS_PROFILE=admin` (13 scripts: `maas-fabric-prune`, `phase-00-teardown-destroy`, `pre-flight-checks`, `maas-role-tags`, `carve-host-interfaces`, `dc-plane-ipam`, `phase-00-teardown-release`, `lib-hosts`, `maas-node-power`, `phase-00-maas-standup`, `reenroll-hosts`, and two more). MEASURED on voffice1: the sole profile `admin` resolves to `http://10.10.0.20:5240/` -- the OFFICE1 region. The two regions hold SEPARATE databases, so during this migration: - a carve recreated against the wrong profile is an idempotent NO-OP that prints PASS (the state was already there -- in the wrong region); - `machine delete` against the wrong profile DESTROYS the real record, unrecoverably. This is the wrong-HOST class this repo has been bitten by three times already (`dc-mirror.sh` run on voffice1; the egress probe run from the rack; P5 probing the wrong filesystem) -- now wrong-REGION, where the destructive direction has no undo. **A count alone is not proof** -- two regions can hold the same number of machines. Rack-controller identity is the discriminator, because a rack registers to exactly one region. **Live verification, both directions** (capture `docs/audit/dc0-region-profile-assert-20260730.txt`): admin -> machines=23 racks=voffice1,vvr1-dc0,vvr1-dc1 (Office1) vr1-dc0-region -> machines=0 racks=hot-kid (new dc0 region) $ maas-profile-assert.sh vr1-dc0-region hot-kid exit=0 $ maas-profile-assert.sh admin voffice1,vvr1-dc0,vvr1-dc1 exit=0 $ maas-profile-assert.sh admin hot-kid NEGATIVE -> exit=1 **Harness 20/20, and the MUTATION PASS FOUND A REAL DEFECT IN MY OWN TEST.** Three mutations were applied to the script and the suite re-run: - A (comparison always true) -- killed 4 tests. Good. - B (empty rack list accepted) -- killed 4 tests. Good. - C (nameless rack entries accepted) -- **suite stayed fully GREEN (19/19)**. The assertion was decoration: a nameless entry fell through to the generic empty-`ACTUAL` guard and refused for a different reason, so deleting the branch it claimed to test changed nothing. Fixed by giving the nameless case its own exit code and refusal message and asserting on THAT; mutation C now kills a test. This is exactly the "prove each new assertion can FAIL" rule, and it caught me writing decoration. **Revert.** `git rm scripts/maas-profile-assert.sh tests/maas-profile-assert/ && git checkout ^ -- .` Nothing else depends on it yet; it is additive. --- ## Item 2 -- named MAAS profile `vr1-dc0-region` on voffice1 (LIVE, config) **What.** `maas login vr1-dc0-region http://127.0.0.1:5241/MAAS/ -`, key streamed from `~/vr1-dc0-creds/maas-region-api-key.txt` on vcloud via stdin, never printed. The existing `admin` profile is UNTOUCHED and still points at Office1. **Why the tunnel.** Neither candidate host has everything the migration needs: | host | repo | maas CLI | reaches new region | reaches NetBox apex | |---|---|---|---|---| | voffice1 | yes | yes | **NO** (curl exit 28) | yes | | dc0 rack | no | yes | yes (200) | **NO** (10.10.1.10:8000 CLOSED) | | region VM | no | yes | yes | not measured | `dc-plane-ipam.sh` derives the v6 plane prefixes from the NetBox apex (D-136 option D), so it must run where NetBox is reachable. An SSH tunnel from voffice1 through the rack (`-L 127.0.0.1:5241:10.12.8.6:5240`) gives voffice1 both. Measured: `localhost:5241` answers **200**. **This does not puncture SEC-010.** SEC-010 forbids the rack FORWARDING between its transit leg and the plane nets. The tunnel is an application-layer, rack-ORIGINATED connection -- exactly the distinction SEC-010's own justification draws ("a MAAS rack proxies at the application layer and needs no kernel forwarding"). No route, no nftables rule, and no libvirt net was changed. The rack's `table inet sec010` drops are intact. **Safe failure direction, verified by construction:** if the tunnel dies, the `vr1-dc0-region` profile becomes unreachable and `maas-profile-assert.sh` REFUSES (exit 2). It cannot silently fall back to Office1, because the fallback would have to be a different URL. **CREDENTIAL RESIDENCY -- SEC-026 addendum, flagged not glossed.** This puts dc0's region admin API key on voffice1, where it was not before. Today that is a strict SUBSET of the blast radius already present: voffice1 holds the Office1 region admin key, which currently administers BOTH DCs' nodes. **After the migration that inverts** -- Office1's key stops covering dc0, and dc0's region key becomes the broader credential on a shared headend. **Obligation recorded: remove the `vr1-dc0-region` profile from voffice1 once dc0's control host is settled.** A security-ledger row update is owed at session close. **Revert.** On voffice1: `maas logout vr1-dc0-region`, and kill the tunnel (`pkill -f '5241:10.12.8.6'`). Both are local to voffice1; no cloud state is touched. --- ## Item 3 -- OPERATING NOTE: MAAS CLI for a per-DC region cannot run from voffice1 **Measured.** voffice1 -> `10.12.8.6:5240` times out (curl exit 28); voffice1 -> `10.10.0.20:5240` is fine. The dc0 rack reaches BOTH (200 / 200). **Consequence.** D-128's "Plane 2 (MAAS/NetBox/inner tofu) executes on voffice1" clause was written when there was ONE region, at Office1. D-132 q1 makes the MAAS control surface DC-local, so for the per-DC regions that clause needs either a tunnel (what this session does) or a DC-side host. **This is a consequence of an already-ruled decision, not a new one** -- recorded here, and owed as a DOCFIX against D-128's Plane-2 wording and the phase-4 runbook's run-location banner (which F5 already flags as wrong for the juju half after D-138). **No revert** -- this item is a record, not a change. --- ## Item 4 -- `lib-hosts.sh`: the power address is now per-REGION (both DCs) **What.** Each VR1 arm gains `VIRSH_POWER_ADDRESS_FROM_OFFICE1` (the rack's transit leg) and `VIRSH_POWER_ADDRESS_FROM_DCREGION` (the rack's metal-admin leg). `VIRSH_POWER_ADDRESS` is now an ALIAS of the Office1 form, byte-for-byte unchanged. The flat/VR0 layer declares both as EMPTY. **Why (measured).** MAAS power ops originate from the REGION, so the reachable rack address depends on which region dials. From `vr1-dc0-maas-01` (10.12.8.6): 172.31.0.2:22 CLOSED <- the transit address lib-hosts has always used 10.12.8.2:22 OPEN <- the rack's metal-admin leg routes: 10.12.4.0/22, 10.12.8.0/22, default via 10.12.4.1 dc1's `10.12.68.2` was MEASURED on the dc1 rack (`ip -4 -o addr` -> `virbr6 inet 10.12.68.2/22`) and corroborated against `dc-rack-net.sh`'s dc1 `LEGS` block. It was NOT inferred from dc0's `.2` by symmetry. **Backward compatibility is the load-bearing half.** `reenroll-hosts.sh:132` and the teardown runbook's `virsh -c "$VIRSH_POWER_ADDRESS"` probes must not silently change target while Office1 still owns dc1's nodes. Flipping the default to the DC-region form is OWED once BOTH DCs have migrated. **Harness `tests/dc-selector` 81 checks ALL PASS, 4 mutations each killed tests:** | mutation | tests killed | |---|---| | A: collapse dc0's DCREGION onto the transit form | 2 | | B: dc1's DCREGION gets dc0's octet (cross-DC contamination) | 2 | | C: `VIRSH_POWER_ADDRESS` silently repointed to the DC-region form | 3 | | D: VR0 inherits a VR1 form | 1 | **Revert.** `git checkout ^ -- scripts/lib-hosts.sh tests/dc-selector/run-tests.sh`. --- ## Item 5 -- `scripts/maas-region-power-key.sh` + harness (NEW), and INSTALLED at dc0 **What.** `check|install `. Installs the per-DC MAAS->libvirt power key into a region's snap and verifies it with a REAL virsh call. Key is read from STDIN, so it never lands on an intermediate host and never appears in a process argument. **Why.** `runbooks/dc-dc-phase2-tofu-dc-substrate.md` prerequisite 5 carries this as PROSE, and prose is not a gate. It must be re-run at every DC standup and -- per that same prerequisite's own FRAGILITY note -- after EVERY MAAS snap refresh, because the key lives under per-revision `/var/snap/maas/current/`. MEASURED before this work: `/var/snap/maas/current/root/.ssh/` did not exist on the new region at all, so commissioning could not have powered a node on. **LIVE at dc0** (capture `docs/audit/dc0-region-power-key-20260730.txt`), 9/9 exit 0, ending on the artifact rather than the config: [ok] key mode 0600 / owned by root / parses as a valid PRIVATE key (SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY) [ok] config binds Host 10.12.8.2 -> id_dc0_power, IdentitiesOnly yes [ok] REAL virsh connect to qemu+ssh://jessea123@10.12.8.2/system succeeds [ok] virsh enumerates domains on the rack (got 12) The key is the SAME dc0-scoped key already authorized on the dc0 rack (fingerprint matched against the rack's `authorized_keys`), so SEC-012's per-DC isolation holds -- this is not a cross-DC reuse. **OWED: remove Office1's copy once dc0's migration completes**, since that region will no longer power dc0 machines. **THREE DEFECTS THIS SCRIPT'S OWN TESTING FOUND, all measured, none by review:** 1. **`ssh-keygen -lf ` reads an ADJACENT `.pub` when one exists** and reports ITS fingerprint. A stale sibling made a freshly-installed key read back as a different one, which would have made the idempotency and cross-DC-swap refusals answer about the wrong key. Fixed with `ssh-keygen -y -f` (derive the public half FROM the private key), which no neighbouring file can fool. **Found by the harness.** 2. **A snap has TWO roots and they are not interchangeable.** `/var/snap//current` is writable data (where `root/.ssh` lives); `/snap//current` is the read-only squashfs (where the binaries live). The first live run REFUSED with "no virsh binary found" while `find /snap/maas/current -name virsh` had already located one. 3. **The snap's binary must be run INSIDE the snap's runtime.** Invoking `/snap/maas/current/usr/bin/virsh` directly fails with `error while loading shared libraries: libvirt-lxc.so.0`. `snap run --shell maas -c 'virsh ...'` executes it with the snap's environment AND its ssh config -- which is exactly the context MAAS's own power driver uses, so this is the correct artifact to grade, not a proxy. **Harness 44/44.** Assertions include every refusal direction (absent key, wrong-host binding, missing `IdentitiesOnly`, unreachable virsh, virsh listing ZERO domains) and the cross-DC-swap refusal, verified not to overwrite. Two harness assertions were found passing FOR THE WRONG REASON during development and were made specific. **A NOTE ON MY OWN INSTRUMENT.** The first live install reported `EXIT=0` while the check had actually returned 2 -- I had read `$?` from the end of a pipeline (`... | tail`), so I measured `tail`'s exit, not the script's. **That is the identical class this repo recorded twice on 2026-07-30** (`| grep -q` under `pipefail`). Fixed by capturing into a variable and reading `$?` directly. **Revert.** `git rm scripts/maas-region-power-key.sh tests/maas-region-power-key/`. To undo the live install: on the region VM, `sudo rm /var/snap/maas/current/root/.ssh/id_dc0_power` and delete the `Host 10.12.8.2` stanza from that dir's `config`. --- ## Item 6 -- RECON: what actually builds the dc0 carve (read-only; changes nothing) A read-only survey mapped each of the seven carve components to its creating tool. **Four findings change the migration plan; all four were verified against the files before being recorded here.** **(a) THE PRE-MIGRATION CAPTURE IS NOT A DIFF TARGET, and I had planned to use it as one.** Its `vlan_id=5189..5193` / `id=6,7,12..15,20..25` / `spaces id=1..6` values are MAAS **database row ids, not 802.1Q tags** -- every named fabric line reads `vid=0`. Row ids are region-local and will differ on a rebuild. Its RESERVED-RANGES section is `subnet reserved-ip-ranges` output, which COALESCES and is state-dependent (metal-internal et al. show a merged `.4-.99 reserved`, while provider-public splits `.4-.49`/`.50-.99` because `.4-.49` there also carries `assigned-ip` from live statics). **A textual diff of a perfectly rebuilt region against this file would NOT match.** The only stable identities are fabric NAME, space NAME, subnet CIDR, tag NAME, node boot MAC, interface NAME -- which is what `scripts/lib-net.sh:8-10` already says repo-wide ("CIDR is the stable key throughout; MAAS subnet IDs drift across cutovers"). The real gate is the `check` actions (`dc-plane-ipam.sh check`, `dc-node-v6-carve.py check`, `maas-role-tags.sh check`), not a file diff. My task-7 plan is corrected accordingly. **(b) `scripts/dc-node-v6-carve.py` IS PROFILE-ENV-BLIND -- verified at the file.** `PROFILE_DEFAULT = "admin"` (`:39`) and `--profile` (`:70`), with NO environment read anywhere. So `export MAAS_PROFILE=vr1-dc0-region` -- the idiom every bash tool here honours -- **silently targets Office1**. Combined with this changelog's own item 1 reasoning (a carve run against the wrong profile is an idempotent no-op that prints PASS), this is a live foot-gun on exactly the tool the migration needs. It must be invoked with an explicit `--profile`, or fixed to read the env. **(c) THREE COMPONENTS HAVE NO REPO TOOL AT ALL.** The named plane fabrics, the six v4 plane subnets, and the per-node v4 NIC carve (60 NIC re-homes + 9 `br-ex` bridges + 54 statics) were all created ad-hoc during the Stage-4 window; the mutations went to `~/as-executed/2026-07-23-stage4-carve.log`, which is NOT in the repo. The `openstack-vr1-dc0` tag has no usable creator either. So the "re-run the generator against the new endpoint" plan is only available for roughly half the work. **(d) TWO EXISTING SCRIPTS MUST NOT BE POINTED AT THE NEW REGION, both verified:** - `scripts/carve-host-interfaces.sh:62-64` REFUSES `vr1-*` by design (it carves the VR0 Pattern-A VLAN-103 stack, superseded for VR1 by D-133). `scripts/reenroll-hosts.sh:53-55` refuses `vr1-*` likewise. - `scripts/phase-00-maas-standup.sh` does NOT refuse, and that is the danger: its parity guard PASSES for `vr1-dc0`, then it creates fabrics named after the SPACE (`provider-public`, not `vr1-dc0-provider-public`, `:192-194`), creates a named `metal-admin` fabric the as-built does not have, and builds metal-internal as a tagged VID-103 VLAN riding metal-admin's fabric -- which D-133 abolished for VR1. It would silently produce a divergent region. **Also recorded:** `runbooks/phase-01-bundle-deploy.md:128` iterates **hardcoded subnet ids** (`for ID in 1 2 6 7 8 9`) and is profile-blind -- in a file that warns against exactly this at `:54` and `:287-291`. `scripts/site-headend-install.sh:528` picks a rack by list INDEX (`[0]['system_id']`), which is meaningless across regions. Both are pre-existing and LOGGED, NOT EXECUTED (hard rule 1). **No revert** -- this item is a record, not a change. --- ## Item 7 -- harness manifest re-recorded (91 harnesses) `scripts/run-tests-all.sh` FAILED with `harness set DRIFTED from tests/HARNESS-MANIFEST`, listing `maas-profile-assert` and `maas-region-power-key` as present-but-unpinned. That is the manifest gate working as designed on two harnesses added this session, not a regression. Re-recorded with `--record-manifest`; 35 -> 37 entries in the diff, 91 total. **Revert.** `git checkout ^ -- tests/HARNESS-MANIFEST`. --- ## Item 8 -- as-executed log NOT used for this window, and why `scripts/run-logged.sh` opens an INTERACTIVE `script(1)` subshell and is unusable from a non-interactive agent session. F6 (2026-07-30) already records the harness classifier refusing the wrapped `script -aqe ... -c` form. Rather than produce a log that looks complete and is not, this session captures to `docs/audit/*` files -- which is what GA-R6 actually requires of a gate -- and says so here. A `logs/as-executed-index.md` row is owed at close.