Changelog 2026-07-30 -- dc0 MAAS region migration (Stage 5 unblock)

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Context: Stage 5 is OPEN and BLOCKED. D-132 q1 (RULED 2026-07-30) puts a MAAS region in each DC; the dc0 region VM is LIVE at 10.12.8.6:5240, but the Office1 region still owns all 9 dc0 role nodes plus the Juju controller. MAAS cannot move machines between regions, so the carve must be RECREATED and the machines re-enrolled. This changelog covers that migration.

Operator authorization for this session, verbatim: "Begin working through all tasks and continue deployment. Work as autonomously as possible, deploy agents as needed." Under blanket approval this changelog IS the review surface (GA-R2): every item states what, why (evidence), and how to revert.

Item 1 -- scripts/maas-profile-assert.sh + tests/maas-profile-assert/ (NEW)

What. A guard that proves WHICH MAAS REGION a CLI profile resolves to, by rack-controller identity, before any command that reads or writes that region. Exit 0 match / 1 wrong region / 2 refuse.

Why (measured, not reasoned). Every MAAS call in this repo defaults to MAAS_PROFILE=admin (13 scripts: maas-fabric-prune, phase-00-teardown-destroy, pre-flight-checks, maas-role-tags, carve-host-interfaces, dc-plane-ipam, phase-00-teardown-release, lib-hosts, maas-node-power, phase-00-maas-standup, reenroll-hosts, and two more). MEASURED on voffice1: the sole profile admin resolves to http://10.10.0.20:5240/ -- the OFFICE1 region. The two regions hold SEPARATE databases, so during this migration:

• a carve recreated against the wrong profile is an idempotent NO-OP that prints PASS (the state was already there -- in the wrong region);
• machine delete against the wrong profile DESTROYS the real record, unrecoverably.

This is the wrong-HOST class this repo has been bitten by three times already (dc-mirror.sh run on voffice1; the egress probe run from the rack; P5 probing the wrong filesystem) -- now wrong-REGION, where the destructive direction has no undo.

A count alone is not proof -- two regions can hold the same number of machines. Rack-controller identity is the discriminator, because a rack registers to exactly one region.

Live verification, both directions (capture docs/audit/dc0-region-profile-assert-20260730.txt):

admin           -> machines=23 racks=voffice1,vvr1-dc0,vvr1-dc1   (Office1)
vr1-dc0-region  -> machines=0  racks=hot-kid                      (new dc0 region)

$ maas-profile-assert.sh vr1-dc0-region hot-kid                    exit=0
$ maas-profile-assert.sh admin voffice1,vvr1-dc0,vvr1-dc1          exit=0
$ maas-profile-assert.sh admin hot-kid            NEGATIVE ->      exit=1

Harness 20/20, and the MUTATION PASS FOUND A REAL DEFECT IN MY OWN TEST. Three mutations were applied to the script and the suite re-run:

• A (comparison always true) -- killed 4 tests. Good.
• B (empty rack list accepted) -- killed 4 tests. Good.
• C (nameless rack entries accepted) -- suite stayed fully GREEN (19/19). The assertion was decoration: a nameless entry fell through to the generic empty-ACTUAL guard and refused for a different reason, so deleting the branch it claimed to test changed nothing. Fixed by giving the nameless case its own exit code and refusal message and asserting on THAT; mutation C now kills a test. This is exactly the "prove each new assertion can FAIL" rule, and it caught me writing decoration.

Revert. git rm scripts/maas-profile-assert.sh tests/maas-profile-assert/ && git checkout <this-commit>^ -- . Nothing else depends on it yet; it is additive.

Item 2 -- named MAAS profile vr1-dc0-region on voffice1 (LIVE, config)

What. maas login vr1-dc0-region http://127.0.0.1:5241/MAAS/ -, key streamed from ~/vr1-dc0-creds/maas-region-api-key.txt on vcloud via stdin, never printed. The existing admin profile is UNTOUCHED and still points at Office1.

Why the tunnel. Neither candidate host has everything the migration needs:

dc-plane-ipam.sh derives the v6 plane prefixes from the NetBox apex (D-136 option D), so it must run where NetBox is reachable. An SSH tunnel from voffice1 through the rack (-L 127.0.0.1:5241:10.12.8.6:5240) gives voffice1 both. Measured: localhost:5241 answers 200.

This does not puncture SEC-010. SEC-010 forbids the rack FORWARDING between its transit leg and the plane nets. The tunnel is an application-layer, rack-ORIGINATED connection -- exactly the distinction SEC-010's own justification draws ("a MAAS rack proxies at the application layer and needs no kernel forwarding"). No route, no nftables rule, and no libvirt net was changed. The rack's table inet sec010 drops are intact.

Safe failure direction, verified by construction: if the tunnel dies, the vr1-dc0-region profile becomes unreachable and maas-profile-assert.sh REFUSES (exit 2). It cannot silently fall back to Office1, because the fallback would have to be a different URL.

CREDENTIAL RESIDENCY -- SEC-026 addendum, flagged not glossed. This puts dc0's region admin API key on voffice1, where it was not before. Today that is a strict SUBSET of the blast radius already present: voffice1 holds the Office1 region admin key, which currently administers BOTH DCs' nodes. After the migration that inverts -- Office1's key stops covering dc0, and dc0's region key becomes the broader credential on a shared headend. Obligation recorded: remove the vr1-dc0-region profile from voffice1 once dc0's control host is settled. A security-ledger row update is owed at session close.

Revert. On voffice1: maas logout vr1-dc0-region, and kill the tunnel (pkill -f '5241:10.12.8.6'). Both are local to voffice1; no cloud state is touched.

Item 3 -- OPERATING NOTE: MAAS CLI for a per-DC region cannot run from voffice1

Measured. voffice1 -> 10.12.8.6:5240 times out (curl exit 28); voffice1 -> 10.10.0.20:5240 is fine. The dc0 rack reaches BOTH (200 / 200).

Consequence. D-128's "Plane 2 (MAAS/NetBox/inner tofu) executes on voffice1" clause was written when there was ONE region, at Office1. D-132 q1 makes the MAAS control surface DC-local, so for the per-DC regions that clause needs either a tunnel (what this session does) or a DC-side host. This is a consequence of an already-ruled decision, not a new one -- recorded here, and owed as a DOCFIX against D-128's Plane-2 wording and the phase-4 runbook's run-location banner (which F5 already flags as wrong for the juju half after D-138).

No revert -- this item is a record, not a change.

Item 4 -- lib-hosts.sh: the power address is now per-REGION (both DCs)

What. Each VR1 arm gains VIRSH_POWER_ADDRESS_FROM_OFFICE1 (the rack's transit leg) and VIRSH_POWER_ADDRESS_FROM_DCREGION (the rack's metal-admin leg). VIRSH_POWER_ADDRESS is now an ALIAS of the Office1 form, byte-for-byte unchanged. The flat/VR0 layer declares both as EMPTY.

Why (measured). MAAS power ops originate from the REGION, so the reachable rack address depends on which region dials. From vr1-dc0-maas-01 (10.12.8.6):

172.31.0.2:22  CLOSED        <- the transit address lib-hosts has always used
10.12.8.2:22   OPEN          <- the rack's metal-admin leg
routes: 10.12.4.0/22, 10.12.8.0/22, default via 10.12.4.1

dc1's 10.12.68.2 was MEASURED on the dc1 rack (ip -4 -o addr -> virbr6 inet 10.12.68.2/22) and corroborated against dc-rack-net.sh's dc1 LEGS block. It was NOT inferred from dc0's .2 by symmetry.

Backward compatibility is the load-bearing half. reenroll-hosts.sh:132 and the teardown runbook's virsh -c "$VIRSH_POWER_ADDRESS" probes must not silently change target while Office1 still owns dc1's nodes. Flipping the default to the DC-region form is OWED once BOTH DCs have migrated.

Harness tests/dc-selector 81 checks ALL PASS, 4 mutations each killed tests:

Revert. git checkout <this-commit>^ -- scripts/lib-hosts.sh tests/dc-selector/run-tests.sh.

Item 5 -- scripts/maas-region-power-key.sh + harness (NEW), and INSTALLED at dc0

What. check|install <site> <rack-virsh-uri>. Installs the per-DC MAAS->libvirt power key into a region's snap and verifies it with a REAL virsh call. Key is read from STDIN, so it never lands on an intermediate host and never appears in a process argument.

Why. runbooks/dc-dc-phase2-tofu-dc-substrate.md prerequisite 5 carries this as PROSE, and prose is not a gate. It must be re-run at every DC standup and -- per that same prerequisite's own FRAGILITY note -- after EVERY MAAS snap refresh, because the key lives under per-revision /var/snap/maas/current/. MEASURED before this work: /var/snap/maas/current/root/.ssh/ did not exist on the new region at all, so commissioning could not have powered a node on.

LIVE at dc0 (capture docs/audit/dc0-region-power-key-20260730.txt), 9/9 exit 0, ending on the artifact rather than the config:

[ok] key mode 0600 / owned by root / parses as a valid PRIVATE key
     (SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY)
[ok] config binds Host 10.12.8.2 -> id_dc0_power, IdentitiesOnly yes
[ok] REAL virsh connect to qemu+ssh://jessea123@10.12.8.2/system succeeds
[ok] virsh enumerates domains on the rack (got 12)

The key is the SAME dc0-scoped key already authorized on the dc0 rack (fingerprint matched against the rack's authorized_keys), so SEC-012's per-DC isolation holds -- this is not a cross-DC reuse. OWED: remove Office1's copy once dc0's migration completes, since that region will no longer power dc0 machines.

THREE DEFECTS THIS SCRIPT'S OWN TESTING FOUND, all measured, none by review:

1. ssh-keygen -lf <privkey> reads an ADJACENT .pub when one exists and reports ITS fingerprint. A stale sibling made a freshly-installed key read back as a different one, which would have made the idempotency and cross-DC-swap refusals answer about the wrong key. Fixed with ssh-keygen -y -f (derive the public half FROM the private key), which no neighbouring file can fool. Found by the harness.
2. A snap has TWO roots and they are not interchangeable. /var/snap/<n>/current is writable data (where root/.ssh lives); /snap/<n>/current is the read-only squashfs (where the binaries live). The first live run REFUSED with "no virsh binary found" while find /snap/maas/current -name virsh had already located one.
3. The snap's binary must be run INSIDE the snap's runtime. Invoking /snap/maas/current/usr/bin/virsh directly fails with error while loading shared libraries: libvirt-lxc.so.0. snap run --shell maas -c 'virsh ...' executes it with the snap's environment AND its ssh config -- which is exactly the context MAAS's own power driver uses, so this is the correct artifact to grade, not a proxy.

Harness 44/44. Assertions include every refusal direction (absent key, wrong-host binding, missing IdentitiesOnly, unreachable virsh, virsh listing ZERO domains) and the cross-DC-swap refusal, verified not to overwrite. Two harness assertions were found passing FOR THE WRONG REASON during development and were made specific.

A NOTE ON MY OWN INSTRUMENT. The first live install reported EXIT=0 while the check had actually returned 2 -- I had read $? from the end of a pipeline (... | tail), so I measured tail's exit, not the script's. That is the identical class this repo recorded twice on 2026-07-30 (| grep -q under pipefail). Fixed by capturing into a variable and reading $? directly.

Revert. git rm scripts/maas-region-power-key.sh tests/maas-region-power-key/. To undo the live install: on the region VM, sudo rm /var/snap/maas/current/root/.ssh/id_dc0_power and delete the Host 10.12.8.2 stanza from that dir's config.

Item 6 -- RECON: what actually builds the dc0 carve (read-only; changes nothing)

A read-only survey mapped each of the seven carve components to its creating tool. Four findings change the migration plan; all four were verified against the files before being recorded here.

(a) THE PRE-MIGRATION CAPTURE IS NOT A DIFF TARGET, and I had planned to use it as one. Its vlan_id=5189..5193 / id=6,7,12..15,20..25 / spaces id=1..6 values are MAAS database row ids, not 802.1Q tags -- every named fabric line reads vid=0. Row ids are region-local and will differ on a rebuild. Its RESERVED-RANGES section is subnet reserved-ip-ranges output, which COALESCES and is state-dependent (metal-internal et al. show a merged .4-.99 reserved, while provider-public splits .4-.49/.50-.99 because .4-.49 there also carries assigned-ip from live statics). A textual diff of a perfectly rebuilt region against this file would NOT match. The only stable identities are fabric NAME, space NAME, subnet CIDR, tag NAME, node boot MAC, interface NAME -- which is what scripts/lib-net.sh:8-10 already says repo-wide ("CIDR is the stable key throughout; MAAS subnet IDs drift across cutovers"). The real gate is the check actions (dc-plane-ipam.sh check, dc-node-v6-carve.py check, maas-role-tags.sh check), not a file diff. My task-7 plan is corrected accordingly.

(b) scripts/dc-node-v6-carve.py IS PROFILE-ENV-BLIND -- verified at the file. PROFILE_DEFAULT = "admin" (:39) and --profile (:70), with NO environment read anywhere. So export MAAS_PROFILE=vr1-dc0-region -- the idiom every bash tool here honours -- silently targets Office1. Combined with this changelog's own item 1 reasoning (a carve run against the wrong profile is an idempotent no-op that prints PASS), this is a live foot-gun on exactly the tool the migration needs. It must be invoked with an explicit --profile, or fixed to read the env.

(c) THREE COMPONENTS HAVE NO REPO TOOL AT ALL. The named plane fabrics, the six v4 plane subnets, and the per-node v4 NIC carve (60 NIC re-homes + 9 br-ex bridges + 54 statics) were all created ad-hoc during the Stage-4 window; the mutations went to ~/as-executed/2026-07-23-stage4-carve.log, which is NOT in the repo. The openstack-vr1-dc0 tag has no usable creator either. So the "re-run the generator against the new endpoint" plan is only available for roughly half the work.

(d) TWO EXISTING SCRIPTS MUST NOT BE POINTED AT THE NEW REGION, both verified:

• scripts/carve-host-interfaces.sh:62-64 REFUSES vr1-* by design (it carves the VR0 Pattern-A VLAN-103 stack, superseded for VR1 by D-133). scripts/reenroll-hosts.sh:53-55 refuses vr1-* likewise.
• scripts/phase-00-maas-standup.sh does NOT refuse, and that is the danger: its parity guard PASSES for vr1-dc0, then it creates fabrics named after the SPACE (provider-public, not vr1-dc0-provider-public, :192-194), creates a named metal-admin fabric the as-built does not have, and builds metal-internal as a tagged VID-103 VLAN riding metal-admin's fabric -- which D-133 abolished for VR1. It would silently produce a divergent region.

Also recorded: runbooks/phase-01-bundle-deploy.md:128 iterates hardcoded subnet ids (for ID in 1 2 6 7 8 9) and is profile-blind -- in a file that warns against exactly this at :54 and :287-291. scripts/site-headend-install.sh:528 picks a rack by list INDEX ([0]['system_id']), which is meaningless across regions. Both are pre-existing and LOGGED, NOT EXECUTED (hard rule 1).

No revert -- this item is a record, not a change.

Item 7 -- harness manifest re-recorded (91 harnesses)

scripts/run-tests-all.sh FAILED with harness set DRIFTED from tests/HARNESS-MANIFEST, listing maas-profile-assert and maas-region-power-key as present-but-unpinned. That is the manifest gate working as designed on two harnesses added this session, not a regression. Re-recorded with --record-manifest; 35 -> 37 entries in the diff, 91 total.

Revert. git checkout <this-commit>^ -- tests/HARNESS-MANIFEST.

Item 8 -- scripts/dc-region-topology.sh + harness (NEW), and APPLIED at dc0

What. check|apply <site> --profile <p> --expect-rack <r> [--commit] over the five named plane fabrics, the six spaces, the six v4 plane subnets on the right VLANs with the ruled gateways, the VLAN->space bindings, and the openstack-<site> tag. This is the tool item 6(c) found missing.

Design decisions worth stating:

• --profile has NO default. The repo-wide default admin resolves to the Office1 region, where this apply would be an idempotent no-op that prints PASS. Every mutating path is gated by maas-profile-assert.sh FIRST.
• metal-admin deliberately keeps MAAS's own auto-created fabric. It is a discovery artifact created when the rack registers, before anything in this repo could name it; Juju binds SPACES, not fabrics. The check asserts only that it does not share a fabric with a named plane (which would collapse two L2 domains).
• No numeric MAAS id is persisted. Everything resolves per-run by name or CIDR.

LIVE at dc0 (capture docs/audit/dc0-region-topology-20260730.txt). The apply created 5 fabrics, 6 spaces, 4 subnets, MOVED 10.12.4.0/22 off the auto fabric-1 onto vr1-dc0-provider-public, bound 6 VLANs to their spaces, and created the site tag. Final named check: 39 passed, 0 failed, exit 0.

A CONCRETE ILLUSTRATION OF WHY ROW IDS ARE NOT PORTABLE, measured in passing: VLAN row id 5005 is dc0's metal-admin VLAN in the Office1 region and vr1-dc0-data-tenant in the new region. Same integer, different object, different DC plane. Anything that carried 5005 across would have mis-targeted silently.

Harness 39/39; 4 mutations each killed tests (region gate removed; profile defaulted to admin; space-binding assertion neutered; fabric-placement assertion neutered). Fixtures were corrected against a live read: MAAS reports an unbound VLAN as the STRING "undefined", not null, so the first fixtures graded a payload MAAS never emits.

Revert. The apply is additive. To undo: delete the five vr1-dc0-* fabrics and the six spaces on the new region, and move 10.12.4.0/22 back to fabric-1. Nothing in the Office1 region was touched by this item.

Item 9 -- metal-admin service config set on the new region (LIVE)

dns_servers=10.12.8.6, allow_dns=false, and the D-134 dynamic range 10.12.8.201-10.12.8.254. Subnet resolved BY CIDR (-> id 1), never from the capture.

The DNS target was MEASURED, not carried over, and my first reading of it was WRONG. The old region set dns_servers=10.12.8.3 (the D-131 rack forwarder). A first probe suggested the new region's own BIND at 10.12.8.6 could not resolve external names, which would have forced keeping the forwarder. That was a cold-cache timeout at +time=3 +tries=1, not a capability gap -- re-probed with +time=5 +tries=3, 10.12.8.6 answers archive.ubuntu.com, streams.canonical.com and api.snapcraft.io correctly, and a full dig shows flags: qr rd ra with ANSWER: 9. I nearly recorded a false finding on one under-timed query. Pointing node DNS at the DC-LOCAL region is the correct end state: it is authoritative for the zone the migrated nodes will live in (its own maas-internal, SOA serial 16) and removes the cross-fiber dependency that D-132 q1 exists to remove.

FOLLOW-UP RECORDED, NOT ACTIONED (hard rule 1): the live rack forwarder is /etc/dnsmasq-dc0-node.conf with no-resolv + server=10.10.0.20 -- it sends EVERYTHING to the Office1 region. After dc0's nodes migrate, that upstream points at a region that no longer owns them. dc-rack-net.sh carries DNS_UPSTREAM="10.10.0.20" for BOTH sites (:66, :82), and dc1's is still correct, so this is a per-site change at each DC's cutover. Also noted: the live unit is named with the BARE dc0 token (dc0-node-dns.service, /etc/dnsmasq-dc0-node.conf) while the current script generates vr1-dc0-* names -- a D-119 retired-token residue from the 2026-07-21 install.

Revert. maas vr1-dc0-region subnet update 1 dns_servers=10.12.8.3 allow_dns=false and delete the dynamic iprange.

Item 10 -- BLOCKED: the DHCP handover (task 5) needs an operator decision

What was attempted. With the topology verified complete (39/0), the next step is the one-way DHCP cutover:

# 1. off at Office1  (measured target, resolved live, not from memory)
maas admin vlan update 4 0 dhcp_on=false
# 2. verify dhcpd actually STOPPED on the rack, BY PROCESS
#    (MAAS self-report is not evidence -- the 2026-07-20 Temporal incident class)
# 3. on at the new region, primary_rack = hot-kid
maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=<hot-kid system_id>
# 4. verify dhcpd RUNNING by process

It was REFUSED by the harness permission classifier. Per the standing rule -- and the precedent recorded in CURRENT-STATE for the four MAAS carve calls refused on 2026-07-30 -- it was NOT retried in an altered shape.

NOTHING IS HALF-APPLIED, verified read-only after the refusal: Office1 still reads 10.12.8.0/22 dhcp_on=True primary_rack=7chphy, and the rack still runs exactly one dhcpd on virbr2. The refusal landed before any mutation.

Measured state at the boundary, so the next session does not re-derive it:

Why the gap is safe to take: all 10 dc0 machines are Ready and POWERED OFF, so no node is mid-DHCP. Two DHCP servers on one segment is the hazard the ordering avoids, which is why off-then-on is one operation and not two independent ones.

No revert needed -- nothing was changed.

Item 11 -- as-executed log NOT used for this window, and why

scripts/run-logged.sh opens an INTERACTIVE script(1) subshell and is unusable from a non-interactive agent session. F6 (2026-07-30) already records the harness classifier refusing the wrapped script -aqe ... -c form. Rather than produce a log that looks complete and is not, this session captures to docs/audit/* files -- which is what GA-R6 actually requires of a gate -- and says so here. A logs/as-executed-index.md row is owed at close.