Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260730-dc0-region-migration.md

Changelog 2026-07-30 -- dc0 MAAS region migration (Stage 5 unblock)

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Context: Stage 5 is OPEN and BLOCKED. D-132 q1 (RULED 2026-07-30) puts a MAAS region in each DC; the dc0 region VM is LIVE at 10.12.8.6:5240, but the Office1 region still owns all 9 dc0 role nodes plus the Juju controller. MAAS cannot move machines between regions, so the carve must be RECREATED and the machines re-enrolled. This changelog covers that migration.

Operator authorization for this session, verbatim: "Begin working through all tasks and continue deployment. Work as autonomously as possible, deploy agents as needed." Under blanket approval this changelog IS the review surface (GA-R2): every item states what, why (evidence), and how to revert.


Item 1 -- scripts/maas-profile-assert.sh + tests/maas-profile-assert/ (NEW)

What. A guard that proves WHICH MAAS REGION a CLI profile resolves to, by rack-controller identity, before any command that reads or writes that region. Exit 0 match / 1 wrong region / 2 refuse.

Why (measured, not reasoned). Every MAAS call in this repo defaults to MAAS_PROFILE=admin (13 scripts: maas-fabric-prune, phase-00-teardown-destroy, pre-flight-checks, maas-role-tags, carve-host-interfaces, dc-plane-ipam, phase-00-teardown-release, lib-hosts, maas-node-power, phase-00-maas-standup, reenroll-hosts, and two more). MEASURED on voffice1: the sole profile admin resolves to http://10.10.0.20:5240/ -- the OFFICE1 region. The two regions hold SEPARATE databases, so during this migration:

  • a carve recreated against the wrong profile is an idempotent NO-OP that prints PASS (the state was already there -- in the wrong region);
  • machine delete against the wrong profile DESTROYS the real record, unrecoverably.

This is the wrong-HOST class this repo has been bitten by three times already (dc-mirror.sh run on voffice1; the egress probe run from the rack; P5 probing the wrong filesystem) -- now wrong-REGION, where the destructive direction has no undo.

A count alone is not proof -- two regions can hold the same number of machines. Rack-controller identity is the discriminator, because a rack registers to exactly one region.

Live verification, both directions (capture docs/audit/dc0-region-profile-assert-20260730.txt):

admin           -> machines=23 racks=voffice1,vvr1-dc0,vvr1-dc1   (Office1)
vr1-dc0-region  -> machines=0  racks=hot-kid                      (new dc0 region)

$ maas-profile-assert.sh vr1-dc0-region hot-kid                    exit=0
$ maas-profile-assert.sh admin voffice1,vvr1-dc0,vvr1-dc1          exit=0
$ maas-profile-assert.sh admin hot-kid            NEGATIVE ->      exit=1

Harness 20/20, and the MUTATION PASS FOUND A REAL DEFECT IN MY OWN TEST. Three mutations were applied to the script and the suite re-run:

  • A (comparison always true) -- killed 4 tests. Good.
  • B (empty rack list accepted) -- killed 4 tests. Good.
  • C (nameless rack entries accepted) -- suite stayed fully GREEN (19/19). The assertion was decoration: a nameless entry fell through to the generic empty-ACTUAL guard and refused for a different reason, so deleting the branch it claimed to test changed nothing. Fixed by giving the nameless case its own exit code and refusal message and asserting on THAT; mutation C now kills a test. This is exactly the "prove each new assertion can FAIL" rule, and it caught me writing decoration.

Revert. git rm scripts/maas-profile-assert.sh tests/maas-profile-assert/ && git checkout <this-commit>^ -- . Nothing else depends on it yet; it is additive.


Item 2 -- named MAAS profile vr1-dc0-region on voffice1 (LIVE, config)

What. maas login vr1-dc0-region http://127.0.0.1:5241/MAAS/ -, key streamed from ~/vr1-dc0-creds/maas-region-api-key.txt on vcloud via stdin, never printed. The existing admin profile is UNTOUCHED and still points at Office1.

Why the tunnel. Neither candidate host has everything the migration needs:

host repo maas CLI reaches new region reaches NetBox apex
voffice1 yes yes NO (curl exit 28) yes
dc0 rack no yes yes (200) NO (10.10.1.10:8000 CLOSED)
region VM no yes yes not measured

dc-plane-ipam.sh derives the v6 plane prefixes from the NetBox apex (D-136 option D), so it must run where NetBox is reachable. An SSH tunnel from voffice1 through the rack (-L 127.0.0.1:5241:10.12.8.6:5240) gives voffice1 both. Measured: localhost:5241 answers 200.

This does not puncture SEC-010. SEC-010 forbids the rack FORWARDING between its transit leg and the plane nets. The tunnel is an application-layer, rack-ORIGINATED connection -- exactly the distinction SEC-010's own justification draws ("a MAAS rack proxies at the application layer and needs no kernel forwarding"). No route, no nftables rule, and no libvirt net was changed. The rack's table inet sec010 drops are intact.

Safe failure direction, verified by construction: if the tunnel dies, the vr1-dc0-region profile becomes unreachable and maas-profile-assert.sh REFUSES (exit 2). It cannot silently fall back to Office1, because the fallback would have to be a different URL.

CREDENTIAL RESIDENCY -- SEC-026 addendum, flagged not glossed. This puts dc0's region admin API key on voffice1, where it was not before. Today that is a strict SUBSET of the blast radius already present: voffice1 holds the Office1 region admin key, which currently administers BOTH DCs' nodes. After the migration that inverts -- Office1's key stops covering dc0, and dc0's region key becomes the broader credential on a shared headend. Obligation recorded: remove the vr1-dc0-region profile from voffice1 once dc0's control host is settled. A security-ledger row update is owed at session close.

Revert. On voffice1: maas logout vr1-dc0-region, and kill the tunnel (pkill -f '5241:10.12.8.6'). Both are local to voffice1; no cloud state is touched.


Item 3 -- OPERATING NOTE: MAAS CLI for a per-DC region cannot run from voffice1

Measured. voffice1 -> 10.12.8.6:5240 times out (curl exit 28); voffice1 -> 10.10.0.20:5240 is fine. The dc0 rack reaches BOTH (200 / 200).

Consequence. D-128's "Plane 2 (MAAS/NetBox/inner tofu) executes on voffice1" clause was written when there was ONE region, at Office1. D-132 q1 makes the MAAS control surface DC-local, so for the per-DC regions that clause needs either a tunnel (what this session does) or a DC-side host. This is a consequence of an already-ruled decision, not a new one -- recorded here, and owed as a DOCFIX against D-128's Plane-2 wording and the phase-4 runbook's run-location banner (which F5 already flags as wrong for the juju half after D-138).

No revert -- this item is a record, not a change.


Item 4 -- as-executed log NOT used for this window, and why

scripts/run-logged.sh opens an INTERACTIVE script(1) subshell and is unusable from a non-interactive agent session. F6 (2026-07-30) already records the harness classifier refusing the wrapped script -aqe ... -c form. Rather than produce a log that looks complete and is not, this session captures to docs/audit/* files -- which is what GA-R6 actually requires of a gate -- and says so here. A logs/as-executed-index.md row is owed at close.