Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260730-dc0-region-migration.md
@JANeumatrix JANeumatrix 23 hours ago 50 KB Plane IPAM complete on the new region: 24/0 PASS, existing tooling only
Raw Blame History

Changelog 2026-07-30 -- dc0 MAAS region migration (Stage 5 unblock)

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Status claims live ONLY in docs/CURRENT-STATE.md.

Context: Stage 5 is OPEN and BLOCKED. D-132 q1 (RULED 2026-07-30) puts a MAAS region in each DC; the dc0 region VM is LIVE at 10.12.8.6:5240, but the Office1 region still owns all 9 dc0 role nodes plus the Juju controller. MAAS cannot move machines between regions, so the carve must be RECREATED and the machines re-enrolled. This changelog covers that migration.

Operator authorization for this session, verbatim: "Begin working through all tasks and continue deployment. Work as autonomously as possible, deploy agents as needed." Under blanket approval this changelog IS the review surface (GA-R2): every item states what, why (evidence), and how to revert.

Item 1 -- scripts/maas-profile-assert.sh + tests/maas-profile-assert/ (NEW)

What. A guard that proves WHICH MAAS REGION a CLI profile resolves to, by rack-controller identity, before any command that reads or writes that region. Exit 0 match / 1 wrong region / 2 refuse.

Why (measured, not reasoned). Every MAAS call in this repo defaults to MAAS_PROFILE=admin (13 scripts: maas-fabric-prune, phase-00-teardown-destroy, pre-flight-checks, maas-role-tags, carve-host-interfaces, dc-plane-ipam, phase-00-teardown-release, lib-hosts, maas-node-power, phase-00-maas-standup, reenroll-hosts, and two more). MEASURED on voffice1: the sole profile admin resolves to http://10.10.0.20:5240/ -- the OFFICE1 region. The two regions hold SEPARATE databases, so during this migration:

  • a carve recreated against the wrong profile is an idempotent NO-OP that prints PASS (the state was already there -- in the wrong region);
  • machine delete against the wrong profile DESTROYS the real record, unrecoverably.

This is the wrong-HOST class this repo has been bitten by three times already (dc-mirror.sh run on voffice1; the egress probe run from the rack; P5 probing the wrong filesystem) -- now wrong-REGION, where the destructive direction has no undo.

A count alone is not proof -- two regions can hold the same number of machines. Rack-controller identity is the discriminator, because a rack registers to exactly one region.

Live verification, both directions (capture docs/audit/dc0-region-profile-assert-20260730.txt):

admin           -> machines=23 racks=voffice1,vvr1-dc0,vvr1-dc1   (Office1)
vr1-dc0-region  -> machines=0  racks=hot-kid                      (new dc0 region)

$ maas-profile-assert.sh vr1-dc0-region hot-kid                    exit=0
$ maas-profile-assert.sh admin voffice1,vvr1-dc0,vvr1-dc1          exit=0
$ maas-profile-assert.sh admin hot-kid            NEGATIVE ->      exit=1

Harness 20/20, and the MUTATION PASS FOUND A REAL DEFECT IN MY OWN TEST. Three mutations were applied to the script and the suite re-run:

  • A (comparison always true) -- killed 4 tests. Good.
  • B (empty rack list accepted) -- killed 4 tests. Good.
  • C (nameless rack entries accepted) -- suite stayed fully GREEN (19/19). The assertion was decoration: a nameless entry fell through to the generic empty-ACTUAL guard and refused for a different reason, so deleting the branch it claimed to test changed nothing. Fixed by giving the nameless case its own exit code and refusal message and asserting on THAT; mutation C now kills a test. This is exactly the "prove each new assertion can FAIL" rule, and it caught me writing decoration.

Revert. git rm scripts/maas-profile-assert.sh tests/maas-profile-assert/ && git checkout <this-commit>^ -- . Nothing else depends on it yet; it is additive.

Item 2 -- named MAAS profile vr1-dc0-region on voffice1 (LIVE, config)

What. maas login vr1-dc0-region http://127.0.0.1:5241/MAAS/ -, key streamed from ~/vr1-dc0-creds/maas-region-api-key.txt on vcloud via stdin, never printed. The existing admin profile is UNTOUCHED and still points at Office1.

Why the tunnel. Neither candidate host has everything the migration needs:

host repo maas CLI reaches new region reaches NetBox apex
voffice1 yes yes NO (curl exit 28) yes
dc0 rack no yes yes (200) NO (10.10.1.10:8000 CLOSED)
region VM no yes yes not measured

dc-plane-ipam.sh derives the v6 plane prefixes from the NetBox apex (D-136 option D), so it must run where NetBox is reachable. An SSH tunnel from voffice1 through the rack (-L 127.0.0.1:5241:10.12.8.6:5240) gives voffice1 both. Measured: localhost:5241 answers 200.

This does not puncture SEC-010. SEC-010 forbids the rack FORWARDING between its transit leg and the plane nets. The tunnel is an application-layer, rack-ORIGINATED connection -- exactly the distinction SEC-010's own justification draws ("a MAAS rack proxies at the application layer and needs no kernel forwarding"). No route, no nftables rule, and no libvirt net was changed. The rack's table inet sec010 drops are intact.

Safe failure direction, verified by construction: if the tunnel dies, the vr1-dc0-region profile becomes unreachable and maas-profile-assert.sh REFUSES (exit 2). It cannot silently fall back to Office1, because the fallback would have to be a different URL.

CREDENTIAL RESIDENCY -- SEC-026 addendum, flagged not glossed. This puts dc0's region admin API key on voffice1, where it was not before. Today that is a strict SUBSET of the blast radius already present: voffice1 holds the Office1 region admin key, which currently administers BOTH DCs' nodes. After the migration that inverts -- Office1's key stops covering dc0, and dc0's region key becomes the broader credential on a shared headend. Obligation recorded: remove the vr1-dc0-region profile from voffice1 once dc0's control host is settled. A security-ledger row update is owed at session close.

Revert. On voffice1: maas logout vr1-dc0-region, and kill the tunnel (pkill -f '5241:10.12.8.6'). Both are local to voffice1; no cloud state is touched.

Item 3 -- OPERATING NOTE: MAAS CLI for a per-DC region cannot run from voffice1

Measured. voffice1 -> 10.12.8.6:5240 times out (curl exit 28); voffice1 -> 10.10.0.20:5240 is fine. The dc0 rack reaches BOTH (200 / 200).

Consequence. D-128's "Plane 2 (MAAS/NetBox/inner tofu) executes on voffice1" clause was written when there was ONE region, at Office1. D-132 q1 makes the MAAS control surface DC-local, so for the per-DC regions that clause needs either a tunnel (what this session does) or a DC-side host. This is a consequence of an already-ruled decision, not a new one -- recorded here, and owed as a DOCFIX against D-128's Plane-2 wording and the phase-4 runbook's run-location banner (which F5 already flags as wrong for the juju half after D-138).

No revert -- this item is a record, not a change.

Item 4 -- lib-hosts.sh: the power address is now per-REGION (both DCs)

What. Each VR1 arm gains VIRSH_POWER_ADDRESS_FROM_OFFICE1 (the rack's transit leg) and VIRSH_POWER_ADDRESS_FROM_DCREGION (the rack's metal-admin leg). VIRSH_POWER_ADDRESS is now an ALIAS of the Office1 form, byte-for-byte unchanged. The flat/VR0 layer declares both as EMPTY.

Why (measured). MAAS power ops originate from the REGION, so the reachable rack address depends on which region dials. From vr1-dc0-maas-01 (10.12.8.6):

172.31.0.2:22  CLOSED        <- the transit address lib-hosts has always used
10.12.8.2:22   OPEN          <- the rack's metal-admin leg
routes: 10.12.4.0/22, 10.12.8.0/22, default via 10.12.4.1

dc1's 10.12.68.2 was MEASURED on the dc1 rack (ip -4 -o addr -> virbr6 inet 10.12.68.2/22) and corroborated against dc-rack-net.sh's dc1 LEGS block. It was NOT inferred from dc0's .2 by symmetry.

Backward compatibility is the load-bearing half. reenroll-hosts.sh:132 and the teardown runbook's virsh -c "$VIRSH_POWER_ADDRESS" probes must not silently change target while Office1 still owns dc1's nodes. Flipping the default to the DC-region form is OWED once BOTH DCs have migrated.

Harness tests/dc-selector 81 checks ALL PASS, 4 mutations each killed tests:

mutation tests killed
A: collapse dc0's DCREGION onto the transit form 2
B: dc1's DCREGION gets dc0's octet (cross-DC contamination) 2
C: VIRSH_POWER_ADDRESS silently repointed to the DC-region form 3
D: VR0 inherits a VR1 form 1

Revert. git checkout <this-commit>^ -- scripts/lib-hosts.sh tests/dc-selector/run-tests.sh.

Item 5 -- scripts/maas-region-power-key.sh + harness (NEW), and INSTALLED at dc0

What. check|install <site> <rack-virsh-uri>. Installs the per-DC MAAS->libvirt power key into a region's snap and verifies it with a REAL virsh call. Key is read from STDIN, so it never lands on an intermediate host and never appears in a process argument.

Why. runbooks/dc-dc-phase2-tofu-dc-substrate.md prerequisite 5 carries this as PROSE, and prose is not a gate. It must be re-run at every DC standup and -- per that same prerequisite's own FRAGILITY note -- after EVERY MAAS snap refresh, because the key lives under per-revision /var/snap/maas/current/. MEASURED before this work: /var/snap/maas/current/root/.ssh/ did not exist on the new region at all, so commissioning could not have powered a node on.

LIVE at dc0 (capture docs/audit/dc0-region-power-key-20260730.txt), 9/9 exit 0, ending on the artifact rather than the config:

[ok] key mode 0600 / owned by root / parses as a valid PRIVATE key
     (SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY)
[ok] config binds Host 10.12.8.2 -> id_dc0_power, IdentitiesOnly yes
[ok] REAL virsh connect to qemu+ssh://jessea123@10.12.8.2/system succeeds
[ok] virsh enumerates domains on the rack (got 12)

The key is the SAME dc0-scoped key already authorized on the dc0 rack (fingerprint matched against the rack's authorized_keys), so SEC-012's per-DC isolation holds -- this is not a cross-DC reuse. OWED: remove Office1's copy once dc0's migration completes, since that region will no longer power dc0 machines.

THREE DEFECTS THIS SCRIPT'S OWN TESTING FOUND, all measured, none by review:

  1. ssh-keygen -lf <privkey> reads an ADJACENT .pub when one exists and reports ITS fingerprint. A stale sibling made a freshly-installed key read back as a different one, which would have made the idempotency and cross-DC-swap refusals answer about the wrong key. Fixed with ssh-keygen -y -f (derive the public half FROM the private key), which no neighbouring file can fool. Found by the harness.
  2. A snap has TWO roots and they are not interchangeable. /var/snap/<n>/current is writable data (where root/.ssh lives); /snap/<n>/current is the read-only squashfs (where the binaries live). The first live run REFUSED with "no virsh binary found" while find /snap/maas/current -name virsh had already located one.
  3. The snap's binary must be run INSIDE the snap's runtime. Invoking /snap/maas/current/usr/bin/virsh directly fails with error while loading shared libraries: libvirt-lxc.so.0. snap run --shell maas -c 'virsh ...' executes it with the snap's environment AND its ssh config -- which is exactly the context MAAS's own power driver uses, so this is the correct artifact to grade, not a proxy.

Harness 44/44. Assertions include every refusal direction (absent key, wrong-host binding, missing IdentitiesOnly, unreachable virsh, virsh listing ZERO domains) and the cross-DC-swap refusal, verified not to overwrite. Two harness assertions were found passing FOR THE WRONG REASON during development and were made specific.

A NOTE ON MY OWN INSTRUMENT. The first live install reported EXIT=0 while the check had actually returned 2 -- I had read $? from the end of a pipeline (... | tail), so I measured tail's exit, not the script's. That is the identical class this repo recorded twice on 2026-07-30 (| grep -q under pipefail). Fixed by capturing into a variable and reading $? directly.

Revert. git rm scripts/maas-region-power-key.sh tests/maas-region-power-key/. To undo the live install: on the region VM, sudo rm /var/snap/maas/current/root/.ssh/id_dc0_power and delete the Host 10.12.8.2 stanza from that dir's config.

Item 6 -- RECON: what actually builds the dc0 carve (read-only; changes nothing)

A read-only survey mapped each of the seven carve components to its creating tool. Four findings change the migration plan; all four were verified against the files before being recorded here.

(a) THE PRE-MIGRATION CAPTURE IS NOT A DIFF TARGET, and I had planned to use it as one. Its vlan_id=5189..5193 / id=6,7,12..15,20..25 / spaces id=1..6 values are MAAS database row ids, not 802.1Q tags -- every named fabric line reads vid=0. Row ids are region-local and will differ on a rebuild. Its RESERVED-RANGES section is subnet reserved-ip-ranges output, which COALESCES and is state-dependent (metal-internal et al. show a merged .4-.99 reserved, while provider-public splits .4-.49/.50-.99 because .4-.49 there also carries assigned-ip from live statics). A textual diff of a perfectly rebuilt region against this file would NOT match. The only stable identities are fabric NAME, space NAME, subnet CIDR, tag NAME, node boot MAC, interface NAME -- which is what scripts/lib-net.sh:8-10 already says repo-wide ("CIDR is the stable key throughout; MAAS subnet IDs drift across cutovers"). The real gate is the check actions (dc-plane-ipam.sh check, dc-node-v6-carve.py check, maas-role-tags.sh check), not a file diff. My task-7 plan is corrected accordingly.

(b) scripts/dc-node-v6-carve.py IS PROFILE-ENV-BLIND -- verified at the file. PROFILE_DEFAULT = "admin" (:39) and --profile (:70), with NO environment read anywhere. So export MAAS_PROFILE=vr1-dc0-region -- the idiom every bash tool here honours -- silently targets Office1. Combined with this changelog's own item 1 reasoning (a carve run against the wrong profile is an idempotent no-op that prints PASS), this is a live foot-gun on exactly the tool the migration needs. It must be invoked with an explicit --profile, or fixed to read the env.

(c) THREE COMPONENTS HAVE NO REPO TOOL AT ALL. The named plane fabrics, the six v4 plane subnets, and the per-node v4 NIC carve (60 NIC re-homes + 9 br-ex bridges + 54 statics) were all created ad-hoc during the Stage-4 window; the mutations went to ~/as-executed/2026-07-23-stage4-carve.log, which is NOT in the repo. The openstack-vr1-dc0 tag has no usable creator either. So the "re-run the generator against the new endpoint" plan is only available for roughly half the work.

(d) TWO EXISTING SCRIPTS MUST NOT BE POINTED AT THE NEW REGION, both verified:

  • scripts/carve-host-interfaces.sh:62-64 REFUSES vr1-* by design (it carves the VR0 Pattern-A VLAN-103 stack, superseded for VR1 by D-133). scripts/reenroll-hosts.sh:53-55 refuses vr1-* likewise.
  • scripts/phase-00-maas-standup.sh does NOT refuse, and that is the danger: its parity guard PASSES for vr1-dc0, then it creates fabrics named after the SPACE (provider-public, not vr1-dc0-provider-public, :192-194), creates a named metal-admin fabric the as-built does not have, and builds metal-internal as a tagged VID-103 VLAN riding metal-admin's fabric -- which D-133 abolished for VR1. It would silently produce a divergent region.

Also recorded: runbooks/phase-01-bundle-deploy.md:128 iterates hardcoded subnet ids (for ID in 1 2 6 7 8 9) and is profile-blind -- in a file that warns against exactly this at :54 and :287-291. scripts/site-headend-install.sh:528 picks a rack by list INDEX ([0]['system_id']), which is meaningless across regions. Both are pre-existing and LOGGED, NOT EXECUTED (hard rule 1).

No revert -- this item is a record, not a change.

Item 7 -- harness manifest re-recorded (91 harnesses)

scripts/run-tests-all.sh FAILED with harness set DRIFTED from tests/HARNESS-MANIFEST, listing maas-profile-assert and maas-region-power-key as present-but-unpinned. That is the manifest gate working as designed on two harnesses added this session, not a regression. Re-recorded with --record-manifest; 35 -> 37 entries in the diff, 91 total.

Revert. git checkout <this-commit>^ -- tests/HARNESS-MANIFEST.

Item 8 -- scripts/dc-region-topology.sh + harness (NEW), and APPLIED at dc0

What. check|apply <site> --profile <p> --expect-rack <r> [--commit] over the five named plane fabrics, the six spaces, the six v4 plane subnets on the right VLANs with the ruled gateways, the VLAN->space bindings, and the openstack-<site> tag. This is the tool item 6(c) found missing.

Design decisions worth stating:

  • --profile has NO default. The repo-wide default admin resolves to the Office1 region, where this apply would be an idempotent no-op that prints PASS. Every mutating path is gated by maas-profile-assert.sh FIRST.
  • metal-admin deliberately keeps MAAS's own auto-created fabric. It is a discovery artifact created when the rack registers, before anything in this repo could name it; Juju binds SPACES, not fabrics. The check asserts only that it does not share a fabric with a named plane (which would collapse two L2 domains).
  • No numeric MAAS id is persisted. Everything resolves per-run by name or CIDR.

LIVE at dc0 (capture docs/audit/dc0-region-topology-20260730.txt). The apply created 5 fabrics, 6 spaces, 4 subnets, MOVED 10.12.4.0/22 off the auto fabric-1 onto vr1-dc0-provider-public, bound 6 VLANs to their spaces, and created the site tag. Final named check: 39 passed, 0 failed, exit 0.

A CONCRETE ILLUSTRATION OF WHY ROW IDS ARE NOT PORTABLE, measured in passing: VLAN row id 5005 is dc0's metal-admin VLAN in the Office1 region and vr1-dc0-data-tenant in the new region. Same integer, different object, different DC plane. Anything that carried 5005 across would have mis-targeted silently.

Harness 39/39; 4 mutations each killed tests (region gate removed; profile defaulted to admin; space-binding assertion neutered; fabric-placement assertion neutered). Fixtures were corrected against a live read: MAAS reports an unbound VLAN as the STRING "undefined", not null, so the first fixtures graded a payload MAAS never emits.

Revert. The apply is additive. To undo: delete the five vr1-dc0-* fabrics and the six spaces on the new region, and move 10.12.4.0/22 back to fabric-1. Nothing in the Office1 region was touched by this item.

Item 9 -- metal-admin service config set on the new region (LIVE)

dns_servers=10.12.8.6, allow_dns=false, and the D-134 dynamic range 10.12.8.201-10.12.8.254. Subnet resolved BY CIDR (-> id 1), never from the capture.

The DNS target was MEASURED, not carried over, and my first reading of it was WRONG. The old region set dns_servers=10.12.8.3 (the D-131 rack forwarder). A first probe suggested the new region's own BIND at 10.12.8.6 could not resolve external names, which would have forced keeping the forwarder. That was a cold-cache timeout at +time=3 +tries=1, not a capability gap -- re-probed with +time=5 +tries=3, 10.12.8.6 answers archive.ubuntu.com, streams.canonical.com and api.snapcraft.io correctly, and a full dig shows flags: qr rd ra with ANSWER: 9. I nearly recorded a false finding on one under-timed query. Pointing node DNS at the DC-LOCAL region is the correct end state: it is authoritative for the zone the migrated nodes will live in (its own maas-internal, SOA serial 16) and removes the cross-fiber dependency that D-132 q1 exists to remove.

FOLLOW-UP RECORDED, NOT ACTIONED (hard rule 1): the live rack forwarder is /etc/dnsmasq-dc0-node.conf with no-resolv + server=10.10.0.20 -- it sends EVERYTHING to the Office1 region. After dc0's nodes migrate, that upstream points at a region that no longer owns them. dc-rack-net.sh carries DNS_UPSTREAM="10.10.0.20" for BOTH sites (:66, :82), and dc1's is still correct, so this is a per-site change at each DC's cutover. Also noted: the live unit is named with the BARE dc0 token (dc0-node-dns.service, /etc/dnsmasq-dc0-node.conf) while the current script generates vr1-dc0-* names -- a D-119 retired-token residue from the 2026-07-21 install.

Revert. maas vr1-dc0-region subnet update 1 dns_servers=10.12.8.3 allow_dns=false and delete the dynamic iprange.

Item 10 -- the DHCP handover (task 5): initially BLOCKED, then EXECUTED

What was attempted. With the topology verified complete (39/0), the next step is the one-way DHCP cutover:

# 1. off at Office1  (measured target, resolved live, not from memory)
maas admin vlan update 4 0 dhcp_on=false
# 2. verify dhcpd actually STOPPED on the rack, BY PROCESS
#    (MAAS self-report is not evidence -- the 2026-07-20 Temporal incident class)
# 3. on at the new region, primary_rack = hot-kid
maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=<hot-kid system_id>
# 4. verify dhcpd RUNNING by process

It was REFUSED by the harness permission classifier. Per the standing rule -- and the precedent recorded in CURRENT-STATE for the four MAAS carve calls refused on 2026-07-30 -- it was NOT retried in an altered shape.

NOTHING IS HALF-APPLIED, verified read-only after the refusal: Office1 still reads 10.12.8.0/22 dhcp_on=True primary_rack=7chphy, and the rack still runs exactly one dhcpd on virbr2. The refusal landed before any mutation.

Measured state at the boundary, so the next session does not re-derive it:

Office1 region new dc0 region
dc0 metal-admin fabric-4 / vid 0 / vlan row 5005 fabric-0 / vid 0 / vlan row 5001
dhcp_on True, primary_rack 7chphy (vvr1-dc0) False, primary_rack None
dns_servers 10.12.8.3 10.12.8.6
rack system_ids mtstwf / 7chphy / nmpcq4 hot-kid

Why the gap is safe to take: all 10 dc0 machines are Ready and POWERED OFF, so no node is mid-DHCP. Two DHCP servers on one segment is the hazard the ordering avoids, which is why off-then-on is one operation and not two independent ones.

RESOLVED AND EXECUTED 2026-07-30 -- see item 15. Everything above stands as the record of the blocked attempt; nothing was changed at the time of the refusal.

THE EXACT UNBLOCK SEQUENCE, every parameter RE-RESOLVED LIVE at session end. Run these on the jumphost with the ! prefix so the output lands in-session. The vr1-dc0-region profile needs the tunnel, which was verified UP at capture time (pgrep -f '5241:10.12.8.6'); if it has since died, re-open it with ssh voffice1 "ssh -f -N -L 127.0.0.1:5241:10.12.8.6:5240 jessea123@172.31.0.2".

# 1. OFF at Office1  (fabric-4 / vid 0 -- dc0 metal-admin, dhcp_on=True, primary_rack=7chphy)
ssh voffice1 'maas admin vlan update 4 0 dhcp_on=false'

# 2. VERIFY BY PROCESS, not by MAAS self-report (the 2026-07-20 Temporal incident class)
ssh vr1-dc0-rack 'pgrep -c dhcpd; ps -ef | grep "[d]hcpd"'      # expect 0 dhcpd

# 3. ON at the new region  (fabric-0 / vid 0, rack hot-kid = c3aqh8 IN THAT REGION)
ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=c3aqh8'

# 4. VERIFY BY PROCESS again -- expect exactly one dhcpd, now owned by the region VM
ssh vr1-dc0-maas 'pgrep -c dhcpd; ps -ef | grep "[d]hcpd"'

# 5. Re-run the named gate; it should stay 40/0
ssh voffice1 'cd ~/openstack-caracal-dc-dc && bash scripts/dc-region-topology.sh \
    check vr1-dc0 --profile vr1-dc0-region --expect-rack hot-kid'

PRE-CUTOVER CHECKS, both done, both were genuinely load-bearing:

  • hot-kid IS eligible as primary_rack for fabric-0/vid 0 -- its enp1s0 is on that VLAN (vlan_row=5001, static 10.12.8.6). If it were not, step 1 would succeed, step 3 would fail, and the segment would have NO DHCP with ten machines about to need PXE. That is the one-way risk in the sequence and it is answerable read-only, so it was.
  • hot-kid's system_id is c3aqh8 in the NEW region and tw7ptw in Office1 -- the same machine, two different ids. One more proof that ids do not cross regions.

Item 12 -- DEFECT IN MY OWN APPLY, found by reading interfaces back, and fixed

What went wrong. dc-region-topology.sh apply CREATED vr1-dc0-provider-public and MOVED 10.12.4.0/22 onto it. MAAS does not bring interface links along on a subnet move. The region VM's own enp2s0 stayed on the old auto fabric's VLAN while its subnet left, so hot-kid was left holding a static 10.12.4.6 on a VLAN with no subnet -- the region's provider-public leg, which it needs for image sync, and the identical under-carve class that cost three bootstrap attempts on the Juju controller.

Why nothing caught it. Live connectivity was UNAFFECTED (the OS netplan is independent of MAAS's model -- default via 10.12.4.1, egress 200), and the check read subnets, fabrics, spaces and tags but never an interface link. It passed 39/39 with the model inconsistent.

Fixed three ways:

  1. Live: subnet returned to vlan 5002, the wrongly-created fabric deleted, fabric-1 RENAMED to vr1-dc0-provider-public, space re-bound. enp2s0 now reads fabric=vr1-dc0-provider-public with its static intact.
  2. In the tool: it now RENAMES an auto fabric that already carries only this plane's subnet -- renaming moves nothing. A MOVE survives only for the case where the fabric carries other planes, and now prints a warning that links do not follow.
  3. In the gate: a new assertion that every interface link sits on its subnet's VLAN. Live re-check is now 40 passed / 0 failed, exit 0.

Harness 47/47; the rename path and the stranded-link check each proven able to fail. One older assertion asserted the CREATE behaviour and was RE-POINTED rather than deleted, per the standing rule.

Item 13 -- dc-node-v6-carve.py: env-blindness and traceback fixed

Item 6(b) recorded this as a foot-gun; it is now fixed rather than only noted, because this migration will invoke it. PROFILE_DEFAULT now reads MAAS_PROFILE (with --profile still overriding), and every run PRINTS which profile it targets and where that came from -- a capture that does not name the profile cannot be audited for wrong-region work. Separately, an absent maas CLI raised FileNotFoundError and exited 1 with a traceback, indistinguishable from a real failure and contrary to this script's own REFUSE contract; it now exits 2. Harness 14/14 (was 9); both fixes mutation-proven.

Item 14 -- the MEASURED VR1 node NIC -> plane order (lib-hosts.sh)

The per-node v4 carve has no repo tool, so the next step was about to re-derive this by hand. MEASURED instead, end to end: the substrate roots pin six MACs per node and the module attaches them in list order, so macs[i] IS enp<i+1>s0. Verified on vr1-dc0-control-01 against the live carve -- enp1s0 metal-admin (PXE), enp2s0 provider-public (the br-ex parent, Pattern A), enp3s0 metal-internal, enp4s0 data-tenant, enp5s0 storage, enp6s0 replication.

THE TRAP, now guarded: this is NOT lib-net.sh's PLANE_CIDRS order, which starts with provider-public. Anything walking PLANE_CIDRS positionally to place NICs swaps enp1s0 and enp2s0 and puts the PXE leg on provider-public -- stranding commissioning on a plane with no DHCP. NIC_PLANE_ORDER + BREX_PARENT_NIC are recorded with the evidence, and dc-selector (88 checks) asserts the two orders differ at position 0, so the guard cannot go vacuous.

STILL OWED (explicit, so nothing is lost)

  1. The DHCP cutover -- item 10, blocked on the permission wall.
  2. scripts/dc-node-carve.sh does not exist -- 60 NIC re-homes + 9 br-ex bridges + 54 v4 statics, the largest remaining no-tool gap. The two hard inputs are now settled (item 14's ordering, and carve-host-interfaces.sh:211-236's build_ovs_brex() call sequence, which the recon flagged as the correct MAAS ordering: unlink commissioning links -> move the parent NIC onto the provider VLAN -> interfaces create-bridge bridge_type=ovs -> link-subnet mode=STATIC on the bridge). Deliberately NOT written blind: it cannot be exercised until the nodes are re-enrolled, and both defects found today (the stranding bug, the fixture mismatch) were caught by LIVE runs, not by fixtures. Write it against a region it can actually run against.
  3. maas-profile-assert.sh is wired into dc-region-topology.sh only. dc-plane-ipam.sh, maas-role-tags.sh and maas-node-power.sh still default to MAAS_PROFILE=admin. A forgotten export on any of them mutates OFFICE1 -- which still owns dc1's nine live nodes, so a mis-targeted maas-role-tags.sh apply would retag the wrong DC. Until they are wired, run the assert in the same command immediately before each and put it in the capture.
  4. The rack DNS forwarder's upstream -- item 9's follow-up.
  5. Remove the vr1-dc0-region profile from voffice1 once dc0's control host settles (SEC-026), and remove Office1's copy of the dc0 power key once the migration lands.

Item 11 -- as-executed log NOT used for this window, and why

scripts/run-logged.sh opens an INTERACTIVE script(1) subshell and is unusable from a non-interactive agent session. F6 (2026-07-30) already records the harness classifier refusing the wrapped script -aqe ... -c form. Rather than produce a log that looks complete and is not, this session captures to docs/audit/* files -- which is what GA-R6 actually requires of a gate -- and says so here. A logs/as-executed-index.md row is owed at close.

Item 15 -- DHCP HANDOVER EXECUTED, and the permission wall diagnosed

Executed in the ruled order, region identity proven first (capture docs/audit/dc0-dhcp-handover-20260730.txt):

step command read-back
1 maas admin vlan update 4 0 dhcp_on=false dhcp_on=False
2 pgrep -c dhcpd on the dc0 rack 0
3 maas vr1-dc0-region vlan update 0 0 dhcp_on=true primary_rack=c3aqh8 dhcp_on=True primary_rack=c3aqh8 fabric=fabric-0 space=metal-admin
4 ps -ef on hot-kid; rack re-checked 2 dhcpd on the region VM; rack still 0

Step 2 is the one that matters: MAAS's self-report is not evidence that dhcpd stopped (the 2026-07-20 Temporal incident class). Exactly ONE DHCP server is now on the segment, owned by the DC-local region. All 10 machines were Ready and POWERED OFF throughout.

Recorded so it is not later read as a fault: the new region also starts dhcpd6, where the Office1 rack ran v4 only. Its v6 metal-admin subnet carries NO dynamic range -- only rfc-4291-2.6.1 and assigned-ip,reserved, identical to the old region -- so it has nothing to offer and nodes keep their ruled MAAS statics.

THE WALL WAS A RULE THAT FAILED TO MATCH, NOT A CLASSIFIER FAULT -- the standing lesson, restated with two new specifics. .claude/settings.json already had Bash(maas admin * update*) in ask, and settings.local.json already had Bash(ssh voffice1 "maas admin subnet *) and siblings. Neither matched, because:

  1. those entries pin the double-quote form (ssh voffice1 "maas ...) while the commands used single quotes; and
  2. they pin the maas admin profile, while the new region needs maas vr1-dc0-region.

So the call fell through to the classifier, which blocked it as unmatched. Operator-approved rules were added to .claude/settings.local.json (gitignored -- committed team policy in settings.json is unchanged): profile-agnostic ask rules Bash(ssh *'maas * update*) plus the double-quote twin and the ipranges pair, and read-only allow rules for pgrep / ps -ef over ssh. The operator chose ask over allow deliberately, keeping hard rule 3 intact for every MAAS mutation.

A NOTE ON THE RULE I ALMOST WROTE: the shape first proposed pinned maas admin * update*, mirroring the existing project rule. It would have matched step 1 and silently failed to match step 3, which runs against maas vr1-dc0-region -- leaving the cutover half done at exactly the two-servers boundary. Widened to maas * update* before use. An ask rule that is too broad only costs extra prompts; one that is too narrow costs a stall mid-cutover.

Revert. ssh voffice1 'maas vr1-dc0-region vlan update 0 0 dhcp_on=false' then ssh voffice1 'maas admin vlan update 4 0 dhcp_on=true primary_rack=7chphy', verifying by process at each step. Remove the added rules from .claude/settings.local.json.

Item 16 -- CANARY re-enrolment, and the ORDERING the migration actually needs

Canary chosen deliberately: xqqwdq / moral-salmon = vr1-dc0-storage-04, a storage node -- the least central of the ten if anything went wrong. The dc1 standup set the precedent of proving enlistment with one machine before touching the rest.

MEASURED SEQUENCE, and it is NOT the obvious one. A freshly-enlisted machine in a new region has NO power configuration, so MAAS cannot power-cycle it, so it CANNOT drive commissioning. Steps 1-5 below are each individually MEASURED, but see item 18: the sequence has NOT yet produced a Ready machine, so it is not a closed invariant and may need a further step. The order so far:

  1. machine delete <id> on the OLD region (the libvirt domain is untouched -- all 12 dc0 domains stayed defined throughout).
  2. Power the VM on by hand (virsh -c qemu:///system start <domain> on the rack). This step exists only because the new region has no power driver for a machine it has never seen. Nothing else can start it.
  3. It PXEs, leases from the new region's dynamic range, and SELF-ENLISTS.
  4. Set power config BEFORE expecting commissioning to progress.
  5. machine abort (-> New), then machine commission. MAAS now powers it on itself.

Skipping step 4 leaves the machine wedged in Commissioning producing ZERO events -- which is exactly what happened here and looks alarming until the cause is named. The enlistment ephemeral powers the node off when it finishes; with no power driver MAAS can never bring it back, so it waits forever. machine abort is the documented way out.

The power command needs BOTH URIs, and maas-node-power.sh already supports it:

MAAS_PROFILE=vr1-dc0-region \
VIRSH_URI=qemu+ssh://jessea123@172.31.0.2/system \    # ENUMERATE from voffice1 (transit)
  bash scripts/maas-node-power.sh --commit \
  qemu+ssh://jessea123@10.12.8.2/system vr1-dc0       # STORE what the REGION dials

VIRSH_URI defaults to the power address but is env-overridable, which is precisely the separation this migration needs: voffice1 can reach the rack only over the transit /30, while the region reaches it only over metal-admin. Conflating them would either fail to enumerate or store an address the region cannot dial.

Result: [ok] civil-bug -> vr1-dc0-storage-04 (power state: off) -- a REAL query-power-state, which proves the region can drive the rack's libvirt over metal-admin using the SEC-012 key installed in its snap (item 5). Power path verified end to end, not inferred. MAAS then powered the VM on for commissioning by itself (vm=running).

IDENTITY PRESERVED, verified by the stable key: the re-enlisted machine carries boot MAC 52:54:00:2b:ed:ab, matching lib-hosts.sh HOST_BOOT_MAC[vr1-dc0-storage-04] exactly. Its system_id changed (xqqwdq -> 6q4syf) and its hostname changed (moral-salmon -> civil-bug), which is why the pre-delete snapshot (docs/audit/dc0-premigration-machine-identity-20260730.txt) records the BOOT MAC as the identity and treats system_id and hostname as disposable.

Revert. Delete the record from the new region and re-enlist into Office1 by re-enabling its DHCP -- possible only while the domain and its pinned MACs survive, which they do.

Item 17 -- DIAGNOSTIC BLIND SPOT: the per-node serial log stopped capturing on 2026-07-21

Measured. /var/lib/libvirt/vr1/staging/vr1-dc0-storage-04-serial.log has mtime 2026-07-21 08:54:43 -- nine days stale. Today's boot wrote NOTHING to it, although the domain XML configures both <serial> and <console> with <log file='...' append='on'/>.

Why this matters more than it looks. The file still CONTAINS a complete, plausible boot: kernel command line, cloud-init stages, "All scripts successfully ran", and a clean reboot: Power down at 138s. I read that as today's boot and reported it as fact before checking the mtime. It was the Stage-4 commissioning of 2026-07-21, betrayed only by its internal dates and by DataSourceMAASLocal [http://10.12.8.2:5248/...] -- the OLD region's metadata path. An append-mode log that has silently stopped appending is worse than an absent one: tail returns confident, well-formed, wrong evidence.

Same class as two other misreadings this session (a +time=3 +tries=1 DNS probe called a capability gap; a head -5-truncated listener list called a missing TFTP bind). All three were instrument errors that produced a clean-looking answer. The rule that catches this one: check the mtime before trusting a log file as current.

LOGGED, NOT ACTIONED (hard rule 1). Not fixed here -- it is a diagnostic aid, not on the deploy path, and the 2026-07-22 dc1 build already recorded a related fault ("serial-log staging dir absent on dc1; queued to standup DoD"). Worth folding into that same DoD item: a serial log configured but not capturing is a standup defect, and it should be verified by mtime at each DC standup rather than assumed from the domain XML.

Item 18 -- RESOLVED: commissioning passed but landed New ONCE; re-issuing reached Ready

OUTCOME: the canary reached Ready at 22:05:33 on the third commission -- 6q4syf/civil-bug, cpu=8 mem=24576 (exact D-121 Option C storage-node shape), power=virsh/off, all six pinned MACs present. Item 16's ordering is therefore a WORKING sequence. What follows is the diagnosis, kept because it refutes two plausible theories and because the intermediate New will recur on the other nine.

MEASURED, from the surface I initially skipped. My first look at node-script-results read returned unparseable output and I moved on -- which is the "could not look is not nothing there" failure, committed against my own rule. Read properly, it is the only direct evidence and it is unambiguous:

result set type status detail
id=2 Commissioning Aborted the enlistment-time run I aborted deliberately
id=3 Commissioning Passed all 13 scripts exit 0; 30-maas-01-bmc-config Skipped (normal for a VM)
id=4 Testing Passed smartctl-validate Skipped on both virtio disks

Hardware discovery populated correctly: cpu_count=8, memory=24576, storage present -- exactly the D-121 Option C storage-node shape. So neither commissioning nor testing failed. The machine transitioned to New at 21:53:08, the same second the testing set completed.

REFUTED: the plausible theory that smartctl-validate against virtio disks was the fault. It did not fail -- it was Skipped, and its result set Passed. Had I re-run with testing_scripts=none as the first move, it would have "fixed" nothing and I would have learned nothing.

ROOT CAUSE: a commission issued immediately after an abort can land in New once. Attempt 2 (issued seconds after my abort) passed every script and still went to New. Attempt 3, issued under IDENTICAL conditions from a settled New, reached Ready in ~3.5 minutes -- matching the dc1 standup timing exactly. It was a ONE-OFF.

The repeat was justified as an EXPERIMENT, not a blind retry. It was the single move that distinguished "one-off" from "systematic", and those two have completely different next steps. Re-running the same thing is only defensible when the re-run is the discriminator; it was here, and it is not in general.

SECOND THEORY ALSO REFUTED, in parallel: a config delta between the two independently configured regions. All 13 commissioning-relevant settings compare IDENTICAL -- enlist_commissioning, commissioning_distro_series, default_osystem, default_distro_series, default_min_hwe_kernel, enable_third_party_drivers, default_storage_layout, node_timeout, enable_disk_erasing_on_release, max_node_commissioning_results, curtin_verbose, force_v1_network_yaml, enable_kernel_crash_dump.

STANDING CONSEQUENCE for the remaining nine: do not read a single New after a passing commission as a fault. Re-issue machine commission once. Only a SECOND New is evidence of a real problem.

Holding the other nine until the canary passed was the right call -- it kept an undiagnosed fault from being multiplied by ten while the return path (re-enable Office1 DHCP, re-enlist) was still cheap. They are now unblocked.

THE CARVE GAP IS NOW CONCRETE. At Ready the canary reads: enp1s0 correctly on fabric-0 (metal-admin -- it PXE'd there) with an AUTO link on 10.12.8.0/22, and enp2s0..enp6s0 each on a FRESH auto-created fabric (fabric-7..11) with link_up and no subnet. That is precisely the 60 NIC re-homes + 9 br-ex bridges + 54 statics task 7 owes, and there is now a stable Ready machine to build dc-node-carve.sh against.

Item 19 -- THE NEW REGION HAD ZERO SSH KEYS (would have broken juju bootstrap)

MEASURED: maas vr1-dc0-region sshkeys read -> count 0, while maas admin sshkeys read -> 1 (SHA256:iUYex2kmvtdGTPiMKRZ2Nfywqa8pWacVV1GEEx6sscI, vr1-office1-svc). A fresh maas init registers no keys, and nothing in the migration sequence adds them.

Why this matters far beyond convenience. MAAS injects its registered SSH keys into every machine it deploys. With none registered, deployed nodes would come up with NO authorized key -- and juju bootstrap needs direct client->machine SSH. That is exactly the failure signature that consumed three bootstrap attempts on 2026-07-30 ("juju then spent 15 minutes unable to SSH the machine"). Task 8 would have hit it again, and the symptom would have looked like a network fault rather than a missing key.

Found by accident, which is worth admitting: I was probing the stalled enlistment nodes and noticed sshd was open but I had no way in. The gap was never on my checklist. It should be a standing DC-standup item -- a new region's SSH key set is empty until someone fills it -- and it belongs next to the power-key item (prerequisite 5) in the phase-2 runbook.

FIXED: the SAME key Office1 carries was imported, deliberately rather than minting a new one -- docs/CURRENT-STATE.md records that deployed nodes carry the MAAS-registered vr1-office1-svc key, and the vr1-dc0-juju / node ssh aliases built earlier this session depend on that identity. Minting a per-DC node key is a defensible SEC-side improvement but would break the existing aliases and tooling, so it is LOGGED as a follow-up rather than done mid-migration.

Revert. maas vr1-dc0-region sshkeys delete <id>.

Item 20 -- enlistment does NOT scale to nine concurrent nodes; batch it

Measured sequence. The canary alone enlisted and commissioned in ~2 min. Nine started together produced: all 9 leasing as maas-enlisting-node, all 9 alive (ping + sshd open, CPU climbing), cloud-init output flowing until 22:12:46 -- and then nothing for 15 minutes while DHCP renewals continued. Zero machine records were created.

Everything downstream was RULED OUT by measurement, not assumed:

suspect measurement verdict
DC edge egress rack -> security.ubuntu.com 200 in 0.9s healthy
MAAS proxy (the only path nodes have -- metal-admin carries NO gateway) rack -> via 10.12.8.6:8000 200 in 1.3s healthy
region VM overload load 0.29, 5 GB free idle
nodes dead ping OK, ssh port OPEN, CPU climbing alive

A NOTE ON MY OWN INSTRUMENT, because I got this wrong twice in five minutes. I first called it stalled off a bad elapsed-time estimate (the last line was 90 seconds old, not minutes). Then, correcting too fast, I counted "104 maas-enlist lines in 3 minutes" and called it recovered -- but those were DHCP lease renewals, not cloud-init progress; filtering for cloud-init specifically returned 0. The stall was real, my first two readings of it were not. Counting lines from a broad grep is not the same as measuring the thing you care about.

ACTION: all nine stopped and restarted in batches of three. Not a fix for the root cause -- which is still uncharacterised beyond "concurrency" -- but the canary is direct evidence that low concurrency works, and the migration does not need the root cause to proceed. LOGGED for the dc1 standup and for Roosevelt: do not power on a whole rack of nodes for enlistment at once.

Item 21 -- BATCHED RE-ENROLMENT WORKS, and it confirms the ordering was the cause

Batch of three (control-01/02/03): enlisted in 2.3 minutes against a 15-minute stall with zero records for nine. Batching is the working procedure.

The more useful result: all three reached Ready on the FIRST commission. The canary needed a second; these did not. The difference is that their power config was set BEFORE commission was issued, whereas the canary was commissioned first and had power set afterwards. So the canary's extra commission was caused by the ordering, not by chance -- a stronger and more useful conclusion than the earlier "one-off", and it supersedes it.

PRECISION, correcting my own first wording of this item. I originally wrote that batch 1 went "straight to Ready with no New detour". That is wrong: I aborted them to New myself, to clear the enlistment-time wedge. EVERY machine passes through New -- either by self-settling there (batch 2 did, before I could abort) or by an explicit abort. The real distinction is whether the commission ISSUED FROM New succeeds on the first try, and that is what power_type decides.

REVISED per-node procedure, now evidenced twice:

delete (old region) -> virsh start -> self-enlist -> SET POWER -> abort -> commission

with the emphasis on power BEFORE commission. Issue abort only to clear the wedge left by the enlistment-time commissioning; once power is set first, commission completes in one pass (~3.5 min for three concurrent).

Standing guidance for dc1 and Roosevelt: enlist in batches of three, and never issue commission on a machine whose power_type is still unset.

Item 22 -- role tags: covered by existing tooling, ordering enforced by the tool itself

The new region carries only openstack-vr1-dc0 (created by dc-region-topology.sh) and MAAS's own virtual. The control / compute / storage / juju-controller-vr1-dc0 tags do NOT exist yet -- but scripts/maas-role-tags.sh CREATES them (ROLES at :38, ROLES+=("juju-controller-$SITE") at :125, creation at :184), so task 7's tagging needs no new tool.

Its own refusal enforces the correct ordering: it refuses if any pinned boot MAC has no MAAS record, so it cannot run until all ten machines are enlisted. That is the gate working in the right direction -- no additional sequencing discipline required from the operator.

Item 23 -- ALL TEN MACHINES Ready IN THE PER-DC REGION (task 6 COMPLETE)

Capture docs/audit/dc0-region-machines-ready-20260730.txt. Enlisted as canary + three batches of three; each batch virsh start -> self-enlist -> maas-node-power.sh --commit -> commission.

Shapes match D-121 Option C exactly, and every machine is matched to its libvirt domain by PINNED BOOT MAC rather than by hostname or system_id (both of which were re-minted):

role count shape machines
control 3 16cpu / 65536 mint-roughy, gentle-raven, square-insect
compute 2 12cpu / 49152 pure-condor, able-puma
storage 4 8cpu / 24576 amused-corgi, civil-bug, known-alpaca, loyal-oyster
juju controller 1 4cpu / 8192 subtle-grouse

All ten carry 6 interfaces. NAMED GATE: maas-role-tags.sh check vr1-dc0 -> 0 role tag(s) missing, 0 node(s) needing a tag, 0 node(s) not in MAAS, PASS, EXIT 0.

A GAP THAT ONLY A DIFF AGAINST THE PRE-MIGRATION SNAPSHOT WOULD CATCH. maas-role-tags.sh reported a clean PASS while the machines were still MISSING the openstack-vr1-dc0 placement tag -- because that tag is not in its ROLES set (control/compute/storage + juju-controller-<site>), and the recon had already noted it has no usable creator anywhere in the repo. A green gate on an incomplete state, exactly the class this repo keeps finding.

Consequences had it shipped: bundle.yaml places units on tags=openstack-vr1-dc0, and dc-node-v6-carve.py:89-93 DIES if no machine carries openstack-<site> -- so both the deploy and the v6 carve would have failed, the second with a refusal that names the tag and the first with an allocation failure that does not.

Applied to the NINE role nodes and deliberately NOT to the juju controller. That asymmetry is not a choice -- it reproduces the Office1 pre-migration snapshot row-for-row (ace-robin = control,openstack-vr1-dc0,virtual; moved-troll = juju-controller-vr1-dc0,virtual). The pre-migration identity snapshot earned its keep here: without it there was nothing to diff the rebuilt region against, and the missing tag would have surfaced as a deploy failure.

Revert. maas vr1-dc0-region tag update-nodes openstack-vr1-dc0 remove=<id> per node.

Item 24 -- plane IPAM completed with EXISTING tooling (no new code)

NAMED GATE: MAAS_PROFILE=vr1-dc0-region dc-plane-ipam.sh check vr1-dc0 -> pass 24 / fail 0, EXIT 0 (capture docs/audit/dc0-region-plane-ipam-20260730.txt). Before: pass 7 / fail 17.

Two applies, both idempotent, dry-run first, every write read back:

  • reserve vr1-dc0 --commit -> 13 applied, 0 errors: the D-134 utility (.4-.49) and VIP (.50-.99) bands on all six planes, plus the FIP pool 10.12.5.0-10.12.7.254 that phase-04-network-verify asserts.
  • carve-v6 vr1-dc0 --commit -> 5 applied, 0 errors: the missing v6 plane subnets, each created on the SAME MAAS vlan as its v4 twin and READ BACK on it.

The tool's own forced sequencing worked and is worth noting as a design that paid off: the reserve run SKIPPED every v6 band with not in MAAS yet -- run carve-v6 first, rather than failing or silently doing nothing. Running the two in the wrong order is therefore safe.

This is also the step that justifies the tunnel architecture (item 2): carve-v6 derives its prefixes from the NetBox apex record, which only voffice1 can reach, while the target region is only reachable from the rack. Neither host alone could have run it.

Revert. Delete the 5 v6 subnets and the 13 ipranges on the new region.