Batch 1 all Ready; the New detour was ORDERING, not chance ...

Batch of three enlisted in 2.3 min against a 15-minute zero-record stall for
nine, and all three went straight to Ready with no New detour -- because
power was set BEFORE commission was issued, where the canary was
commissioned first and powered afterwards.

That supersedes the earlier 'one-off' reading with a stronger and more
useful conclusion: the detour is caused by ordering. Revised standing
procedure is delete -> virsh start -> self-enlist -> SET POWER -> abort ->
commission, and never commission a machine whose power_type is unset.

Region now holds 4 machines all Ready, each matched to its libvirt domain by
pinned MAC.

Role tags need no new tool: maas-role-tags.sh creates the four missing ones
and refuses while any pinned MAC lacks a record, so it enforces the ordering
itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Nine records deleted; found a fresh region has ZERO SSH keys ...

All nine remaining Office1 dc0 records deleted individually. A scripted loop
over them was correctly refused by the guard -- hard rule 3 says destructive
steps are never batched, and I had written exactly that. Office1 is now at
13 machines with zero dc0-tagged.

Found by accident while probing stalled nodes: the new region has no SSH
keys at all. MAAS injects registered keys into every machine it deploys and
juju bootstrap needs client->machine SSH, so this is the same signature that
consumed three bootstrap attempts today and would have recurred at task 8
looking like a network fault. Imported the same vr1-office1-svc key Office1
carries, deliberately rather than minting a per-DC one, because the node ssh
aliases built this session depend on that identity.

Enlistment does not scale to nine concurrent nodes: all alive and leasing,
cloud-init silent for 15 minutes, zero records created. Egress, MAAS proxy,
region load and node liveness all ruled out by measurement. Restarted in
batches of three.

I misread the stall twice in five minutes -- first off a bad elapsed-time
estimate, then by counting 104 enlist lines that were DHCP renewals rather
than cloud-init progress. Counting lines from a broad grep is not measuring
the thing you care about.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Canary reaches Ready; two theories refuted by measurement ...

6q4syf/civil-bug Ready at 22:05:33 -- cpu=8 mem=24576 (exact D-121 Option C
storage shape), power=virsh, all six pinned MACs. The re-enrolment ordering
is now a working sequence.

Root cause of the earlier New: a commission issued immediately after an
abort can land in New once; re-issuing from a settled New reached Ready in
~3.5 min, matching the dc1 timing. One-off, not systematic. The repeat was
justified as the experiment that distinguished those two cases -- not as a
blind retry.

Refuted by measurement, recorded so neither is re-proposed: smartctl on
virtio disks (it was Skipped and its result set Passed, so
testing_scripts=none would have taught nothing), and a config delta between
the independently-configured regions (all 13 commissioning-relevant
settings identical).

Standing consequence for the remaining nine: a single New after a passing
commission is not a fault -- re-issue once; only a second New is evidence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Correct an over-claim: the re-enrolment ordering has NOT reached Ready ...

I wrote the five-step re-enrolment sequence into CURRENT-STATE and the
changelog as a standing invariant. Each step is individually measured, but
the sequence has not yet produced a Ready machine, so calling it a working
invariant was ahead of the evidence. Reworded to separate what is proven
(delete leaves the domain; manual power-on is required; self-enlistment
carries the pinned MAC; power config is a prerequisite, and without it the
machine wedges in Commissioning with zero events; abort+commission has MAAS
power it on) from what is not (that it reaches Ready).

Opens item 18 with the real diagnostic, which I had skipped:
node-script-results returned unparseable on first read and I moved on --
"could not look" treated as "nothing there", against my own rule. Read
properly it is unambiguous: commissioning result set Passed with all 13
scripts exit 0, testing set Passed with smartctl-validate Skipped on both
virtio disks, hardware discovery correct at 8 cpu / 24576 MiB. Neither
phase failed; the machine went to New anyway.

This REFUTES the smartctl-on-virtio theory. Re-running with
testing_scripts=none would have fixed nothing and taught nothing.

The other nine machines stay untouched until the canary is Ready, and
dc-node-carve.sh stays unwritten because a New machine's interfaces are
rediscovered on the next commission.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Canary re-enrolment: the ordering a fresh region actually needs ...

Canary xqqwdq/moral-salmon = vr1-dc0-storage-04 (a storage node, least
central of the ten). Office1 23 -> 22; it re-enlisted into the new region
carrying boot MAC 52:54:00:2b:ed:ab -- the machine that arrived is the
machine that left. system_id AND hostname are both re-minted
(xqqwdq -> 6q4syf, moral-salmon -> civil-bug), so the boot MAC is the only
stable key; the pre-delete snapshot records it as the identity.

MEASURED ORDERING, not the obvious one: a freshly-enlisted machine has no
power config, so MAAS cannot power-cycle it and cannot drive commissioning.
delete -> power on BY HAND via virsh -> self-enlist -> SET POWER -> abort ->
commission. Skipping the power step wedges the machine in Commissioning
producing zero events, because the enlistment ephemeral powers the node off
and nothing can bring it back.

maas-node-power.sh already supports the split the migration needs: VIRSH_URI
enumerates from voffice1 over the transit /30 while the positional address
stores what the REGION dials over metal-admin. Result was a real
query-power-state, proving the region drives the rack's libvirt with the
SEC-012 key in its snap.

Found: the per-node serial log stopped capturing on 2026-07-21 while the
domain XML still configures append-mode logging. It contains a complete,
plausible boot that I read as current and reported as fact before checking
the mtime. An append log that silently stopped appending returns confident,
well-formed, wrong evidence. Logged, not actioned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

DHCP handover EXECUTED: dc0 metal-admin now served by the DC-local region ...

Ruled order, region identity proven first. Office1 vlan update 4 0
dhcp_on=false -> read back False; verified STOPPED BY PROCESS (pgrep -c
dhcpd = 0 on the rack, because MAAS self-report is not evidence -- the
2026-07-20 Temporal incident class); new region vlan update 0 0
dhcp_on=true primary_rack=c3aqh8 -> read back True on fabric-0 / space
metal-admin; verified RUNNING BY PROCESS on hot-kid with the rack
re-checked at 0. Exactly one DHCP server on the segment. All 10 machines
Ready and powered off throughout.

Recorded so it is not later read as a fault: the new region also starts
dhcpd6 where the Office1 rack ran v4 only. Its v6 metal-admin subnet has
NO dynamic range -- only rfc-4291-2.6.1 and assigned-ip/reserved, same as
the old region -- so it has nothing to hand out and nodes keep their ruled
statics.

The permission wall was a rule that FAILED TO MATCH, not a classifier
fault. Existing ask rules pin the double-quote form while the commands used
single quotes, and pin the maas admin profile while the new region needs
maas vr1-dc0-region. Operator-approved ask rules added to
settings.local.json (gitignored; committed team policy unchanged),
profile-agnostic so both regions match, plus read-only pgrep/ps allow
rules. The first rule shape I proposed pinned maas admin and would have
matched step 1 while silently missing step 3 -- stalling the cutover at
exactly the two-servers boundary.

Capture docs/audit/dc0-dhcp-handover-20260730.txt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Session record: dc0 region topology 40/0; cutover blocked with exact commands ...

Records the fabric-rename remediation (changelog item 12), the two
pre-cutover checks (hot-kid IS eligible as primary_rack for fabric-0 -- had
it not been, DHCP-off would have succeeded and DHCP-on failed, leaving the
segment with no DHCP and ten machines about to PXE), the measured NIC ->
plane order, and the exact unblock sequence with every parameter
re-resolved live.

hot-kid is c3aqh8 in the new region and tw7ptw in Office1 -- the same
machine, two ids.

Capture updated with the corrected 40 passed / 0 failed, exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

dc0 region plane topology BUILT and verified (39/0); DHCP cutover blocked ...

dc-region-topology.sh applied to the new dc0 region: 5 named plane fabrics,
6 spaces, 4 plane subnets, 10.12.4.0/22 MOVED off the auto fabric-1 onto
vr1-dc0-provider-public, 6 VLAN->space bindings, openstack-vr1-dc0 tag.
Named check now reads 39 passed / 0 failed, exit 0.

metal-admin service config set: dns_servers=10.12.8.6, allow_dns=false,
D-134 dynamic range .201-.254, subnet resolved by CIDR.

The DNS target was measured rather than carried over, and my first reading
was wrong: a +time=3 +tries=1 probe suggested the new region's BIND could
not resolve externally. That was a cold-cache timeout. At +time=5 +tries=3
it answers archive.ubuntu.com, streams.canonical.com and api.snapcraft.io
with flags qr rd ra. Pointing node DNS at the DC-local region is correct --
it is authoritative for the zone the migrated nodes will live in and drops
the cross-fiber dependency D-132 q1 exists to remove.

Measured in passing, and worth keeping: VLAN row id 5005 is dc0's
metal-admin VLAN in Office1 and vr1-dc0-data-tenant in the new region.
Same integer, different DC plane.

The DHCP cutover was REFUSED by the harness classifier and was not retried
in an altered shape. Verified read-only afterwards that nothing is
half-applied: Office1 still reads dhcp_on=True primary_rack=7chphy and the
rack still runs exactly one dhcpd on virbr2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Per-region power address, region power-key tool, and two red gates fixed ...

lib-hosts.sh: MAAS power ops originate from the REGION, so the reachable
rack address depends on which region dials. Measured from vr1-dc0-maas-01:
the rack's transit leg 172.31.0.2:22 is CLOSED, its metal-admin leg
10.12.8.2:22 is OPEN. Adds VIRSH_POWER_ADDRESS_FROM_{OFFICE1,DCREGION} for
both DCs (dc1's 10.12.68.2 measured on the dc1 rack, not inferred from
dc0's .2). VIRSH_POWER_ADDRESS still aliases the Office1 form so existing
callers are byte-for-byte unchanged. dc-selector 81 checks; 4 mutations
each killed tests.

New scripts/maas-region-power-key.sh (harness 44/44) installs and verifies
the per-DC MAAS->libvirt key inside a region's snap, ending on a real virsh
call rather than on config. Runbook prerequisite 5 carried this as prose,
and prose is not a gate -- it must also be re-asserted after every snap
refresh. LIVE at dc0: 9/9 exit 0, virsh enumerates 12 domains. The key is
the SEC-012 dc0 key already authorized on that rack, so per-DC isolation
holds.

Three defects its own testing found: ssh-keygen -lf reads an adjacent .pub
and reports ITS fingerprint; a snap's data root (/var/snap) and exec root
(/snap) are not interchangeable; the snap's virsh must run inside the snap
runtime or it fails on missing libraries.

tests/node-vm T8/T9 were RED at HEAD before any of this session's work
(proven by stashing and re-running): commits 086c827 + 447315f added the
region VMs, moving the dc0 root to 11 macs lists / 66 MAC literals against
assertions reading 10 / 60, and the gauntlet was not re-run. Re-pointed to
the new invariant, reason recorded in-file, re-proven able to fail.

HARNESS-MANIFEST re-recorded (91) for the two new harnesses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Region-resolution gate: maas-profile-assert.sh + named dc0 region profile ...

The wrong-REGION hazard is the migration's unrecoverable failure mode: 13
repo scripts default to MAAS_PROFILE=admin, which on voffice1 resolved to
the OFFICE1 region. Against the wrong profile a carve recreate is a silent
no-op that prints PASS, and machine delete destroys the real record.

New scripts/maas-profile-assert.sh asserts which region a profile resolves
to by rack-controller IDENTITY (a machine count is not proof -- two regions
can hold the same number). Harness 20/20; mutation pass found one of its own
assertions could not fail (nameless rack entries were caught by a different
branch) and it was made branch-specific.

Live both directions: admin -> 23 machines / office1 trio (exit 0);
vr1-dc0-region -> 0 machines / hot-kid (exit 0); admin asserted as hot-kid
(exit 1). Capture docs/audit/dc0-region-profile-assert-20260730.txt.

Tunnel voffice1 -> rack -> 10.12.8.6:5240 because no single host has repo +
CLI + new-region reach + NetBox apex reach. Application-layer and
rack-originated, so SEC-010 is not punctured; nothing was rerouted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf