| 2026-07-30 |
Batch 1 all Ready; the New detour was ORDERING, not chance
...
Batch of three enlisted in 2.3 min against a 15-minute zero-record stall for
nine, and all three went straight to Ready with no New detour -- because
power was set BEFORE commission was issued, where the canary was
commissioned first and powered afterwards.
That supersedes the earlier 'one-off' reading with a stronger and more
useful conclusion: the detour is caused by ordering. Revised standing
procedure is delete -> virsh start -> self-enlist -> SET POWER -> abort ->
commission, and never commission a machine whose power_type is unset.
Region now holds 4 machines all Ready, each matched to its libvirt domain by
pinned MAC.
Role tags need no new tool: maas-role-tags.sh creates the four missing ones
and refuses while any pinned MAC lacks a record, so it enforces the ordering
itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Nine records deleted; found a fresh region has ZERO SSH keys
...
All nine remaining Office1 dc0 records deleted individually. A scripted loop
over them was correctly refused by the guard -- hard rule 3 says destructive
steps are never batched, and I had written exactly that. Office1 is now at
13 machines with zero dc0-tagged.
Found by accident while probing stalled nodes: the new region has no SSH
keys at all. MAAS injects registered keys into every machine it deploys and
juju bootstrap needs client->machine SSH, so this is the same signature that
consumed three bootstrap attempts today and would have recurred at task 8
looking like a network fault. Imported the same vr1-office1-svc key Office1
carries, deliberately rather than minting a per-DC one, because the node ssh
aliases built this session depend on that identity.
Enlistment does not scale to nine concurrent nodes: all alive and leasing,
cloud-init silent for 15 minutes, zero records created. Egress, MAAS proxy,
region load and node liveness all ruled out by measurement. Restarted in
batches of three.
I misread the stall twice in five minutes -- first off a bad elapsed-time
estimate, then by counting 104 enlist lines that were DHCP renewals rather
than cloud-init progress. Counting lines from a broad grep is not measuring
the thing you care about.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

lib-hosts: the node carve is ROLE-DEPENDENT, not uniform
...
Measured from the pre-migration capture. Role nodes carve all six planes
plus an OVS br-ex parented on enp2s0 carrying the provider-public static,
with enp2s0 itself holding no L3 link. The juju controller carves only TWO
planes -- enp1s0 metal-admin and enp2s0 provider-public as a raw NIC WITH
the gateway -- has no br-ex, and leaves enp3s0..enp6s0 on their
auto-created VLANs.
Evidence: 7n87bt (controller) reads enp3-6 on vlans 5200-5203, which are
auto fabrics; 8mtpxq (storage-02) reads enp3-6 on 5190-5193, the named
planes, plus br-ex.
This matters because assuming uniformity fails in both directions: it would
either strand the controller without a default route -- the under-carve that
cost three bootstrap attempts on 2026-07-30 -- or give it four legs it must
not have. The node carve has no repo tool yet, so this was about to be
re-derived by hand.
dc-selector 95 checks; the depths-must-differ guard proven able to fail, and
the controller-presence assertion checked inside a per-DC subshell because
the main shell still holds the flat VR0 fleet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Canary reaches Ready; two theories refuted by measurement
...
6q4syf/civil-bug Ready at 22:05:33 -- cpu=8 mem=24576 (exact D-121 Option C
storage shape), power=virsh, all six pinned MACs. The re-enrolment ordering
is now a working sequence.
Root cause of the earlier New: a commission issued immediately after an
abort can land in New once; re-issuing from a settled New reached Ready in
~3.5 min, matching the dc1 timing. One-off, not systematic. The repeat was
justified as the experiment that distinguished those two cases -- not as a
blind retry.
Refuted by measurement, recorded so neither is re-proposed: smartctl on
virtio disks (it was Skipped and its result set Passed, so
testing_scripts=none would have taught nothing), and a config delta between
the independently-configured regions (all 13 commissioning-relevant
settings identical).
Standing consequence for the remaining nine: a single New after a passing
commission is not a fault -- re-issue once; only a second New is evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Correct an over-claim: the re-enrolment ordering has NOT reached Ready
...
I wrote the five-step re-enrolment sequence into CURRENT-STATE and the
changelog as a standing invariant. Each step is individually measured, but
the sequence has not yet produced a Ready machine, so calling it a working
invariant was ahead of the evidence. Reworded to separate what is proven
(delete leaves the domain; manual power-on is required; self-enlistment
carries the pinned MAC; power config is a prerequisite, and without it the
machine wedges in Commissioning with zero events; abort+commission has MAAS
power it on) from what is not (that it reaches Ready).
Opens item 18 with the real diagnostic, which I had skipped:
node-script-results returned unparseable on first read and I moved on --
"could not look" treated as "nothing there", against my own rule. Read
properly it is unambiguous: commissioning result set Passed with all 13
scripts exit 0, testing set Passed with smartctl-validate Skipped on both
virtio disks, hardware discovery correct at 8 cpu / 24576 MiB. Neither
phase failed; the machine went to New anyway.
This REFUTES the smartctl-on-virtio theory. Re-running with
testing_scripts=none would have fixed nothing and taught nothing.
The other nine machines stay untouched until the canary is Ready, and
dc-node-carve.sh stays unwritten because a New machine's interfaces are
rediscovered on the next commission.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Canary re-enrolment: the ordering a fresh region actually needs
...
Canary xqqwdq/moral-salmon = vr1-dc0-storage-04 (a storage node, least
central of the ten). Office1 23 -> 22; it re-enlisted into the new region
carrying boot MAC 52:54:00:2b:ed:ab -- the machine that arrived is the
machine that left. system_id AND hostname are both re-minted
(xqqwdq -> 6q4syf, moral-salmon -> civil-bug), so the boot MAC is the only
stable key; the pre-delete snapshot records it as the identity.
MEASURED ORDERING, not the obvious one: a freshly-enlisted machine has no
power config, so MAAS cannot power-cycle it and cannot drive commissioning.
delete -> power on BY HAND via virsh -> self-enlist -> SET POWER -> abort ->
commission. Skipping the power step wedges the machine in Commissioning
producing zero events, because the enlistment ephemeral powers the node off
and nothing can bring it back.
maas-node-power.sh already supports the split the migration needs: VIRSH_URI
enumerates from voffice1 over the transit /30 while the positional address
stores what the REGION dials over metal-admin. Result was a real
query-power-state, proving the region drives the rack's libvirt with the
SEC-012 key in its snap.
Found: the per-node serial log stopped capturing on 2026-07-21 while the
domain XML still configures append-mode logging. It contains a complete,
plausible boot that I read as current and reported as fact before checking
the mtime. An append log that silently stopped appending returns confident,
well-formed, wrong evidence. Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

DHCP handover EXECUTED: dc0 metal-admin now served by the DC-local region
...
Ruled order, region identity proven first. Office1 vlan update 4 0
dhcp_on=false -> read back False; verified STOPPED BY PROCESS (pgrep -c
dhcpd = 0 on the rack, because MAAS self-report is not evidence -- the
2026-07-20 Temporal incident class); new region vlan update 0 0
dhcp_on=true primary_rack=c3aqh8 -> read back True on fabric-0 / space
metal-admin; verified RUNNING BY PROCESS on hot-kid with the rack
re-checked at 0. Exactly one DHCP server on the segment. All 10 machines
Ready and powered off throughout.
Recorded so it is not later read as a fault: the new region also starts
dhcpd6 where the Office1 rack ran v4 only. Its v6 metal-admin subnet has
NO dynamic range -- only rfc-4291-2.6.1 and assigned-ip/reserved, same as
the old region -- so it has nothing to hand out and nodes keep their ruled
statics.
The permission wall was a rule that FAILED TO MATCH, not a classifier
fault. Existing ask rules pin the double-quote form while the commands used
single quotes, and pin the maas admin profile while the new region needs
maas vr1-dc0-region. Operator-approved ask rules added to
settings.local.json (gitignored; committed team policy unchanged),
profile-agnostic so both regions match, plus read-only pgrep/ps allow
rules. The first rule shape I proposed pinned maas admin and would have
matched step 1 while silently missing step 3 -- stalling the cutover at
exactly the two-servers boundary.
Capture docs/audit/dc0-dhcp-handover-20260730.txt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 bookend: dc0 region topology 40/0; cutover blocked on a permission wall
...
Ledger at 265 lines, under the 300 cap. Gauntlet ALL GREEN (92),
repo-lint 0 fail. Stage 5 remains OPEN and blocked; the exact 4-command
unblock sequence with live-resolved parameters is in the changelog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Session record: dc0 region topology 40/0; cutover blocked with exact commands
...
Records the fabric-rename remediation (changelog item 12), the two
pre-cutover checks (hot-kid IS eligible as primary_rack for fabric-0 -- had
it not been, DHCP-off would have succeeded and DHCP-on failed, leaving the
segment with no DHCP and ten machines about to PXE), the measured NIC ->
plane order, and the exact unblock sequence with every parameter
re-resolved live.
hot-kid is c3aqh8 in the new region and tw7ptw in Office1 -- the same
machine, two ids.
Capture updated with the corrected 40 passed / 0 failed, exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

lib-hosts: record the MEASURED VR1 node NIC -> plane order
...
The substrate roots pin six MACs per node and the module attaches them in
list order, so macs[i] is enp<i+1>s0. Verified end to end on
vr1-dc0-control-01 against the live carve: enp1s0 metal-admin (PXE),
enp2s0 provider-public (br-ex parent), enp3s0 metal-internal, enp4s0
data-tenant, enp5s0 storage, enp6s0 replication.
This is NOT lib-net.sh's PLANE_CIDRS order, which starts with
provider-public. Anything walking PLANE_CIDRS positionally to place NICs
swaps enp1s0 and enp2s0 and puts the PXE leg on provider-public, stranding
commissioning on a plane with no DHCP. The node carve has no repo tool yet,
so this ordering was about to be re-derived by hand; recording it measured,
with a guard that the two orders differ, is the cheap half of that tool.
dc-selector 88 checks; the trap guard proven able to fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc-node-v6-carve.py: honour MAAS_PROFILE; refuse instead of tracebacking
...
The script was env-blind -- default 'admin', --profile only, no environment
read anywhere. Every bash tool here honours MAAS_PROFILE, so
`export MAAS_PROFILE=vr1-dc0-region` left THIS tool silently targeting the
Office1 region. During the per-DC region migration that is the
unrecoverable direction: against the wrong region a carve is an idempotent
no-op that prints PASS while the region you meant stays untouched.
It now reads MAAS_PROFILE, keeps --profile as an override, and prints which
region it targets and where that came from -- a capture that does not name
the profile cannot be audited for wrong-region work.
An absent maas CLI raised FileNotFoundError and exited 1 with a traceback,
indistinguishable from a real failure and contrary to this script's own
REFUSE contract. It now refuses with exit 2.
Harness 14/14 (was 9); both fixes proven able to fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc-region-topology: RENAME the auto fabric instead of moving the subnet
...
Moving a subnet between fabrics does NOT bring interface links along. The
first apply created vr1-dc0-provider-public and moved 10.12.4.0/22 onto it,
leaving the region VM's enp2s0 holding a static 10.12.4.6 on a VLAN whose
subnet had left -- the region's provider-public leg, which it needs for
image sync, and the same under-carve class that cost three bootstrap
attempts on the Juju controller. Live connectivity was unaffected because
the OS netplan is independent of MAAS's model, so every other assertion
passed 39/39 with the model inconsistent.
Renaming the auto fabric that already carries the subnet moves nothing. A
MOVE is kept only for the case where the fabric carries other planes, and
now prints a warning that links do not follow.
Adds the assertion that would have caught it: every interface link must be
on its subnet's VLAN. Harness 47/47; the rename path and the stranded-link
check each proven able to fail.
Live remediation already applied to the dc0 region: subnet returned to
vlan 5002, the wrongly-created fabric deleted, fabric-1 renamed, space
re-bound. enp2s0 now reads fabric=vr1-dc0-provider-public with its static
intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc0 region plane topology BUILT and verified (39/0); DHCP cutover blocked
...
dc-region-topology.sh applied to the new dc0 region: 5 named plane fabrics,
6 spaces, 4 plane subnets, 10.12.4.0/22 MOVED off the auto fabric-1 onto
vr1-dc0-provider-public, 6 VLAN->space bindings, openstack-vr1-dc0 tag.
Named check now reads 39 passed / 0 failed, exit 0.
metal-admin service config set: dns_servers=10.12.8.6, allow_dns=false,
D-134 dynamic range .201-.254, subnet resolved by CIDR.
The DNS target was measured rather than carried over, and my first reading
was wrong: a +time=3 +tries=1 probe suggested the new region's BIND could
not resolve externally. That was a cold-cache timeout. At +time=5 +tries=3
it answers archive.ubuntu.com, streams.canonical.com and api.snapcraft.io
with flags qr rd ra. Pointing node DNS at the DC-local region is correct --
it is authoritative for the zone the migrated nodes will live in and drops
the cross-fiber dependency D-132 q1 exists to remove.
Measured in passing, and worth keeping: VLAN row id 5005 is dc0's
metal-admin VLAN in Office1 and vr1-dc0-data-tenant in the new region.
Same integer, different DC plane.
The DHCP cutover was REFUSED by the harness classifier and was not retried
in an altered shape. Verified read-only afterwards that nothing is
half-applied: Office1 still reads dhcp_on=True primary_rack=7chphy and the
rack still runs exactly one dhcpd on virbr2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-region-topology: explicit dry-run placeholders instead of empty ids
...
An empty `vlan=` in a dry run reads like a defect and would hide a real
one. On a from-scratch build the target fabric does not exist yet when the
MOVE and space-bind lines are planned, so their ids cannot resolve; they
now print <vlan-of-NAME> / <fabric-for-PLANE>. Four assertions added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc-region-topology.sh: the plane-topology builder the repo never had
...
A read-only survey found THREE dc0 carve components with no repo tool at
all -- the named plane fabrics, the six v4 plane subnets, and the site
placement tag were created ad-hoc in the Stage-4 window, with the mutations
landing only in ~/as-executed/2026-07-23-stage4-carve.log, which is not in
the repo. When D-132 q1 required rebuilding that carve on a new region
there was nothing to re-run. This closes the gap so dc1 and every Roosevelt
DC standup rebuild from a tool rather than a transcript.
check/apply over: five named plane fabrics, six spaces, six v4 plane
subnets on the right VLANs with the ruled gateways, VLAN->space bindings,
and the openstack-<site> tag. Expected values derive from lib-net.sh;
fabric names derive from the site token. Numeric MAAS ids are resolved
per-run by name or CIDR and never persisted -- the pre-migration capture's
id= values are database row ids, not 802.1Q tags, and nothing reads them.
metal-admin deliberately keeps MAAS's own auto-created fabric: it is a
discovery artifact created when the rack registers, Juju binds spaces
rather than fabrics, and renaming it would fight MAAS for ownership of its
own object for no functional gain. The check asserts only that it does not
share a fabric with a named plane.
Every mutating path is gated by maas-profile-assert.sh first -- against
the wrong profile this apply is an idempotent no-op that prints PASS.
--profile has NO default, because the repo-wide default 'admin' resolves
to the Office1 region.
Harness 35/35; 4 mutations each killed tests (region gate removed, profile
defaulted to admin, space-binding assertion, fabric-placement assertion).
Fixtures corrected against a live read: MAAS reports an unbound VLAN as
the string "undefined", not null, so the first fixtures graded a payload
MAAS never emits.
Live read-only check against the new dc0 region: 5 passed, 22 failed,
exit 1 -- the expected shape of a region that has only auto-discovered its
own two NICs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Per-region power address, region power-key tool, and two red gates fixed
...
lib-hosts.sh: MAAS power ops originate from the REGION, so the reachable
rack address depends on which region dials. Measured from vr1-dc0-maas-01:
the rack's transit leg 172.31.0.2:22 is CLOSED, its metal-admin leg
10.12.8.2:22 is OPEN. Adds VIRSH_POWER_ADDRESS_FROM_{OFFICE1,DCREGION} for
both DCs (dc1's 10.12.68.2 measured on the dc1 rack, not inferred from
dc0's .2). VIRSH_POWER_ADDRESS still aliases the Office1 form so existing
callers are byte-for-byte unchanged. dc-selector 81 checks; 4 mutations
each killed tests.
New scripts/maas-region-power-key.sh (harness 44/44) installs and verifies
the per-DC MAAS->libvirt key inside a region's snap, ending on a real virsh
call rather than on config. Runbook prerequisite 5 carried this as prose,
and prose is not a gate -- it must also be re-asserted after every snap
refresh. LIVE at dc0: 9/9 exit 0, virsh enumerates 12 domains. The key is
the SEC-012 dc0 key already authorized on that rack, so per-DC isolation
holds.
Three defects its own testing found: ssh-keygen -lf reads an adjacent .pub
and reports ITS fingerprint; a snap's data root (/var/snap) and exec root
(/snap) are not interchangeable; the snap's virsh must run inside the snap
runtime or it fails on missing libraries.
tests/node-vm T8/T9 were RED at HEAD before any of this session's work
(proven by stashing and re-running): commits 086c827 + 447315f added the
region VMs, moving the dc0 root to 11 macs lists / 66 MAC literals against
assertions reading 10 / 60, and the gauntlet was not re-run. Re-pointed to
the new invariant, reason recorded in-file, re-proven able to fail.
HARNESS-MANIFEST re-recorded (91) for the two new harnesses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Region-resolution gate: maas-profile-assert.sh + named dc0 region profile
...
The wrong-REGION hazard is the migration's unrecoverable failure mode: 13
repo scripts default to MAAS_PROFILE=admin, which on voffice1 resolved to
the OFFICE1 region. Against the wrong profile a carve recreate is a silent
no-op that prints PASS, and machine delete destroys the real record.
New scripts/maas-profile-assert.sh asserts which region a profile resolves
to by rack-controller IDENTITY (a machine count is not proof -- two regions
can hold the same number). Harness 20/20; mutation pass found one of its own
assertions could not fail (nameless rack entries were caught by a different
branch) and it was made branch-specific.
Live both directions: admin -> 23 machines / office1 trio (exit 0);
vr1-dc0-region -> 0 machines / hot-kid (exit 0); admin asserted as hot-kid
(exit 1). Capture docs/audit/dc0-region-profile-assert-20260730.txt.
Tunnel voffice1 -> rack -> 10.12.8.6:5240 because no single host has repo +
CLI + new-region reach + NetBox apex reach. Application-layer and
rack-originated, so SEC-010 is not punctured; nothing was rerouted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

GA-R4 session close: Stage 5 opened, D-138 + D-132 ruled, dc0 region live
...
Sweep first: every in-session finding grepped against CURRENT-STATE and the
changelog. THREE lived only in the transcript and would have been lost -- the
absent openstack CLI on the DC client host (blocks Step 7+), the metal-admin
v6 rack-leg gap, and the plane-fabric mtu=1500 vs libvirt 9000 mismatch. All
now in docs/audit/queued-findings-20260730-stage5.txt (F1-F12), with six tool
traps collected as platform-traps candidates.
F6 recorded honestly: the as-executed log for this window is PARTIAL. The
classifier refused several `script -aqe ... -c` wrapped forms while the plain
ssh form passed, so those calls ran unwrapped. The index row says so -- a log
that silently looks complete is worse than one declaring its gap.
Bookend appended (14 lines, under the GA-R4 cap; ledger 250/300 so no
rotation owed). Counters: 3 decisions, SEC 23, D 139 / DOCFIX 206 /
BUNDLEFIX 053.
GA-R7 memory review: one entry added (as-executed vs classifier), indexed.
Nothing in memory claims operator policy.
Stage 5 remains OPEN. Nothing is half-applied -- Office1 still owns all 9 dc0
nodes, correctly carved and Ready.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Region images synced (jammy+noble); pre-migration carve captured
...
Both releases now Synced on the dc0 region, 6 rows each, matching Office1.
jammy is NOT seeded by `maas init` (noble only) and the OpenStack nodes
require it. The first import raced the selection's creation and returned
noble-only while reporting is-importing=false -- assert on Synced ROWS, never
on the import call returning.
Measured and recorded so the two are not conflated: MAAS boot images do NOT
come from the DC mirror. The region pulls images.maas.io; the mirror 404s on
every simplestreams index path because dc-mirror.sh mirrors debmirror APT
packages only. Three artifact classes, one local. Consequence: the
D-135/D-107 egress narrowing would BREAK image sync and juju bootstrap, not
just slow them, and nothing sequences it after them.
Pre-migration carve captured to docs/audit (179 lines): fabrics, VLANs,
spaces, subnets, D-134 bands, tags, per-node identity/power and the full
per-interface carve. MAAS cannot move machines between regions, so the
migration must recreate all of it.
The destructive half has NOT started -- the Office1 region still owns all 9
dc0 nodes, correctly carved and Ready. Nothing is half-applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc0 MAAS region LIVE; three creds minted, consolidated and REGISTERED (SEC-027)
...
Region: noble + MAAS 3.7.2 + PostgreSQL 16.14 snaps, byte-identical versions
to the Office1 region (checked, not assumed). regiond/rackd/apiserver ACTIVE,
dhcpd INACTIVE so there is no DHCP conflict with the Office1 rack still
serving that VLAN. API answers 200 on 10.12.8.6:5240 -- the address jujud
could not reach at Office1, which is the whole point of D-132's amendment.
Carve verified BOTH legs (the lesson of today): metal-admin 10.12.8.6 + v6
::6, provider-public 10.12.4.6 with gw 10.12.4.1.
Credentials, per operator direction: DB password, region admin password and
admin API key minted ON the VM, never printed, consolidated to
~/vr1-dc0-creds/ at 0600 with sha256 digests compared to prove the copies are
identical, API key format-verified parts=3. SEC-027 opened (open SEC 22->23),
recording the residual argv exposure honestly -- `maas init` has no stdin or
file option for --database-uri.
The creds MATRIX now expects them at BOTH DCs: 5 rows per DC, a new `region`
host-role token in the enum, 4 vm-secret-locations rows (definition-of-done
for a new mint site), dc0's manifest declares its three. dc1's three now FAIL
S2 BY DESIGN -- the forward register making dc1's absence detectable, which is
what D-137 exists for. Harness 65/65 PASS, repo-lint 0 fail.
Also: durable ssh aliases for DC nodes. Deployed nodes carry the
MAAS-registered vr1-office1-svc key, not the rack service key, so each hop
needs its own identity -- a bare `ssh -i <key> -J voffice1,<rack>` fails
because -i forces one key on every hop. No credential was copied anywhere.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc0 MAAS region VM APPLIED, MACs pinned, converged; commissioning trap recorded
...
Capacity gate re-measured PASS (RAM 870/1024 = 85%, FIT, 154 GiB headroom).
Plan asserted on content -- exactly two resources, both the new VM. Applied
2 added / 0 changed / 0 destroyed; MACs measured and pinned; converged ZERO
DIFF with live virsh domiflist byte-identical to the pin; virsh power set.
Enlisted as hot-kid / tw7ptw.
Corrected against measurement: the "1 add" figure in both new comments and in
D-104's amendment prose is wrong at resource level -- node-vm creates a
domain AND a volume, so it is 2 to add. The half that mattered, 0 change /
0 destroy, held.
Ordering trap measured and recorded as a standing rule: the first apply sets
running=true, so the VM PXE-enlists and auto-commissions within ~90s, and
pinning MACs is an in-place libvirt_domain update that BOUNCES the guest --
stalling that commission at "Loading ephemeral". After pinning MACs on a
freshly-applied VM, assume commissioning was interrupted and re-commission
deliberately. Recovery is abort -> New -> commission.
dc1's region VM is authored but NOT applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pin vr1-dc0-maas-01 MACs from measurement, pre-enlistment
...
Measured with virsh domiflist on vvr1-dc0 in the same session as the first
apply, before the VM enlists -- the boundary modules/node-vm warns about and
the one the 2026-07-20 MAC regeneration made expensive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Author the per-DC MAAS region VM into both substrate roots (D-132 addendum)
...
vr1-dc0-maas-01 and vr1-dc1-maas-01: 4 vCPU / 8 GiB / 150 GiB, macs = []
for first apply. Additive on a for_each-keyed caller, so each plans
1 add / 0 change / 0 destroy.
Sizing is the shape the capacity gate actually modelled
(dc-dc-whole-host-budget.py --containment-overhead-mem-gib 32
--containment-overhead-vcpu 12 -> RAM 870/1024 = 85 pct, FIT, 154 GiB
headroom). Disk is 150 not 100 because this VM stores the region's
PostgreSQL AND its boot-image set.
Both entries carry the carve requirement in the comment: metal-admin AND
provider-public at enlistment. Carving metal-admin alone is exactly the
under-carve that cost three bootstrap attempts on the juju controller VM
today -- no default route, no image sync.
MACs deliberately empty, to be pinned from measurement immediately after
first apply, per the same rule that the 2026-07-20 MAC regeneration
established.
opentofu-validate: root + 12 modules + both extra roots all PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-132 placement RULED: per-DC MAAS region VM at utility .6; D-134 map extended
...
GA-R5. Operator utterance: "Dedicated region VM per DC at utility .6
(Recommended)".
The region DB must not share fate with the hypervisor running every node it
manages -- also a precondition for D-132 q3 cross-site backup custody, which
stays pinned. The D-134 standing octet map now reads .4 artifact service /
.5 juju controller / .6 MAAS region, binding at every future DC standup.
Build notes carried, not ruled: author BOTH metal-admin and provider-public
legs at creation (the identical under-carve on the juju controller VM cost
three bootstrap attempts today); re-measure capacity before applying.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-132 q1 RULED for VR1: a MAAS region controller per DC; Stage 5 blocked
...
GA-R5, before dependent work. Operator utterance: "Put a MAAS region
controller in each DC".
Third bootstrap got PAST the agent fetch (downloaded attempt 1 -- the
controller carve closed that failure class) and failed at the next layer:
jujud, running ON the node, cannot reach the MAAS region API at
10.10.0.20:5240. Measured: rack reaches it (200, rack-originated, permitted
by SEC-010); the node cannot (forwarded, blocked); and the rack does NOT
proxy it -- nginx listens on 5248 only. All six required flows enumerated;
exactly one crosses the boundary.
The finding underneath: SEC-010 as written is incompatible with the
deployment's own control-plane topology (D-104 controller in-DC + MAAS
region at Office1). D-138 was necessary but NOT sufficient -- it moved the
cloud-facing client into the DC, and jujud is itself a provider client whose
MAAS dependency I did not enumerate when scoping it.
Per-DC regions remove the requirement rather than excepting it, so SEC-010
is PRESERVED UNAMENDED. Single, not HA (utterance is singular); D-132 q2/q3
stay pinned to the next deployment.
Consequence stated plainly: this reopens the MAAS layer of Stages 3 and 4,
both closed and merged. A region owns its own DB; nothing migrates between
regions. Open sub-question blocking the build: where each DC's region lives.
Also landed: both controllers carved symmetric with gw-bearing
provider-public legs. Trap recorded -- MAAS `interface update vlan=`
returned a full JSON object while changing nothing on a Deployed machine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

G17 dc0 CAPTURED (both assertions PASS); bootstrap root cause = under-carved controller
...
G17's one-shot first-boot window opened during the second bootstrap attempt and
was taken rather than lost. Capture docs/audit/g17-dc0-firstboot-20260730.txt,
exit 0: mirror asserted on FOUR body fields of a real package path (not the
autoindex root), and the node time source is the MAAS rack 10.12.8.2 chaining
to the region, not the DC edge. Scope stated: the node is the dc0 CONTROLLER
VM, not a role node; G17's dc1 half stays OPEN.
Two defects in G17's own text, both measured: it names `chronyc sources` and
chrony is NOT installed on the MAAS jammy image (node runs systemd-timesyncd),
so the gate as written can only REFUSE; and a first pass false-FAILED on
`printf | grep -q` under pipefail -- the same trap recorded earlier the same
day. Capture uses here-strings.
Root cause of both bootstrap failures is measured and is NOT architectural:
the controller VMs are UNDER-CARVED. D-138's path works -- the second attempt
reached "Connected to 10.12.8.5" -- then failed fetching the juju agent
because the node has NO DEFAULT ROUTE. Role nodes carry provider-public with
gw 10.12.4.1; the controllers carry metal-admin only, because they were added
after both carves and Stage 4's 90 NIC re-homes covered the 18 role nodes.
Owned: this session's egress re-probe was run FROM THE RACK and reported the
window open. The rack holds a default route nodes lack, so it did not
discharge the caveat for a node-side fetch. Same wrong-host class as the
dc-mirror instrument note, made after writing that note.
Blocked on a harness permission wall for four MAAS interface calls, not on a
decision. Not retried in altered shapes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-138 ADOPTED: the cloud-facing client lives IN the DC; SEC-026 opened
...
GA-R5, before dependent work. Operator utterance: "Move the cloud-facing
client into the DC (Recommended)".
Resolves the contradiction the Stage-5 bootstrap failure exposed: SEC-010 +
D-052 make metal-admin DC-local and forbid the route juju needs, while D-100
names Juju as fiber traffic and D-128 puts the client on voffice1. SEC-010's
"pinning is free" was priced against tools that proxy at the app layer; juju
dials the machine at L3 and was not in scope.
The client moves; the boundary does not. SEC-010, D-052 and D-125 are
UNCHANGED -- nothing punctured, no plane opened. D-128 is amended to exclude
cloud-facing tools. Scope was set by enumeration first: keystone's VIP is on
provider-public, so routing would have opened two planes per DC across a
dozen ports.
SEC-026 opened for the consequence: a MAAS admin-scoped key becomes resident
on a DC-local host, over a region shared by both DCs. Isolation is the
control -- each DC's client host gets ONLY its own credential.
Counters: 22 open SEC, next-free D 139.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Stage 5 BLOCKED: no juju-client->node-plane path; D-134 .5 now BUILT both DCs
...
Bootstrap attempted and failed. Machine selection was correct (7n87bt, the
tagged VM) and MAAS deployed jammy end to end -- confirming the
--bootstrap-base ubuntu@22.04 pin against a live deploy. juju then could not
SSH the machine and released it.
Root cause: voffice1 has NO route to any DC node plane and two deliberate
controls forbid one -- SEC-010's transit FORWARD-drop on the rack, and
libvirt's blanket reject into the isolated plane bridges. The DC edge has no
metal-admin leg. Nothing regressed; this path never existed.
This is a contradiction between RULED surfaces: SEC-010/D-052 make
metal-admin DC-local and forbid the region routing to 10.12.8.0/22, while
D-100 says the fiber carries Juju traffic and D-128 puts the juju client on
voffice1. SEC-010's "pinning is free" justification was priced against MAAS,
which proxies at the app layer; juju dials the machine at L3 and was not in
scope. Needs a ruling, not a firewall edit. No reachability change made.
BUILT, both DCs (operator: "Fix now: static .5 + v6, re-bootstrap"): the
controller VMs held AUTO v4-only addresses; now static 10.12.8.5 +
fd50:840e:74e2:220::5 and 10.12.68.5 + fd50:840e:74e2:320::5. v6 prefixes
confirmed by VLAN pairing, not inferred. D-134's amendment was
ruled-but-not-built until now.
Capture: docs/audit/stage5-bootstrap-reachability-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 5: Step 1 + 2.0 gates passed; bootstrap constraint-flag ruled
...
GA-R5, before dependent work. Operator utterance: "Use both flags".
juju 3.6.27's own help assigns machine-targeting to --bootstrap-constraints
and model-defaulting to --constraints; the runbook and D-104's mechanism
sentence both name only --constraints. D-104 is NOT amended -- only the flag
implementing it is clarified. DOCFIX owed, recorded in the changelog.
Read-only gates, all on voffice1: selectors exit 0 (6 planes, 10 hosts);
credential gate outcome 1 of 3 so the mint is skipped (SEC-018 sprawl);
exactly one machine carries juju-controller-vr1-dc0, Ready.
Artifact sources PASS both DCs; egress re-probed from both racks with
--noproxy '*' -- bootstrap window OPEN. Instrument note recorded: the DC
checkers run ON the rack host and read all-MISS when run from voffice1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

STAGE 5 OPEN: P5 accepted by ruling, entry gate captured on voffice1
...
Opens Stage 5 / Phase 4 (Juju controller + OpenStack bundle, per DC) on the
standing operator directive. GA-R1/C1: the status change and CURRENT-STATE
land in one commit. GA-R5: the P5 ruling is committed and pushed BEFORE any
dependent work.
Entry gate, measured ON voffice1 (the only host whose reading counts):
P1/P2/P3/P4 PASS, P7 PASS 37/0 with the literal zone line, P5 FAIL on 6
pre-existing credential-register findings. Capture
docs/audit/stage5-preflight-dc0-20260730.txt.
Operator ruling, exact utterance: "Accept and proceed to deploy (Recommended)".
The acceptance covers those six findings, enumerated, and nothing else.
Logged not executed (hard rule 1): bundle.yaml:592 gives ceph-osd the stale
VR0 constraint tags=openstack, a tag measured absent from the VR1 region.
Decided at Step 4.2's dry-run, where its impact is first observable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|