| 2026-07-31 |
skill: /savegame -- the end-of-session bookend + transcript sweep
...
Four operator rulings: prepare-then-stop (never commits), archive at stage
close, sweep by model-enumerate-then-grep, repo-specific. Named savegame at
operator request.
The sweep is mechanical: every candidate is grepped against the repo, so
FIRST SURFACE items are proven rather than asserted. Five structurally
invisible classes always swept, including gitignored permission rules and
measurements that contradict a standing doc.
Two live conditions found while writing it: the ledger is at ~295 against the
300-line cap so the next close breaches it, and GA-R4 rule 2's literal text
(body moves at session close) diverges from ruled practice (stage close) --
DOCFIX owed, logged not taken.
repo-lint L9 caught the first draft hardcoding the repo name (D-110); fixed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Session close: GA-R4 bookend, memory review, gauntlet 93 green
...
dc0 node carve complete (v4 134/0, v6 54/54, diff vs pre-migration EMPTY);
juju controller vr1-dc0-controller LIVE on the fourth attempt; model, spaces
gate and apt-mirror done. Bundle deploy attempt 1 failed on prefer-ipv6 with
nothing half-applied; operator ruled research-first, so R2 is NOT amended and
Stage 5 is blocked on that question.
SEC-028 + SEC-029 opened and registered before use. DOCFIX-206 fixed a
credential gate that was not region-scoped. preflight region-blindness logged.
GA-R7 memory review: instrument-currency memory updated with three new
instances and a new DETECTOR -- an implausibly uniform result is itself the
tell. Other three memories reviewed, accurate, none claim operator intent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Bundle deploy attempt 1 FAILED: prefer-ipv6 unsupported on 6 of 13 VIP charms
...
ERROR cannot deploy application 'barbican': unknown option 'prefer-ipv6'.
Model measured EMPTY afterwards -- juju validated and aborted atomically, so
nothing is half-applied and Steps 1-3.5 remain good.
The dry-run passed and the deploy did not: --dry-run resolves charms and plans
relations but does NOT validate config option names. A green dry-run is not a
gate on option validity.
Measured against charmhub's own config-yaml per pinned channel: 7 of 13 VIP
charms declare prefer-ipv6, 6 do not. barbican never had it at any channel, so
it was not an obsoleted option -- R2's uniform application was never valid.
Not fixed: invariant 9 couples prefer-ipv6 to the v6 VIP legs, so this needs a
GA-R5 ruling, not a mid-deploy edit.
Owned: my first charmhub query used a wrong field and returned a uniform
'no config' across 13 charms -- implausible, which is what caught it.
Capture: docs/audit/stage5-dc0-deploy-attempt1-20260731.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
permissions: 12 read-only allow rules for the per-DC MAAS/PKI checkers
...
Scanned 50 recent transcripts. Most high-frequency read-only commands were
already covered (repo-lint, ledger-scan, run-tests-all, tests/*, preflight,
opentofu-validate, tofu validate/state list). These twelve were not.
Every rule is pinned to a read-only SUBCOMMAND -- check/verify/assert/plan/
--list -- so apply, reserve, reissue and --run stay gated. Nothing that mutates
the cloud was added, and no interpreter or ssh wildcard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage deploy artifacts on the rack; SEC-029 for the PKI overlay residency
...
Operator-ruled: 'Copy dc0's PKI overlay to the dc0 rack (Recommended)'.
bundle.yaml + machines + vips staged with digests compared; the octavia-pki
overlay copied 0600 with sha256 equal end to end.
D-138 means the deploy cannot run from voffice1 at all -- no L3 path to the
controller API on the provider plane -- so the overlay must be DC-resident.
Per-DC isolation holds by construction: each rack gets only its own overlay.
SEC-029 records that reissue must now update BOTH copies or the rack deploys a
superseded cert. dc1's rows are the forward register.
creds-matrix 65/65; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Step 3.5 complete: model vr1-dc0, spaces gate PASS, apt-mirror set
...
add-model guarded; spaces gate 0 fatal, all six ruled spaces present;
apt-mirror key VERIFIED against the live client (not guessed) and set to
http://10.12.8.4/ubuntu, asserted on content via dists/jammy/Release.
The mirror gate was RED on content -- one package, a transient upstream 500 on
linux-tools-intel-iotg. debmirror's all-or-nothing exit failed the whole run.
Re-sync took 15s and the gate is green. This is the assert-on-content fix
working as designed.
Owned: my first wait-loop polled systemctl immediately after an async start and
reported the sync done while the status file said RUNNING. Re-polled on content.
Recorded: D-138 moved the client to the rack but the rack has no repo clone --
same class as queued F1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
FINDING: preflight.sh is region-blind, 19 false negatives for dc0
...
DC=vr1-dc0 preflight reports every machine 'not enrolled in MAAS' while
resolving profile=admin -- the OFFICE1 region, which no longer owns them.
Re-run with MAAS_PROFILE=vr1-dc0-region: zero node failures, RED on P5 only.
The message reads as a deployment-state problem when the true statement is
'wrong region'. Fourth consumer of the MAAS_PROFILE=admin default.
Logged not fixed: a blanket refusal of 'admin' for vr1-* would be WRONG because
dc1's nodes still live in Office1. Needs per-DC resolution plus a region-identity
assert, with its own harness.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
JUJU CONTROLLER BOOTSTRAPPED: vr1-dc0-controller live, active/idle
...
Fourth attempt, first success. Each earlier failure was a real defect one layer
deeper and all three are now closed: D-138 (client into the DC), the controller
under-carve fix (default route), and D-132 q1 (per-DC MAAS region).
controller/0 active/idle on juju-controller rev 311; machine 0 started
ubuntu@22.04 as subtle-grouse; MAAS reads arfr7p Deployed ubuntu/jammy.
Transcript evidence: agent binaries resolved on attempt 1 (339-retry class stays
closed); targeting took the tagged VM not a role node; juju offered the v6 leg
restored earlier this session before connecting on metal-admin.
Capture: docs/audit/stage5-bootstrap-dc0-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Juju re-pointed at the dc0 region: credential minted, proven, registered (SEC-028)
...
Operator-ruled: 'Mint juju-vr1-dc0 on the new region (Recommended)' and
'Add it to allow'. juju-vr1-dc0 minted in the dc0 region as a superuser,
mirroring the measured Office1 shape. Never printed; moved host-to-host with
all three sha256 digests compared; staging shredded.
Proven to AUTHENTICATE before bootstrap -- login + users/machines/rack reads
confirming superuser scope and DC-local region identity. Cloud re-pointed to
10.12.8.6:5240 and read back; the stale Office1-keyed credential removed first.
DOCFIX-206: Step 2.0's credential gate was not region-scoped, and I hit it --
all three checks passed while the user lived in Office1, so the SKIP branch
would have led to an auth failure that reads as a network fault.
Two snap-confinement traps recorded: private /tmp, and the home interface
refusing dot-directories. Neither error names confinement.
Registered BEFORE bootstrap: SEC-028, 8 matrix rows, dc0 manifest, and the
first 'rack' rows in vm-secret-locations. creds-matrix 65/65; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
v6 carve complete 54/54; controller v6 restored; Juju re-point blocked on a credential mint
...
v6: applied=54 errors=0, read back 54/54. Ran after the v4 carve because the
provider v6 static lands on br-ex. The tool walks only the nine tagged role
nodes, so the controller's metal-admin v6 was applied separately -- restoring
ruled state the re-enrolment dropped. Its provider GUA stays deliberately absent.
Juju survey on the rack: cloud still points at the OFFICE1 region, which is what
killed bootstrap 3. juju unregister is NOT needed -- no controllers registered.
Blocker: the new region has no juju-vr1-dc0 MAAS user. Re-establishing it mints
a new API key, so it is operator-gated. Nothing was minted; the refusal was a
permission rule failing to match, not a classifier fault.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 v4 node carve COMPLETE: 10/10 machines, gate 134/0, diff vs pre-migration EMPTY
...
Canary-first then node by node, never batched. 11 + 13x8 + 4 mutations, every one
read back. Nine role nodes on six planes with OVS br-ex; the juju controller on
two planes raw with no br-ex and enp3-6 left on auto VLANs.
Verified by diffing against the Office1 pre-migration capture: every node's six
v4 legs match address for address. Compared by LEG SET, since system_id and
hostname are both re-minted at re-enlistment.
Instrument note: the first extraction read the wrong capture field and returned
'-' for every old address, which would have read as 'the old region had no
statics'. Caught by the implausible shape, not by the parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-node-carve: unlink reason is a parameter, not a constant
...
The plan for the metal-admin leg read 'L3 moves to br-ex', which is false --
do_unlink is called for two different reasons and had one hardcoded. A mutation
log that states the wrong reason is the confident-but-wrong evidence class this
repo keeps paying for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-node-carve: convert the commissioning auto link, found by a live dry-run
...
The first live plan refused enp1s0 on every machine: it holds an `auto` link on
metal-admin out of enlistment, and the "wrong address on the right subnet is a
FAILURE" rule was too broad. A commissioning link is the expected starting
state, not a conflict -- so the tool could not carve the metal-admin leg at all.
Narrowed BY MODE: auto/dhcp/link_up gets unlinked and re-created as STATIC
(MAAS will not convert a link's mode in place); a `static` link with the wrong
address is still refused, because something deliberate put it there.
The `raw` fixture modelled the auto link correctly -- what was missing was any
assertion that `apply` over the raw state EXITS 0. T21-T24 only grepped for
planned strings, so a refusal on the very first NIC read as green. T26/T27 now
cover both directions and both are mutation-proven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-30 |
dc-node-carve.sh: the v4 node interface carve, mutation-proven 40/40
...
The last no-tool gap between the dc0 region migration and the Juju deploy:
60 NIC re-homes, 9 OVS br-ex bridges, 54 statics. The Stage-4 equivalent was
done ad-hoc and logged outside the repo, so there was nothing to re-run when
D-132 q1 forced a re-enrolment.
Proved the four-call MAAS cycle on ONE live interface (canary 6q4syf enp3s0)
before writing the tool -- nothing had exercised `interface update vlan=` on a
link_up-only interface on an auto fabric, which is the state all 60 NICs are in.
Measured in passing: link-subnet REPLACES a link_up link, so only the br-ex
member needs an explicit unlink.
No default profile; maas-profile-assert.sh before any read or write; every id
resolved live by stable key (boot MAC / CIDR / interface name); every mutation
read back and compared. Role asymmetry derived from the host token, and the
controller's deliberately-uncarved NICs are asserted ABSENT.
Mutation pass found the br-ex static compare could not fail -- Pattern B was
being caught by the type/parent assertions instead. Added the brexip fixture
and re-proved the kill. Also found that neutering an assertion's MESSAGE only
proves it exists, not that its predicate works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Close sweep: queued findings F1-F10, memory review, skill invariants
...
Sweep capture docs/audit/queued-findings-20260730-dc0-region-migration.txt.
Three items lived ONLY in the transcript and would have been lost on a
clear:
F1 -- this session's permission rules exist only in the GITIGNORED
.claude/settings.local.json, so their verbatim text is now recorded,
including a broad Bash(ssh vr1-dc0-rack *) that an interactive approval
auto-added and which deserves review.
F2 -- the Office1 region STILL registers a rack controller on the dc0 rack
(vvr1-dc0/7chphy) and still holds the dc0 region VM's machine record. Not
harmful (Office1's DHCP for that VLAN is off, verified by process) but
dual-registration the operator should rule on.
F10 -- the five instrument errors of this session as one pattern, with the
rule that catches them.
F9 carries the next session's first commands and every measured input the
carve tool needs.
GA-R7 memory review: sharpened the classifier memory with the profile-token
lesson (a rule pinning 'maas admin' would have matched step 1 of the DHCP
cutover and silently missed step 3), and added an instrument-currency
memory. Both graduate to repo surfaces; neither is the only copy.
Skill swept: the per-DC region migration invariants folded in -- region
identity by rack, ids do not cross regions, the per-node ordering, batches
of three, zero SSH keys, the role-dependent carve, the role-tags gate that
passes on an incomplete tag set, and the two scripts that must not be
pointed at a new region.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 bookend: dc0 migrated to its own MAAS region, 10/10 Ready
...
Supersedes the part-4 bookend, which closed with the cutover blocked.
Office1 holds zero dc0 machines; the per-DC region holds all ten Ready with
shapes exact to D-121 Option C. Four named gates green: topology 40/0,
power key 9/0, role tags 0/0/0, plane IPAM 24/0.
Ledger at 278 lines, under the 300 cap. Gauntlet ALL GREEN (92), repo-lint
0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Plane IPAM complete on the new region: 24/0 PASS, existing tooling only
...
dc-plane-ipam.sh check vr1-dc0 goes from pass 7 / fail 17 to pass 24 /
fail 0, exit 0. Two idempotent applies, dry-run first, every write read
back: 13 D-134 reserved bands plus the FIP pool, then the 5 missing v6
plane subnets each created on the same MAAS vlan as its v4 twin.
No new code was needed. The tool's forced sequencing also proved itself --
the reserve run skipped every v6 band with 'not in MAAS yet, run carve-v6
first' rather than failing or silently no-opping, so running the two out of
order is safe.
This is the step that justifies the tunnel architecture: carve-v6 derives
its prefixes from the NetBox apex, reachable only from voffice1, while the
target region is reachable only via the rack. Neither host alone could have
run it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
All ten dc0 machines Ready in the per-DC region; task 6 complete
...
Enlisted as canary + three batches of three. Shapes match D-121 Option C
exactly (3x16/64, 2x12/48, 4x8/24, 1x4/8), all with 6 interfaces, every one
matched to its libvirt domain by pinned boot MAC -- hostname and system_id
were both re-minted.
maas-role-tags.sh check: 0 tags missing, 0 nodes needing a tag, 0 not in
MAAS, exit 0.
But it reported that clean PASS while the machines were still missing the
openstack-vr1-dc0 placement tag, which is not in its ROLES set and has no
creator anywhere in the repo. bundle.yaml places on that tag and
dc-node-v6-carve.py dies without it, so both the deploy and the v6 carve
would have failed. Caught only by diffing against the pre-migration
snapshot. Applied to the nine role nodes and deliberately not to the
controller -- reproducing the Office1 shape row-for-row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Correct my own wording: every machine passes through New
...
I wrote that batch 1 went 'straight to Ready with no New detour'. Wrong --
I aborted them to New myself to clear the enlistment-time wedge. Every
machine passes through New, either by self-settling there (batch 2 did,
before I could abort) or by an explicit abort.
The real distinction is whether the commission ISSUED FROM New succeeds on
the first try, and that is what power_type decides. The conclusion stands;
the phrasing overstated it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Batch 1 all Ready; the New detour was ORDERING, not chance
...
Batch of three enlisted in 2.3 min against a 15-minute zero-record stall for
nine, and all three went straight to Ready with no New detour -- because
power was set BEFORE commission was issued, where the canary was
commissioned first and powered afterwards.
That supersedes the earlier 'one-off' reading with a stronger and more
useful conclusion: the detour is caused by ordering. Revised standing
procedure is delete -> virsh start -> self-enlist -> SET POWER -> abort ->
commission, and never commission a machine whose power_type is unset.
Region now holds 4 machines all Ready, each matched to its libvirt domain by
pinned MAC.
Role tags need no new tool: maas-role-tags.sh creates the four missing ones
and refuses while any pinned MAC lacks a record, so it enforces the ordering
itself.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Nine records deleted; found a fresh region has ZERO SSH keys
...
All nine remaining Office1 dc0 records deleted individually. A scripted loop
over them was correctly refused by the guard -- hard rule 3 says destructive
steps are never batched, and I had written exactly that. Office1 is now at
13 machines with zero dc0-tagged.
Found by accident while probing stalled nodes: the new region has no SSH
keys at all. MAAS injects registered keys into every machine it deploys and
juju bootstrap needs client->machine SSH, so this is the same signature that
consumed three bootstrap attempts today and would have recurred at task 8
looking like a network fault. Imported the same vr1-office1-svc key Office1
carries, deliberately rather than minting a per-DC one, because the node ssh
aliases built this session depend on that identity.
Enlistment does not scale to nine concurrent nodes: all alive and leasing,
cloud-init silent for 15 minutes, zero records created. Egress, MAAS proxy,
region load and node liveness all ruled out by measurement. Restarted in
batches of three.
I misread the stall twice in five minutes -- first off a bad elapsed-time
estimate, then by counting 104 enlist lines that were DHCP renewals rather
than cloud-init progress. Counting lines from a broad grep is not measuring
the thing you care about.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
lib-hosts: the node carve is ROLE-DEPENDENT, not uniform
...
Measured from the pre-migration capture. Role nodes carve all six planes
plus an OVS br-ex parented on enp2s0 carrying the provider-public static,
with enp2s0 itself holding no L3 link. The juju controller carves only TWO
planes -- enp1s0 metal-admin and enp2s0 provider-public as a raw NIC WITH
the gateway -- has no br-ex, and leaves enp3s0..enp6s0 on their
auto-created VLANs.
Evidence: 7n87bt (controller) reads enp3-6 on vlans 5200-5203, which are
auto fabrics; 8mtpxq (storage-02) reads enp3-6 on 5190-5193, the named
planes, plus br-ex.
This matters because assuming uniformity fails in both directions: it would
either strand the controller without a default route -- the under-carve that
cost three bootstrap attempts on 2026-07-30 -- or give it four legs it must
not have. The node carve has no repo tool yet, so this was about to be
re-derived by hand.
dc-selector 95 checks; the depths-must-differ guard proven able to fail, and
the controller-presence assertion checked inside a per-DC subshell because
the main shell still holds the flat VR0 fleet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Canary reaches Ready; two theories refuted by measurement
...
6q4syf/civil-bug Ready at 22:05:33 -- cpu=8 mem=24576 (exact D-121 Option C
storage shape), power=virsh, all six pinned MACs. The re-enrolment ordering
is now a working sequence.
Root cause of the earlier New: a commission issued immediately after an
abort can land in New once; re-issuing from a settled New reached Ready in
~3.5 min, matching the dc1 timing. One-off, not systematic. The repeat was
justified as the experiment that distinguished those two cases -- not as a
blind retry.
Refuted by measurement, recorded so neither is re-proposed: smartctl on
virtio disks (it was Skipped and its result set Passed, so
testing_scripts=none would have taught nothing), and a config delta between
the independently-configured regions (all 13 commissioning-relevant
settings identical).
Standing consequence for the remaining nine: a single New after a passing
commission is not a fault -- re-issue once; only a second New is evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Correct an over-claim: the re-enrolment ordering has NOT reached Ready
...
I wrote the five-step re-enrolment sequence into CURRENT-STATE and the
changelog as a standing invariant. Each step is individually measured, but
the sequence has not yet produced a Ready machine, so calling it a working
invariant was ahead of the evidence. Reworded to separate what is proven
(delete leaves the domain; manual power-on is required; self-enlistment
carries the pinned MAC; power config is a prerequisite, and without it the
machine wedges in Commissioning with zero events; abort+commission has MAAS
power it on) from what is not (that it reaches Ready).
Opens item 18 with the real diagnostic, which I had skipped:
node-script-results returned unparseable on first read and I moved on --
"could not look" treated as "nothing there", against my own rule. Read
properly it is unambiguous: commissioning result set Passed with all 13
scripts exit 0, testing set Passed with smartctl-validate Skipped on both
virtio disks, hardware discovery correct at 8 cpu / 24576 MiB. Neither
phase failed; the machine went to New anyway.
This REFUTES the smartctl-on-virtio theory. Re-running with
testing_scripts=none would have fixed nothing and taught nothing.
The other nine machines stay untouched until the canary is Ready, and
dc-node-carve.sh stays unwritten because a New machine's interfaces are
rediscovered on the next commission.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Canary re-enrolment: the ordering a fresh region actually needs
...
Canary xqqwdq/moral-salmon = vr1-dc0-storage-04 (a storage node, least
central of the ten). Office1 23 -> 22; it re-enlisted into the new region
carrying boot MAC 52:54:00:2b:ed:ab -- the machine that arrived is the
machine that left. system_id AND hostname are both re-minted
(xqqwdq -> 6q4syf, moral-salmon -> civil-bug), so the boot MAC is the only
stable key; the pre-delete snapshot records it as the identity.
MEASURED ORDERING, not the obvious one: a freshly-enlisted machine has no
power config, so MAAS cannot power-cycle it and cannot drive commissioning.
delete -> power on BY HAND via virsh -> self-enlist -> SET POWER -> abort ->
commission. Skipping the power step wedges the machine in Commissioning
producing zero events, because the enlistment ephemeral powers the node off
and nothing can bring it back.
maas-node-power.sh already supports the split the migration needs: VIRSH_URI
enumerates from voffice1 over the transit /30 while the positional address
stores what the REGION dials over metal-admin. Result was a real
query-power-state, proving the region drives the rack's libvirt with the
SEC-012 key in its snap.
Found: the per-node serial log stopped capturing on 2026-07-21 while the
domain XML still configures append-mode logging. It contains a complete,
plausible boot that I read as current and reported as fact before checking
the mtime. An append log that silently stopped appending returns confident,
well-formed, wrong evidence. Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DHCP handover EXECUTED: dc0 metal-admin now served by the DC-local region
...
Ruled order, region identity proven first. Office1 vlan update 4 0
dhcp_on=false -> read back False; verified STOPPED BY PROCESS (pgrep -c
dhcpd = 0 on the rack, because MAAS self-report is not evidence -- the
2026-07-20 Temporal incident class); new region vlan update 0 0
dhcp_on=true primary_rack=c3aqh8 -> read back True on fabric-0 / space
metal-admin; verified RUNNING BY PROCESS on hot-kid with the rack
re-checked at 0. Exactly one DHCP server on the segment. All 10 machines
Ready and powered off throughout.
Recorded so it is not later read as a fault: the new region also starts
dhcpd6 where the Office1 rack ran v4 only. Its v6 metal-admin subnet has
NO dynamic range -- only rfc-4291-2.6.1 and assigned-ip/reserved, same as
the old region -- so it has nothing to hand out and nodes keep their ruled
statics.
The permission wall was a rule that FAILED TO MATCH, not a classifier
fault. Existing ask rules pin the double-quote form while the commands used
single quotes, and pin the maas admin profile while the new region needs
maas vr1-dc0-region. Operator-approved ask rules added to
settings.local.json (gitignored; committed team policy unchanged),
profile-agnostic so both regions match, plus read-only pgrep/ps allow
rules. The first rule shape I proposed pinned maas admin and would have
matched step 1 while silently missing step 3 -- stalling the cutover at
exactly the two-servers boundary.
Capture docs/audit/dc0-dhcp-handover-20260730.txt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 bookend: dc0 region topology 40/0; cutover blocked on a permission wall
...
Ledger at 265 lines, under the 300 cap. Gauntlet ALL GREEN (92),
repo-lint 0 fail. Stage 5 remains OPEN and blocked; the exact 4-command
unblock sequence with live-resolved parameters is in the changelog.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Session record: dc0 region topology 40/0; cutover blocked with exact commands
...
Records the fabric-rename remediation (changelog item 12), the two
pre-cutover checks (hot-kid IS eligible as primary_rack for fabric-0 -- had
it not been, DHCP-off would have succeeded and DHCP-on failed, leaving the
segment with no DHCP and ten machines about to PXE), the measured NIC ->
plane order, and the exact unblock sequence with every parameter
re-resolved live.
hot-kid is c3aqh8 in the new region and tw7ptw in Office1 -- the same
machine, two ids.
Capture updated with the corrected 40 passed / 0 failed, exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
lib-hosts: record the MEASURED VR1 node NIC -> plane order
...
The substrate roots pin six MACs per node and the module attaches them in
list order, so macs[i] is enp<i+1>s0. Verified end to end on
vr1-dc0-control-01 against the live carve: enp1s0 metal-admin (PXE),
enp2s0 provider-public (br-ex parent), enp3s0 metal-internal, enp4s0
data-tenant, enp5s0 storage, enp6s0 replication.
This is NOT lib-net.sh's PLANE_CIDRS order, which starts with
provider-public. Anything walking PLANE_CIDRS positionally to place NICs
swaps enp1s0 and enp2s0 and puts the PXE leg on provider-public, stranding
commissioning on a plane with no DHCP. The node carve has no repo tool yet,
so this ordering was about to be re-derived by hand; recording it measured,
with a guard that the two orders differ, is the cheap half of that tool.
dc-selector 88 checks; the trap guard proven able to fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-node-v6-carve.py: honour MAAS_PROFILE; refuse instead of tracebacking
...
The script was env-blind -- default 'admin', --profile only, no environment
read anywhere. Every bash tool here honours MAAS_PROFILE, so
`export MAAS_PROFILE=vr1-dc0-region` left THIS tool silently targeting the
Office1 region. During the per-DC region migration that is the
unrecoverable direction: against the wrong region a carve is an idempotent
no-op that prints PASS while the region you meant stays untouched.
It now reads MAAS_PROFILE, keeps --profile as an override, and prints which
region it targets and where that came from -- a capture that does not name
the profile cannot be audited for wrong-region work.
An absent maas CLI raised FileNotFoundError and exited 1 with a traceback,
indistinguishable from a real failure and contrary to this script's own
REFUSE contract. It now refuses with exit 2.
Harness 14/14 (was 9); both fixes proven able to fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
