| 2026-07-31 |
Bundle deploy attempt 1 FAILED: prefer-ipv6 unsupported on 6 of 13 VIP charms
...
ERROR cannot deploy application 'barbican': unknown option 'prefer-ipv6'.
Model measured EMPTY afterwards -- juju validated and aborted atomically, so
nothing is half-applied and Steps 1-3.5 remain good.
The dry-run passed and the deploy did not: --dry-run resolves charms and plans
relations but does NOT validate config option names. A green dry-run is not a
gate on option validity.
Measured against charmhub's own config-yaml per pinned channel: 7 of 13 VIP
charms declare prefer-ipv6, 6 do not. barbican never had it at any channel, so
it was not an obsoleted option -- R2's uniform application was never valid.
Not fixed: invariant 9 couples prefer-ipv6 to the v6 VIP legs, so this needs a
GA-R5 ruling, not a mid-deploy edit.
Owned: my first charmhub query used a wrong field and returned a uniform
'no config' across 13 charms -- implausible, which is what caught it.
Capture: docs/audit/stage5-dc0-deploy-attempt1-20260731.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
JUJU CONTROLLER BOOTSTRAPPED: vr1-dc0-controller live, active/idle
...
Fourth attempt, first success. Each earlier failure was a real defect one layer
deeper and all three are now closed: D-138 (client into the DC), the controller
under-carve fix (default route), and D-132 q1 (per-DC MAAS region).
controller/0 active/idle on juju-controller rev 311; machine 0 started
ubuntu@22.04 as subtle-grouse; MAAS reads arfr7p Deployed ubuntu/jammy.
Transcript evidence: agent binaries resolved on attempt 1 (339-retry class stays
closed); targeting took the tagged VM not a role node; juju offered the v6 leg
restored earlier this session before connecting on metal-admin.
Capture: docs/audit/stage5-bootstrap-dc0-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 v4 node carve COMPLETE: 10/10 machines, gate 134/0, diff vs pre-migration EMPTY
...
Canary-first then node by node, never batched. 11 + 13x8 + 4 mutations, every one
read back. Nine role nodes on six planes with OVS br-ex; the juju controller on
two planes raw with no br-ex and enp3-6 left on auto VLANs.
Verified by diffing against the Office1 pre-migration capture: every node's six
v4 legs match address for address. Compared by LEG SET, since system_id and
hostname are both re-minted at re-enlistment.
Instrument note: the first extraction read the wrong capture field and returned
'-' for every old address, which would have read as 'the old region had no
statics'. Caught by the implausible shape, not by the parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-30 |
Close sweep: queued findings F1-F10, memory review, skill invariants
...
Sweep capture docs/audit/queued-findings-20260730-dc0-region-migration.txt.
Three items lived ONLY in the transcript and would have been lost on a
clear:
F1 -- this session's permission rules exist only in the GITIGNORED
.claude/settings.local.json, so their verbatim text is now recorded,
including a broad Bash(ssh vr1-dc0-rack *) that an interactive approval
auto-added and which deserves review.
F2 -- the Office1 region STILL registers a rack controller on the dc0 rack
(vvr1-dc0/7chphy) and still holds the dc0 region VM's machine record. Not
harmful (Office1's DHCP for that VLAN is off, verified by process) but
dual-registration the operator should rule on.
F10 -- the five instrument errors of this session as one pattern, with the
rule that catches them.
F9 carries the next session's first commands and every measured input the
carve tool needs.
GA-R7 memory review: sharpened the classifier memory with the profile-token
lesson (a rule pinning 'maas admin' would have matched step 1 of the DHCP
cutover and silently missed step 3), and added an instrument-currency
memory. Both graduate to repo surfaces; neither is the only copy.
Skill swept: the per-DC region migration invariants folded in -- region
identity by rack, ids do not cross regions, the per-node ordering, batches
of three, zero SSH keys, the role-dependent carve, the role-tags gate that
passes on an incomplete tag set, and the two scripts that must not be
pointed at a new region.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Plane IPAM complete on the new region: 24/0 PASS, existing tooling only
...
dc-plane-ipam.sh check vr1-dc0 goes from pass 7 / fail 17 to pass 24 /
fail 0, exit 0. Two idempotent applies, dry-run first, every write read
back: 13 D-134 reserved bands plus the FIP pool, then the 5 missing v6
plane subnets each created on the same MAAS vlan as its v4 twin.
No new code was needed. The tool's forced sequencing also proved itself --
the reserve run skipped every v6 band with 'not in MAAS yet, run carve-v6
first' rather than failing or silently no-opping, so running the two out of
order is safe.
This is the step that justifies the tunnel architecture: carve-v6 derives
its prefixes from the NetBox apex, reachable only from voffice1, while the
target region is reachable only via the rack. Neither host alone could have
run it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
All ten dc0 machines Ready in the per-DC region; task 6 complete
...
Enlisted as canary + three batches of three. Shapes match D-121 Option C
exactly (3x16/64, 2x12/48, 4x8/24, 1x4/8), all with 6 interfaces, every one
matched to its libvirt domain by pinned boot MAC -- hostname and system_id
were both re-minted.
maas-role-tags.sh check: 0 tags missing, 0 nodes needing a tag, 0 not in
MAAS, exit 0.
But it reported that clean PASS while the machines were still missing the
openstack-vr1-dc0 placement tag, which is not in its ROLES set and has no
creator anywhere in the repo. bundle.yaml places on that tag and
dc-node-v6-carve.py dies without it, so both the deploy and the v6 carve
would have failed. Caught only by diffing against the pre-migration
snapshot. Applied to the nine role nodes and deliberately not to the
controller -- reproducing the Office1 shape row-for-row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Canary re-enrolment: the ordering a fresh region actually needs
...
Canary xqqwdq/moral-salmon = vr1-dc0-storage-04 (a storage node, least
central of the ten). Office1 23 -> 22; it re-enlisted into the new region
carrying boot MAC 52:54:00:2b:ed:ab -- the machine that arrived is the
machine that left. system_id AND hostname are both re-minted
(xqqwdq -> 6q4syf, moral-salmon -> civil-bug), so the boot MAC is the only
stable key; the pre-delete snapshot records it as the identity.
MEASURED ORDERING, not the obvious one: a freshly-enlisted machine has no
power config, so MAAS cannot power-cycle it and cannot drive commissioning.
delete -> power on BY HAND via virsh -> self-enlist -> SET POWER -> abort ->
commission. Skipping the power step wedges the machine in Commissioning
producing zero events, because the enlistment ephemeral powers the node off
and nothing can bring it back.
maas-node-power.sh already supports the split the migration needs: VIRSH_URI
enumerates from voffice1 over the transit /30 while the positional address
stores what the REGION dials over metal-admin. Result was a real
query-power-state, proving the region drives the rack's libvirt with the
SEC-012 key in its snap.
Found: the per-node serial log stopped capturing on 2026-07-21 while the
domain XML still configures append-mode logging. It contains a complete,
plausible boot that I read as current and reported as fact before checking
the mtime. An append log that silently stopped appending returns confident,
well-formed, wrong evidence. Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DHCP handover EXECUTED: dc0 metal-admin now served by the DC-local region
...
Ruled order, region identity proven first. Office1 vlan update 4 0
dhcp_on=false -> read back False; verified STOPPED BY PROCESS (pgrep -c
dhcpd = 0 on the rack, because MAAS self-report is not evidence -- the
2026-07-20 Temporal incident class); new region vlan update 0 0
dhcp_on=true primary_rack=c3aqh8 -> read back True on fabric-0 / space
metal-admin; verified RUNNING BY PROCESS on hot-kid with the rack
re-checked at 0. Exactly one DHCP server on the segment. All 10 machines
Ready and powered off throughout.
Recorded so it is not later read as a fault: the new region also starts
dhcpd6 where the Office1 rack ran v4 only. Its v6 metal-admin subnet has
NO dynamic range -- only rfc-4291-2.6.1 and assigned-ip/reserved, same as
the old region -- so it has nothing to hand out and nodes keep their ruled
statics.
The permission wall was a rule that FAILED TO MATCH, not a classifier
fault. Existing ask rules pin the double-quote form while the commands used
single quotes, and pin the maas admin profile while the new region needs
maas vr1-dc0-region. Operator-approved ask rules added to
settings.local.json (gitignored; committed team policy unchanged),
profile-agnostic so both regions match, plus read-only pgrep/ps allow
rules. The first rule shape I proposed pinned maas admin and would have
matched step 1 while silently missing step 3 -- stalling the cutover at
exactly the two-servers boundary.
Capture docs/audit/dc0-dhcp-handover-20260730.txt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Session record: dc0 region topology 40/0; cutover blocked with exact commands
...
Records the fabric-rename remediation (changelog item 12), the two
pre-cutover checks (hot-kid IS eligible as primary_rack for fabric-0 -- had
it not been, DHCP-off would have succeeded and DHCP-on failed, leaving the
segment with no DHCP and ten machines about to PXE), the measured NIC ->
plane order, and the exact unblock sequence with every parameter
re-resolved live.
hot-kid is c3aqh8 in the new region and tw7ptw in Office1 -- the same
machine, two ids.
Capture updated with the corrected 40 passed / 0 failed, exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 region plane topology BUILT and verified (39/0); DHCP cutover blocked
...
dc-region-topology.sh applied to the new dc0 region: 5 named plane fabrics,
6 spaces, 4 plane subnets, 10.12.4.0/22 MOVED off the auto fabric-1 onto
vr1-dc0-provider-public, 6 VLAN->space bindings, openstack-vr1-dc0 tag.
Named check now reads 39 passed / 0 failed, exit 0.
metal-admin service config set: dns_servers=10.12.8.6, allow_dns=false,
D-134 dynamic range .201-.254, subnet resolved by CIDR.
The DNS target was measured rather than carried over, and my first reading
was wrong: a +time=3 +tries=1 probe suggested the new region's BIND could
not resolve externally. That was a cold-cache timeout. At +time=5 +tries=3
it answers archive.ubuntu.com, streams.canonical.com and api.snapcraft.io
with flags qr rd ra. Pointing node DNS at the DC-local region is correct --
it is authoritative for the zone the migrated nodes will live in and drops
the cross-fiber dependency D-132 q1 exists to remove.
Measured in passing, and worth keeping: VLAN row id 5005 is dc0's
metal-admin VLAN in Office1 and vr1-dc0-data-tenant in the new region.
Same integer, different DC plane.
The DHCP cutover was REFUSED by the harness classifier and was not retried
in an altered shape. Verified read-only afterwards that nothing is
half-applied: Office1 still reads dhcp_on=True primary_rack=7chphy and the
rack still runs exactly one dhcpd on virbr2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Per-region power address, region power-key tool, and two red gates fixed
...
lib-hosts.sh: MAAS power ops originate from the REGION, so the reachable
rack address depends on which region dials. Measured from vr1-dc0-maas-01:
the rack's transit leg 172.31.0.2:22 is CLOSED, its metal-admin leg
10.12.8.2:22 is OPEN. Adds VIRSH_POWER_ADDRESS_FROM_{OFFICE1,DCREGION} for
both DCs (dc1's 10.12.68.2 measured on the dc1 rack, not inferred from
dc0's .2). VIRSH_POWER_ADDRESS still aliases the Office1 form so existing
callers are byte-for-byte unchanged. dc-selector 81 checks; 4 mutations
each killed tests.
New scripts/maas-region-power-key.sh (harness 44/44) installs and verifies
the per-DC MAAS->libvirt key inside a region's snap, ending on a real virsh
call rather than on config. Runbook prerequisite 5 carried this as prose,
and prose is not a gate -- it must also be re-asserted after every snap
refresh. LIVE at dc0: 9/9 exit 0, virsh enumerates 12 domains. The key is
the SEC-012 dc0 key already authorized on that rack, so per-DC isolation
holds.
Three defects its own testing found: ssh-keygen -lf reads an adjacent .pub
and reports ITS fingerprint; a snap's data root (/var/snap) and exec root
(/snap) are not interchangeable; the snap's virsh must run inside the snap
runtime or it fails on missing libraries.
tests/node-vm T8/T9 were RED at HEAD before any of this session's work
(proven by stashing and re-running): commits 086c827 + 447315f added the
region VMs, moving the dc0 root to 11 macs lists / 66 MAC literals against
assertions reading 10 / 60, and the gauntlet was not re-run. Re-pointed to
the new invariant, reason recorded in-file, re-proven able to fail.
HARNESS-MANIFEST re-recorded (91) for the two new harnesses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Region-resolution gate: maas-profile-assert.sh + named dc0 region profile
...
The wrong-REGION hazard is the migration's unrecoverable failure mode: 13
repo scripts default to MAAS_PROFILE=admin, which on voffice1 resolved to
the OFFICE1 region. Against the wrong profile a carve recreate is a silent
no-op that prints PASS, and machine delete destroys the real record.
New scripts/maas-profile-assert.sh asserts which region a profile resolves
to by rack-controller IDENTITY (a machine count is not proof -- two regions
can hold the same number). Harness 20/20; mutation pass found one of its own
assertions could not fail (nameless rack entries were caught by a different
branch) and it was made branch-specific.
Live both directions: admin -> 23 machines / office1 trio (exit 0);
vr1-dc0-region -> 0 machines / hot-kid (exit 0); admin asserted as hot-kid
(exit 1). Capture docs/audit/dc0-region-profile-assert-20260730.txt.
Tunnel voffice1 -> rack -> 10.12.8.6:5240 because no single host has repo +
CLI + new-region reach + NetBox apex reach. Application-layer and
rack-originated, so SEC-010 is not punctured; nothing was rerouted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 session close: Stage 5 opened, D-138 + D-132 ruled, dc0 region live
...
Sweep first: every in-session finding grepped against CURRENT-STATE and the
changelog. THREE lived only in the transcript and would have been lost -- the
absent openstack CLI on the DC client host (blocks Step 7+), the metal-admin
v6 rack-leg gap, and the plane-fabric mtu=1500 vs libvirt 9000 mismatch. All
now in docs/audit/queued-findings-20260730-stage5.txt (F1-F12), with six tool
traps collected as platform-traps candidates.
F6 recorded honestly: the as-executed log for this window is PARTIAL. The
classifier refused several `script -aqe ... -c` wrapped forms while the plain
ssh form passed, so those calls ran unwrapped. The index row says so -- a log
that silently looks complete is worse than one declaring its gap.
Bookend appended (14 lines, under the GA-R4 cap; ledger 250/300 so no
rotation owed). Counters: 3 decisions, SEC 23, D 139 / DOCFIX 206 /
BUNDLEFIX 053.
GA-R7 memory review: one entry added (as-executed vs classifier), indexed.
Nothing in memory claims operator policy.
Stage 5 remains OPEN. Nothing is half-applied -- Office1 still owns all 9 dc0
nodes, correctly carved and Ready.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Region images synced (jammy+noble); pre-migration carve captured
...
Both releases now Synced on the dc0 region, 6 rows each, matching Office1.
jammy is NOT seeded by `maas init` (noble only) and the OpenStack nodes
require it. The first import raced the selection's creation and returned
noble-only while reporting is-importing=false -- assert on Synced ROWS, never
on the import call returning.
Measured and recorded so the two are not conflated: MAAS boot images do NOT
come from the DC mirror. The region pulls images.maas.io; the mirror 404s on
every simplestreams index path because dc-mirror.sh mirrors debmirror APT
packages only. Three artifact classes, one local. Consequence: the
D-135/D-107 egress narrowing would BREAK image sync and juju bootstrap, not
just slow them, and nothing sequences it after them.
Pre-migration carve captured to docs/audit (179 lines): fabrics, VLANs,
spaces, subnets, D-134 bands, tags, per-node identity/power and the full
per-interface carve. MAAS cannot move machines between regions, so the
migration must recreate all of it.
The destructive half has NOT started -- the Office1 region still owns all 9
dc0 nodes, correctly carved and Ready. Nothing is half-applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
G17 dc0 CAPTURED (both assertions PASS); bootstrap root cause = under-carved controller
...
G17's one-shot first-boot window opened during the second bootstrap attempt and
was taken rather than lost. Capture docs/audit/g17-dc0-firstboot-20260730.txt,
exit 0: mirror asserted on FOUR body fields of a real package path (not the
autoindex root), and the node time source is the MAAS rack 10.12.8.2 chaining
to the region, not the DC edge. Scope stated: the node is the dc0 CONTROLLER
VM, not a role node; G17's dc1 half stays OPEN.
Two defects in G17's own text, both measured: it names `chronyc sources` and
chrony is NOT installed on the MAAS jammy image (node runs systemd-timesyncd),
so the gate as written can only REFUSE; and a first pass false-FAILED on
`printf | grep -q` under pipefail -- the same trap recorded earlier the same
day. Capture uses here-strings.
Root cause of both bootstrap failures is measured and is NOT architectural:
the controller VMs are UNDER-CARVED. D-138's path works -- the second attempt
reached "Connected to 10.12.8.5" -- then failed fetching the juju agent
because the node has NO DEFAULT ROUTE. Role nodes carry provider-public with
gw 10.12.4.1; the controllers carry metal-admin only, because they were added
after both carves and Stage 4's 90 NIC re-homes covered the 18 role nodes.
Owned: this session's egress re-probe was run FROM THE RACK and reported the
window open. The rack holds a default route nodes lack, so it did not
discharge the caveat for a node-side fetch. Same wrong-host class as the
dc-mirror instrument note, made after writing that note.
Blocked on a harness permission wall for four MAAS interface calls, not on a
decision. Not retried in altered shapes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 5 BLOCKED: no juju-client->node-plane path; D-134 .5 now BUILT both DCs
...
Bootstrap attempted and failed. Machine selection was correct (7n87bt, the
tagged VM) and MAAS deployed jammy end to end -- confirming the
--bootstrap-base ubuntu@22.04 pin against a live deploy. juju then could not
SSH the machine and released it.
Root cause: voffice1 has NO route to any DC node plane and two deliberate
controls forbid one -- SEC-010's transit FORWARD-drop on the rack, and
libvirt's blanket reject into the isolated plane bridges. The DC edge has no
metal-admin leg. Nothing regressed; this path never existed.
This is a contradiction between RULED surfaces: SEC-010/D-052 make
metal-admin DC-local and forbid the region routing to 10.12.8.0/22, while
D-100 says the fiber carries Juju traffic and D-128 puts the juju client on
voffice1. SEC-010's "pinning is free" justification was priced against MAAS,
which proxies at the app layer; juju dials the machine at L3 and was not in
scope. Needs a ruling, not a firewall edit. No reachability change made.
BUILT, both DCs (operator: "Fix now: static .5 + v6, re-bootstrap"): the
controller VMs held AUTO v4-only addresses; now static 10.12.8.5 +
fd50:840e:74e2:220::5 and 10.12.68.5 + fd50:840e:74e2:320::5. v6 prefixes
confirmed by VLAN pairing, not inferred. D-134's amendment was
ruled-but-not-built until now.
Capture: docs/audit/stage5-bootstrap-reachability-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
STAGE 5 OPEN: P5 accepted by ruling, entry gate captured on voffice1
...
Opens Stage 5 / Phase 4 (Juju controller + OpenStack bundle, per DC) on the
standing operator directive. GA-R1/C1: the status change and CURRENT-STATE
land in one commit. GA-R5: the P5 ruling is committed and pushed BEFORE any
dependent work.
Entry gate, measured ON voffice1 (the only host whose reading counts):
P1/P2/P3/P4 PASS, P7 PASS 37/0 with the literal zone line, P5 FAIL on 6
pre-existing credential-register findings. Capture
docs/audit/stage5-preflight-dc0-20260730.txt.
Operator ruling, exact utterance: "Accept and proceed to deploy (Recommended)".
The acceptance covers those six findings, enumerated, and nothing else.
Logged not executed (hard rule 1): bundle.yaml:592 gives ceph-osd the stale
VR0 constraint tags=openstack, a tag measured absent from the VR1 region.
Decided at Step 4.2's dry-run, where its impact is first observable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Q2 withdrawn: D-137 fork 1 was ruled 2026-07-25, not open
...
Third stale-premise item in queued-findings-20260730.txt (after Q1). Q2 opens
'D-137 OPEN FORK 1 (enforcement strength) -- still unruled'. Sub-ruling 1 ruled
it 2026-07-25 at option (b), utterance 'Blocking in preflight', and its
CONSEQUENCE block explicitly declines option (c): 'NOT adopted: the PreToolUse
guard (so an agent is not constrained at write time, only at the deploy gate)'.
So guard-destructive.py is NOT D-137's enforcement point -- it is the older
CLAUDE.md/DOCFIX-006 mechanism, and hardening it needs no ruling to proceed.
This was verified BEFORE the reissue minter was built, which is why 'Full script
minter now' was consistent with the ruling rather than a side-effect decision.
The misfire evidence stays valuable and the tally is now SEVEN. The two newest
were collected today: the guard blocked a command whose only purpose was to TEST
it (it read nothing), and then blocked the heredoc writing this very correction
into the audit file -- documentation prose, no file read. Both are the same class
as the earlier five: the matcher cannot distinguish reading secret material from
merely naming it.
Correction APPENDED, not rewritten (the stage4-mirror-gate precedent).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F9 CLOSED: controller certs reissued into their own zones, both DCs
...
Executed on voffice1 via Step 1.0-REISSUE at HEAD 7b6a2e4. Operator-gated PER DC
('Go ahead' each), never batched; dc0 verified clean before dc1 was touched.
Capture: docs/audit/octavia-reissue-executed-20260730.txt
vr1-dc0 -> omega.dc0.vr1.cloud.neumatrix.local, IP 10.12.4.57 + f02:11::57,
serial ...250D -> ...250E
vr1-dc1 -> omega.dc1.vr1.cloud.neumatrix.local, IP 10.12.64.57 + f03:11::57,
serial ...674DD -> ...674DE
dc1's OLD cert carried dc0's LABEL as well as the wrong region -- the cross-DC
case A11 exists for. A11 confirms per-DC independence on both DCs after the mint.
Both: fresh EC P-256 key with the rotation PROVEN (public key differs from the
outgoing cert's), neither CA regenerated so the amphora trust domain is intact,
overlay surgery measured other_values_identical=4 changed_lines=1 line_count=8.
ACCEPTANCE was the literal positive line, not the verdict: while os-public-hostname
is unset, A12/A13 RECORD rather than fail, so PASS alone is compatible with a wrong
certificate. Both DCs now carry 'DNS SANs are all in this DC's expected zone'.
verify PASS 37/0 both DCs (was 29/0; +8 from the new assertions). creds-matrix:
same 5 pre-existing findings, no new finding, no S4 drift. No staging residue,
neither run reached exit 5.
The hardened re-run guard proved itself live: re-running dc0 immediately after
returned exit 4 'already correct ... and verify returns clean' -- the fix for the
defect that had it refusing to repair certs verify was failing.
STILL OWED, operator-gated: backup custody. Two archives on the headend hold BOTH
CA private keys, BOTH passphrases and the overlay; 1.0-REISSUE.4 pulls them to the
jumphost creds folder and removes the headend copy. They are also an unregistered
credential class (no creds-matrix row, no retention rule). Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 sweep: correct the artifact that filed Q1, and one stale residue
...
Advisor review caught that correcting CURRENT-STATE and the ledger was not
enough: docs/audit/queued-findings-20260730.txt -- the sweep capture the 07-30
bookend points at -- still presented Q1 as an open ruling that 'blocks any FQDN
cert', enforced by an A12 refusal that no longer exists. A next session reading
it would re-derive Q1 exactly as this one did, reopening the loop just closed.
- queued-findings-20260730.txt: APPENDED correction (not a rewrite; the
stage4-mirror-gate-20260727.txt precedent), naming the two claims that are now
false and pointing at the evidence. Q2 explicitly untouched.
- dc-dc-deployment-workflow.md:257: described phase-6 as proposing the retired
dc1/dc2-hostnames.yaml spelling. MEASURED: the runbook itself was already
corrected to the region-qualified form by an earlier session -- only the
workflow doc's description of it lagged. Under D-119 IS vr1-dc0, so the
-hostnames.yaml interpolation used elsewhere is correct as written.
- The DOCFIX-205 capture's section 1 quoted a 3-line span starting mid-sentence
and left a clause orphaned. Repaired to two verbatim quotes, both re-verified
present in design-decisions.md. CURRENT-STATE cites this file as evidence and
this repo's discipline is exact captures.
octavia-pki 23/23; repo-lint 0 fail (L10 gated this commit correctly -- the
docs/audit/ change set required CURRENT-STATE to move with it).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 capture: the D-117 supersession, annotation coverage 0/4 -> 4/4, live A12
...
Evidence file cited by CURRENT-STATE's superseding block. Carries the quoted
D-117 Status line, D-119's selector-only discharge, measured annotation coverage,
the phase-6 expansion old-vs-new in a real shell, live A12 on the headend
(PASS 29/0 both DCs, both INERT), and both hosts' gate results.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-bookend sweep: the operator's working commands were NOT in the repo
...
Operator: "I'm worried that the fixes we made to the defective commands will be lost when we
close this session ... Complete a full sweep before the bookend."
THE CONCERN WAS WELL FOUNDED. The corrected Step 5/6 commands -- the ones that actually produced
both DCs' live PKI -- had been reviewed in conversation, judged equivalent, and never folded in.
Four items now landed:
1. Portable base64. GEN.d shipped the GNU-only `-w0` flag; what ran was
`base64 < f | tr -d '\r\n'`. The runbook described a command nobody executed.
2. The operator's VIP_OVERLAY:? guard in GEN.c.
3. The heredoc column-1 requirement, as a CAUTION heading rather than a footnote, because that
failure is silent: no alt_names section yields a certificate with NO SANs while every
openssl command still prints OK.
4. ${DC_LABEL:?} at both CA call sites -- F8's sibling, where an unset label bakes a DC-less
subject into a 10-year CA with no error anywhere.
Not done, with the reason recorded: the heredoc was NOT rewritten as printf lines. More
paste-proof, but an untested rewrite of a step that mints 10-year CA material and cannot be
exercised end-to-end from an agent session. Risk bounded instead by A9 plus harness T2.
F9 is now structural rather than remembered: A12 arms itself from os-public-hostname appearing as
a real option key. Live PASS 29/0 both DCs, harness 21/21.
A new ruling-shaped gap, now blocking rather than filed: D-008's shape plus D-106:2563's VR1
instantiation do not say whether substrate vr1-dc1 is dc1 by token or dc2 by position -- the
DC1/DC2 ambiguity item 3.1 retired elsewhere, here deciding certificate identity. Both live certs
carry dc0.vr0, the VR0 region, wrong under either reading. A12 refuses rather than picking.
A precedence bug the harness caught: REFUSE was checked before FAIL, so once A12 armed a
CONFIRMED wrong-region SAN reported as "could not evaluate". A known defect outranks an
unevaluated one.
Sweep capture docs/audit/queued-findings-20260730.txt carries F10-F13 and two ruling-shaped
questions. F10 is the one worth acting on: mint-ref line numbers drift silently and S4 cannot see
it, since it asserts only within-EOF. That bit three times in two sessions and was caught by hand
every time, never by a gate. All 24 octavia refs re-anchored in a single-pass mapping keyed by row
id -- 391 was simultaneously an old and a new value, so sequential seds would have corrupted it.
repo-lint 0 fail; octavia-pki 21/21; creds-matrix 65/65.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |
Per-DC Octavia PKI generated: independence proven, identity unverified, 12 real findings
...
The mint was operator-executed on the headend; the session guard hard-blocks it and that was
not worked around.
PER-DC INDEPENDENCE PROVEN, which is the whole point of F1. Both DCs hold 12 files across 3
dirs, and sha256 DIFFERS between DCs for both encrypted CA keys, the controller cert and the
overlay. Before F1 those columns would have been identical, because the second generation would
have overwritten the first. Both overlays are mode 600, carry exactly five lb-mgmt-* keys, are
ASCII clean and are gitignored -- the F4 gate held.
NOT CLAIMED AS DONE: the four CA subjects, the controller SAN sets and the chain verifications
are unread, because the guard blocks openssl against those paths. Recorded as structurally
complete with identity unverified. A cert can be well-formed and name the wrong DC.
P5 on the headend went 6 -> 18 findings and all 12 new ones are genuine.
E2 x4: the CA certificates are mode 664. This session had explicitly called that harmless "for
public certs", reasoning only about confidentiality. That was wrong -- 664 is group-WRITABLE, so
a group member can replace a trust anchor. Integrity, not secrecy, is the exposure.
E3 x8: four generator outputs per DC (the controller certificate, the CA serial file, the
openssl config and the signing request) exist with no matrix row. The 18-row set landed earlier
today was incomplete -- "enumerate what exists, not only what is declared", applied to this
session's own work.
Validation worth recording: all 12 would have been INVISIBLE before today's F2 and F6 fixes,
since the rows were singleton and the locations resolved on the jumphost. The register found
real defects in exactly the mint it was repaired to observe.
F8: the controller-cert step's heredoc is a live paste hazard -- the operator hit it. An
indented terminator can yield a config with no alt_names section, producing a certificate with
NO SANs while every step still reports OK, and nothing asserts the SAN set.
F9: the controller certificate's DNS SANs are coupled to D-106 and nothing recorded it. They
are inert only because os-public-hostname is set nowhere (B5 IP-ONLY; R5 refused setting it at
Stage 5 as a D-019 repeat). When Stage 7 turns FQDN endpoints on, both DCs' certificates
carrying dc0.vr0 becomes wrong exactly when it starts mattering; likely disposition is reissue
inside D-106's own FQDN-SAN step.
Remediation logged not executed: tighten the six certificate modes; and for E3 either declare
all four artifacts per DC or have the generator drop the two build intermediates and declare the
two that must persist.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F7: office1-tailscale is labelled a subnet router and advertises no routes
...
Raised from an operator report of Permission denied (publickey) reaching voffice1 "from my
workstation via tailscale". Measured rather than assumed, and the premise did not survive.
tailscale is absent from BOTH voffice1 and vcloud -- no binary in any standard location, no
state dir, no interface, service not running. Checked with explicit path probes after an
initial `command -v`, because tool-absent-from-PATH reported as feature-absent is a trap this
repo has hit repeatedly.
The tailnet node is a separate LXD guest INSIDE voffice1: office1-tailscale, RUNNING,
100.64.0.53, site leg 10.10.1.11. Its tailscale status reports AdvertisedRoutes: <none>. So
nothing on the tailnet can reach 10.10.0.20 -- there is no route and there never was.
Two records made the false belief look supported: the ssh stanza labels it "Office1 subnet
router (D-107)", and the operating skill's routing table says the workstation reaches voffice1
over the tailnet, citing D-107 -- which is titled "Airgap posture, per-DC artifact mirror, and
NTP (VR1)" and rules nothing of the sort. A real citation that does not support its claim.
RULED (operator): option (a), ProxyJump through vcloud. Option (b), advertising 10.10.0.0/24
onto the tailnet, was presented and not taken -- it exposes the isolated site net, a posture
change against the airgap subject D-107 actually governs.
Fixed operationally outside the repo: vcloud's ssh config now matches Host voffice1 10.10.0.20,
so the raw IP resolves to the office1 service key with IdentitiesOnly yes instead of falling
through to the id_* defaults, which was the actual cause. Both forms verified, with a negative
control proving the key is not offered to unrelated hosts.
Third wrong record this session, after D-124's MAC-sourcing instruction and the skill citation.
Two of the three surfaced only because something adjacent was being fixed; nothing here gates
prose, so no check could have caught them.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Five Stage-5 findings raised, none executed: R7's per-DC Octavia PKI is half built
...
F1 (HIGH): the D-109 amendment rules per-DC independent Octavia CAs, and the
2026-07-29 work correctly made the CA subject and the VIP gate per-DC -- but $DC
appears in no PATH. phase-01-bundle-deploy.md:293 is a fixed $HOME/octavia-pki and
:473 writes a fixed overlays/octavia-pki.yaml, so generating dc1 after dc0
overwrites dc0's issuing-CA key, controller-CA key and both passphrases, and a
later dc0 redeploy reads the fixed overlay name and applies dc1's CA to dc0 -- the
cross-DC shared amphora root R7 explicitly refused. The existence checks at
phase-01:86 and pre-flight-checks.sh:65 cannot detect it: they ask whether the file
exists, never whose CA it is. The gap is in R7's own "Work implied" list.
F2: creds-matrix.tsv:103-111 declares all nine Octavia credentials `singleton`,
while creds-matrix.py:368-398 enforces both-DC existence only for `per-DC` rows --
so the register behind the BLOCKING P5 gate is structurally blind to a missing
second PKI set. Baseline captured (82 rows, 7 findings, exit 1) so the delta is
attributable. Must land before generation.
F3: both ~/octavia-pki/ and overlays/octavia-pki.yaml are ABSENT (existence only,
no contents read), so F1/F2 are fixable at zero risk right now; the window closes
when the first DC's PKI is generated.
F4: .gitignore:40 is an exact path, so a per-DC overlay rename would un-ignore a
file holding CA key blobs plus a plaintext passphrase, in a repo SEC-004 records as
PUBLIC. The F1 fix must widen the glob in the same commit.
F5: preflight.sh:99 sets DC without exporting it and pre-flight-checks.sh never
reads DC at all, so a dc1 invocation runs a DC-aware P2 beside a dc0-frozen P4
under one combined verdict.
Logged not executed per hard rule 1. repo-lint 0 fail / 615 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 session close: bounded bookend + the 2026-07-29 findings sweep
...
Bookend is 14 content lines against the 15-line cap, measured. Ledger 263
lines against the 300 cap, so no rotation is owed this close.
Close sweep captured docs/audit/queued-findings-20260729.txt -- transcript-
only material preserved before compaction, per the queued-findings-20260726
and -20260727 precedent. The substantive content:
- A strict-bash quoting trap FAMILY with three measured instances in one
session: IFS=$'\n\t' defeats word-splitting on a space-separated string
(killed pre-flight-checks CHECK 1 silently); a bare under
inherit_errexit ABORTS the run instead of reaching a refusal branch; and an
apostrophe inside the single-quoted shell string wrapping embedded python
makes bash parse python.
- Two greps that looked like findings and were not: 'grep mac' matches
type_machine, and grep -c 'macs = [' counts COMMENTS -- the same class as
the ledger-scan defect where a doc quoting a token inflated a counter.
- Five MAAS behaviours, including that enlistment is NOT commissioning and
that MAAS auto-configures power only for IPMI machines, so a virsh VM must
be power-configured out of band first.
- Two guards defeatable by accident: lib_hosts_select_dc's one-DC-per-shell
guard muted by 2>&1, and NetBox's ?site= filter silently not filtering.
- A process note on two parallel live-ops agents: safety came from agents
working the voffice1 clone while the orchestrator edited vcloud's, so no git
race was possible -- at the cost that agents barred from repo writes cannot
leave captures, and dc1's evidence had to be promoted by hand from /tmp.
The skill sweep stays DEFERRED to the Stage-5 close per the 2026-07-27
operator direction; this capture is its input, not a substitute.
Gauntlet ALL GREEN (87) on vcloud; repo-lint 0 fail / 614 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: OSD carve + D-104 controllers complete across both DCs (two agents)
...
Operator directed two background agents: one to finish dc0 storage-02/03/04,
one for all of dc1. Both completed.
Eight re-commissions, every one the same single-change diff: vdb at
536870912000 bytes (500 GiB) added; links 12 -> 12; per-interface MACs and
link lists identical. dc0 at 204/196/202s, dc1 all ~182s. dc1's substrate
applied 6 add / 4 change / 0 destroy with MAC drift 0 and re-converged to
ZERO DIFF.
I verified every claim from MAAS directly rather than accepting the reports:
fleet 22 (18 Ready + 2 Deployed + 2 New), 216 links all static, both DCs
blockdevs {1:5, 2:4}, tagcount {2:9}. All matched. I also diffed a raw dc1
before/after pair myself -- exactly the vdb hunk plus one power_state line.
Captures promoted from voffice1:/tmp (which does not survive a reboot) to
docs/audit/osd-carve-20260729/. Both agents were barred from repo writes, so
dc1 would otherwise have had no audit record at all.
Two unprompted agent findings worth keeping: MAAS tags SURVIVED
re-commissioning on all 18 nodes (checked because the snapshot code
structurally could not see tags -- removes a would-be Stage-5 placement
blocker); and lib_hosts_select_dc's one-DC-per-shell guard fires correctly
but can be muted with 2>&1, silently yielding one DC's data labelled as the
other's.
One disclosed judgment call, and it was correct: three dc1 pairs also showed
power_state moving off/error. Agent B declared a STOP, re-measured, then
excluded power_state from its verdict while still printing it, because 8 of 9
dc1 nodes read error BEFORE any mutation. Vindicated -- all 18 role nodes now
read off, including five dc1 nodes never commissioned. A transient MAAS
power-query flap.
Still blocking for both controllers: power_type is EMPTY, the state that
stalled all nine role nodes on 2026-07-20. Now fixable since both are in
lib-hosts at the ruled .5 with MACs pinned.
Gauntlet ALL GREEN (87) on vcloud; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: canary re-commission closes R1 precondition 3 by measurement
...
Operator: 'Commission storage-01 as the canary'. maas admin machine
commission kghggm skip_networking=1, sysid matched by pinned boot MAC never
by name. Commissioning -> Ready in ~200s, no timeout, no SERVFAIL.
The full before/after diff of status, power, block devices, interfaces, MACs
and links is EXACTLY ONE HUNK: vdb (536870912000 bytes = 500 GiB) added to
blockdevices. links 12 -> 12; MACs identical; links identical. All twelve
static links survive with addresses, modes and subnets intact, including the
D-134 octet .150 mirrored across all six planes in both families -- the v6
host part ::150 being the 2026-07-27 D-136 mirror ruling visible in live
data.
So skip_networking=1 preserves the network EXACTLY while storage is still
re-scanned, which is the entire point. Precondition 3 was previously answered
from the API help text; it is now answered by measurement. The amendment's
concern was legitimate -- the default path does reconfigure networking -- and
the vendor's own control is sufficient.
Remaining: dc0 storage-02/03/04, then all of dc1.
Capture: docs/audit/dc0-canary-commission-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: dc0 substrate applied -- R1 OSD volumes + the D-104 controller VM
...
Operator approval: "Apply dc0 first". Executed from voffice1 via a SAVED
PLAN with a same-session pre-apply re-verify (G8 precedent), so what was
applied is exactly what was reviewed.
Apply complete: 6 added, 4 changed, 0 destroyed. Four libvirt_volume.osd +
the vr1-dc0-juju-01 domain and its disk; the four storage domains updated
in-place to attach vdb.
THE CHECKPOINT HELD -- MAC DRIFT COUNT 0. Verified by virsh domiflist
against lib-hosts' pinned MACs BEFORE anything touched MAAS: all 9 nodes
pinned==live, 6 NICs each. That sequencing is the point. 2026-07-20 became a
fleet-wide outage because MAAS was told to re-commission while its records
were already stale, after a 0/9/0 in-place apply regenerated every MAC
without the plan showing it. So the clean plan was necessary and explicitly
not sufficient. First live evidence that the 2026-07-21 MAC pinning survives
an in-place domain update.
Convergence restored (precondition 1): the dc0 inner root re-plans ZERO
DIFF. MAAS undisturbed -- 20 machines (18 Ready + 2 Deployed), 216 links,
every one still mode=static. MAAS still sees ONE block device per dc0 node,
which is expected and is why the re-commission is genuinely required.
Also recorded: the saved plan's MAC audit returned one hit that was a false
positive -- type_machine = "q35" inside the juju-01 create block, since
"mac" is a substring of "machine".
NOT DONE, gated separately: the re-commission with skip_networking=1, and
dc1 entirely. Capture: docs/audit/dc0-osd-juju-apply-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R1 precondition 3 VERIFIED read-only from voffice1: skip_networking=1 is the control
...
The amendment required verifying, BEFORE the apply, that re-commissioning
preserves the D-134 statics and the pinned MACs. Both halves now answered
from the headend, read-only, no mutation.
HALF 1 -- baseline measured: 18 Ready nodes, 7 interfaces, 12 links each,
mode=static on every link (6 planes x 2 families). MAAS 3.7's commission API
carries skip_networking: 'Whether to skip re-configuring the networking on
the machine after the commissioning has completed.' The option exists BECAUSE
the default reconfigures, which is exactly the flagged risk. The combination
R1 needs is skip_networking=1 with skip_storage UNSET -- network preserved,
storage re-scanned so the new /dev/vdb enters inventory.
HALF 2 -- the tofu change does not touch NICs. Both inner roots plan
6 add / 4 change / 0 destroy; the four in-place diffs change devices.disks
ONLY, gaining vdb. A grep for mac/interface across the whole diff returns one
hit and it is inside the NEW juju-01 create block. Risk is materially lower
than 2026-07-20 because MACs are now config-pinned -- the control introduced
after that incident precisely because the plan was untrustworthy.
NOT overclaimed: a clean plan was necessary but not sufficient in 2026-07-20,
when 0/9/0 in-place regenerated every MAC. So the sequence puts a MAC
verification BETWEEN the apply and the re-commission -- that incident became
fleet-wide because MAAS was told to re-commission while its records were
already stale.
Capture: docs/audit/r1-precondition3-verify-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
