| 2026-07-31 |
>>> BUNDLE DEPLOY SUCCEEDED: "Deploy of bundle completed." exit 0 <<<
...
Option D half 2. The 23 application definitions were removed INDIVIDUALLY,
never batched (hard rule 3), each read back, with the precondition re-verified
immediately before the first removal and the model read back empty afterwards.
Step-3.5 state survived the clear -- apt-mirror still set, all six spaces still
bound with both address families, checked not assumed.
MEASURED IMMEDIATELY AFTER: juju 56 applications / 9 machines / 33 units,
machines pending+allocating; MAAS 10 machines -- 9 Deploying, 1 Deployed (the
controller).
Three defects closed in the order they were hit: attempt 1's unknown option
prefer-ipv6 (D-101 ruling note); attempt 2a's missing policies/overrides.zip on
the client host; attempt 2b's barbican 265-vs-261 downgrade refusal (explicit
base on all 56 AND a clean model).
Operational note, because the first removal read as a failure: juju
remove-application PROMPTS by default and aborts on non-interactive stdin.
--no-prompt is required.
WHAT IS NOT CLAIMED: this means juju ACCEPTED and QUEUED the bundle. It does
NOT mean the cloud is up. 9 machines are allocating, 33 units pending; the
settle takes hours and nothing here asserts unit health or any service verdict.
G17's dc0 half is now genuinely armable -- the existing capture was taken on
the CONTROLLER VM and says so. The nine role nodes are booting for the first
time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Deploy attempt 2: prefer-ipv6 fixed, two new defects, model partially populated
...
The prefer-ipv6 failure did NOT recur. Two new ones did, and for the first time
the model is not empty: MEASURED 23 applications, 0 machines, 0 units.
Attempt 1 aborted during VALIDATION; this one reached EXECUTION, which is why
it left residue. NOTHING IS PROVISIONED -- no MAAS machine left Ready, no disk
written.
DEFECT 1: the rack's ~/repo-stage is a PARTIAL COPY of the repo with no
policies/, and bundle.yaml:214 is the only local-file reference in the deploy
input. juju resolves it relative to the bundle, so it must exist ON THE HOST
THAT DEPLOYS. Nothing caught it: --dry-run does not upload resources (the
SECOND --dry-run blind spot in two days), attempt 1 aborted before resource
upload, and preflight runs on voffice1 where policies/ DOES exist -- the same
wrong-host instrument class already recorded twice, and the second instance
this session. Fixed forward, both files sha256-verified.
DEFECT 2: a bundle relying on default-base is deployable exactly ONCE. The
re-run failed on barbican 265-vs-261. Measured: 265 is the ubuntu@22.04
revision, 261 the ubuntu@24.04 one. The 23 existing apps are all 22.04, so run
2a honoured bundle.yaml:85 -- but on a re-run, for an app that ALREADY EXISTS,
juju resolves without that default. A model-level default-base has NO EFFECT
(tested, then RESET). An explicit per-application base WORKS (tested read-only
on a throwaway; the error moved to the next app).
OWNED: two of those read-only tests were staged under /tmp and failed with
"no charm was found" -- the juju snap's PRIVATE /tmp, a trap this repo has
already recorded in CURRENT-STATE and the phase-4 runbook.
The rollback decision tree is written for tofu apply and defaults to
fix-forward, which is measured not to work here. A partial juju deploy is a
failure mode the repo does not cover. Recovery is an operator decision;
options are enumerated in the capture and the model residue is untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-deploy: rack input refreshed + hash-verified, preflight captured, dry-run green
...
(a) The dc0 rack's ~/repo-stage is a COPY, not a clone -- nothing updates it, and
it still held the pre-ruling vips overlay. Exactly one of four files differed;
copied, then ALL FOUR hash-compared against repo HEAD. The gitignored
octavia-pki overlay was deliberately not touched (SEC-029) and verifies
byte-identical at 5fc117f1, still 0600. A sweep is evidence about the deploy
only if the swept bytes ARE the deployed bytes.
(b) Preflight captured (242 lines, exit 1), run with
MAAS_PROFILE=vr1-dc0-region -- without it preflight is REGION-BLIND and emits 19
false negatives. Instrument proven current first: maas-profile-assert.sh
vr1-dc0-region hot-kid exits 0. P1-P4 and P7 PASS, incl. aligned VIPs 13 OK / 0
bad against the RE-RENDERED overlay. P5 FAIL, 11 findings.
(c) The P5 delta is ENUMERATED. The 2026-07-30 acceptance covered six findings
and says it covers "these six, enumerated, and nothing else". All six are still
present; FIVE NEW are all vr1-dc1 S2 rows (SEC-027/-028) -- the D-137 forward
register correctly reporting that dc1's region is not built. Diff appended to
the capture. Needs its own GA-R5 exchange before the deploy.
(d) Step 4.2 --dry-run against the fixed input: exit 0, 56 apps / 108 relations
/ 33 unit placements. The ceph-osd tags=openstack exposure is now MEASURED, not
reasoned: the plan places ceph-osd/0..3 on machines 5-8 by explicit id, so the
absent tag never has to match. The residual (a later UNPLACED add-unit) stands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Sweep both deploy inputs for option NAMES and VALUE TYPES: clean, and proven able to fail
...
The two extensions owed after the ruling, now run:
dc0 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
dc1 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
Both EXIT 0. dc1 introduces no new charm@channel pair -- checked, not assumed.
PROOF OF TEETH, because a clean sweep across two inputs is exactly the
implausibly-uniform result this repo has been burned by. Three defects planted
in a THROWAWAY copy, all three caught, EXIT 1:
[FAIL name] barbican does not declare option 'prefer-ipv6' (attempt-1 defect)
[FAIL type] keystone 'prefer-ipv6': declared boolean, got str ('true')
[note] glance 'vip': declared string, value is unquoted float
Type judgement is deliberately conservative: a mismatch is reported only where
juju's own coercion cannot save it. A scalar where a STRING is declared is a
NOTE, not a failure -- but it is still surfaced, because an unquoted value is
how a version string silently becomes a float. An UNRECOGNISED declared type
REFUSES rather than passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
prefer-ipv6 research: size option (b) and scope the sweep claim
...
Two corrections to my own framing, both measured:
(1) Option (b) is NOT a values-file edit. 'prefer-ipv6: true' is INJECTED
UNCONDITIONALLY by scripts/render-dc-overlays.py:243-246 whenever fam ==
"dual", and 'family: dual' is a single TOP-LEVEL key in
render/values/vr1-dc{0,1}-vips.yaml:3 -- not per-app data. Implementing (b)
means teaching the renderer which charms declare the option, re-pointing
tests/render-drift, and replacing invariant 9: a code change with a harness.
Recorded before the fork is put to the operator so the ruling is taken on the
real cost.
(2) The option-name sweep's claim is scoped inline. It checked NAMES, dc0
only. Option VALUE TYPES (the 'type:' field in every schema already
downloaded) and the dc1 input are both owed before the redeploy, and each
could cost an attempt the same way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
prefer-ipv6 research DONE: the option never drove the v6 HAProxy bind
...
Answers the 2026-07-31 ruled question ("Research what those 6 charms do with
v6 VIPs first"). Measured by downloading each charm at its pinned channel for
amd64/ubuntu@22.04 and reading its templates and code -- not Charmhub's
config-yaml, which answers only whether the knob is declared.
FINDING: in BOTH template families the ':::port' frontend bind is gated on
ipv6_enabled = not is_ipv6_disabled(), a read of the kernel
net.ipv6.conf.all.disable_ipv6 sysctl -- MEASURED 0 on a live MAAS-deployed
jammy node in this DC. The v6 VIP is assigned by pacemaker on per-address
family detection. vault ships no haproxy template at all; its listener is the
hardcoded literal [::]:8200. prefer-ipv6 is a UNIT ADDRESS-FAMILY switch, and
the one value it sets that sounds like a bind address (haproxy_host) is
consumed by no template in any charm downloaded.
So option (b) -- drop the option from the six, keep their v6 VIP legs -- is
what the measurement supports. NOT TAKEN: the ruling is the operator's, the
overlay is untouched and R2 is unamended.
Also recorded: provider-bundle-check invariant 9's mechanism claim is wrong
and must be REPLACED (never deleted) when a ruling lands; a new and separate
question about the SEVEN charms that do declare the option; a full
option-name sweep of the dc0 deploy input (56 apps / 81 assignments / 20
schemas / 0 unresolvable) showing prefer-ipv6 on the six is the ONLY unknown
option, so nothing else is waiting for attempt 2; and a runbook defect --
the dc0 deploy block omits vr1-dc0-machines.yaml, which is NOT a no-op (it
carries ovn-chassis bridge-interface-mappings).
Read-only session: no cloud mutation, no overlay edit, no ruling adopted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Bundle deploy attempt 1 FAILED: prefer-ipv6 unsupported on 6 of 13 VIP charms
...
ERROR cannot deploy application 'barbican': unknown option 'prefer-ipv6'.
Model measured EMPTY afterwards -- juju validated and aborted atomically, so
nothing is half-applied and Steps 1-3.5 remain good.
The dry-run passed and the deploy did not: --dry-run resolves charms and plans
relations but does NOT validate config option names. A green dry-run is not a
gate on option validity.
Measured against charmhub's own config-yaml per pinned channel: 7 of 13 VIP
charms declare prefer-ipv6, 6 do not. barbican never had it at any channel, so
it was not an obsoleted option -- R2's uniform application was never valid.
Not fixed: invariant 9 couples prefer-ipv6 to the v6 VIP legs, so this needs a
GA-R5 ruling, not a mid-deploy edit.
Owned: my first charmhub query used a wrong field and returned a uniform
'no config' across 13 charms -- implausible, which is what caught it.
Capture: docs/audit/stage5-dc0-deploy-attempt1-20260731.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
JUJU CONTROLLER BOOTSTRAPPED: vr1-dc0-controller live, active/idle
...
Fourth attempt, first success. Each earlier failure was a real defect one layer
deeper and all three are now closed: D-138 (client into the DC), the controller
under-carve fix (default route), and D-132 q1 (per-DC MAAS region).
controller/0 active/idle on juju-controller rev 311; machine 0 started
ubuntu@22.04 as subtle-grouse; MAAS reads arfr7p Deployed ubuntu/jammy.
Transcript evidence: agent binaries resolved on attempt 1 (339-retry class stays
closed); targeting took the tagged VM not a role node; juju offered the v6 leg
restored earlier this session before connecting on metal-admin.
Capture: docs/audit/stage5-bootstrap-dc0-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 v4 node carve COMPLETE: 10/10 machines, gate 134/0, diff vs pre-migration EMPTY
...
Canary-first then node by node, never batched. 11 + 13x8 + 4 mutations, every one
read back. Nine role nodes on six planes with OVS br-ex; the juju controller on
two planes raw with no br-ex and enp3-6 left on auto VLANs.
Verified by diffing against the Office1 pre-migration capture: every node's six
v4 legs match address for address. Compared by LEG SET, since system_id and
hostname are both re-minted at re-enlistment.
Instrument note: the first extraction read the wrong capture field and returned
'-' for every old address, which would have read as 'the old region had no
statics'. Caught by the implausible shape, not by the parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-30 |
Close sweep: queued findings F1-F10, memory review, skill invariants
...
Sweep capture docs/audit/queued-findings-20260730-dc0-region-migration.txt.
Three items lived ONLY in the transcript and would have been lost on a
clear:
F1 -- this session's permission rules exist only in the GITIGNORED
.claude/settings.local.json, so their verbatim text is now recorded,
including a broad Bash(ssh vr1-dc0-rack *) that an interactive approval
auto-added and which deserves review.
F2 -- the Office1 region STILL registers a rack controller on the dc0 rack
(vvr1-dc0/7chphy) and still holds the dc0 region VM's machine record. Not
harmful (Office1's DHCP for that VLAN is off, verified by process) but
dual-registration the operator should rule on.
F10 -- the five instrument errors of this session as one pattern, with the
rule that catches them.
F9 carries the next session's first commands and every measured input the
carve tool needs.
GA-R7 memory review: sharpened the classifier memory with the profile-token
lesson (a rule pinning 'maas admin' would have matched step 1 of the DHCP
cutover and silently missed step 3), and added an instrument-currency
memory. Both graduate to repo surfaces; neither is the only copy.
Skill swept: the per-DC region migration invariants folded in -- region
identity by rack, ids do not cross regions, the per-node ordering, batches
of three, zero SSH keys, the role-dependent carve, the role-tags gate that
passes on an incomplete tag set, and the two scripts that must not be
pointed at a new region.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Plane IPAM complete on the new region: 24/0 PASS, existing tooling only
...
dc-plane-ipam.sh check vr1-dc0 goes from pass 7 / fail 17 to pass 24 /
fail 0, exit 0. Two idempotent applies, dry-run first, every write read
back: 13 D-134 reserved bands plus the FIP pool, then the 5 missing v6
plane subnets each created on the same MAAS vlan as its v4 twin.
No new code was needed. The tool's forced sequencing also proved itself --
the reserve run skipped every v6 band with 'not in MAAS yet, run carve-v6
first' rather than failing or silently no-opping, so running the two out of
order is safe.
This is the step that justifies the tunnel architecture: carve-v6 derives
its prefixes from the NetBox apex, reachable only from voffice1, while the
target region is reachable only via the rack. Neither host alone could have
run it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
All ten dc0 machines Ready in the per-DC region; task 6 complete
...
Enlisted as canary + three batches of three. Shapes match D-121 Option C
exactly (3x16/64, 2x12/48, 4x8/24, 1x4/8), all with 6 interfaces, every one
matched to its libvirt domain by pinned boot MAC -- hostname and system_id
were both re-minted.
maas-role-tags.sh check: 0 tags missing, 0 nodes needing a tag, 0 not in
MAAS, exit 0.
But it reported that clean PASS while the machines were still missing the
openstack-vr1-dc0 placement tag, which is not in its ROLES set and has no
creator anywhere in the repo. bundle.yaml places on that tag and
dc-node-v6-carve.py dies without it, so both the deploy and the v6 carve
would have failed. Caught only by diffing against the pre-migration
snapshot. Applied to the nine role nodes and deliberately not to the
controller -- reproducing the Office1 shape row-for-row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Canary re-enrolment: the ordering a fresh region actually needs
...
Canary xqqwdq/moral-salmon = vr1-dc0-storage-04 (a storage node, least
central of the ten). Office1 23 -> 22; it re-enlisted into the new region
carrying boot MAC 52:54:00:2b:ed:ab -- the machine that arrived is the
machine that left. system_id AND hostname are both re-minted
(xqqwdq -> 6q4syf, moral-salmon -> civil-bug), so the boot MAC is the only
stable key; the pre-delete snapshot records it as the identity.
MEASURED ORDERING, not the obvious one: a freshly-enlisted machine has no
power config, so MAAS cannot power-cycle it and cannot drive commissioning.
delete -> power on BY HAND via virsh -> self-enlist -> SET POWER -> abort ->
commission. Skipping the power step wedges the machine in Commissioning
producing zero events, because the enlistment ephemeral powers the node off
and nothing can bring it back.
maas-node-power.sh already supports the split the migration needs: VIRSH_URI
enumerates from voffice1 over the transit /30 while the positional address
stores what the REGION dials over metal-admin. Result was a real
query-power-state, proving the region drives the rack's libvirt with the
SEC-012 key in its snap.
Found: the per-node serial log stopped capturing on 2026-07-21 while the
domain XML still configures append-mode logging. It contains a complete,
plausible boot that I read as current and reported as fact before checking
the mtime. An append log that silently stopped appending returns confident,
well-formed, wrong evidence. Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DHCP handover EXECUTED: dc0 metal-admin now served by the DC-local region
...
Ruled order, region identity proven first. Office1 vlan update 4 0
dhcp_on=false -> read back False; verified STOPPED BY PROCESS (pgrep -c
dhcpd = 0 on the rack, because MAAS self-report is not evidence -- the
2026-07-20 Temporal incident class); new region vlan update 0 0
dhcp_on=true primary_rack=c3aqh8 -> read back True on fabric-0 / space
metal-admin; verified RUNNING BY PROCESS on hot-kid with the rack
re-checked at 0. Exactly one DHCP server on the segment. All 10 machines
Ready and powered off throughout.
Recorded so it is not later read as a fault: the new region also starts
dhcpd6 where the Office1 rack ran v4 only. Its v6 metal-admin subnet has
NO dynamic range -- only rfc-4291-2.6.1 and assigned-ip/reserved, same as
the old region -- so it has nothing to hand out and nodes keep their ruled
statics.
The permission wall was a rule that FAILED TO MATCH, not a classifier
fault. Existing ask rules pin the double-quote form while the commands used
single quotes, and pin the maas admin profile while the new region needs
maas vr1-dc0-region. Operator-approved ask rules added to
settings.local.json (gitignored; committed team policy unchanged),
profile-agnostic so both regions match, plus read-only pgrep/ps allow
rules. The first rule shape I proposed pinned maas admin and would have
matched step 1 while silently missing step 3 -- stalling the cutover at
exactly the two-servers boundary.
Capture docs/audit/dc0-dhcp-handover-20260730.txt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Session record: dc0 region topology 40/0; cutover blocked with exact commands
...
Records the fabric-rename remediation (changelog item 12), the two
pre-cutover checks (hot-kid IS eligible as primary_rack for fabric-0 -- had
it not been, DHCP-off would have succeeded and DHCP-on failed, leaving the
segment with no DHCP and ten machines about to PXE), the measured NIC ->
plane order, and the exact unblock sequence with every parameter
re-resolved live.
hot-kid is c3aqh8 in the new region and tw7ptw in Office1 -- the same
machine, two ids.
Capture updated with the corrected 40 passed / 0 failed, exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 region plane topology BUILT and verified (39/0); DHCP cutover blocked
...
dc-region-topology.sh applied to the new dc0 region: 5 named plane fabrics,
6 spaces, 4 plane subnets, 10.12.4.0/22 MOVED off the auto fabric-1 onto
vr1-dc0-provider-public, 6 VLAN->space bindings, openstack-vr1-dc0 tag.
Named check now reads 39 passed / 0 failed, exit 0.
metal-admin service config set: dns_servers=10.12.8.6, allow_dns=false,
D-134 dynamic range .201-.254, subnet resolved by CIDR.
The DNS target was measured rather than carried over, and my first reading
was wrong: a +time=3 +tries=1 probe suggested the new region's BIND could
not resolve externally. That was a cold-cache timeout. At +time=5 +tries=3
it answers archive.ubuntu.com, streams.canonical.com and api.snapcraft.io
with flags qr rd ra. Pointing node DNS at the DC-local region is correct --
it is authoritative for the zone the migrated nodes will live in and drops
the cross-fiber dependency D-132 q1 exists to remove.
Measured in passing, and worth keeping: VLAN row id 5005 is dc0's
metal-admin VLAN in Office1 and vr1-dc0-data-tenant in the new region.
Same integer, different DC plane.
The DHCP cutover was REFUSED by the harness classifier and was not retried
in an altered shape. Verified read-only afterwards that nothing is
half-applied: Office1 still reads dhcp_on=True primary_rack=7chphy and the
rack still runs exactly one dhcpd on virbr2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Per-region power address, region power-key tool, and two red gates fixed
...
lib-hosts.sh: MAAS power ops originate from the REGION, so the reachable
rack address depends on which region dials. Measured from vr1-dc0-maas-01:
the rack's transit leg 172.31.0.2:22 is CLOSED, its metal-admin leg
10.12.8.2:22 is OPEN. Adds VIRSH_POWER_ADDRESS_FROM_{OFFICE1,DCREGION} for
both DCs (dc1's 10.12.68.2 measured on the dc1 rack, not inferred from
dc0's .2). VIRSH_POWER_ADDRESS still aliases the Office1 form so existing
callers are byte-for-byte unchanged. dc-selector 81 checks; 4 mutations
each killed tests.
New scripts/maas-region-power-key.sh (harness 44/44) installs and verifies
the per-DC MAAS->libvirt key inside a region's snap, ending on a real virsh
call rather than on config. Runbook prerequisite 5 carried this as prose,
and prose is not a gate -- it must also be re-asserted after every snap
refresh. LIVE at dc0: 9/9 exit 0, virsh enumerates 12 domains. The key is
the SEC-012 dc0 key already authorized on that rack, so per-DC isolation
holds.
Three defects its own testing found: ssh-keygen -lf reads an adjacent .pub
and reports ITS fingerprint; a snap's data root (/var/snap) and exec root
(/snap) are not interchangeable; the snap's virsh must run inside the snap
runtime or it fails on missing libraries.
tests/node-vm T8/T9 were RED at HEAD before any of this session's work
(proven by stashing and re-running): commits 086c827 + 447315f added the
region VMs, moving the dc0 root to 11 macs lists / 66 MAC literals against
assertions reading 10 / 60, and the gauntlet was not re-run. Re-pointed to
the new invariant, reason recorded in-file, re-proven able to fail.
HARNESS-MANIFEST re-recorded (91) for the two new harnesses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Region-resolution gate: maas-profile-assert.sh + named dc0 region profile
...
The wrong-REGION hazard is the migration's unrecoverable failure mode: 13
repo scripts default to MAAS_PROFILE=admin, which on voffice1 resolved to
the OFFICE1 region. Against the wrong profile a carve recreate is a silent
no-op that prints PASS, and machine delete destroys the real record.
New scripts/maas-profile-assert.sh asserts which region a profile resolves
to by rack-controller IDENTITY (a machine count is not proof -- two regions
can hold the same number). Harness 20/20; mutation pass found one of its own
assertions could not fail (nameless rack entries were caught by a different
branch) and it was made branch-specific.
Live both directions: admin -> 23 machines / office1 trio (exit 0);
vr1-dc0-region -> 0 machines / hot-kid (exit 0); admin asserted as hot-kid
(exit 1). Capture docs/audit/dc0-region-profile-assert-20260730.txt.
Tunnel voffice1 -> rack -> 10.12.8.6:5240 because no single host has repo +
CLI + new-region reach + NetBox apex reach. Application-layer and
rack-originated, so SEC-010 is not punctured; nothing was rerouted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 session close: Stage 5 opened, D-138 + D-132 ruled, dc0 region live
...
Sweep first: every in-session finding grepped against CURRENT-STATE and the
changelog. THREE lived only in the transcript and would have been lost -- the
absent openstack CLI on the DC client host (blocks Step 7+), the metal-admin
v6 rack-leg gap, and the plane-fabric mtu=1500 vs libvirt 9000 mismatch. All
now in docs/audit/queued-findings-20260730-stage5.txt (F1-F12), with six tool
traps collected as platform-traps candidates.
F6 recorded honestly: the as-executed log for this window is PARTIAL. The
classifier refused several `script -aqe ... -c` wrapped forms while the plain
ssh form passed, so those calls ran unwrapped. The index row says so -- a log
that silently looks complete is worse than one declaring its gap.
Bookend appended (14 lines, under the GA-R4 cap; ledger 250/300 so no
rotation owed). Counters: 3 decisions, SEC 23, D 139 / DOCFIX 206 /
BUNDLEFIX 053.
GA-R7 memory review: one entry added (as-executed vs classifier), indexed.
Nothing in memory claims operator policy.
Stage 5 remains OPEN. Nothing is half-applied -- Office1 still owns all 9 dc0
nodes, correctly carved and Ready.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Region images synced (jammy+noble); pre-migration carve captured
...
Both releases now Synced on the dc0 region, 6 rows each, matching Office1.
jammy is NOT seeded by `maas init` (noble only) and the OpenStack nodes
require it. The first import raced the selection's creation and returned
noble-only while reporting is-importing=false -- assert on Synced ROWS, never
on the import call returning.
Measured and recorded so the two are not conflated: MAAS boot images do NOT
come from the DC mirror. The region pulls images.maas.io; the mirror 404s on
every simplestreams index path because dc-mirror.sh mirrors debmirror APT
packages only. Three artifact classes, one local. Consequence: the
D-135/D-107 egress narrowing would BREAK image sync and juju bootstrap, not
just slow them, and nothing sequences it after them.
Pre-migration carve captured to docs/audit (179 lines): fabrics, VLANs,
spaces, subnets, D-134 bands, tags, per-node identity/power and the full
per-interface carve. MAAS cannot move machines between regions, so the
migration must recreate all of it.
The destructive half has NOT started -- the Office1 region still owns all 9
dc0 nodes, correctly carved and Ready. Nothing is half-applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
G17 dc0 CAPTURED (both assertions PASS); bootstrap root cause = under-carved controller
...
G17's one-shot first-boot window opened during the second bootstrap attempt and
was taken rather than lost. Capture docs/audit/g17-dc0-firstboot-20260730.txt,
exit 0: mirror asserted on FOUR body fields of a real package path (not the
autoindex root), and the node time source is the MAAS rack 10.12.8.2 chaining
to the region, not the DC edge. Scope stated: the node is the dc0 CONTROLLER
VM, not a role node; G17's dc1 half stays OPEN.
Two defects in G17's own text, both measured: it names `chronyc sources` and
chrony is NOT installed on the MAAS jammy image (node runs systemd-timesyncd),
so the gate as written can only REFUSE; and a first pass false-FAILED on
`printf | grep -q` under pipefail -- the same trap recorded earlier the same
day. Capture uses here-strings.
Root cause of both bootstrap failures is measured and is NOT architectural:
the controller VMs are UNDER-CARVED. D-138's path works -- the second attempt
reached "Connected to 10.12.8.5" -- then failed fetching the juju agent
because the node has NO DEFAULT ROUTE. Role nodes carry provider-public with
gw 10.12.4.1; the controllers carry metal-admin only, because they were added
after both carves and Stage 4's 90 NIC re-homes covered the 18 role nodes.
Owned: this session's egress re-probe was run FROM THE RACK and reported the
window open. The rack holds a default route nodes lack, so it did not
discharge the caveat for a node-side fetch. Same wrong-host class as the
dc-mirror instrument note, made after writing that note.
Blocked on a harness permission wall for four MAAS interface calls, not on a
decision. Not retried in altered shapes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 5 BLOCKED: no juju-client->node-plane path; D-134 .5 now BUILT both DCs
...
Bootstrap attempted and failed. Machine selection was correct (7n87bt, the
tagged VM) and MAAS deployed jammy end to end -- confirming the
--bootstrap-base ubuntu@22.04 pin against a live deploy. juju then could not
SSH the machine and released it.
Root cause: voffice1 has NO route to any DC node plane and two deliberate
controls forbid one -- SEC-010's transit FORWARD-drop on the rack, and
libvirt's blanket reject into the isolated plane bridges. The DC edge has no
metal-admin leg. Nothing regressed; this path never existed.
This is a contradiction between RULED surfaces: SEC-010/D-052 make
metal-admin DC-local and forbid the region routing to 10.12.8.0/22, while
D-100 says the fiber carries Juju traffic and D-128 puts the juju client on
voffice1. SEC-010's "pinning is free" justification was priced against MAAS,
which proxies at the app layer; juju dials the machine at L3 and was not in
scope. Needs a ruling, not a firewall edit. No reachability change made.
BUILT, both DCs (operator: "Fix now: static .5 + v6, re-bootstrap"): the
controller VMs held AUTO v4-only addresses; now static 10.12.8.5 +
fd50:840e:74e2:220::5 and 10.12.68.5 + fd50:840e:74e2:320::5. v6 prefixes
confirmed by VLAN pairing, not inferred. D-134's amendment was
ruled-but-not-built until now.
Capture: docs/audit/stage5-bootstrap-reachability-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
STAGE 5 OPEN: P5 accepted by ruling, entry gate captured on voffice1
...
Opens Stage 5 / Phase 4 (Juju controller + OpenStack bundle, per DC) on the
standing operator directive. GA-R1/C1: the status change and CURRENT-STATE
land in one commit. GA-R5: the P5 ruling is committed and pushed BEFORE any
dependent work.
Entry gate, measured ON voffice1 (the only host whose reading counts):
P1/P2/P3/P4 PASS, P7 PASS 37/0 with the literal zone line, P5 FAIL on 6
pre-existing credential-register findings. Capture
docs/audit/stage5-preflight-dc0-20260730.txt.
Operator ruling, exact utterance: "Accept and proceed to deploy (Recommended)".
The acceptance covers those six findings, enumerated, and nothing else.
Logged not executed (hard rule 1): bundle.yaml:592 gives ceph-osd the stale
VR0 constraint tags=openstack, a tag measured absent from the VR1 region.
Decided at Step 4.2's dry-run, where its impact is first observable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Q2 withdrawn: D-137 fork 1 was ruled 2026-07-25, not open
...
Third stale-premise item in queued-findings-20260730.txt (after Q1). Q2 opens
'D-137 OPEN FORK 1 (enforcement strength) -- still unruled'. Sub-ruling 1 ruled
it 2026-07-25 at option (b), utterance 'Blocking in preflight', and its
CONSEQUENCE block explicitly declines option (c): 'NOT adopted: the PreToolUse
guard (so an agent is not constrained at write time, only at the deploy gate)'.
So guard-destructive.py is NOT D-137's enforcement point -- it is the older
CLAUDE.md/DOCFIX-006 mechanism, and hardening it needs no ruling to proceed.
This was verified BEFORE the reissue minter was built, which is why 'Full script
minter now' was consistent with the ruling rather than a side-effect decision.
The misfire evidence stays valuable and the tally is now SEVEN. The two newest
were collected today: the guard blocked a command whose only purpose was to TEST
it (it read nothing), and then blocked the heredoc writing this very correction
into the audit file -- documentation prose, no file read. Both are the same class
as the earlier five: the matcher cannot distinguish reading secret material from
merely naming it.
Correction APPENDED, not rewritten (the stage4-mirror-gate precedent).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F9 CLOSED: controller certs reissued into their own zones, both DCs
...
Executed on voffice1 via Step 1.0-REISSUE at HEAD 7b6a2e4. Operator-gated PER DC
('Go ahead' each), never batched; dc0 verified clean before dc1 was touched.
Capture: docs/audit/octavia-reissue-executed-20260730.txt
vr1-dc0 -> omega.dc0.vr1.cloud.neumatrix.local, IP 10.12.4.57 + f02:11::57,
serial ...250D -> ...250E
vr1-dc1 -> omega.dc1.vr1.cloud.neumatrix.local, IP 10.12.64.57 + f03:11::57,
serial ...674DD -> ...674DE
dc1's OLD cert carried dc0's LABEL as well as the wrong region -- the cross-DC
case A11 exists for. A11 confirms per-DC independence on both DCs after the mint.
Both: fresh EC P-256 key with the rotation PROVEN (public key differs from the
outgoing cert's), neither CA regenerated so the amphora trust domain is intact,
overlay surgery measured other_values_identical=4 changed_lines=1 line_count=8.
ACCEPTANCE was the literal positive line, not the verdict: while os-public-hostname
is unset, A12/A13 RECORD rather than fail, so PASS alone is compatible with a wrong
certificate. Both DCs now carry 'DNS SANs are all in this DC's expected zone'.
verify PASS 37/0 both DCs (was 29/0; +8 from the new assertions). creds-matrix:
same 5 pre-existing findings, no new finding, no S4 drift. No staging residue,
neither run reached exit 5.
The hardened re-run guard proved itself live: re-running dc0 immediately after
returned exit 4 'already correct ... and verify returns clean' -- the fix for the
defect that had it refusing to repair certs verify was failing.
STILL OWED, operator-gated: backup custody. Two archives on the headend hold BOTH
CA private keys, BOTH passphrases and the overlay; 1.0-REISSUE.4 pulls them to the
jumphost creds folder and removes the headend copy. They are also an unregistered
credential class (no creds-matrix row, no retention rule). Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 sweep: correct the artifact that filed Q1, and one stale residue
...
Advisor review caught that correcting CURRENT-STATE and the ledger was not
enough: docs/audit/queued-findings-20260730.txt -- the sweep capture the 07-30
bookend points at -- still presented Q1 as an open ruling that 'blocks any FQDN
cert', enforced by an A12 refusal that no longer exists. A next session reading
it would re-derive Q1 exactly as this one did, reopening the loop just closed.
- queued-findings-20260730.txt: APPENDED correction (not a rewrite; the
stage4-mirror-gate-20260727.txt precedent), naming the two claims that are now
false and pointing at the evidence. Q2 explicitly untouched.
- dc-dc-deployment-workflow.md:257: described phase-6 as proposing the retired
dc1/dc2-hostnames.yaml spelling. MEASURED: the runbook itself was already
corrected to the region-qualified form by an earlier session -- only the
workflow doc's description of it lagged. Under D-119 IS vr1-dc0, so the
-hostnames.yaml interpolation used elsewhere is correct as written.
- The DOCFIX-205 capture's section 1 quoted a 3-line span starting mid-sentence
and left a clause orphaned. Repaired to two verbatim quotes, both re-verified
present in design-decisions.md. CURRENT-STATE cites this file as evidence and
this repo's discipline is exact captures.
octavia-pki 23/23; repo-lint 0 fail (L10 gated this commit correctly -- the
docs/audit/ change set required CURRENT-STATE to move with it).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 capture: the D-117 supersession, annotation coverage 0/4 -> 4/4, live A12
...
Evidence file cited by CURRENT-STATE's superseding block. Carries the quoted
D-117 Status line, D-119's selector-only discharge, measured annotation coverage,
the phase-6 expansion old-vs-new in a real shell, live A12 on the headend
(PASS 29/0 both DCs, both INERT), and both hosts' gate results.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-bookend sweep: the operator's working commands were NOT in the repo
...
Operator: "I'm worried that the fixes we made to the defective commands will be lost when we
close this session ... Complete a full sweep before the bookend."
THE CONCERN WAS WELL FOUNDED. The corrected Step 5/6 commands -- the ones that actually produced
both DCs' live PKI -- had been reviewed in conversation, judged equivalent, and never folded in.
Four items now landed:
1. Portable base64. GEN.d shipped the GNU-only `-w0` flag; what ran was
`base64 < f | tr -d '\r\n'`. The runbook described a command nobody executed.
2. The operator's VIP_OVERLAY:? guard in GEN.c.
3. The heredoc column-1 requirement, as a CAUTION heading rather than a footnote, because that
failure is silent: no alt_names section yields a certificate with NO SANs while every
openssl command still prints OK.
4. ${DC_LABEL:?} at both CA call sites -- F8's sibling, where an unset label bakes a DC-less
subject into a 10-year CA with no error anywhere.
Not done, with the reason recorded: the heredoc was NOT rewritten as printf lines. More
paste-proof, but an untested rewrite of a step that mints 10-year CA material and cannot be
exercised end-to-end from an agent session. Risk bounded instead by A9 plus harness T2.
F9 is now structural rather than remembered: A12 arms itself from os-public-hostname appearing as
a real option key. Live PASS 29/0 both DCs, harness 21/21.
A new ruling-shaped gap, now blocking rather than filed: D-008's shape plus D-106:2563's VR1
instantiation do not say whether substrate vr1-dc1 is dc1 by token or dc2 by position -- the
DC1/DC2 ambiguity item 3.1 retired elsewhere, here deciding certificate identity. Both live certs
carry dc0.vr0, the VR0 region, wrong under either reading. A12 refuses rather than picking.
A precedence bug the harness caught: REFUSE was checked before FAIL, so once A12 armed a
CONFIRMED wrong-region SAN reported as "could not evaluate". A known defect outranks an
unevaluated one.
Sweep capture docs/audit/queued-findings-20260730.txt carries F10-F13 and two ruling-shaped
questions. F10 is the one worth acting on: mint-ref line numbers drift silently and S4 cannot see
it, since it asserts only within-EOF. That bit three times in two sessions and was caught by hand
every time, never by a gate. All 24 octavia refs re-anchored in a single-pass mapping keyed by row
id -- 391 was simultaneously an old and a new value, so sequential seds would have corrupted it.
repo-lint 0 fail; octavia-pki 21/21; creds-matrix 65/65.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |
Per-DC Octavia PKI generated: independence proven, identity unverified, 12 real findings
...
The mint was operator-executed on the headend; the session guard hard-blocks it and that was
not worked around.
PER-DC INDEPENDENCE PROVEN, which is the whole point of F1. Both DCs hold 12 files across 3
dirs, and sha256 DIFFERS between DCs for both encrypted CA keys, the controller cert and the
overlay. Before F1 those columns would have been identical, because the second generation would
have overwritten the first. Both overlays are mode 600, carry exactly five lb-mgmt-* keys, are
ASCII clean and are gitignored -- the F4 gate held.
NOT CLAIMED AS DONE: the four CA subjects, the controller SAN sets and the chain verifications
are unread, because the guard blocks openssl against those paths. Recorded as structurally
complete with identity unverified. A cert can be well-formed and name the wrong DC.
P5 on the headend went 6 -> 18 findings and all 12 new ones are genuine.
E2 x4: the CA certificates are mode 664. This session had explicitly called that harmless "for
public certs", reasoning only about confidentiality. That was wrong -- 664 is group-WRITABLE, so
a group member can replace a trust anchor. Integrity, not secrecy, is the exposure.
E3 x8: four generator outputs per DC (the controller certificate, the CA serial file, the
openssl config and the signing request) exist with no matrix row. The 18-row set landed earlier
today was incomplete -- "enumerate what exists, not only what is declared", applied to this
session's own work.
Validation worth recording: all 12 would have been INVISIBLE before today's F2 and F6 fixes,
since the rows were singleton and the locations resolved on the jumphost. The register found
real defects in exactly the mint it was repaired to observe.
F8: the controller-cert step's heredoc is a live paste hazard -- the operator hit it. An
indented terminator can yield a config with no alt_names section, producing a certificate with
NO SANs while every step still reports OK, and nothing asserts the SAN set.
F9: the controller certificate's DNS SANs are coupled to D-106 and nothing recorded it. They
are inert only because os-public-hostname is set nowhere (B5 IP-ONLY; R5 refused setting it at
Stage 5 as a D-019 repeat). When Stage 7 turns FQDN endpoints on, both DCs' certificates
carrying dc0.vr0 becomes wrong exactly when it starts mattering; likely disposition is reissue
inside D-106's own FQDN-SAN step.
Remediation logged not executed: tighten the six certificate modes; and for E3 either declare
all four artifacts per DC or have the generator drop the two build intermediates and declare the
two that must persist.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F7: office1-tailscale is labelled a subnet router and advertises no routes
...
Raised from an operator report of Permission denied (publickey) reaching voffice1 "from my
workstation via tailscale". Measured rather than assumed, and the premise did not survive.
tailscale is absent from BOTH voffice1 and vcloud -- no binary in any standard location, no
state dir, no interface, service not running. Checked with explicit path probes after an
initial `command -v`, because tool-absent-from-PATH reported as feature-absent is a trap this
repo has hit repeatedly.
The tailnet node is a separate LXD guest INSIDE voffice1: office1-tailscale, RUNNING,
100.64.0.53, site leg 10.10.1.11. Its tailscale status reports AdvertisedRoutes: <none>. So
nothing on the tailnet can reach 10.10.0.20 -- there is no route and there never was.
Two records made the false belief look supported: the ssh stanza labels it "Office1 subnet
router (D-107)", and the operating skill's routing table says the workstation reaches voffice1
over the tailnet, citing D-107 -- which is titled "Airgap posture, per-DC artifact mirror, and
NTP (VR1)" and rules nothing of the sort. A real citation that does not support its claim.
RULED (operator): option (a), ProxyJump through vcloud. Option (b), advertising 10.10.0.0/24
onto the tailnet, was presented and not taken -- it exposes the isolated site net, a posture
change against the airgap subject D-107 actually governs.
Fixed operationally outside the repo: vcloud's ssh config now matches Host voffice1 10.10.0.20,
so the raw IP resolves to the office1 service key with IdentitiesOnly yes instead of falling
through to the id_* defaults, which was the actual cause. Both forms verified, with a negative
control proving the key is not offered to unrelated hosts.
Third wrong record this session, after D-124's MAC-sourcing instruction and the skill citation.
Two of the three surfaced only because something adjacent was being fixed; nothing here gates
prose, so no check could have caught them.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
