| 2026-07-30 |
G17 dc0 CAPTURED (both assertions PASS); bootstrap root cause = under-carved controller
...
G17's one-shot first-boot window opened during the second bootstrap attempt and
was taken rather than lost. Capture docs/audit/g17-dc0-firstboot-20260730.txt,
exit 0: mirror asserted on FOUR body fields of a real package path (not the
autoindex root), and the node time source is the MAAS rack 10.12.8.2 chaining
to the region, not the DC edge. Scope stated: the node is the dc0 CONTROLLER
VM, not a role node; G17's dc1 half stays OPEN.
Two defects in G17's own text, both measured: it names `chronyc sources` and
chrony is NOT installed on the MAAS jammy image (node runs systemd-timesyncd),
so the gate as written can only REFUSE; and a first pass false-FAILED on
`printf | grep -q` under pipefail -- the same trap recorded earlier the same
day. Capture uses here-strings.
Root cause of both bootstrap failures is measured and is NOT architectural:
the controller VMs are UNDER-CARVED. D-138's path works -- the second attempt
reached "Connected to 10.12.8.5" -- then failed fetching the juju agent
because the node has NO DEFAULT ROUTE. Role nodes carry provider-public with
gw 10.12.4.1; the controllers carry metal-admin only, because they were added
after both carves and Stage 4's 90 NIC re-homes covered the 18 role nodes.
Owned: this session's egress re-probe was run FROM THE RACK and reported the
window open. The rack holds a default route nodes lack, so it did not
discharge the caveat for a node-side fetch. Same wrong-host class as the
dc-mirror instrument note, made after writing that note.
Blocked on a harness permission wall for four MAAS interface calls, not on a
decision. Not retried in altered shapes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 5 BLOCKED: no juju-client->node-plane path; D-134 .5 now BUILT both DCs
...
Bootstrap attempted and failed. Machine selection was correct (7n87bt, the
tagged VM) and MAAS deployed jammy end to end -- confirming the
--bootstrap-base ubuntu@22.04 pin against a live deploy. juju then could not
SSH the machine and released it.
Root cause: voffice1 has NO route to any DC node plane and two deliberate
controls forbid one -- SEC-010's transit FORWARD-drop on the rack, and
libvirt's blanket reject into the isolated plane bridges. The DC edge has no
metal-admin leg. Nothing regressed; this path never existed.
This is a contradiction between RULED surfaces: SEC-010/D-052 make
metal-admin DC-local and forbid the region routing to 10.12.8.0/22, while
D-100 says the fiber carries Juju traffic and D-128 puts the juju client on
voffice1. SEC-010's "pinning is free" justification was priced against MAAS,
which proxies at the app layer; juju dials the machine at L3 and was not in
scope. Needs a ruling, not a firewall edit. No reachability change made.
BUILT, both DCs (operator: "Fix now: static .5 + v6, re-bootstrap"): the
controller VMs held AUTO v4-only addresses; now static 10.12.8.5 +
fd50:840e:74e2:220::5 and 10.12.68.5 + fd50:840e:74e2:320::5. v6 prefixes
confirmed by VLAN pairing, not inferred. D-134's amendment was
ruled-but-not-built until now.
Capture: docs/audit/stage5-bootstrap-reachability-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
STAGE 5 OPEN: P5 accepted by ruling, entry gate captured on voffice1
...
Opens Stage 5 / Phase 4 (Juju controller + OpenStack bundle, per DC) on the
standing operator directive. GA-R1/C1: the status change and CURRENT-STATE
land in one commit. GA-R5: the P5 ruling is committed and pushed BEFORE any
dependent work.
Entry gate, measured ON voffice1 (the only host whose reading counts):
P1/P2/P3/P4 PASS, P7 PASS 37/0 with the literal zone line, P5 FAIL on 6
pre-existing credential-register findings. Capture
docs/audit/stage5-preflight-dc0-20260730.txt.
Operator ruling, exact utterance: "Accept and proceed to deploy (Recommended)".
The acceptance covers those six findings, enumerated, and nothing else.
Logged not executed (hard rule 1): bundle.yaml:592 gives ceph-osd the stale
VR0 constraint tags=openstack, a tag measured absent from the VR1 region.
Decided at Step 4.2's dry-run, where its impact is first observable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Q2 withdrawn: D-137 fork 1 was ruled 2026-07-25, not open
...
Third stale-premise item in queued-findings-20260730.txt (after Q1). Q2 opens
'D-137 OPEN FORK 1 (enforcement strength) -- still unruled'. Sub-ruling 1 ruled
it 2026-07-25 at option (b), utterance 'Blocking in preflight', and its
CONSEQUENCE block explicitly declines option (c): 'NOT adopted: the PreToolUse
guard (so an agent is not constrained at write time, only at the deploy gate)'.
So guard-destructive.py is NOT D-137's enforcement point -- it is the older
CLAUDE.md/DOCFIX-006 mechanism, and hardening it needs no ruling to proceed.
This was verified BEFORE the reissue minter was built, which is why 'Full script
minter now' was consistent with the ruling rather than a side-effect decision.
The misfire evidence stays valuable and the tally is now SEVEN. The two newest
were collected today: the guard blocked a command whose only purpose was to TEST
it (it read nothing), and then blocked the heredoc writing this very correction
into the audit file -- documentation prose, no file read. Both are the same class
as the earlier five: the matcher cannot distinguish reading secret material from
merely naming it.
Correction APPENDED, not rewritten (the stage4-mirror-gate precedent).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F9 CLOSED: controller certs reissued into their own zones, both DCs
...
Executed on voffice1 via Step 1.0-REISSUE at HEAD 7b6a2e4. Operator-gated PER DC
('Go ahead' each), never batched; dc0 verified clean before dc1 was touched.
Capture: docs/audit/octavia-reissue-executed-20260730.txt
vr1-dc0 -> omega.dc0.vr1.cloud.neumatrix.local, IP 10.12.4.57 + f02:11::57,
serial ...250D -> ...250E
vr1-dc1 -> omega.dc1.vr1.cloud.neumatrix.local, IP 10.12.64.57 + f03:11::57,
serial ...674DD -> ...674DE
dc1's OLD cert carried dc0's LABEL as well as the wrong region -- the cross-DC
case A11 exists for. A11 confirms per-DC independence on both DCs after the mint.
Both: fresh EC P-256 key with the rotation PROVEN (public key differs from the
outgoing cert's), neither CA regenerated so the amphora trust domain is intact,
overlay surgery measured other_values_identical=4 changed_lines=1 line_count=8.
ACCEPTANCE was the literal positive line, not the verdict: while os-public-hostname
is unset, A12/A13 RECORD rather than fail, so PASS alone is compatible with a wrong
certificate. Both DCs now carry 'DNS SANs are all in this DC's expected zone'.
verify PASS 37/0 both DCs (was 29/0; +8 from the new assertions). creds-matrix:
same 5 pre-existing findings, no new finding, no S4 drift. No staging residue,
neither run reached exit 5.
The hardened re-run guard proved itself live: re-running dc0 immediately after
returned exit 4 'already correct ... and verify returns clean' -- the fix for the
defect that had it refusing to repair certs verify was failing.
STILL OWED, operator-gated: backup custody. Two archives on the headend hold BOTH
CA private keys, BOTH passphrases and the overlay; 1.0-REISSUE.4 pulls them to the
jumphost creds folder and removes the headend copy. They are also an unregistered
credential class (no creds-matrix row, no retention rule). Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 sweep: correct the artifact that filed Q1, and one stale residue
...
Advisor review caught that correcting CURRENT-STATE and the ledger was not
enough: docs/audit/queued-findings-20260730.txt -- the sweep capture the 07-30
bookend points at -- still presented Q1 as an open ruling that 'blocks any FQDN
cert', enforced by an A12 refusal that no longer exists. A next session reading
it would re-derive Q1 exactly as this one did, reopening the loop just closed.
- queued-findings-20260730.txt: APPENDED correction (not a rewrite; the
stage4-mirror-gate-20260727.txt precedent), naming the two claims that are now
false and pointing at the evidence. Q2 explicitly untouched.
- dc-dc-deployment-workflow.md:257: described phase-6 as proposing the retired
dc1/dc2-hostnames.yaml spelling. MEASURED: the runbook itself was already
corrected to the region-qualified form by an earlier session -- only the
workflow doc's description of it lagged. Under D-119 IS vr1-dc0, so the
-hostnames.yaml interpolation used elsewhere is correct as written.
- The DOCFIX-205 capture's section 1 quoted a 3-line span starting mid-sentence
and left a clause orphaned. Repaired to two verbatim quotes, both re-verified
present in design-decisions.md. CURRENT-STATE cites this file as evidence and
this repo's discipline is exact captures.
octavia-pki 23/23; repo-lint 0 fail (L10 gated this commit correctly -- the
docs/audit/ change set required CURRENT-STATE to move with it).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 capture: the D-117 supersession, annotation coverage 0/4 -> 4/4, live A12
...
Evidence file cited by CURRENT-STATE's superseding block. Carries the quoted
D-117 Status line, D-119's selector-only discharge, measured annotation coverage,
the phase-6 expansion old-vs-new in a real shell, live A12 on the headend
(PASS 29/0 both DCs, both INERT), and both hosts' gate results.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-bookend sweep: the operator's working commands were NOT in the repo
...
Operator: "I'm worried that the fixes we made to the defective commands will be lost when we
close this session ... Complete a full sweep before the bookend."
THE CONCERN WAS WELL FOUNDED. The corrected Step 5/6 commands -- the ones that actually produced
both DCs' live PKI -- had been reviewed in conversation, judged equivalent, and never folded in.
Four items now landed:
1. Portable base64. GEN.d shipped the GNU-only `-w0` flag; what ran was
`base64 < f | tr -d '\r\n'`. The runbook described a command nobody executed.
2. The operator's VIP_OVERLAY:? guard in GEN.c.
3. The heredoc column-1 requirement, as a CAUTION heading rather than a footnote, because that
failure is silent: no alt_names section yields a certificate with NO SANs while every
openssl command still prints OK.
4. ${DC_LABEL:?} at both CA call sites -- F8's sibling, where an unset label bakes a DC-less
subject into a 10-year CA with no error anywhere.
Not done, with the reason recorded: the heredoc was NOT rewritten as printf lines. More
paste-proof, but an untested rewrite of a step that mints 10-year CA material and cannot be
exercised end-to-end from an agent session. Risk bounded instead by A9 plus harness T2.
F9 is now structural rather than remembered: A12 arms itself from os-public-hostname appearing as
a real option key. Live PASS 29/0 both DCs, harness 21/21.
A new ruling-shaped gap, now blocking rather than filed: D-008's shape plus D-106:2563's VR1
instantiation do not say whether substrate vr1-dc1 is dc1 by token or dc2 by position -- the
DC1/DC2 ambiguity item 3.1 retired elsewhere, here deciding certificate identity. Both live certs
carry dc0.vr0, the VR0 region, wrong under either reading. A12 refuses rather than picking.
A precedence bug the harness caught: REFUSE was checked before FAIL, so once A12 armed a
CONFIRMED wrong-region SAN reported as "could not evaluate". A known defect outranks an
unevaluated one.
Sweep capture docs/audit/queued-findings-20260730.txt carries F10-F13 and two ruling-shaped
questions. F10 is the one worth acting on: mint-ref line numbers drift silently and S4 cannot see
it, since it asserts only within-EOF. That bit three times in two sessions and was caught by hand
every time, never by a gate. All 24 octavia refs re-anchored in a single-pass mapping keyed by row
id -- 391 was simultaneously an old and a new value, so sequential seds would have corrupted it.
repo-lint 0 fail; octavia-pki 21/21; creds-matrix 65/65.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |
Per-DC Octavia PKI generated: independence proven, identity unverified, 12 real findings
...
The mint was operator-executed on the headend; the session guard hard-blocks it and that was
not worked around.
PER-DC INDEPENDENCE PROVEN, which is the whole point of F1. Both DCs hold 12 files across 3
dirs, and sha256 DIFFERS between DCs for both encrypted CA keys, the controller cert and the
overlay. Before F1 those columns would have been identical, because the second generation would
have overwritten the first. Both overlays are mode 600, carry exactly five lb-mgmt-* keys, are
ASCII clean and are gitignored -- the F4 gate held.
NOT CLAIMED AS DONE: the four CA subjects, the controller SAN sets and the chain verifications
are unread, because the guard blocks openssl against those paths. Recorded as structurally
complete with identity unverified. A cert can be well-formed and name the wrong DC.
P5 on the headend went 6 -> 18 findings and all 12 new ones are genuine.
E2 x4: the CA certificates are mode 664. This session had explicitly called that harmless "for
public certs", reasoning only about confidentiality. That was wrong -- 664 is group-WRITABLE, so
a group member can replace a trust anchor. Integrity, not secrecy, is the exposure.
E3 x8: four generator outputs per DC (the controller certificate, the CA serial file, the
openssl config and the signing request) exist with no matrix row. The 18-row set landed earlier
today was incomplete -- "enumerate what exists, not only what is declared", applied to this
session's own work.
Validation worth recording: all 12 would have been INVISIBLE before today's F2 and F6 fixes,
since the rows were singleton and the locations resolved on the jumphost. The register found
real defects in exactly the mint it was repaired to observe.
F8: the controller-cert step's heredoc is a live paste hazard -- the operator hit it. An
indented terminator can yield a config with no alt_names section, producing a certificate with
NO SANs while every step still reports OK, and nothing asserts the SAN set.
F9: the controller certificate's DNS SANs are coupled to D-106 and nothing recorded it. They
are inert only because os-public-hostname is set nowhere (B5 IP-ONLY; R5 refused setting it at
Stage 5 as a D-019 repeat). When Stage 7 turns FQDN endpoints on, both DCs' certificates
carrying dc0.vr0 becomes wrong exactly when it starts mattering; likely disposition is reissue
inside D-106's own FQDN-SAN step.
Remediation logged not executed: tighten the six certificate modes; and for E3 either declare
all four artifacts per DC or have the generator drop the two build intermediates and declare the
two that must persist.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F7: office1-tailscale is labelled a subnet router and advertises no routes
...
Raised from an operator report of Permission denied (publickey) reaching voffice1 "from my
workstation via tailscale". Measured rather than assumed, and the premise did not survive.
tailscale is absent from BOTH voffice1 and vcloud -- no binary in any standard location, no
state dir, no interface, service not running. Checked with explicit path probes after an
initial `command -v`, because tool-absent-from-PATH reported as feature-absent is a trap this
repo has hit repeatedly.
The tailnet node is a separate LXD guest INSIDE voffice1: office1-tailscale, RUNNING,
100.64.0.53, site leg 10.10.1.11. Its tailscale status reports AdvertisedRoutes: <none>. So
nothing on the tailnet can reach 10.10.0.20 -- there is no route and there never was.
Two records made the false belief look supported: the ssh stanza labels it "Office1 subnet
router (D-107)", and the operating skill's routing table says the workstation reaches voffice1
over the tailnet, citing D-107 -- which is titled "Airgap posture, per-DC artifact mirror, and
NTP (VR1)" and rules nothing of the sort. A real citation that does not support its claim.
RULED (operator): option (a), ProxyJump through vcloud. Option (b), advertising 10.10.0.0/24
onto the tailnet, was presented and not taken -- it exposes the isolated site net, a posture
change against the airgap subject D-107 actually governs.
Fixed operationally outside the repo: vcloud's ssh config now matches Host voffice1 10.10.0.20,
so the raw IP resolves to the office1 service key with IdentitiesOnly yes instead of falling
through to the id_* defaults, which was the actual cause. Both forms verified, with a negative
control proving the key is not offered to unrelated hosts.
Third wrong record this session, after D-124's MAC-sourcing instruction and the skill citation.
Two of the three surfaced only because something adjacent was being fixed; nothing here gates
prose, so no check could have caught them.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Five Stage-5 findings raised, none executed: R7's per-DC Octavia PKI is half built
...
F1 (HIGH): the D-109 amendment rules per-DC independent Octavia CAs, and the
2026-07-29 work correctly made the CA subject and the VIP gate per-DC -- but $DC
appears in no PATH. phase-01-bundle-deploy.md:293 is a fixed $HOME/octavia-pki and
:473 writes a fixed overlays/octavia-pki.yaml, so generating dc1 after dc0
overwrites dc0's issuing-CA key, controller-CA key and both passphrases, and a
later dc0 redeploy reads the fixed overlay name and applies dc1's CA to dc0 -- the
cross-DC shared amphora root R7 explicitly refused. The existence checks at
phase-01:86 and pre-flight-checks.sh:65 cannot detect it: they ask whether the file
exists, never whose CA it is. The gap is in R7's own "Work implied" list.
F2: creds-matrix.tsv:103-111 declares all nine Octavia credentials `singleton`,
while creds-matrix.py:368-398 enforces both-DC existence only for `per-DC` rows --
so the register behind the BLOCKING P5 gate is structurally blind to a missing
second PKI set. Baseline captured (82 rows, 7 findings, exit 1) so the delta is
attributable. Must land before generation.
F3: both ~/octavia-pki/ and overlays/octavia-pki.yaml are ABSENT (existence only,
no contents read), so F1/F2 are fixable at zero risk right now; the window closes
when the first DC's PKI is generated.
F4: .gitignore:40 is an exact path, so a per-DC overlay rename would un-ignore a
file holding CA key blobs plus a plaintext passphrase, in a repo SEC-004 records as
PUBLIC. The F1 fix must widen the glob in the same commit.
F5: preflight.sh:99 sets DC without exporting it and pre-flight-checks.sh never
reads DC at all, so a dc1 invocation runs a DC-aware P2 beside a dc0-frozen P4
under one combined verdict.
Logged not executed per hard rule 1. repo-lint 0 fail / 615 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 session close: bounded bookend + the 2026-07-29 findings sweep
...
Bookend is 14 content lines against the 15-line cap, measured. Ledger 263
lines against the 300 cap, so no rotation is owed this close.
Close sweep captured docs/audit/queued-findings-20260729.txt -- transcript-
only material preserved before compaction, per the queued-findings-20260726
and -20260727 precedent. The substantive content:
- A strict-bash quoting trap FAMILY with three measured instances in one
session: IFS=$'\n\t' defeats word-splitting on a space-separated string
(killed pre-flight-checks CHECK 1 silently); a bare under
inherit_errexit ABORTS the run instead of reaching a refusal branch; and an
apostrophe inside the single-quoted shell string wrapping embedded python
makes bash parse python.
- Two greps that looked like findings and were not: 'grep mac' matches
type_machine, and grep -c 'macs = [' counts COMMENTS -- the same class as
the ledger-scan defect where a doc quoting a token inflated a counter.
- Five MAAS behaviours, including that enlistment is NOT commissioning and
that MAAS auto-configures power only for IPMI machines, so a virsh VM must
be power-configured out of band first.
- Two guards defeatable by accident: lib_hosts_select_dc's one-DC-per-shell
guard muted by 2>&1, and NetBox's ?site= filter silently not filtering.
- A process note on two parallel live-ops agents: safety came from agents
working the voffice1 clone while the orchestrator edited vcloud's, so no git
race was possible -- at the cost that agents barred from repo writes cannot
leave captures, and dc1's evidence had to be promoted by hand from /tmp.
The skill sweep stays DEFERRED to the Stage-5 close per the 2026-07-27
operator direction; this capture is its input, not a substitute.
Gauntlet ALL GREEN (87) on vcloud; repo-lint 0 fail / 614 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: OSD carve + D-104 controllers complete across both DCs (two agents)
...
Operator directed two background agents: one to finish dc0 storage-02/03/04,
one for all of dc1. Both completed.
Eight re-commissions, every one the same single-change diff: vdb at
536870912000 bytes (500 GiB) added; links 12 -> 12; per-interface MACs and
link lists identical. dc0 at 204/196/202s, dc1 all ~182s. dc1's substrate
applied 6 add / 4 change / 0 destroy with MAC drift 0 and re-converged to
ZERO DIFF.
I verified every claim from MAAS directly rather than accepting the reports:
fleet 22 (18 Ready + 2 Deployed + 2 New), 216 links all static, both DCs
blockdevs {1:5, 2:4}, tagcount {2:9}. All matched. I also diffed a raw dc1
before/after pair myself -- exactly the vdb hunk plus one power_state line.
Captures promoted from voffice1:/tmp (which does not survive a reboot) to
docs/audit/osd-carve-20260729/. Both agents were barred from repo writes, so
dc1 would otherwise have had no audit record at all.
Two unprompted agent findings worth keeping: MAAS tags SURVIVED
re-commissioning on all 18 nodes (checked because the snapshot code
structurally could not see tags -- removes a would-be Stage-5 placement
blocker); and lib_hosts_select_dc's one-DC-per-shell guard fires correctly
but can be muted with 2>&1, silently yielding one DC's data labelled as the
other's.
One disclosed judgment call, and it was correct: three dc1 pairs also showed
power_state moving off/error. Agent B declared a STOP, re-measured, then
excluded power_state from its verdict while still printing it, because 8 of 9
dc1 nodes read error BEFORE any mutation. Vindicated -- all 18 role nodes now
read off, including five dc1 nodes never commissioned. A transient MAAS
power-query flap.
Still blocking for both controllers: power_type is EMPTY, the state that
stalled all nine role nodes on 2026-07-20. Now fixable since both are in
lib-hosts at the ruled .5 with MACs pinned.
Gauntlet ALL GREEN (87) on vcloud; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: canary re-commission closes R1 precondition 3 by measurement
...
Operator: 'Commission storage-01 as the canary'. maas admin machine
commission kghggm skip_networking=1, sysid matched by pinned boot MAC never
by name. Commissioning -> Ready in ~200s, no timeout, no SERVFAIL.
The full before/after diff of status, power, block devices, interfaces, MACs
and links is EXACTLY ONE HUNK: vdb (536870912000 bytes = 500 GiB) added to
blockdevices. links 12 -> 12; MACs identical; links identical. All twelve
static links survive with addresses, modes and subnets intact, including the
D-134 octet .150 mirrored across all six planes in both families -- the v6
host part ::150 being the 2026-07-27 D-136 mirror ruling visible in live
data.
So skip_networking=1 preserves the network EXACTLY while storage is still
re-scanned, which is the entire point. Precondition 3 was previously answered
from the API help text; it is now answered by measurement. The amendment's
concern was legitimate -- the default path does reconfigure networking -- and
the vendor's own control is sufficient.
Remaining: dc0 storage-02/03/04, then all of dc1.
Capture: docs/audit/dc0-canary-commission-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: dc0 substrate applied -- R1 OSD volumes + the D-104 controller VM
...
Operator approval: "Apply dc0 first". Executed from voffice1 via a SAVED
PLAN with a same-session pre-apply re-verify (G8 precedent), so what was
applied is exactly what was reviewed.
Apply complete: 6 added, 4 changed, 0 destroyed. Four libvirt_volume.osd +
the vr1-dc0-juju-01 domain and its disk; the four storage domains updated
in-place to attach vdb.
THE CHECKPOINT HELD -- MAC DRIFT COUNT 0. Verified by virsh domiflist
against lib-hosts' pinned MACs BEFORE anything touched MAAS: all 9 nodes
pinned==live, 6 NICs each. That sequencing is the point. 2026-07-20 became a
fleet-wide outage because MAAS was told to re-commission while its records
were already stale, after a 0/9/0 in-place apply regenerated every MAC
without the plan showing it. So the clean plan was necessary and explicitly
not sufficient. First live evidence that the 2026-07-21 MAC pinning survives
an in-place domain update.
Convergence restored (precondition 1): the dc0 inner root re-plans ZERO
DIFF. MAAS undisturbed -- 20 machines (18 Ready + 2 Deployed), 216 links,
every one still mode=static. MAAS still sees ONE block device per dc0 node,
which is expected and is why the re-commission is genuinely required.
Also recorded: the saved plan's MAC audit returned one hit that was a false
positive -- type_machine = "q35" inside the juju-01 create block, since
"mac" is a substring of "machine".
NOT DONE, gated separately: the re-commission with skip_networking=1, and
dc1 entirely. Capture: docs/audit/dc0-osd-juju-apply-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R1 precondition 3 VERIFIED read-only from voffice1: skip_networking=1 is the control
...
The amendment required verifying, BEFORE the apply, that re-commissioning
preserves the D-134 statics and the pinned MACs. Both halves now answered
from the headend, read-only, no mutation.
HALF 1 -- baseline measured: 18 Ready nodes, 7 interfaces, 12 links each,
mode=static on every link (6 planes x 2 families). MAAS 3.7's commission API
carries skip_networking: 'Whether to skip re-configuring the networking on
the machine after the commissioning has completed.' The option exists BECAUSE
the default reconfigures, which is exactly the flagged risk. The combination
R1 needs is skip_networking=1 with skip_storage UNSET -- network preserved,
storage re-scanned so the new /dev/vdb enters inventory.
HALF 2 -- the tofu change does not touch NICs. Both inner roots plan
6 add / 4 change / 0 destroy; the four in-place diffs change devices.disks
ONLY, gaining vdb. A grep for mac/interface across the whole diff returns one
hit and it is inside the NEW juju-01 create block. Risk is materially lower
than 2026-07-20 because MACs are now config-pinned -- the control introduced
after that incident precisely because the plan was untrustworthy.
NOT overclaimed: a clean plan was necessary but not sufficient in 2026-07-20,
when 0/9/0 in-place regenerated every MAC. So the sequence puts a MAC
verification BETWEEN the apply and the re-commission -- that incident became
fleet-wide because MAAS was told to re-commission while its records were
already stale.
Capture: docs/audit/r1-precondition3-verify-20260729.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-28 |
LIVE: node IPv6 carve both DCs -- 108 links, IPv4 untouched, 18 Ready unchanged
...
Operator-gated dc0 then dc1. 54 applied per DC, 0 errors, read-back 54/54 each.
IPv6 links 0 -> 108; IPv4 links 108 unchanged; 18 Ready unchanged. Both
dc-node-v6-carve gates PASS and dc-plane-ipam still reads 24/24 on both DCs.
The octet mirror holds across all six planes per node -- superb-piglet .121 ->
::121 everywhere, big-trout .100 -> ::100. enp2s0 correctly received nothing: the
D-100 raw provider NIC has no v4 link to mirror, and br-ex carries provider-public
instead, on the node /64 rather than the VIP /64. That falls out of deriving from
"already carries v4" rather than naming planes in a table.
The gate discriminated rather than agreeing with what it found -- it flipped dc0
to PASS while dc1 still read FAIL, before dc1 was carved.
This closes the node half of step 3, which was the correction the operator's
octet-mirror question surfaced: v6 does not arrive with the subnet, because MAAS
mode=static means an explicitly configured address.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-101 governing rationale recorded; node v6 carve scoped; two of my claims corrected
...
The operator's question -- "are we mirroring the assigned IPv6 octet with the last
IPv4 octet like we did before?" -- exposed two errors in what this session recorded.
(i) I said MAAS static assignment "needs only the subnet, which now exists". Wrong:
mode=static means an EXPLICITLY CONFIGURED address, which is how the Stage-4 carve
set the 90 v4 links. Measured -- all 18 Ready nodes carry ZERO IPv6 links, while
their v4 side is correctly octet-mirrored. So VIPs are mirrored (156 apex objects)
and nodes are not; 108 assignments are owed. (ii) v6 gateway_ip and dns_servers are
unset on all 12 v6 subnets, unnamed before I called step 3 complete. That
"complete" is withdrawn -- the MAAS/apex/lib-net population stands, the node layer
is the remaining half.
D-101 gains its GOVERNING RATIONALE, quoted verbatim, because it existed in no repo
surface: IPv6 unless IPv4 is NECESSARY, driven by real IPv4 sizing constraints in
future expansion -- a commercial requirement, not a preference. It records that
v4-first was DELIBERATE RISK REDUCTION so a future session does not read v4
surfaces as neglect and "correct" them; that Roosevelt has full v4 and v6 edge
transport; and that NAT64/DNS64 was considered and REJECTED on that basis -- a shim
with no Roosevelt analog, against ULA planes that are internal by design.
Node v6 carve scoped in docs/audit/node-v6-carve-scope-20260727.md: 108 assignments
mirroring each node's live v4 octet, prior art measured v4-only, blast radius
per-link and reversible, and the operator's own rationale arguing to carve BEFORE
the deploy since v6-unfriendly modules surface far more cheaply on a static
read-back than mid-bundle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: D-134's bands now EXIST -- both DCs at pass=24 fail=0, from a 6/18 baseline
...
Operator-gated, dependency-ordered: dc1 v6 carve (6 subnets), dc0 reserve (13
ranges), dc1 reserve (12 ranges). Each pre-apply re-verified in the same session;
every write read back. Subnets 18 -> 30, ipranges 3 -> 28. Machines 18 Ready + 2
Deployed unchanged throughout -- no node, no tfstate, no running service touched.
This closes the "RULED IS NOT BUILT" finding for D-134: its band table had been
prose since 2026-07-23 while MAAS held zero reserved ranges.
FINDING -- MAAS already reserves the entire low IPv6 block. Every explicit v6 band
create failed with "Requested reserved range conflicts with an existing range".
Measured via `subnet reserved-ip-ranges`: MAAS auto-reserves ::1-::ffff:ffff on
every IPv6 subnet plus :: per RFC 4291 s2.6.1, with allocatable space starting at
<prefix>:0:1::. The ruled bands sit entirely inside it, so the write is impossible
AND unnecessary. R4's "v6 bands as a second pass" is not executable in MAAS and
need not be; the tool now verifies coverage instead of writing. Execution-level
correction -- R4's intent is satisfied. Same on ULA and GUA.
Harness T19 asserted a literal string the fix removed. Re-pointed at the surviving
invariant (textual 4-99, never hex 4-63) rather than deleted, per the standing rule
against going green by removing an assertion. 25/25, gauntlet ALL GREEN (83).
dc1's FIP pool still correctly REFUSED -- unset by design, so R4 cannot close for
dc1 until ruled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
LIVE: dc0 v6 plane carve applied -- 6 MAAS subnets, each read back
...
Operator-gated ("Run carve-v6 vr1-dc0 --commit"). dc0 only; dc1 not authorised
and not touched.
Pre-apply re-verified in the same session per the G8 precedent -- plan unchanged
at 6/1/0 -- then applied: 6 applied, 1 skipped, 0 errors, every create READ BACK
on its intended vlan. MAAS subnets 18 -> 24, v6 1 -> 7.
Each v6 plane landed on the SAME vlan as its v4 twin, so the plane is genuinely
dual-stack on one L2 rather than a parallel fabric. The provider GUA VIP /64 was
deliberately not created -- it holds hacluster-managed API VIPs, not node
addresses, so MAAS never allocates from it.
Idempotency proven LIVE rather than only in fixture: --commit ran twice (the
second to read the script's true exit code rather than a pipeline's) and the
post-state carries zero duplicate CIDRs.
Nothing else moved: 18 Ready + 2 Deployed unchanged, no node touched, no tfstate
involved, dc1 still measures 6 v6 planes absent. The gate now reports dc0's six
v6 planes [ok]; its remaining reds are the 12 unreserved D-134 bands.
Changelog item 9 carries the revert, with the 2026-07-21 pod-delete cascade
lesson attached: read occupancy BEFORE deleting a subnet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Step 3: dc-plane-ipam.sh check -- D-134's missing executable gate, with a measured baseline
...
R4's D-134 amendment ruled a site-keyed tool that "also gives D-134 an EXECUTABLE
gate instead of prose". This is that gate. Read-only half only; the mutating
actions are deliberately not in this commit (hard rule 3 -- the gate ships and
runs first, so the mutation is planned against a MEASURED baseline).
Expected state is derived, never hardcoded: v4 planes from lib-net's DC selector,
v6 planes from the NetBox apex record (D-136 option (D) applied -- the apex is the
source, so this does not create a second hand-maintained table), bands from the
D-134 2026-07-23 amendment.
LIVE BASELINE, both DCs, symmetric at 6 pass / 18 fail: all six v4 planes present;
ZERO of six v6 planes present; ZERO of twelve D-134 bands reserved. MAAS holds 17
v4 subnets and exactly one v6, which is Office1's -- a third-layer confirmation of
U17. Cloud-wide there are 3 ipranges, all dynamic, matching R4's measurement.
Proven able to BOTH fail and pass: every live run fails because everything it
asserts is absent, so T6 drives a fully-provisioned fixture to green. A gate only
ever observed failing is as untrustworthy as one only ever observed passing. It
REFUSES rather than reporting clean on an unreachable MAAS, an unreadable apex, or
a band of unrecognised type, and distinguishes an ABSENT maas binary from an
unreachable MAAS -- the misdiagnosis class this audit found three times.
The provider GUA VIP /64 is reported but NOT asserted as a MAAS subnet: it holds
hacluster-managed VIPs, not node addresses, so MAAS cannot hand one out. Whether it
should also be a MAAS subnet is left open rather than decided by a gate.
Harness 14/14. Gauntlet ALL GREEN (83) -- the R15(2) manifest gate correctly caught
the new harness as unpinned drift and was re-recorded deliberately, which is the
mechanism working end to end in its first real use.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
VIP/band reconciliation measured; two ruled decisions found to have an unruled gap
...
Step 4 prep for the render pipeline. Read-only enumeration of the four
disagreeing VIP surfaces, with every consequential claim re-verified directly
before recording. Nothing adopted.
NEW GAP: R11 ruled three gate changes (band .60->.99, VIP_COUNT_EXPECT 11->13,
FAIL on hacluster-without-vip) but NOT the arity change R2's dual-stack forces.
provider-bundle-check.py:137 requires exactly 3 addresses; a dual-family vip is
6, so under R2 the checker fails every application. :149's octet extraction also
returns the whole address string on a v6 literal, and a v6 address tests False
against an IPv4Network without raising -- so a naive arity fix just relocates the
failure. Two RULED decisions whose combined end state the gate cannot express.
TRAP: EXPECT_PUBLIC_VIP must STAY 11 while VIP_COUNT_EXPECT goes to 13. Measured
against bundle.yaml -- neither vault nor designate carries a `public` binding, so
neither joins that count. Reading "11 -> 13" and bumping both breaks the gate.
The L3-9 overlay collision is worse than recorded and the dangerous merge order
is the GREEN one: vips-last exits 0 while dropping every IPv6 leg, and
prefer-ipv6: true survives as a separate key -- charms would bind :::port with no
v6 VIP for pacemaker. Measured through the checker's merge MIRROR, not juju
(absent here); flagged for --dry-run confirmation rather than asserted.
CORRECTED one of the agent's own conclusions: dc1 "inheriting dc0's band bounds"
is not a defect. The octet band is DC-invariant by design, so lib-net.sh:157
correctly unsets the prefixes and keeps the bounds. Acting on the uncorrected
framing would have invented per-DC bounds that do not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-27 |
Phase 0 delivery cleanup: session changelog with reverts, and a mis-filed finding corrected
...
Three gaps in the Phase 0 delivery, plus one new measured finding.
1. SESSION CHANGELOG ADDED (GA-R2/D1) -- docs/changelog-20260727-stage5-phase0.md.
It was missing, and it is the only revert surface: no logged window was
opened for this session, so the changelog + capture are the entire
as-executed record of five live mutations (two branch deletions on origin,
two on voffice1, a 68-package install on the region host). Each item now
carries WHAT / WHY / HOW TO REVERT with the pre-state hashes recorded
(39e8988, 61c416e, 57836b1) and the apt purge scoped with a caution that
68 packages were pulled and some are shared.
2. CURRENT-STATE section 7 no longer carries two rows for one component. The
prior "ABSENT ON BOTH HOSTS" state is folded into the single pin row as
history, following the Juju row's in-row supersession precedent. A pin
table asserting both presence and absence of the same client is the same
stale-surface class the audit close-out sweep found the audit itself
creating.
3. A FINDING I MIS-FILED IS CORRECTED BY MEASUREMENT. P4's
"metal-admin gateway=none (want 10.12.8.1)" was dismissed as covered by
readiness item 3.7. It is not -- 3.7 is specifically the VID-103 /
br-internal assertions. Measured from the dc0 rack: 10.12.8.1 answers 0/2
pings with an INCOMPLETE ARP entry (never resolved -- nothing holds it),
against a control ping to the edge at 10.12.4.1 at 0% loss. So MAAS is
right to carry no gateway and lib-net.sh:37's expectation is the defect, a
VR0 inheritance; the D-134 carve ruled .1 gateways for the PROVIDER subnets
only. Logged as P0-5, not fixed (hard rule 1). The dc1 arm (lib-net.sh:149)
carries the same shape at 10.12.68.1.
Also queued for one GA-R5 exchange, no D-number assigned: which host is
authoritative for the gates. R10 (preflight on voffice1) and R15(3) (preflight
stops failing open) executed together make preflight-on-voffice1 permanently
unpassable given P0-2, and R15(2)'s manifest does not reach P0-1 -- so the R15
execution scope needs this answered first.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 Phase 0 EXECUTED: clone advanced, refs pruned, openstack client installed
...
Readiness-doc preconditions 0.1/0.2/0.3, operator-approved step by step
("Approve A, B, and C, and retire the audit branch"). NO STAGE OPENED.
0.1 voffice1 is at main (HEAD == origin/main == 6495cfb, asserted). The real
risk was that both DCs' inner tfstate -- the substrate's state-of-record --
lives inside that working tree; every state artifact was proven gitignored and
origin/main proven to track an identical file set at those paths BEFORE the
switch. Both tfstate sha256s are byte-identical after. The two dc1 overlays are
now present and bundle.yaml is the 9-node role-separated layout.
0.3 three stale remote-tracking refs pruned, two stale local branches deleted
(the record named one), containment proven first. The audit branch was retired
on origin BEFORE the fetch so the clone could not be handed a fresh stale ref.
0.2 python3-openstackclient 6.6.0-0ubuntu2 on voffice1, verified behaviourally.
The snap was refuted by measurement: no Caracal channel (newest stable zed,
2023), plus the home-only confinement already recorded at design-decisions:638.
Two NEW findings from the payoff runs, logged not fixed (hard rule 1):
- the gauntlet is HOST-DEPENDENT (2/81 on voffice1 vs ALL GREEN on vcloud, same
commit); both failures are host-portability defects, one of them an aggregate
FAIL over all-passing sub-checks
- preflight P5 is HOST-BLIND (34 findings vs 7), and between the two hosts there
is no single host on which preflight is currently correct
R10's ruled consequence is confirmed: P3 now verifies all 33 charm-channel pins
(previously zero) and P4 reports MAAS reachable.
Capture: docs/audit/stage5-phase0-20260727.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Session-close sweep: three stale surfaces the audit itself created, plus transcript-only facts
...
Operator direction before the bookend: "complete a session sweep for anything
missed that should be committed that would be lost on session end". Precedent:
queued-findings-20260726.txt and -20260727.txt. Capture:
docs/audit/queued-findings-20260727-stage5-audit.txt.
THE SWEEP FOUND THREE STALE SURFACES THIS AUDIT ITSELF CREATED, which is the
exact defect class it was convened to find:
1. docs/audit/stage5-readiness-20260727.md -- the "read this first" artifact --
still carried THIRTEEN NEEDS-RULING markers after every one of R1-R15 had been
ruled or withdrawn. At that moment it was the single most misleading surface in
the repo. Fixed with a supersession banner that names the three authorities in
order and states what in the document is STILL true and worth reading: the
ordered precondition sequence, the evidence column, the what-breaks column. The
row-level markers were deliberately NOT rewritten -- they are the record of what
was owed at the time, and rewriting them would destroy that record.
2. queued-rulings-20260727.md still opened with "Nothing here is adopted" after
fourteen adoptions. Fixed with a STATUS block naming design-decisions.md and
CURRENT-STATE.md as the ruling authorities rather than that file; the original
sentence is struck through, not deleted.
3. SIX sections of that file had BLANK OPERATOR UTTERANCE lines -- R8, R9, R10,
R12, R13, R15 -- for rulings that WERE properly recorded in design-decisions
and CURRENT-STATE. GA-R5 was satisfied; the question sheet was not, and a
future session reading only that file would have believed six questions were
still open. All six filled with the exact utterance and where the ruling lives.
R14-ORIGINAL's blank line is CORRECT and left alone -- it is superseded text
kept for the audit trail.
TRANSCRIPT-ONLY OPERATIONAL FACTS NOW CAPTURED: the repo-lint L5 heading trap (a
heading LEADING with a D-number counts as a second definition unless the line also
contains AMENDMENT or RESOLVED -- it bit this session twice and is on no
author-facing surface); the commit-gated-on-lint discipline adopted after a red
commit was pushed, which then caught the second occurrence before it landed; the
read-only live-apex poll procedure that keeps the token in env and prints nothing;
that the office1 VMs answer from vcloud but NOT via voffice1, so "unreachable" from
the wrong host must never be recorded as "absent"; and that creds-matrix.tsv is
SPACE-aligned despite the .tsv extension, so awk -F'\t' returns zero rows and looks
like a real result.
META-FINDING RECORDED FOR THE NEXT COMMITTEE: the lenses' OBSERVATIONS were
reliable, their CONCLUSIONS repeatedly were not. Three of this audit's own premises
needed correcting by measurement -- R2a (literals already assigned under D-111), R9
(one failure mode not two, two scripts not twenty), R14 (the register already
attributed and explained the findings, and already warned against the suppression
the question contemplated) -- and R8's entire evidence base collapsed on reading
the cited bugs. A lens finding is an OBSERVATION, not a conclusion; measure before
putting it to the operator as a question.
VERIFIED DURING THE SWEEP, after an API disconnect mid-turn: nothing was lost or
corrupted. The earlier lens-5 API casualty was fully recovered -- its 18 findings
and section header are in the committed record, and all seven lenses are present.
The disconnect this turn fell cleanly between edits and a commit; the two modified
files parsed intact and lint was red only for the L10 this commit satisfies.
NOT DONE AND NOT OWED YET: the skill sweep. This session closed no STAGE, so the
stage-close fold-in and snapshot regeneration are not due. B1/C1/C2 in the capture
are the candidates when Stage 5 closes.
Revert: git revert this commit; the capture is new and the two repairs are
additive (banner, status block, filled utterance lines).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R14 WITHDRAWN: the register already does everything the question assumed it could not
...
Withdrawn without a ruling, following the R2a precedent. R14 asked whether the
credential matrix needs a ruled-exception field, on the premise that three S5
power-key asymmetries were "ruled correct by SEC-016" and so reported a permanent
red a reader learns to ignore.
All three parts of that premise fail on measurement:
1. They are NOT ruled correct. SEC-016 ruled per-DC ISOLATION -- dc1 gets its own
dedicated power key rather than reusing dc0's -- which is satisfied. It never
blessed the filename, host and custody divergence. That is SEC-021(b), an OPEN
ledger defect whose own disposition reads "needs a naming/custody reconciliation
to the dc1 shape", and whose stated complaint was "Per-DC rows that should be
symmetric are not, and nothing compares them". S5 is the thing that now compares
them, so the finding is the register working as designed.
2. The register already ATTRIBUTES them: those rows carry sec-ref=SEC-021 and
notes-ref=n-dc0-power-key-divergence.
3. The register already EXPLAINS them: creds-matrix-notes.md carries the
n-dc0-power-key-divergence note describing the name and host-role divergence and
why the expected jumphost rows fail EXPECTED-BUT-ABSENT.
And the notes file already warns against the exact move this question contemplated.
The line immediately preceding that note reads: "failure, not a matrix error: do
not delete the row to make the checker green."
So no schema change is needed and none should be made. Adding a suppression
mechanism would have HIDDEN an open security-ledger item -- the opposite of what
the register exists for. The red clears when SEC-021(b) is remediated, which is the
intended behaviour.
This is the third premise this audit has had to correct in its own framing, after
R2a (literals already assigned under D-111) and R9 (one failure mode, not two).
Worth noting the pattern: the audit's lens findings were sound as observations and
repeatedly wrong as conclusions, and measurement caught it every time.
Residual recorded so it is not mistaken for an oversight: whether a ruled-exception
field is EVER needed is now hypothetical. If one arises, notes-ref and sec-ref are
the place to start, not a new column.
Revert: git revert this commit; the R14 entry is rewritten in place with the
original retained beneath as R14-ORIGINAL for the audit trail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Final pre-ruling measurements R9/R10/R12-R15: two premises corrected
...
Operator direction: gather everything remaining so the last decisions can be
worked through without stopping to measure. Capture:
docs/audit/r9-r15-final-measurements-20260727.txt.
TWO OF THESE CHANGED MATERIALLY, and both are corrections to the audit's OWN
earlier framing rather than new findings.
R9 -- there is only ONE failure mode, not two, and the Stage-5 blast radius is
TWO scripts rather than twenty. lib-net.sh:76-79 states the design explicitly:
sourcing without the selector "continues to populate PLANE_CIDRS ... exactly as
above (VR0/DC0's real, measured values) -- every existing script that sources
lib-net.sh keeps working completely unchanged". So the unset block fires ONLY
for selector-CALLERS. The 20 non-selector consumers fail SILENTLY with dc0
literals; the 8 correctly-updated ones are the ones that break loudly under
set -u. That inverts the intuitive reading -- the scripts nobody fixed are the
dangerous ones. And grepping the two runbooks Stage 5 actually uses, only
phase-03-core-verify.sh and deploy-watch.sh are invoked there; the rest bite at
later stages.
R14 -- THE PREMISE IS WRONG and the question largely dissolves. I framed it as
"the matrix cannot express a RULED exception, so three S5 asymmetries ruled
correct by SEC-016 report as a permanent red". Measured: they are NOT ruled
correct. SEC-016 ruled per-DC ISOLATION -- dc1 gets its own dedicated power key
rather than reusing dc0's -- which is satisfied. It never blessed the filename,
host and custody divergence. THAT is SEC-021(b), an OPEN defect whose own
recorded disposition reads "needs a naming/custody reconciliation to the dc1
shape", and whose stated complaint was "Per-DC rows that should be symmetric are
not, and nothing compares them". S5 is the thing that now compares them. The
register is RIGHT and the finding is REAL. Suppressing it would have hidden an
open security-ledger item.
R15 refinement: the gauntlet ALREADY has a zero-floor -- run-tests-all.sh:35
exits 2 when RAN is 0. What is missing is a MINIMUM-count floor, since 81 is
pinned nowhere executable. repo_lint.py:132-135 has NEITHER a valid-root check
nor a files-scanned floor. The two gates need different fixes, which the earlier
framing conflated.
R12 and R10 confirmed as previously stated, with exact line citations.
Revert: git revert this commit; the capture is new and CURRENT-STATE additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
G18 OPENED: IPAM apex completeness for the Octavia lb-mgmt plane (blocking, deferred)
...
Operator direction, verbatim: "leave this as an open decision that will need a
ruling once we have the cloud live and we have a better read on the network and
how everything is functioning with the addition of the new IPv6 configurations.
Make this a gated decision so we cannot close the project (or whatever phase you
think it best ruled in) without a ruling on this item."
WHAT IT IS. R8 ruled that Octavia creates and owns its own IPv6 lb-mgmt network.
The measured consequence is that the octavia charm exposes NO CIDR, address-family
or router configuration option -- create-mgmt-network (default True) is the only
related option -- so the prefix is CHARM-GENERATED and cannot come from the D-111
carve. NetBox is therefore knowingly incomplete for exactly one plane. That is the
authority-inversion concern the audit's lens 7 raised: the apex being back-filled
to match a deploy rather than driving it.
WHY DEFERRED RATHER THAN RULED NOW. It is not answerable from artifacts -- it
needs the cloud live and an observed read on IPv6 behaviour. It is also adjacent
to the UNRULED D-136 NetBox-coupled render pipeline, which covers the same
apex-authority ground, so ruling it early would pre-empt that decision.
PLACEMENT. Recorded as gate G18 in CURRENT-STATE section 6, the gate authority.
[R] ruling-type: closes ONLY on a GA-R5 recorded ruling with the exact utterance,
dated, committed and pushed. ANSWERABLE from Stage 5 onward, since the prefix
exists once Octavia deploys. BLOCKING at the FINAL stage close / project close --
the deployment may not be declared complete while it is open. That placement gives
the widest window in which live evidence can accumulate while still guaranteeing
the question cannot be lost.
Options are pre-recorded in the gate row so a future session does not have to
re-derive them: (a) back-fill the charm-created prefix into NetBox post-deploy as
a documented record; (b) record the plane as charm-owned and explicitly out of
apex scope; (c) fold it into D-136's render-pipeline ruling if that is taken first.
ALSO CROSS-RECORDED, because it is exactly the kind of thing that gets "fixed" by
mistake: the absence of an lb-mgmt :x80 prefix in the VR1 ULA carve is CORRECT
under R8, not a gap. Noted in the D-101 R8 ruling note and in the G18 row.
Mirrored into docs/audit/queued-rulings-20260727.md as R16 (gated, deferred) with
CURRENT-STATE named as the authority, so the queue and the gate table agree.
Revert: git revert this commit; the gate row and cross-references are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R8 RULED: Octavia creates and owns its own IPv6 network (D-101 note; sub-ruling closed)
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "I want to take a Octavia creates and owns its own IPv6
network." Supporting reasoning recorded because it is load-bearing: "We have to
research to troubleshoot if we run into MTU bug issues down the road. We have
already deployed using this topology in the v1 DC test deployments that got us to
this point."
This CLOSES the octavia-family sub-ruling that the 2026-07-25 D-101 note left
explicitly open.
THE RULING REQUIRES NO ARTIFACT CHANGE, and the operator's reasoning checks out on
the artifacts: create-mgmt-network is set NOWHERE in bundle.yaml or any overlay, so
the charm default True has always applied, and Octavia's whole options block is
debug, openstack-origin, amp-image-tag, vip. VR0 deployed Octavia on precisely this
shape. This ratifies the existing topology rather than changing it.
FAMILY AGREES THREE WAYS. The charm's default lb-mgmt-subnet is IPv6 (LP #1897418,
verbatim: "By default, Octavia charm uses ipv6 for its lb-mgmt-subnet") and it is a
ULA -- the amphora address quoted in LP #1911788 is fc00:fa21:3d5c:9cfd:..., i.e.
fc00::/7. That is exactly D-101's "IPv6-only ULA ... Octavia lb-mgmt ... Internal,
no external clients".
THE APPARENT VR0 GUA CONTRADICTION WAS A PAPER ALLOCATION, checked on operator
instruction: lib-net.sh gives VR0 six IPv4 planes and no lbaas plane; lbaas is
listed in STALE_SPACES; maas-as-built-reference records that NIC as "idle
(undefined; ex-lbaas), raw NIC, no link"; and D-101's own context says VR0 is
IPv4-only. The whole VR0 v6 tree in the apex is a design record.
CONSEQUENCE FOR D-111, worth recording so it is not later "fixed": the absence of
an lb-mgmt :x80 prefix in the VR1 ULA carve is CORRECT, not a gap. The charm
exposes no CIDR option, so the prefix is charm-generated and cannot come from the
apex.
STANDING OBLIGATION carried by the operator's own reasoning: LP #2018998 (o-hm0 vs
lb-mgmt-net MTU, charm-octavia, High) is Fix Released across our lineage but
recurred 2025-12-31 against octavia 14.0.0 / 2024.1 stable -- our exact pin. A
jumbo lb-mgmt-net beside a 1500 o-hm0 silently drops health messages over 1500
bytes and causes spurious failovers, which is the direct interaction with R3's
jumbo-underlay ruling. Owed at the Octavia step: verify o-hm0's MTU MATCHES
lb-mgmt-net's by measurement, not by trusting the charm.
NOT ASSERTED: whether the charm attaches an external gateway to the Neutron router
it creates. No config surface tells it to and ULA is not globally routable, but
isolation was not proven from the documentation. Settle by inspecting the router at
deploy time; queued as an observation, not a blocker.
Revert: git revert this commit; the ruling note and research sections are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R8 research: both in-repo citations fail; charm default is IPv6; new MTU risk found
...
Operator declined to rule on the overlay's own citations -- "I would like
documentation and vendor specific information rather than conjecture. Research is
cheap, guesswork is expensive" -- then directed a full read of two bugs weighed
against current versions. That was the right call and it inverted the answer.
BOTH CITATIONS IN overlays/dc-dc-ipv6-family-matrix.yaml FAIL ON INSPECTION:
- LP #1913409 is Fix Released (2021) against kolla-ansible, a DIFFERENT
INSTALLER. No bearing on a Juju/charm deployment.
- LP #1911788 is Incomplete and a DUPLICATE of LP #1896630, and is NOT an IPv6
defect. Its diagnosed cause is an OVN port-binding hostname mismatch --
binding_host_id carrying the shortname instead of the FQDN for LXD containers
on MAAS, giving binding_failed instead of ovs. The IPv6 address in the timeout
was incidental; the port never bound, so nothing would have worked over any
family.
VERSION WEIGHING, verified rather than asserted. #1896630's primary fix is
ovs-record-hostname.service, which the bug places in OVS 2.15 (Focal shipped
2.13). Rather than assume, I queried THIS DEPLOYMENT'S OWN dc0 mirror for what a
jammy node would actually install: openvswitch-switch 2.17.0-0ubuntu1 (jammy) and
2.17.9-0ubuntu0.22.04.2 (jammy-updates). 2.17.9 >> 2.15, so the fix is present on
every node this deployment will build. The bug's era was Juju 2.7.8, charms 20.08
Ussuri, Focal/OVS 2.13; we are Juju 3.6.27, charms 2024.1, jammy/OVS 2.17.9. The
charm-layer-ovn task -- the one closest to this repo's OVN usage -- is closed
Invalid.
AND THE DEFAULT RUNS THE OTHER WAY: LP #1897418 records, verbatim, "By default,
Octavia charm uses ipv6 for its lb-mgmt-subnet", with a reporter noting success
"using the charm default (ipv6) network but using ipv4 everywhere else". Upstream
Octavia documents that "IPv6 subnets can be used for the LB Network". So D-101's
IPv6-only placement of lb-mgmt is ALIGNED with the charm default, and a v4-only
lb-mgmt-net would be the DEPARTURE requiring justification -- the opposite of how
R8 was first framed.
NEW LIVE RISK FOUND, COUPLED TO THE R3 RULING: LP #2018998 "MTU mismatch between
o-hm0 and lb-mgmt-net" (charm-octavia, High, Edward Hope-Morley). A jumbo
lb-mgmt-net beside a 1500 o-hm0 silently drops health messages over 1500 bytes
and triggers SPURIOUS load-balancer failovers -- evidenced in the bug as "UDP,
length 1534 > 1500 and o-hm0 never receives them" and "o-hm0: dropped over-mtu
packet: 1744 > 1500". Fix Released across our lineage, BUT a recurrence was
reported 2025-12-31 against octavia 14.0.0 / 2024.1 stable, the exact channel
bundle.yaml pins. R3 ruled the underlay be finished to jumbo, which is precisely
this bug's precondition. OWED at the Octavia step of Stage 5: verify o-hm0's MTU
MATCHES lb-mgmt-net's after deploy, not merely that the charm claims to set it.
GENERALISABLE LESSON, recorded in the capture: a bug NUMBER in a comment is an
existence claim; what the bug SAYS is the content, and only content is evidence.
Same class as the two ruled-but-never-built decisions this audit already found --
a plausible in-repo statement that nothing verifies.
R8 is NOT ruled by this commit. It will be re-presented on corrected evidence.
Revert: git revert this commit; the capture is new and CURRENT-STATE additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
