| 2026-07-27 |
R8 RULED: Octavia creates and owns its own IPv6 network (D-101 note; sub-ruling closed)
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "I want to take a Octavia creates and owns its own IPv6
network." Supporting reasoning recorded because it is load-bearing: "We have to
research to troubleshoot if we run into MTU bug issues down the road. We have
already deployed using this topology in the v1 DC test deployments that got us to
this point."
This CLOSES the octavia-family sub-ruling that the 2026-07-25 D-101 note left
explicitly open.
THE RULING REQUIRES NO ARTIFACT CHANGE, and the operator's reasoning checks out on
the artifacts: create-mgmt-network is set NOWHERE in bundle.yaml or any overlay, so
the charm default True has always applied, and Octavia's whole options block is
debug, openstack-origin, amp-image-tag, vip. VR0 deployed Octavia on precisely this
shape. This ratifies the existing topology rather than changing it.
FAMILY AGREES THREE WAYS. The charm's default lb-mgmt-subnet is IPv6 (LP #1897418,
verbatim: "By default, Octavia charm uses ipv6 for its lb-mgmt-subnet") and it is a
ULA -- the amphora address quoted in LP #1911788 is fc00:fa21:3d5c:9cfd:..., i.e.
fc00::/7. That is exactly D-101's "IPv6-only ULA ... Octavia lb-mgmt ... Internal,
no external clients".
THE APPARENT VR0 GUA CONTRADICTION WAS A PAPER ALLOCATION, checked on operator
instruction: lib-net.sh gives VR0 six IPv4 planes and no lbaas plane; lbaas is
listed in STALE_SPACES; maas-as-built-reference records that NIC as "idle
(undefined; ex-lbaas), raw NIC, no link"; and D-101's own context says VR0 is
IPv4-only. The whole VR0 v6 tree in the apex is a design record.
CONSEQUENCE FOR D-111, worth recording so it is not later "fixed": the absence of
an lb-mgmt :x80 prefix in the VR1 ULA carve is CORRECT, not a gap. The charm
exposes no CIDR option, so the prefix is charm-generated and cannot come from the
apex.
STANDING OBLIGATION carried by the operator's own reasoning: LP #2018998 (o-hm0 vs
lb-mgmt-net MTU, charm-octavia, High) is Fix Released across our lineage but
recurred 2025-12-31 against octavia 14.0.0 / 2024.1 stable -- our exact pin. A
jumbo lb-mgmt-net beside a 1500 o-hm0 silently drops health messages over 1500
bytes and causes spurious failovers, which is the direct interaction with R3's
jumbo-underlay ruling. Owed at the Octavia step: verify o-hm0's MTU MATCHES
lb-mgmt-net's by measurement, not by trusting the charm.
NOT ASSERTED: whether the charm attaches an external gateway to the Neutron router
it creates. No config surface tells it to and ULA is not globally routable, but
isolation was not proven from the documentation. Settle by inspecting the router at
deploy time; queued as an observation, not a blocker.
Revert: git revert this commit; the ruling note and research sections are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R8 research: both in-repo citations fail; charm default is IPv6; new MTU risk found
...
Operator declined to rule on the overlay's own citations -- "I would like
documentation and vendor specific information rather than conjecture. Research is
cheap, guesswork is expensive" -- then directed a full read of two bugs weighed
against current versions. That was the right call and it inverted the answer.
BOTH CITATIONS IN overlays/dc-dc-ipv6-family-matrix.yaml FAIL ON INSPECTION:
- LP #1913409 is Fix Released (2021) against kolla-ansible, a DIFFERENT
INSTALLER. No bearing on a Juju/charm deployment.
- LP #1911788 is Incomplete and a DUPLICATE of LP #1896630, and is NOT an IPv6
defect. Its diagnosed cause is an OVN port-binding hostname mismatch --
binding_host_id carrying the shortname instead of the FQDN for LXD containers
on MAAS, giving binding_failed instead of ovs. The IPv6 address in the timeout
was incidental; the port never bound, so nothing would have worked over any
family.
VERSION WEIGHING, verified rather than asserted. #1896630's primary fix is
ovs-record-hostname.service, which the bug places in OVS 2.15 (Focal shipped
2.13). Rather than assume, I queried THIS DEPLOYMENT'S OWN dc0 mirror for what a
jammy node would actually install: openvswitch-switch 2.17.0-0ubuntu1 (jammy) and
2.17.9-0ubuntu0.22.04.2 (jammy-updates). 2.17.9 >> 2.15, so the fix is present on
every node this deployment will build. The bug's era was Juju 2.7.8, charms 20.08
Ussuri, Focal/OVS 2.13; we are Juju 3.6.27, charms 2024.1, jammy/OVS 2.17.9. The
charm-layer-ovn task -- the one closest to this repo's OVN usage -- is closed
Invalid.
AND THE DEFAULT RUNS THE OTHER WAY: LP #1897418 records, verbatim, "By default,
Octavia charm uses ipv6 for its lb-mgmt-subnet", with a reporter noting success
"using the charm default (ipv6) network but using ipv4 everywhere else". Upstream
Octavia documents that "IPv6 subnets can be used for the LB Network". So D-101's
IPv6-only placement of lb-mgmt is ALIGNED with the charm default, and a v4-only
lb-mgmt-net would be the DEPARTURE requiring justification -- the opposite of how
R8 was first framed.
NEW LIVE RISK FOUND, COUPLED TO THE R3 RULING: LP #2018998 "MTU mismatch between
o-hm0 and lb-mgmt-net" (charm-octavia, High, Edward Hope-Morley). A jumbo
lb-mgmt-net beside a 1500 o-hm0 silently drops health messages over 1500 bytes
and triggers SPURIOUS load-balancer failovers -- evidenced in the bug as "UDP,
length 1534 > 1500 and o-hm0 never receives them" and "o-hm0: dropped over-mtu
packet: 1744 > 1500". Fix Released across our lineage, BUT a recurrence was
reported 2025-12-31 against octavia 14.0.0 / 2024.1 stable, the exact channel
bundle.yaml pins. R3 ruled the underlay be finished to jumbo, which is precisely
this bug's precondition. OWED at the Octavia step of Stage 5: verify o-hm0's MTU
MATCHES lb-mgmt-net's after deploy, not merely that the charm claims to set it.
GENERALISABLE LESSON, recorded in the capture: a bug NUMBER in a comment is an
existence claim; what the bug SAYS is the content, and only content is evidence.
Same class as the two ruled-but-never-built decisions this audit already found --
a plausible in-repo statement that nothing verifies.
R8 is NOT ruled by this commit. It will be re-presented on corrected evidence.
Revert: git revert this commit; the capture is new and CURRENT-STATE additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R7 RULED: per-DC independent Octavia PKI (D-109 amendment)
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "Per-DC independent Octavia PKI; fix the generator first
(Recommended)".
Recorded as a D-109 AMENDMENT. D-109 established per-DC INDEPENDENT Vault roots
with a DR-honest rationale but never mentioned the Octavia amphora
control-plane PKI, which is a SEPARATE trust domain with its own CA generated
outside Vault by phase-01 step 1.0-GEN. This extends the same posture to it.
MEASURED REFINEMENT THAT NARROWS THE WORK. The generator is dc0-frozen in two
ways of DIFFERENT severity, and only one is a design problem:
- The CA SUBJECT is a baked literal, "/CN=VR0 DC0 Omega Cloud Octavia Controller
CA/O=Neumatrix". Reusing it roots dc1's amphora chain in a VR0-DC0-named CA.
- The controller cert's SAN is ALREADY DERIVED per-DC by design -- the step
states it carries the controller FQDN, octavia API FQDN and the Octavia API VIP
"derived from the bundle at generation time -- DOCFIX-067; never a baked
literal". That half needs no change at all.
- The actual blocker is the VIP gate, grep -qE '^10\.12\.4\.', which accepts only
dc0's band while dc1's octavia VIP is 10.12.64.57 and lives in an overlay.
A DOCFIX was owed regardless of the ruling: the generator cannot produce a dc1
artifact as written, and phase-01:144-145 hard-ABORTS the deploy when
overlays/octavia-pki.yaml is absent, which it is. Option (c) would therefore have
deferred only the part D-109 already answers by precedent while leaving the
required work untouched.
REUSE REFUSED ON POSTURE. The overlay carries CA private keys plus a plaintext
issuing-CA passphrase inside the repo clone, and SEC-004 records the repo as
still PUBLIC. Sharing one amphora control-plane CA private key across two clouds
D-100 defines as independent would widen an existing exposure rather than
contain it, in a commercial multi-tenant cloud with hard tenant isolation.
Independent per-DC CAs keep that blast radius to one DC.
CAVEAT CARRIED FORWARD, because it will bite otherwise: the generator's VIP gate
must be re-pointed at the MERGED deploy input rather than bundle.yaml. Left
reading the base bundle it breaks again the moment ruling-3's VIP extraction and
R11's new .61/.62 allocations land.
Roosevelt analog: per-DC amphora CAs match per-DC IPMI/BMC credentials and the
per-DC MAAS power keys of SEC-012/-016 -- the same no-cross-DC-shared-secret
principle already ruled twice in this deployment.
Revert: git revert this commit; the amendment is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R11 RULED: vault .61 and designate .62, dual-family triples (D-020 amendment)
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "Both full triples (.61 vault, .62 designate), dual-family,
and fix the gate (Recommended)".
THE FINDING THAT MEASURING FIRST PRODUCED: vault was ALREADY RULED and never
built. D-020's decision text enumerates vault BY NAME among the clustered
applications carrying both a provider and a metal VIP; measured, base vault is
num_units:1 with an EMPTY options block. This is a conformance repair of a
2026-era decision, not a new choice -- and it is the SECOND ruled decision this
audit has found unimplemented, after D-134's address bands (R4). Worth stating
plainly: the pattern is ruled-but-never-built, and nothing in the repo could
detect either case.
designate is genuinely new -- absent from D-020's enumeration -- so it needed a
ruling rather than a repair, and this amendment adds it. Its dnsaas endpoint is
already bound provider-public, so a provider leg is coherent for it.
Shape is the ESTABLISHED triple, not a new form. The measured octet map is
consecutive: keystone .50 through ceph-radosgw .60, all provider/admin/internal
triples. vault takes .61, designate .62, both dual-family per R2 -- adding
v4-only now and re-doing them later would mean re-issuing certificate SANs on a
live cloud, and for vault that is 23 certificate relations.
Option (b), vault metal-only per the 2026-07-25 expansion review, was refused and
the conflict is RECORDED so that proposal is not later mistaken for the ruled
position: it has a real technical argument (all 23 vault consumers are internal)
but it contradicts D-020's own enumeration and would leave vault the single
non-triple in the bundle, a permanent special case for the checker.
Mechanical consequences, no choice in them: .61/.62 are legal under D-134's
amended .50-.99 band but REJECTED by the gate today -- provider-bundle-check.py
holds OCTET_LO/HI = 50,60 and lib-net.sh holds VIP_OCTET_MAX=60. These are
SEPARATELY NAMED constants in two files, so widening the band is a two-file
change (L3-7). VIP_COUNT_EXPECT moves 11 -> 13.
Gate hardening ruled IN SCOPE rather than deferred: the checker learns to FAIL on
an application with an hacluster relation and no vip. Justification is measured --
grep -rn cluster_count scripts/ tests/ returns NOTHING, and a constructed overlay
rewriting all 20 cluster_count values 3 -> 1 yields a byte-identical PASS (L4-3).
That is exactly why decorative HA was found by an audit instead of by a gate.
Per the R6 ruling, these VIPs land BEFORE dc-ha-scaleup.yaml is applied.
Execution is a separate gated step.
Revert: git revert this commit; the amendment is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R6 RULED: close the two VIP gaps, then apply the HA overlay whole (D-121 note)
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "Close the two VIP gaps first, then apply the overlay whole
(Recommended)".
THE TENSION IT RESOLVES. D-121 is titled "VR1 makes HA real -- scale the
decorative single-unit control plane to 3". Deferring the overlay would have
deployed VR1 in exactly the shape D-121 was written to retire. Applying it as-is
would have shipped, for vault, precisely the defect D-121 exists to remove -- a
pacemaker cluster with nothing to manage. Only the ruled sequence satisfies the
decision rather than half of it.
MEASURED BEFORE PRESENTING, and it reframed the question. The overlay scales 14
applications and moves all 12 base hacluster subordinates to cluster_count 3, but
ELEVEN OF TWELVE principals already carry proper VIP triples. The gap is exactly
two applications, and they are not the same shape:
- designate is in the base bundle WITH an hacluster and an ha binding, no vip.
- vault has NO hacluster in base at all and an EMPTY options block. The overlay
INTRODUCES the vault-hacluster application, its cluster_count 3, and the
vault:ha relation -- and still no vip.
Vault is the deciding case because of blast radius: 23 relations consume
vault:certificates, plus barbican-vault:secrets-storage. With no VIP every
consumer binds a unit address and there is nothing for a failover to move.
Remediating that on a live 3-unit vault means re-pointing 23 certificate
relations and re-issuing SANs on a running cloud, which is why the faster option
was refused.
SEQUENCING CONSEQUENCE: R11 becomes a HARD Stage-5 precondition ordered BEFORE
the overlay, not a parallel item.
Scope is tractable and the pattern is already in-repo: octavia carries a correct
triple (10.12.4.57 10.12.8.57 10.12.12.57), so R11 has a shape to copy rather
than design.
NOT resolved here, tracked separately: the Stage-6 radosgw multisite path is
single-unit-shaped while this scales ceph-radosgw to 3; and
provider-bundle-check.py checks cluster_count NOWHERE, which is precisely why
decorative HA was found by an audit instead of by a gate.
Address demand is not an argument against this -- R4's band ruling already covers
the overlay's growth from 27 to ~55 LXD units.
Execution is a separate gated step, not authorised by this ruling.
Revert: git revert this commit; the ruling note is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-ruling measurements for R6-R15: measure before asking, all ten
...
Operator: "Measure all the rest then we can work through them with relevant
data at hand." Taken directly by the session rather than delegated, after the
UNMEASURED-gap sweep showed lens findings need measuring before they become
questions. Capture: docs/audit/r6-r15-measurements-20260727.txt.
THE TWO RESULTS THAT CHANGE A QUESTION'S SHAPE:
R6/R11 -- vault's ENTIRE HA apparatus exists only in dc-ha-scaleup.yaml. Base
vault is num_units:1 with an EMPTY options block: no vip, no hacluster, no
relation. The overlay INTRODUCES the vault-hacluster application (not present
in base at all), sets cluster_count:3, adds the vault:ha relation -- and still
no vip. Applying the overlay therefore creates a 3-node pacemaker cluster with
nothing to manage. The blast radius is what makes it sharp: 23 relations
consume vault:certificates, plus barbican-vault:secrets-storage. Vault is the
CA for the whole cloud and every consumer binds a unit address. Of the 12 base
hacluster subordinates exactly ONE principal lacks a vip (designate), and
octavia carries a proper triple -- so this is a 2-app gap, not a pattern.
R10 -- the question largely DISSOLVES. Preflight has been run on the wrong
host. Measured: vcloud has neither maas nor juju nor openstack; voffice1 has
maas AND juju. So P3's 33 warns and P4's "MAAS unreachable" both clear simply
by running preflight on the D-128 Plane-2 host where it belongs. Only the
octavia-pki absence and the 7 credential findings are host-independent. Same
"tool absence reported as something else" class as U6, now three instances.
OTHERS: R7 the octavia CA subject is a baked VR0-DC0 literal while the SAN is
already derived by design (DOCFIX-067) -- the two halves differ in severity.
R8 the v4-only lb-mgmt shape is already pre-analysed in-repo with LP #1911788
and #1913409 cited and a drafted block ready (NOT verified upstream by me).
R9 only 8 of 28 lib-net consumers call the DC selector; the 20 that do not
include the entire phase-02..phase-06 family. R12 chronyc appears ZERO times
in CURRENT-STATE, confirming G17 omits the time check. R13 30 rows / 16 ids
are operator-terminal, of which SEVEN are keypairs whose loss is
unrecoverable-in-place. R14 the matrix has no exception field at all. R15 81
harnesses on disk and 81 reported, with 81 pinned nowhere executable.
METHOD NOTE, recorded because it nearly published a wrong figure: parsing
creds-matrix.tsv with awk -F'\t' returned ZERO operator-terminal rows,
contradicting lens 7's 30. The file is SPACE-ALIGNED, not tab-separated.
Re-measured correctly it is 30/16 and lens 7 was right. A disagreement with a
prior finding was treated as a reason to re-check my instrument rather than to
publish the new number.
Revert: git revert this commit; the capture is new and CURRENT-STATE additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Fix L5 collision introduced by the R5 ruling-note heading
...
repo-lint L5 failed on the previous commit and I pushed it anyway -- I ran lint
in the same compound command as the commit but did not GATE on its result.
Owned, and fixed forward rather than by rewriting pushed history.
Root cause, read from the rule rather than guessed: repo_lint.py:355 matches
`^##+\s+(D-\d{3,})\b(?!.*AMENDMENT|.*RESOLVED)` -- ANY heading level that LEADS
with a D-number counts as a definition unless the line also contains AMENDMENT
or RESOLVED. My heading "### D-106 RULING NOTE 2026-07-27 -- ..." led with the
D-number and contained neither, so it registered as a second definition of
D-106 and collided with "## D-106: Designate reactivation (VR1)".
The established convention already handles this and I should have followed it:
the existing D-101 ruling notes read "### RULING NOTE 2026-07-25 -- D-101 ...",
placing the D-number AFTER the words so the regex does not match. The heading is
now "### RULING NOTE 2026-07-27 -- D-106: ...", matching that pattern. Content
is unchanged.
Why the R1 and R4 amendments did not trip this: both contain the literal word
AMENDMENT, which the negative lookahead excludes.
Process fix for the rest of this session: commits are now gated on repo-lint
exiting clean, not merely preceded by a lint run whose output I read afterwards.
Revert: git revert this commit (restores the colliding heading and a red lint).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R5 RULED: designate deploys at Stage 5, is configured at Stage 7 (D-106 note)
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "Accept at Stage 5; rewrite Stage 7 Step 5 to
configure-not-deploy (Recommended)".
CORRECTION RECORDED IN THE DECISION ITSELF, not just this message. When R5 was
first put to the operator, the audit claimed this option "inverts D-106's
bootstrap order, which puts os-public-hostname + FQDN-SAN certs BEFORE
Designate". That was WRONG. D-106's order is a CONFIGURATION sequence -- static
hosts, then os-public-hostname, then Vault FQDN-SAN certs, then zones and A/AAAA,
then neutron, then tenant subnets. It governs when the DNS wiring happens, not
when the charm is installed. Deploying the app earlier does not invert it: the
zones still follow the certs. Conflating "install the charm" with "do D-106's
work" nearly cost a decision made on a false constraint, so the correction is
recorded in D-106 where a future session will meet it.
MEASURED BEFORE PRESENTING. All four designate applications and all EIGHT
relations are deploy-ready at Stage 5 -- every relation peer (mysql-innodb-cluster,
keystone, rabbitmq-server, vault, memcached, designate-bind) is created by Stage
5. But they will be FUNCTIONALLY INERT: os-public-hostname is set in NO deploy
artifact, and bundle.yaml:11 records the current posture as IP-ONLY with the dual
VIPs as the catalog endpoint. The charm runs; the feature does not. That is
exactly the state Stage 7 exists to resolve.
NEW SURFACE DEFECT FOUND BY THE MEASUREMENT: the phase-6 runbook contradicts
ITSELF. Line 172 states "There is no designate: or designate-bind: application
block anywhere in ..."; line 178 states "Designate is deployed in-bundle in each
DC". Both cannot be true. The bundle settles it -- DOCFIX-167 put designate there
on 2026-07-10 -- and the runbook needs rewriting regardless of this ruling.
Option (c), setting os-public-hostname at Stage 5, was refused and the reason is
worth keeping: it is the ONE branch that genuinely collides with D-106. Publishing
a public FQDN endpoint before Vault has issued FQDN-SAN certs recreates the exact
D-019 root cause D-106 was written to remove -- metal-only charms pulling a public
FQDN endpoint they cannot resolve, which is also the D-021 amphora constraint.
Option (b), suppressing designate at Stage 5, was refused because it needs
machinery that does not exist and partially undoes DOCFIX-167 -- building a
mechanism to satisfy a stale gate rather than fixing the gate.
Execution is a DOCFIX against the phase-6 runbook (Step 1 inventory prose, Step 5
verb and gate), logged to the Phase-3 batch, not executed under this ruling.
Revert: git revert this commit; the ruling note is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R4 RULED: reserved bands become enforced in MAAS via a DC-aware tool (D-134 amendment)
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "Build a DC-aware tool; full v4 scheme + FIP now, v6 bands
after the carve (Recommended)".
MEASURED BEFORE ASKING, which sharpened the question considerably.
The defect is worse than "bands not enforced": D-134's band table has never
existed anywhere but prose. maas admin ipranges read returns THREE ranges
cloud-wide, ALL type=dynamic, ZERO reserved.
The collision is now QUANTIFIED rather than hypothetical. MAAS's own
subnet unreserved-ip-ranges for dc1 metal-admin reports its lowest free span as
10.12.68.5-.99 (95 addrs) -- precisely the .4-.49 utility band and the whole
.50-.99 VIP band. Against that, the dc1 bundle places 27 LXD units in base form
and ~55 once dc-ha-scaleup scales 14 applications, each needing a MAAS-allocated
address, and the VIP band is where the 33 per-DC VIPs live. Zero 10.12.*
addresses are allocated today, so this is a PRE-EMPTION and not an incident.
MAAS's exact allocation ORDER is deliberately NOT asserted -- lens 2 was right to
refuse that claim and I have not added it.
Confirmed there is genuinely NO VR1 path: only site-headend-install.sh
(office1-only) and phase-00-maas-standup.sh can create an iprange, and the latter
REFUSES for any non-VR0 DC. That refusal is CORRECT -- it is the cross-DC mixing
guard lib-net's selector exists to enforce. The gap is that nothing replaced it.
NEW ARCHITECTURAL CONTENT, and it exists because of R2: D-134's bands are v4-only.
Ruling dual-stack made the 33 VIPs dual-family, which left the v6 planes with no
band discipline at all and the same collision waiting in the other family. The
amendment establishes that the v6 planes inherit an equivalent scheme. The exact
v6 octet mapping is NOT ruled -- mirroring the v4 host-part layout is the obvious
mechanical default, not a fresh design question.
The v6 pass following the carve is FORCED SEQUENCING, not a deferral: a reserved
range cannot be created on a subnet that does not yet exist, and the v6 plane
subnets are not in MAAS.
Worth noting what the tool buys beyond the reservation itself: a `check` arm gives
D-134 an EXECUTABLE gate. Today nothing in the repo can detect that the bands are
unenforced, which is why this survived from 2026-07-23 to now.
Execution is a separate gated step. Standard delivery discipline applies to the
tool (harness green, gauntlet, repo-lint, changelog with revert); the 12-subnet
reservation pass is an operator-gated live MAAS mutation and is not batched with it.
Revert: git revert this commit; the amendment is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R3 RULED: finish the jumbo underlay; first recorded MTU budget verdict
...
GA-R5: question and exact utterance quoted, dated, pushed before dependent work.
Operator utterance: "Raise the two lagging segments to 9000 (Recommended)".
Recorded as a D-101 RULING NOTE. D-102 is the original MTU sub-policy but is
MERGED INTO D-101 and its body directs amendments there.
MEASURED BEFORE ASKING, which materially re-framed the question. D-101 requires
"the measured underlay MTU is a Phase-0 gate -- do not assume jumbo", and
scripts/dc-dc-mtu-geneve-budget.sh had existed without ever being run to a
recorded verdict. Running it both ways, plus measuring the host bridges, showed
the jumbo branch is nearly complete already rather than a large project:
- Every vcloud MESH leg is ALREADY 9000, including mesh-vr1-dc0-vr1-dc1
(virbr5), the inter-DC path itself. All six plane bridges on both racks: 9000.
- The four 1500 legs are the D-125 SIMULATED-ISP uplinks. They model the
internet, D-125's egress gate is defined against them, and they must STAY
1500. Not a defect.
- Exactly TWO segments lag: enp1s0 inside both containment VMs, and all 17 MAAS
VLAN records.
The MAAS record is the one that bites silently: MAAS renders VLAN MTU into node
netplan, so a jumbo bridge beneath a 1500 record still yields 1500 node
interfaces. Jumbo bridges alone do not deliver a jumbo underlay.
Why (a): tenant MTU stays 1500, so no per-charm MTU coordination is needed at
all. Option (b) required ovn geneve + tenant-network MTU + amphora to agree
exactly and permanently across both DCs, and NOTHING in this repo checks MTU --
neither cloud-assert.sh nor provider-bundle-check.py has any MTU assertion -- so
drift would be silent, which is precisely what D-101 calls "the classic
nested-OpenStack failure mode". Option (c) was the worst: jumbo at the plane
layer with a still-capped transit throttles D-108 rbd-mirror and radosgw
multisite behind a chokepoint invisible where an operator would look.
COUPLING TO R2, recorded because it changes the arithmetic: the 56-byte overhead
is the IPv6 figure and applies BECAUSE dual-stack was ruled. Under v4-only it
would have been 42, and the 1500-underlay tenant MTU 1458. R2 and R3 are not
independent.
Execution is a SEPARATE gated step and is NOT authorised by this ruling. The
verification owed is BEHAVIOURAL: an end-to-end large-frame test with DF set
across the inter-DC path. An interface claiming 9000 is not proof a 9000-byte
frame survives the path -- the assert-on-content rule applied to MTU.
Also worth recording: invoked bare, the budget script correctly REFUSES ("FAIL:
--underlay-mtu is REQUIRED -- no default, measure it this session"). It does not
guess. That discipline is why this verdict is trustworthy.
Revert: git revert this commit; the ruling note and capture are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Second gap sweep: U15-U17 closed; DC data path has no IPv6 at ANY layer
...
Operator: "Close the remaining gaps". All three measurable items are now
measured; what remains open is blocked by a known missing credential or is a
deliberate deferral, and both are stated with cause rather than left implicit.
U15 -- voffice1 transit addressing is REBOOT-DURABLE. A positive result, which
is worth recording as clearly as a defect: live enp2s0 172.31.0.1/30 (dc0 leg)
and enp3s0 172.31.0.5/30 (dc1 leg), both netplan-persistent via
/etc/netplan/60-transit.yaml and 61-transit-dc1.yaml. The D-128 Plane-2 path
survives a headend reboot and no site-baseleg row is owed for it.
U16 -- RETROFIT_WAIT=30m has NO recorded provenance. git log -S traces it to a
single bulk commit ("New Phase Scripts") with no rationale, and a repo-wide grep
finds no constant anywhere in scripts/ documented as nested-virt or depth-4
calibrated. It is an inherited default that has never been validated against the
nested I/O it will actually run on at Stage 5 Step 9. Log-only, but it should
not be mistaken for a tuned value. Bonus finding from reading the file: that
script's own preconditions require BOTH the openstack and juju clients, and no
host currently has both -- vcloud has neither, voffice1 has juju only. It cannot
run anywhere today, which reinforces S-1.
U17 -- the DC data path carries NO IPv6 at any layer measured. On BOTH racks:
zero global v6 addresses on any plane bridge, no v6 default route, and
accept_ra=1 (so they would accept a router advertisement; none is arriving).
Combined with the existing measurements -- no v6 subnet on any of the 12 MAAS
plane fabrics, no v6 link on any of the 18 nodes -- this is a THREE-LAYER
confirmation of L2-3. It also WIDENS the R2 propagation task: it is not only
"carve into MAAS", the rack bridges have no v6 either.
STILL OPEN, with cause rather than silence: the two DC edges' own interface-level
v6 configuration. dc0 is blocked by a known missing credential, not by a failure
to look -- ~/vr1-dc0-creds/ contains no opnsense-api.txt while ~/vr1-dc1-creds/
does, which is SEC-021(a) directly visible on disk, the residual item the
2026-07-27 consolidation batch deliberately excluded because the re-mint is a
live edge mutation. dc1's API is credentialed but measured unreachable from
vcloud (timeout; the path runs from the rack, where the edge creds are correctly
not staged per SEC-015). U17 answers the substantive question from the rack side.
repo-lint/gauntlet ON voffice1 remain deliberately deferred until precondition
0.1 advances that clone -- running them against a 105-commit-stale tree would
produce a number that means nothing. That is a judgement, not an omission, and
it is recorded as such.
Revert: git revert this commit; register and CURRENT-STATE changes are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
UNMEASURED-gap sweep: 14 deferred items closed; juju restore-backup CONFIRMED absent
...
Operator challenge: did the committee actually measure live state, or take
shortcuts? Answer: some of both. This commit is the accounting, plus the sweep
that closes what was closeable.
WHO DID WHAT, precisely, because the accounting matters:
- The apex was never polled by ANY lens. Lens 2 declared it out of scope and did
so HONESTLY -- its UNMEASURED note explicitly warned "D-101's literals may
exist in NetBox and simply not be carved into MAAS". No overclaim. But it is a
key system that lens had reach and time to measure. Under-scoped, not dishonest.
- The two-day-old dump was MINE, in the correction, not an agent's.
- Sharpest instance of the general problem: lens 6 declared juju restore-backup
unverifiable because "no Juju client exists on this host", while juju 3.6.27
was installed on voffice1 and lens 5 was running juju help against it in the
same session. One lens called impossible what another was measuring.
CLOSED BY THE SWEEP (14 items). The consequential ones:
- LIVE apex polled via netbox/office1-record-dump.py: 139 prefixes, 103 IPv6,
ZERO added, ZERO removed vs the dump. The conclusion was right; the method was
not, and that distinction is the whole point.
- juju restore-backup DOES NOT EXIST on 3.6.27 -- "not a juju command ... Did you
mean: create-backup". L6-14 goes from flagged RISK to CONFIRMED DEFECT: Stage 6
Step 9's D-104 restore drill and its DoD bullet are unsatisfiable as written.
- juju bind DOES exist as documented -- lens 6 was RIGHT to exclude it from L6-7
rather than assert it wrong. A correctly-handled uncertainty.
- dc0 compute provider MACs measured (52:54:00:1b:19:e6 / 52:54:00:18:ab:b4),
matching lib-hosts and CONFIRMING L3-8.
- curl present on both racks, so L4-13's hole is theoretical, not live.
- The pinned charm channels DO resolve (2024.1, 2.4, squid all present via juju
info on voffice1) -- proving L4-2's root cause and its fix: P3's 33 warns are
purely the missing juju binary on vcloud, not a charmhub problem.
NEW FINDING: preflight's "MAAS unreachable: 'maas admin subnets read' failed" is
a MISDIAGNOSIS -- the maas binary is simply ABSENT on vcloud. Together with the
absent juju (L4-2) and absent openstack (S-1), THREE separate preflight/deploy
failures on this jumphost are all "the client is not installed", and each is
reported as something else. A tool-absence must never be reported as a
target-unreachable. Its own DOCFIX.
STILL-OPEN gaps that are measurable and were NOT done are listed in the
register's section 3 explicitly, not buried: DC edge v6 state, repo-lint/gauntlet
on voffice1 (deferred on purpose -- that clone is 105 commits stale, so running
them today would measure a stale tree), voffice1 transit reboot-durability, and
the provenance of RETROFIT_WAIT=30m.
STANDING LESSON added for the next committee: before declaring an item
unmeasurable, VERIFY THE CONSTRAINT YOU ARE ASSERTING. The UNMEASURED escape
hatch is load-bearing for honesty and must not become a way to avoid looking.
Revert: git revert this commit; the register is new and the CURRENT-STATE
paragraph is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CORRECTION: the v6 literals are ASSIGNED (D-111), not pending -- R2a withdrawn
...
The operator asked whether the Office1 NetBox apex had actually been polled. It
had not. It has now been read, and the answer inverts a claim I made one commit
ago and a question I put in front of them.
WHAT IS TRUE: the v6 literals are assigned, ratified and recorded, and have been
since 2026-07-11 under D-111 (ADOPTED). Measured from
netbox/draft/vr1-office1-current-20260725.json -- 139 prefixes, 103 IPv6, every
relevant row tagged D-101/D-111: ULA fd50:840e:74e2::/48 with DC0 planes at
:220/:221/:230/:240/:250::/64 and DC1 at :320/:321/:330/:340/:350::/64; GUA
provider-public DC0 2602:f3e2:f02:10::/64 + VIP f02:11::/64, DC1
2602:f3e2:f03:10::/64 + VIP f03:11::/64.
WHAT I GOT WRONG: the R2 ruling note claimed D-101's "Remaining open item" (the
org ULA /48 and per-DC GUA carve) had become a Stage-5 precondition because
"dual-stack cannot deploy against literals that do not exist". They exist. R2a
("which literals") is WITHDRAWN as never having been open; no operator utterance
is owed on it.
THE REAL PRECONDITION IS NARROWER AND BETTER: propagation, not assignment, and it
needs no ruling. The ratified values are absent from the two places Stage 5
actually reads -- scripts/lib-net.sh carries no v6 arm at all, and MAAS carries
no v6 on any of the 12 DC plane fabrics. Both are mechanical copies from an
authoritative source, so both move from Part A (needs a decision) to the Phase-3
mechanical batch.
Note CURRENT-STATE's "DRIFT-FREE across netbox/lib-net/artifacts" is NOT
contradicted: drift-free means values appearing in more than one place agree, not
that coverage is complete. lib-net.sh simply never gained a v6 arm.
PROCESS FAILURE, OWNED AND RECORDED IN THE DECISION ITSELF: the audit's own lens 2
listed the apex as UNMEASURED and warned, in terms, that "D-101's literals may
exist in NetBox and simply not be carved into MAAS". That warning was correct and
available, and I put a question to the operator anyway on the strength of D-101's
stale prose. Trusting stale decision prose over an available measurement is
exactly what this audit was convened to catch.
NEW DOCFIX-CLASS FINDING, logged not fixed (hard rule 1): D-101's "Remaining open
item" paragraph still reads "pending NetBox assignment (gap #3)" for literals
D-111 adopted on 2026-07-11 -- a decision's own prose contradicted by a later
ruling, DOCFIX-200/204 class, and the direct cause of this error. Queued in the
Phase-3 batch.
The R2 ruling itself STANDS -- dual-stack as ruled. Only its stated effect changed.
Revert: git revert this commit to restore the prior (incorrect) framing; the R2
ruling in the preceding commit is independent and should not be reverted with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R2 RULED: dual-stack as originally decided; the v6 literals move onto the critical path
...
GA-R5: question and exact utterance quoted in the Status block, dated, pushed
before dependent work.
Operator utterance: "Carve v6 and deploy dual-stack as ruled (Recommended)".
Recorded as a D-101 RULING NOTE -- a re-confirmation, amending nothing, in the
same shape as the 2026-07-25 note it follows.
WHY IT WAS A LIVE QUESTION AT ALL: the 07-25 ruling was measured against the
substrate for the first time by this audit and found unimplemented. Exactly ONE
IPv6 subnet exists cloud-wide and it is on the Office1 base fabric; none of the
12 DC plane fabrics carries one; zero v6 links across all 18 nodes; every node
reports default_gateways.ipv6 = NONE. D-101's matrix requires ULA on
data-tenant/storage/replication plus legs on metal-admin/metal-internal, with
nothing to bind against.
THE CONSEQUENTIAL EFFECT, and the reason this commit matters beyond recording a
choice: D-101's own "Remaining open item" -- the org ULA /48 and per-DC GUA carve
-- has been carried since authoring as "pending NetBox assignment ... not a
ratification question", i.e. non-blocking. It is now a STAGE-5 PRECONDITION.
Dual-stack cannot be deployed against literals that do not exist.
Downstream, all now inheriting dual-family: R9 (where dc1's literals live), R11
(vault/designate VIPs, which must be dual-family from the outset or their cert
SANs get re-issued on a live cloud), and the L3-9 overlay collision, which must
be reconciled BEFORE either authority location is populated. Worth restating: of
the two merge orders the audit measured, the DANGEROUS one is the one that
PASSES -- vips-overlay-last silently replaces every v6 leg and reports green.
R8 (octavia's family) is explicitly NOT resolved by this ruling.
NEW SUB-QUESTION OPENED, queued not ruled: R2a, which literals. The ruling
directs that they be assigned; it does not assign them. I have deliberately not
proposed prefixes -- that is the operator's address space, and hard rule 2
forbids inventing a literal. Only the SHAPE is presented.
Revert: git revert this commit; the ruling note and R2a are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R1 RULED: ceph-osd gets a real second disk on the storage nodes (D-121 amendment)
...
GA-R5: question as presented and the operator's exact utterance are both quoted
in the Status block, dated, and pushed BEFORE any dependent work.
Operator utterance: "Add an OSD volume to node-vm (Recommended)".
Recorded as a D-121 AMENDMENT rather than a new D-number. D-121 is the decision
that ratifies modules/node-vm sizing/count, and its own capacity re-validation
already records "Ceph disk re-run for 4 storage/DC = PASS 5.31 TiB" -- the second
disk was BUDGETED and never BUILT. GA-R3: architectural consequence, but it
amends a ruled node layout rather than establishing new ground.
TWO SCOPE CORRECTIONS landed with the ruling, both of which narrow it:
- It is EIGHT volumes, not eighteen. ceph-osd is placed on the four storage
nodes per DC (bundle.yaml:556-557, to: ["5","6","7","8"]). The other five
nodes per DC neither run ceph-osd nor need a second device. My earlier framing
said "all 18 nodes" -- true of the measured defect, wrong about the fix.
- Rack headroom measured before asserting feasibility: dc0 2.0T available, dc1
2.9T available under /var/lib/libvirt, thin-provisioned qcow2.
Why (a) over (b): MINIMIZE DELTA TO ROOSEVELT -- Roosevelt storage nodes have
real dedicated disks. Option (b) was also not safe to pick as presented: whether
ceph-osd at the pinned squid/stable supports a directory- or partition-backed
OSD was explicitly UNRESEARCHED, and the audit refused to assert it either way.
THE APPLY IS NOT AUTHORISED BY THIS RULING. Four preconditions recorded, the
sharpest being that re-commissioning is required for MAAS to see the new device,
and whether that preserves the D-134 statics and pinned MACs is UNVERIFIED --
the 2026-07-20 MAC-regeneration incident is the precedent for exactly that class.
modules/node-vm is shared by both DCs, so the change must be additive/opt-in or
it plans against all 18 domains.
Revert: git revert this commit; the amendment is additive and no artifact changed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CURRENT-STATE: land the dc1 egress measurement (repairs the prior commit's L10)
...
The previous commit (c1f5dc8) touched docs/audit/ without the paired
CURRENT-STATE update that GA-R1 amendment C1 requires, so repo-lint L10 failed
on it. Owned rather than silently amended: the commit is pushed, so this is a
fix-forward.
The measurement genuinely belongs in CURRENT-STATE under amendment C2. The
Stage-5 bootstrap entry has carried the hard ordering caveat -- bootstrap fetches
the juju agent stream and juju/juju-db snaps pre-apt, outside the apt proxy, so
it works ONLY while the dc1 edge egress is open -- but it carried no measurement
of whether egress actually IS open. It now does, probed from the dc1 rack with
--noproxy so an apt-cacher hit could not fake it: streams.canonical.com/juju/tools/
returns 200, api.snapcraft.io answers, archive.ubuntu.com 200, ping 1.1.1.1 0%
loss, default route via 10.12.64.1.
Stated with its shelf life: re-probe immediately before bootstrap, because the
D-135/D-107 narrowing is the closing mutation and nothing prevents it landing
first.
Revert: git revert this commit; the edit is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: post-review corrections + dc1 egress MEASURED
...
Three fixes from a final review pass, one of which is a new measurement.
MEASUREMENT -- dc1 edge egress is OPEN, probed rather than assumed. Lens 5 had
correctly listed this as UNMEASURED (it rested on a record). Probed directly
from the dc1 rack with --noproxy so an apt-cacher hit could not fake it:
streams.canonical.com/juju/tools/ -> 200 (the agent stream bootstrap actually
needs), api.snapcraft.io reachable, archive.ubuntu.com -> 200, ping 1.1.1.1 0%
loss, default route via 10.12.64.1. So Phase 5.2 stays a standing caveat rather
than becoming a blocker -- but it is now measured, and the row says to re-probe
immediately before bootstrap because the measurement has a shelf life.
DELIVERABLE CORRECTIONS:
- The readiness doc said "Eleven decisions are yours", which reads as the total.
It is 11 blocking (R1-R11) plus 4 standing (R12-R15) = 15, now stated as such,
with the GA-R5 one-at-a-time rule restated at the point of use.
- Phase 3's intro still said the runbook fails "in at least five distinct ways"
from before lens 5 added ten more rows. Corrected to 21 listed corrections and
nine independent failure paths, with the rows named.
- R9 and R11 are NOT independent of R2 and were presented as if they were. Both
now carry an explicit "ANSWER R2 FIRST" dependency, matching R8. R2 decides
what the VIP literals ARE; R9 only asks where they live, and R11's VIPs are
single- or dual-family depending on it. Under dual-stack, the L3-9 overlay
collision must be solved BEFORE either authority is populated.
Also recorded for future sessions: there is no Bash(timeout *) allow rule, which
is why lens 1's `timeout 240 bash scripts/preflight.sh` was denied while the bare
command is allowed -- a failed-to-MATCH, same class as the 2026-07-26 quoted-sudo
finding, not a classifier override.
Revert: git revert this commit; all changes are to audit artifacts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: session bookend (GA-R4) + ledger rotation
...
Close bookend for the autonomous Stage-5 grounding audit. Bounded ledger
summary (9 bullets, cap 15); full body archived in the same commit as
docs/archive/changelogs/changelog-20260727-stage5-grounding-audit.md, the ONE
changelog for this session per GA-R2/D1, carrying a per-item revert.
Ledger rotation: the bookend pushed the live ledger to 305 lines against the
300-line cap, so the oldest remaining live summary (2026-07-24, caveman
guardrails + DC1 proxy-build start) moved VERBATIM to
docs/archive/session-ledger-rotated-20260727.md as a clearly-labelled second
rotation pass. Live ledger now 289 lines. Sessions from 2026-07-25 remain live.
Reconciled against repo ground truth at close: ledger-scan reports 21 open SEC
rows and next-free D 138 / DOCFIX 205 / BUNDLEFIX 053 -- all UNCHANGED, which
is correct: this session assigned no number because it remediated nothing. The
machine-derived block needed no re-seed.
The changelog records, prominently, that a SESSION-SCOPED PERMISSION BLOCK is
live in the gitignored .claude/settings.local.json and must be removed at
close; the scope doc's section 6 is its only durable record.
Revert: git revert this commit; then restore the 2026-07-24 summary from the
rotation archive if the ledger rotation is also being undone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: lens 5 + the three deliverables
...
VERDICT: Stage 5 would NOT run error-free today, and would fail early. The
substrate underneath is excellent -- all three OpenTofu roots ZERO DIFF, 18
nodes Ready exact to D-121 Option C, MACs and power addresses matching
lib-hosts, D-134 statics perfect, 17 fabrics, zero orphaned interfaces, both
artifact paths serving, gauntlet ALL GREEN (81). What is not ready is the
layer between the substrate and the deploy.
LENS 5 (relaunched after an API error) returned the ordered precondition list
and the largest runbook defect found: Step 4 says "follow phase-01 verbatim",
and phase-01 ACTIVELY REFUSES dc1 -- its VIP guard greps bundle.yaml for
eleven 10.12.4.x VIPs, dc1's live in an overlay on 10.12.64/68/72, so it
takes the "ABORT: VIP guard failed" branch. After the ruled VIP extraction it
aborts for dc0 too. It also carries hardcoded VR0 system_ids, a jumphost-local
libvirt loop over disks that do not exist, and a 4-machine plan gate against a
9-machine bundle.
Three Stage-5 gate commands cannot execute at all, measured against the juju
actually installed (3.6.27): juju run used for a shell command when it is the
action runner; download-backup given a backup-id when it takes a controller
path; and the geneve gate grepping ovn-central for a config key the same
runbook says does not exist. Four of the five VERIFY-LIVE gates the record
says Stage 5 owes have NO step in the runbook -- including the keystone
policyd-override check, which is RULED.
Two convergent confirmations raise confidence in the whole set: the Ceph OSD
blocker was found independently by two lenses using different methods, and the
stale-clone blocker independently by this session and lens 5.
DELIVERABLES:
- docs/audit/stage5-readiness-20260727.md -- ordered precondition checklist,
READ FIRST. 5 phases, each row with status/evidence/what-breaks.
- docs/audit/stage5-committee-raw-20260727.md -- all 7 lenses verbatim.
- docs/audit/queued-rulings-20260727.md -- 11 Stage-5-blocking + 4 standing
questions, GA-R5 shape, one exchange each, blank utterance lines. NONE
adopted; a batch answer rules NOTHING.
MECHANICAL FIX TAKEN (exactly one, deliberately): the G3 gate row read OPEN
with a standing FREEZE while its own cited evidence file records G3 CLOSED and
the freeze lifted. Left standing, that clause would have blocked the very
DOCFIX batch this audit queues. No ruling was required -- two surfaces already
declared it closed.
The 21-item DOCFIX remediation batch is LOGGED NOT EXECUTED. Nearly every
runbook fix interlocks with an unanswered ruling, so landing them now would
encode assumptions about questions the operator has not answered.
Next-free numbers unchanged (D 138 / DOCFIX 205 / BUNDLEFIX 053) -- no number
was assigned, correctly, since nothing was remediated.
Revert: git revert this commit; the deliverables are new files and the
CURRENT-STATE edits are additive plus the one G3 correction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: lens 4 (gate integrity) + privileged creds re-run
...
LENS 4 broke seven gates with CONSTRUCTED, MEASURED fixtures. Independently
re-verified by this session before landing:
- repo-lint reports "PASS (0 fail, 0 warn)" over ZERO files when given a
one-character typo of either its documented flag or the repo path. This is
the gate whose 0-fail every GA-R6 stage-close commit cites; the verdict is
not evidence the repo was read. Reproduced here exactly.
- provider-bundle-check PASSES on decorative HA: `grep -rn cluster_count
scripts/ tests/` returns NOTHING, while the overlay carries 20 such lines.
A 3->1 rewrite of every cluster_count yields a byte-identical PASS.
- preflight's aggregator ignores every sub-gate exit code that is not 1 or 2,
so a gate that crashes (127) leaves "PREFLIGHT: PASS -- clear to deploy".
- P3 verified ZERO of 33 charm-channel pins TODAY because juju is not on the
host's PATH, and a missing BINARY is folded into the same WARN as an
unreachable charmhub. This CORRECTS this session's own earlier capture,
which waved the 33-warn line through as "expected" -- the count was
expected, the cause was not.
- G17's named check cannot fail: curl -sI exits 0 on 404/500 and the dc0 URL
is an autoindex root that answers 200 with nothing behind it.
Lens 4 also returned an 11-item COULD-NOT-BREAK list; notably the 2026-07-27
dc-mirror last-sync fix HOLDS under all five status arms, and creds-matrix's
tier-2 probe states could not be made to turn "could not look" into green.
CREDS: privileged sweep re-run (operator-authorized). Exit 1, still exactly
the same 7 findings -- no drift in a day. All three root-owned locations read
via sudo -n escalation, metadata only; ZERO E3 findings. Recorded as a
RE-CONFIRMATION of the 2026-07-26 privileged run, not a newly-closed item.
Six of seven lenses are now in docs/audit/stage5-committee-raw-20260727.md.
Lens 5 (ordered preconditions) is still running after an API-error relaunch.
Revert: git revert this commit; all files are audit artifacts plus additive
CURRENT-STATE paragraphs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: live measurement sweep + first five lens returns
...
MEASUREMENTS (capture docs/audit/stage5-live-measurement-20260727.txt):
- ALL THREE OpenTofu roots plan ZERO DIFF -- outer on vcloud, and for the first
time both inner roots on voffice1 in the same session. Validity of the inner
pair was established FIRST by proving those roots are byte-identical to main
despite the voffice1 clone being 105 commits behind; a non-empty diff would
have made them UNMEASURED, not green.
- Gauntlet ALL GREEN (81). repo-lint 0 fail / 1 warn (the L1 legacy carve-out).
- preflight exits 1 for EXACTLY the known reasons -- nothing new has joined.
- dc0 full mirror and dc1 caching proxy both genuinely serving; dc0 sync
attested OK 2026-07-27T08:43:46Z.
CURRENT-STATE corrections under GA-R1 amendment C2 (measurement wins, corrected
in the capture's own commit -- this is also what repo-lint L10 requires):
- Juju on the headend measured 3.6.27, not the recorded 3.6.25. In-channel
patch refresh, which D-071 permits, so policy-compliant drift, not an
incident -- but it is the client that will bootstrap the controller.
- NEW pin row: the openstack client is ABSENT on BOTH hosts while ten
Stage-5/6/7 scripts invoke it. No prior surface carried this.
- Section 5 convergence history extended with the three-root re-confirmation.
OWNED MEASUREMENT ERROR, recorded rather than quietly corrected: the artifact
checkers were first run LOCALLY on vcloud and printed a comprehensive FAIL with
every probe MISS -- output indistinguishable from "never installed". Both
scripts are designed to be PIPED TO THE RACK. Corrected by direct functional
probe; lens 2 independently ran the canonical piped form and got PASS on all
three. The scripts are not at fault.
FIVE LENS RETURNS landed verbatim in docs/audit/stage5-committee-raw-20260727.md
(lenses 1, 2, 3, 6, 7). Findings are NOT yet verified -- verification is the next
step and a refuted finding stays in that file with its refutation, never deleted.
Lens 5 died on an API error and was relaunched; lens 4 still running.
Revert: git revert this commit. The two CURRENT-STATE edits are additive; the
audit files are new.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: land the scope charter before spawning any agent
...
Operator directed a full reconciliation + grounding committee from a fresh
session before Stage 5 opens, running autonomously while they are away, with
rulings QUEUED not adopted (GA-R5).
This commit lands the charter FIRST so a session that dies mid-run still
leaves the plan behind. It fixes up front: the three deliverables (ordered
readiness checklist / committee record / GA-R5-shaped queued rulings), the
seven read-only lenses, the GA-F06 bound on the record-integrity lens
(record-churn is this project's measured failure mode -- a defect is only a
finding if it changes a Stage-5 decision or contradicts a RULED decision),
the mechanical-vs-ruling discriminator (does the fix require a choice?), and
the subagent contract (read-only; a denied call is UNMEASURED with its exact
command, never folded into a negative finding).
Section 6 records the session permission posture. The operator offered a
broad exception; a SCOPED READ-ONLY expansion was taken instead -- the task
is read-only by construction and the ask rules ARE the enforcement of hard
rule 3. Design: allow enumerates READ verbs, ask enumerates MUTATING verbs,
anything matching neither falls through and FAILS SAFE as UNMEASURED. The
settings file itself is gitignored, so section 6 is its only durable record;
the block is removed at session close.
Revert: git revert this commit (the file is new and nothing references it yet).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
close-out durability sweep: 4 transcript-only items landed; dc1-is-not-a-mirror warning
...
Operator-requested sweep for content that existed only in the session transcript, before
clearing. Precedent: docs/audit/queued-findings-20260726.txt. Capture:
docs/audit/queued-findings-20260727.txt.
THE CONSEQUENTIAL ONE. scripts/dc-mirror.sh's dc1 site row carried NO warning that dc1 is no
longer a mirror site, so `dc-mirror.sh install dc1` would have SILENTLY REBUILT everything the
same-day teardown removed -- units, nginx vhost, sync helper, and an ENABLED daily debmirror
timer that would start a fresh ~950G pull on a rack whose ruled artifact path is the proxy. That
is precisely the command someone would reach for after treating a failing `check dc1` as a
regression. The row now states dc1 is proxy-only per the D-135 amendment, that `check dc1` FAILS
BY DESIGN, and that `install dc1` is a deliberate strategy change and never a repair. The row is
RETAINED so its measured values survive a legitimate rebuild.
QUEUED, not built (hard rule 1): a RUNTIME guard in do_install for a site whose ruled path is
not the mirror. A comment is strictly weaker than a guard, and prose-only prevention has already
failed twice on this project (DoD item 8 missed dc1; D-137's wiring sat as prose and missed BOTH
DC standups). Recording that the interim is an interim.
A WRONG CLAIM CORRECTED. docs/audit/stage4-mirror-gate-20260727.txt said `systemctl reset-failed`
would re-arm the paused dc1 debmirror. It cannot: reset-failed on the SERVICE does not start an
INACTIVE TIMER. The real vector was a REBOOT -- `enabled` starts the timer at boot and
`Persistent=yes` fires it IMMEDIATELY because its window had passed. Correction APPENDED rather
than edited in place: the file is dated evidence, so a wrong analytical claim is superseded
openly, not quietly rewritten.
THE GENERALISABLE LESSON, previously unrecorded, is now platform-traps section 5:
5a "stopped is not dormant across a reboot" -- enabled + inactive + EMPTY NextElapse looks
harmless and is not; plus reset-failed does not re-arm, and Result=signal/status=15 on a
oneshot means it was SIGTERM'd and never wrote a terminal state (so a stale RUNNING is a
corpse, not progress).
5b a `oneshot` with RemainAfterExit=yes does NOT undo its work on stop, so stopping it proves
NOTHING about dependents -- which is exactly why the dc1 teardown test deleted the live
address and route instead of stopping the mirror net unit.
Three rows added to the verbatim-error index. Note dc0-mirror-sync.timer is still enabled with
Persistent=yes, so 5a describes dc0 today -- benign while its syncs succeed, same shape.
HARD RULE 2. dc-cache-proxy.sh stated that the two net units coexist safely on one host as
FACT. It is REASONED (idempotent replace verbs, identical measured values), never MEASURED --
the proxy has only run on dc1, where the mirror unit was then removed, so nobody has run both.
Marked accordingly, with an instruction to verify rather than trust the sentence.
VERIFICATION: dangling-reference sweep over docs/, runbooks/, scripts/ and the skill -- every
path this session introduced or cited RESOLVES; all pre-existing dangles are legitimate
(deleted-as-history, not-yet-built, or the deliberately-absent octavia PKI overlay preflight P4
fails on). ledger-scan reconciled against the machine-derived block. Gauntlet ALL GREEN (81),
repo-lint 0-fail, dc-mirror 24/24, dc-cache-proxy 20/20.
Revert: text + comments only; the capture correction is an append.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
session-ledger: rotate to restore the GA-R4 cap (317 -> 260); re-seed the scan header
...
CORRECTION to the previous commit's message, which said "ledger 317 lines, under the 300-line
cap". That is wrong and self-contradictory -- 317 is OVER the 300-line cap, so GA-R4 F1
rotation was OWED, not deferred. Recording the correction rather than quietly fixing it,
because mis-stating an owed obligation as satisfied is the failure mode the cap exists to
prevent (the 2026-07-25 close had to record exactly this breach as OWED).
ROTATED oldest-first per GA-R4 rule 3, matching the 2026-07-26 precedent: the three oldest live
summaries (all 2026-07-23 -- G12 dc1 edge->commission+merge, Stage 4 OPEN+carve+mirrors, queue
pass + D-068 rulings) moved VERBATIM to docs/archive/session-ledger-rotated-20260727.md. Each
still points at its own archived full body; only the summaries moved. Live ledger 317 -> 260,
under the cap. Sessions from 2026-07-24 onward remain live.
Re-seeded the machine-derived block header, which still credited a 2026-07-26 scan while
carrying 2026-07-27 values. Verified against `bash scripts/ledger-scan.sh` this session:
21 open SEC rows, D next-free 138, DOCFIX 205, BUNDLEFIX 053 -- all match.
repo-lint 0-fail. The session is now durably bookended: Stage 4 CLOSED and merged (6f5701d),
ledger under cap, scan values reconciled, every finding on a surface rather than in a commit
message. Safe to clear.
Revert: text-only; the rotation is a verbatim move.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
session bookend: correct the stale merge status + land a finding that lived only in a commit
...
Two defects in the durable record, both because the bookend was written BEFORE the merge.
Fixing them before the session is cleared, since the ledger is what the next session reads.
1. The bookend said "Merge to `main` is the ONLY remaining step and is operator-gated" -- stale
the moment the merge landed. Now records the actual end state: merge commit 6f5701d (2
parents, not squashed, 77 commits), branch retired local + remote, post-merge gauntlet ALL
GREEN (81) + repo-lint 0-fail ON main, close recorded by 1023596.
2. A finding was logged only in commit c29059b's MESSAGE, which is findable but is not a surface
any session reads at open: `tests/creds-matrix` T24's finding-class baseline covers TIER 1
ONLY, so tier-2/3 classes (E1/E3/E4/V1/V2) have no baselined red state and a future false
green there would not turn the gauntlet red. That is the same class as the two false greens
this session fixed, so burying it in a commit message was the wrong home. Now in the
bookend, together with the still-open `creds-mint.sh` advice -- worth ruling BEFORE Stage 5,
which is the largest credential-minting event in the deployment.
Bookend stays within the GA-R4 15-line cap (9 body lines); ledger 317 lines, under the 300-line
cap's rotation trigger for the next close to handle. repo-lint 0-fail.
Revert: text-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CURRENT-STATE: Stage 4 CLOSED 2026-07-27 -- merged 6f5701d, branch retired
...
GA-R1 C1: the merge is a status change, so the status authority records it in the same change
set. Stage 4 / Phase 3 flips OPEN -> CLOSED with its evidence: operator-gated merge to `main`
as merge commit 6f5701d (2 parents, NOT squashed; 77 commits), branch
`dc-dc-stage4-phase3-maas-deploy` retired local + remote after confirming containment with
`git branch --merged main`. Post-merge verification ON main: gauntlet ALL GREEN (81),
repo-lint 0-fail.
Recorded as travelling forward BY DESIGN rather than silently dropped: gate G17 (node-side
artifact reachability, triggers at Stage 5 first boot), 21 open SEC rows, and 7 residual
credential-register findings including the dc0 edge-API re-mint that was deliberately excluded
from the consolidation batch.
The next stage branches off post-merge `main`.
Revert: text-only. The merge itself is 6f5701d.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 4 close-out set: GA-R2 consolidation, skill sweep, snapshot, GA-R7 review
...
Every substantive Stage 4 item is closed or split to gate row G17. This is the GA-R6 close-out
set; only the operator-gated merge to `main` remains.
GA-R2 CONSOLIDATION: the 7 stage changelogs archived to docs/archive/changelogs/ (top-level
docs/ 25 -> 18, 113 archived total) and the stage record written at
docs/archive/stage-records/vr1-stage4-record.md. It records what the stage actually delivered
(READY handoff, not deployed nodes), the three things a future session should read first, and
the false-green pattern this stage kept surfacing. 12 stale `docs/changelog-*` paths in live
surfaces rewritten to the archive -- including several already dangling from the G12 close, so
every changelog reference in CURRENT-STATE / session-ledger / security-ledger now resolves.
SKILL SWEEP: three new INVARIANTS folded in --
- per-DC ARTIFACT DELIVERY is a per-DC STRATEGY (dc0 full mirror / dc1 caching proxy, D-135
amendments); checking dc1 as a mirror fails BY DESIGN. Corollary, learned the hard way: a
per-DC utility service OWNS ITS OWN network prerequisites -- never let one strategy's
teardown be able to break the other's.
- Stage 4 hands off READY nodes, NOT deployed ones (DOCFIX-200). Any surface saying "nodes
Deployed" as a Stage-4 exit is a defect; four were found and fixed. Consequence: nothing
node-side is probeable in Stage 4, which is why G17 exists.
- "A CHECKER THAT CANNOT FAIL IS NOT A GATE", with the working rules that came out of two
measured false greens: assert on CONTENT not existence; an unrecognised state REFUSES;
"could not look" is never "nothing there"; ENUMERATE what exists rather than only auditing
what is declared, because a register cannot report the absence of a row it does not have;
and when a test asserts a literal finding string, REPLACE the assertion, never delete it.
Plus two routing rows (per-DC artifact checks; D-137 as the credential-lifecycle authority).
SNAPSHOT REGENERATED: .claude/skills/openstack-cloud-ops-consolidated-20260727.md (1589 lines,
ASCII + LF byte-verified, all 7 sections). The superseded 20260725 snapshot is REMOVED rather
than kept alongside -- a stale snapshot being uploaded is the exact failure
docs/audit/skill-divergence-20260725.md records (a failed clone + a mis-shaped handoff), and it
is a derived artifact regenerable from any commit.
GA-R7 MEMORY REVIEW: NO new memory written -- everything durable from this stage graduated to
the skill and the repo, which is the correct GA-R7 outcome, not an omission. Both existing
entries verified against the repo (the finding-class baseline they cite does exist) and extended
with the two reasoning traps this stage produced: the register cannot see a credential it has no
ROW for, so enumerate what exists; and diagnose a denied call by checking whether an existing
rule failed to MATCH before blaming the classifier (the measured 2026-07-26 root cause). Also
marked the ask-vs-allow precedence claim UNVERIFIED so no future session relies on it.
Noted, not actioned (hard rule 1): T24's finding-class baseline covers TIER 1 only, so tier-2/3
classes have no baselined red state and a future false green there would not be caught.
Gauntlet ALL GREEN (81), repo-lint 0-fail.
Revert: git revert this commit; the changelog moves are renames and the snapshot is derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
set-interface-v4: reload the pf filter on --commit (RULED); appendix-A rewritten
...
RULING (GA-R5) 2026-07-27. Question as presented: (a) add `configctl filter reload` to the
script's --commit path; (b) leave the script and rely on DC standup DoD item 8; (c) put it behind
an opt-in flag. Operator answer, exact utterance: "a". OPS under GA-R3 -- a script fix completing
an already-documented operation, governed by D-113; no new D-number.
DEFECT (appendix-A, measured on the fresh dc1 edge 2026-07-23): --commit ran `configctl interface
reconfigure`, which applies the address to the kernel but does NOT regenerate the pf ruleset, so
OPNsense never creates its automatic outbound-NAT rules for the (re-)addressed LAN. Forwarded LAN
packets left the WAN un-NAT'd and replies never returned -- the edge passed NO LAN traffic while
looking perfectly configured. Self-masking: any later plugin install or config apply reloads the
filter as a side effect, so dc0's 07-20 qga work hid it there until dc1 exposed it.
Option (b) rejected on evidence, not taste: DoD item 8 already existed and did NOT fire at dc1,
and prose-only controls have a measured failure record here (D-137's own enforcement sat as prose
at phase-3:498 and missed BOTH DC standups).
PLACEMENT IS THE SUBSTANCE OF THE FIX, and my first attempt got it wrong -- caught before
shipping. Putting the reload on the next line of the `interface reconfigure` heredoc would have
been a fix that fixes nothing exactly when it matters: re-addressing the interface you arrived on
drops the session DURING the reconfigure, so a following command in that stream never executes --
and that drop case IS this incident's own scenario (dc1's LAN moved 192.168.1.1 -> 10.12.64.1
over the LAN itself). As shipped the reload runs over a FRESH connection to PROBE_HOST.
Also added a `pfctl -s nat` read-back with the `proto carp` boilerplate filtered out, since
boilerplate-only is precisely appendix-A's broken-state signature. REPORTED, not gated: an edge's
first addressing may legitimately have no gateway and so no NAT to generate, and hard-failing
would block a correct run. The hard gate stays address-on-the-kernel.
Harness 59/59 (was 53), cases 15-15f. Case 15f caught a real inconsistency -- appendix-A still
said "not yet ruled". Case 15e caught a bug in my own first draft, which scanned to EOF and so
failed the script for having the very hard gate the case exists to preserve.
appendix-A rewritten: the FIX stays for an edge already in this state, plus a PREVENTED-IN-THE-TOOL
block recording the placement detail and why prose prevention was rejected.
Gauntlet ALL GREEN (81), repo-lint 0-fail. No live edge touched -- both DC edges are already
addressed, so this affects only the NEXT edge addressing.
With this, every substantive Stage 4 in-stage item is closed or split to gate row G17; only the
GA-R6 close-out set remains.
Revert: git revert this commit's script + harness + appendix-A changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
SEC-024 retention RULED: keep both state-surgery snapshots (GA-R5)
...
Question as presented 2026-07-27: delete both `pre-*` state-surgery snapshots (cleanest, closes
SEC-024 fully), keep both and let preflight P5 keep watching them, or keep pre-G6 and delete
pre-G16 -- noting both gates are CLOSED, both carry the Office1 MAAS API key in plaintext per
DOCFIX-175, and deletion is irreversible with no other copy of that pre-surgery state.
Operator answer, exact utterance: "Keep both".
opentofu/terraform.tfstate.pre-G6-20260719 and .pre-G16-20260721 are RETAINED at 0600, neither
git-tracked, as the only record of state prior to the two direct tfstate edits (G6 autostart
injection, G16 channels reconcile).
SEC-024 stays OPEN as a standing WATCH rather than an open remediation: the mode defect is
fixed, but the umask CAUSE was out of ruled scope so a future apply may rewrite 0664, and P5 is
the detection. Recorded in the SEC-024 disposition cell and in CURRENT-STATE. Open SEC count
unchanged at 21.
repo-lint 0-fail.
Revert: text-only; the ruling record reverts with git revert. No file was deleted (that is the
point of the ruling).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 4 cleanup: carve residue 108 fabrics -> 17; dc1 leftover nginx purged
...
Both items were queued to the stage close at the 2026-07-23 carve.
CARVE RESIDUE. Deleted subnet id=8 192.168.1.0/24 (the superseded OPNsense FACTORY LAN -- both
edges were long since re-addressed to 10.12.4.1 / 10.12.64.1), then fabric-5, then the 90
auto-created empties (ids 6..95). 108 fabrics -> 17.
The audit caught an ORDERING DEPENDENCY the original flag did not state: 192.168.1.0/24 was the
ONLY occupant of fabric-5, so deleting the fabric first would have cascaded the subnet away
instead of removing it cleanly. Subnet first, then the fabric.
METHOD, chosen because of this repo's own history: a MAAS fabric delete cascades to its VLANs,
and the 2026-07-21 pod-delete incident cost 9 machine records precisely because the association
check ran AFTER the delete. So emptiness was PROVEN per fabric -- zero subnets, zero ipranges,
zero node interfaces across all its VLANs -- against a fresh occupancy snapshot (fabrics +
subnets + ipranges + every machine's interface_set) taken immediately before the batch, and the
cascade check was re-run afterwards too.
KEPT, all verified occupied: office1 base + GUA (fabric-0), office1 compose with its 2 LXD
guests (fabric-1), both transits (fabric-2 / fabric-141), both metal-admin/boot fabrics with 9
nodes each (fabric-4 / fabric-142), libvirt default (fabric-3), and the 10 named plane fabrics.
POST-STATE: 18 Ready + 2 Deployed office1 guests, all 18 still power_type=virsh, 7 interfaces
each (6 flat planes per D-133 + br-ex), ZERO orphaned interfaces, placement tags 9+9 intact,
both DCs' artifact paths re-verified PASS.
NGINX ON dc1. Purged nginx + nginx-common (dry-run first: nothing else depended on them, no
nginx-core was installed), plus autoremove of liblockfile-simple-perl and liblog-agent-perl --
debmirror's orphaned perl deps. /etc/nginx gone, nothing on :80, proxy still PASS on :3142. dc0
keeps its nginx and was re-verified unaffected (it serves the mirror).
Capture docs/audit/stage4-carve-residue-cleanup-20260727.txt. repo-lint 0-fail.
Revert: fabrics/subnets are MAAS records, not config -- no undo, and nothing needs them (90
were auto-created empties; 192.168.1.0/24 was a superseded factory address). nginx reverts with
apt-get install nginx, which dc1 has no use for while it is the proxy DC.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
