| 2026-08-01 |

GA-R5: D-139 gains an OOB plane, dual-stack; 10.12.60.0/22 ruled back into force
...
Operator utterances: "make sure to include oob in the dual stack recordings" and
"Dual-stack; rule 10.12.60.0/22 back into force for OOB".
Measured gap: D-139's CARVE table carried 10,11,20,21,30,40,50,80 and no 0xf0 (OOB)
or 0xe0 (VPN) -- design-decisions.md:3066 declared all three out of scope of the
six-plane tool, D-139 brought :80 back and left the other two. The step-1 push would
have written a carve incomplete against the org standard ruling B cites. Held, not run.
The ruling knowingly diverges from both precedents: VR0 and Willamette carry OOB
IPv6-only and a sweep of every apex prefix returns ZERO v4 rows with role oob at any
site. Dual-stack is a deliberate Roosevelt correction (BMC/IPMI is v4 in practice).
Supersedes the "OOB n/a / bare-metal-only concern" row at design-decisions.md:93.
D-058 remains SUPERSEDED in full; only that one row's VALUE is reinstated, under
D-139's authority. Block verified free before ruling. Per-DC v4 split is UNRULED and
deliberately not inferred; the v6 half is complete and the v6-only push is unblocked.
VPN (:e0) is the same omission, left open rather than folded in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R5 ruling: the D-139 GUA carve is sequenced BEFORE the Stage-5 deploy
...
Operator answer, exact utterance: "Apex push AND the MAAS/node GUA carve, then deploy".
D-139 execution steps 1-3 run before the bundle deploy, so the cloud is deployed once
on its final GUA addresses and nothing is re-addressed underneath it. The 2026-07-30
standing directive is satisfied in substance rather than literally.
Recorded, not inferred: steps 4-6 were NOT put, and step 6 has a deploy coupling this
ruling does not settle -- the per-DC VIP overlays carry v6 VIPs in the ULA range, so
deploying after steps 1-3 alone leaves GUA nodes with ULA VIPs. Own exchange owed
before Step 4. Neither ruling A nor B is touched; no new D-number (GA-R3: OPS).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
L10 correction: cite the preflight capture in CURRENT-STATE (cde7441 pushed lint-red)
...
cde7441 added docs/audit/stage5-preflight-dc0-20260801.txt without touching
CURRENT-STATE.md in the same commit, which L10 (GA-R1 rule 8 / C1) forbids. The lint
was piped through `tail` so its exit code never gated the push -- the same masking
recorded at the 2026-07-31 close. Corrected forward, not by rewriting pushed history.
Records the post-fix re-run: P8 WARNs off-substrate, [FAIL] count 12 -> 11, all eleven
the ruled-accepted P5 set; preflight still exits 1, which both P5 rulings anticipate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Capture: Stage-5 preflight dc0 2026-08-01, post-P8-fix (11 FAIL = the ruled-accepted P5 set)
...
Named-check capture for the Stage-5 entry gate, run on voffice1 at HEAD db15666 with
the region profile. P8 now WARNs correctly off-substrate; the 12th FAIL is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

P8 host guard: && -> || (live false FAIL on voffice1); dc0 MAAS path restored
...
P8's guard warned only when BOTH terraform.tfstate and opentofu/.terraform were
absent. voffice1 has no state file but does have .terraform -- D-128 has it run the
INNER tofu roots, and `tofu init` leaves a provider cache -- so neither warn branch
fired, P8 ran a real `tofu plan`, and it died on a gitignored tfvar -> exit 1 -> hard
FAIL on the very host the Stage-5 runbook designates for preflight. terraform.tfstate
is the OWNERSHIP marker; .terraform is only a cache. Both must be present to evaluate.
No test caught it because T38 removes the whole opentofu/ directory, so both markers
vanish together; the fixture itself sat in the defective state and passed only because
the && was wrong. Fixture now creates both markers; new T39 covers voffice1's real
shape (cache present, state absent -> WARN). Mutation: reverting || to && turns T39
and only T39 red; script restored sha256-identical.
Also: the dc0 MAAS path was down -- the vr1-dc0-region profile points at an SSH
forward (127.0.0.1:5241) that died with the 05:48 rack reboot. Restored and proven
(plan vr1-dc0 PASS exit 0, was REFUSE exit 3). Not an SEC-010 puncture: an SSH forward
originates on the rack. voffice1 pulled to HEAD. Preflight re-run: 12 [FAIL] lines,
11 = the ruled-accepted P5 set compared by identity, 1 = P8.
tests/preflight 38 -> 39/39; gauntlet ALL GREEN (96); repo-lint 0 fail / 1 warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

CORRECTION: no autostart setting was changed, and D-127 already rules it
...
Operator asked directly whether I had changed autostart. I had NOT -- item 21
logged it under hard rule 1 and started the two VMs by hand. The question, asked
to scope a power-settings-matrix session, prompted the check that shows I FRAMED
THE FINDING WRONGLY.
D-127 ("VR1 host-level VM autostart policy") already rules this, autostart is
TOFU-MANAGED, and every measured value is an explicit per-instance decision with
its reason in-line: main.tf:413/:540 containment VMs false ("MANUAL (gated
bring-up), never on host boot"); main.tf:102/:178 office1-opnsense and voffice1
true ("foundational"); DC edges true ("comes up with its containment VM"); node
VMs false ("MAAS-power-controlled"). The module variable's own description forbids
relying on its default. Calling it "a standing exposure" implied nobody had
decided. A session told to "fix autostart" would fight a ruling -- and now that P8
exists, a virsh autostart change would also surface as substrate DRIFT.
WHAT SURVIVES, narrower and real: vr1-dc0-maas-01 and vr1-dc0-juju-01 are created
by the vr1_dc0_node for_each, so they inherit the NODE justification
"MAAS-power-controlled". TRUE for the juju controller (subtle-grouse, a dc0-region
machine, power_type=virsh). CIRCULAR for the MAAS region VM: hot-kid's power is
owned by the OFFICE1 region -- the region D-132 q1 is migrating away from. Retire
Office1 and nothing owns the dc0 region VM's power while it does not autostart.
That is an ordering exposure in the migration, not an autostart bug. LOGGED, NOT
FIXED.
WHY THE OPERATOR'S MATRIX IS THE RIGHT INSTRUMENT: autostart is a boolean that
cannot express WHO OWNS a VM's power or WHAT HAPPENS WHEN THAT OWNER GOES AWAY --
the questions that surface the circularity, and they recur at every DC standup.
Seed inventory measured: 4 outer domains, 12 dc0 inner, 11 dc1 inner; exactly four
carry autostart=enable (voffice1, office1-opnsense, both DC edges).
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Both DC VMs 416 -> 480 GiB; 128 GiB swap; new preflight gate P8 (substrate drift)
...
POST-BOOKEND: the GA-R4 bookend (4b8ba3c) was committed BEFORE this work, so the
ledger close summary and the 08-01 sweep do not cover it. The ledger line is
AMENDED in this commit rather than left stale.
Operator: "Option 2 look sgood me. I approve the sequencing, process as
autonomously as possible" -- +64 GiB to EACH DC with swap first, over my
recommendation of +48. The deciding arithmetic, put up before the ruling: at +64
each, 994 GiB allocated against a 1007.4 GiB host = 13.4 GiB residual while the
host's OWN measured footprint is 18.0 GiB, with 1 GiB swap free. Swap is what
makes +64 safe, hence the sequencing.
PROBLEM FIXED, MEASURED: dc0's rack ran 402 GiB of inner guests in a 409 GiB host
(98.3%), leaving 7 GiB for the rack OS, mirror, snap proxy and page cache. Each
rack now has ~71 GiB.
SWAP: operator-run -- sudo -n is NOT available on vcloud so that half could not be
automated. /swap2.img 128 GiB, 600 root:root, live + fstab. 135 GiB total.
CHANGED THROUGH TOFU, the only correct place: a virsh setmaxmem would have been
reverted by the next apply. The derivation now lives IN the variable comment.
PLAN ASSERTED ON CONTENT (2026-07-20: an in-place apply silently regenerated 9
node MACs): per DC 1 in-place, 0 create/destroy/replace, 0 MAC changes, memory the
only changed attribute. A THIRD resource appeared and was resolved BEFORE
applying: module.office1_opnsense ~ id = 2 -> 11 sits under "has changed outside
of OpenTofu" (drift OBSERVED), not "will be updated in-place" (action PLANNED) --
libvirt's domain id is a runtime value. It was not touched.
dc1 FIRST AS CANARY, and it answered the open question: the in-place update
BOUNCES the guest (domain id 9 -> 12, uptime 0 min). Guest sees 472.2 GiB.
>>> FINDING: vr1-dc0-maas-01 (MAAS region) and vr1-dc0-juju-01 (juju controller)
have autostart=disable. <<< Only the edge auto-recovers. Checking first is the
only reason the dc0 bounce did not come back looking catastrophic. Both started by
hand and verified. LOGGED NOT FIXED: a host reboot leaves dc0 with no region and
no controller -- a standing exposure independent of this change.
RESULT: host used 809 -> 63 GiB, available 197 -> 944 GiB; the restarts also
released the stranded RSS (vvr1-dc1 354 -> 12 GiB), discharging 08-01 sweep F1 as
a side effect. Recovery verified end to end: MAAS region API 000 -> 502 -> 200 (a
real boot progression), snap proxy listening, mirror 200, juju controller model
"Last connection: just now".
GATE P8 -- substrate drift. Built because answering "does tofu need updates
logged" produced a measurement: preflight.sh and pre-flight-checks.sh contained
ZERO tofu plan checks and the opnsense drift had sat unseen. DESIGN POINT:
-detailed-exitcode returns 2 for BOTH a pending change and a harmless observation,
so P8 asserts on CONTENT -- pending ACTION FAILS, observed-only drift WARNS,
unrecognised shape REFUSES. A gate permanently red on benign drift gets ignored.
It WARNS rather than fails when it cannot look, since the outer root lives on
vcloud and "not the substrate host" is a legitimate state.
Harness 33 -> 38 (T34-T38). Baseline proven by stashing: my first draft's 7
failures were MINE, not pre-existing. Three mutations each killed a NAMED test,
script restored sha256-identical. T38 exercises the absence by removing the
fixture STATE, never the fake tofu from fakebin -- the real tofu is on the system
PATH and a "closed" PATH would fall through to the LIVE substrate. PROVEN LIVE:
tofu exit=0, 0 pending, 0 drift -> [ok] zero diff.
SCOPE: tofu owns the SUBSTRATE only. MAAS carves, rack services and the v6 node
carve are script-driven by design (Model B / D-123); P8 does not check them.
gauntlet ALL GREEN (96); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

SESSION CLOSE 2026-08-01 (GA-R4 bookend): snap proxy LIVE, D-139, IPv6 proven
...
DURABILITY TRIAD: vcloud 0 uncommitted / 0 unpushed at 844b2e4. voffice1 was 36
commits BEHIND and has been brought current. The dc0 rack's ~/repo-stage (D-138
client input, NO git) was DIGEST-COMPARED not assumed: bundle.yaml and both
non-PKI overlays sha256-MATCH the repo, octavia-pki correctly absent (gitignored).
The provenance gap is real; the drift is not.
GATES: repo-lint 0 fail / 1 warn (legacy carve-out), 649 files; gauntlet ALL GREEN
(96 harnesses); ledger-scan 3 open decisions, 26 open SEC (none opened this
session), next-free D-140 / DOCFIX-207 / BUNDLEFIX-053. RECONCILED: D moved
139->140 matching the one D-number assigned; SEC unchanged matching zero opened.
SWEEP: docs/audit/queued-findings-20260801-stage5-ipv6-d139.txt -- SEVEN FIRST
SURFACE items, each grep-proven absent from every repo surface before writing.
F1 is highest-consequence: the capacity headroom the next deploy needs is HELD BY
TWO IDLE VMs (vvr1-dc0 434 GB RSS, vvr1-dc1 371 GB, ~805 of 1007 GB) with dc0's
nodes powered off and dc1 carrying no model -- and this document's own capacity
record was computed against ALLOCATION, not residency. Also: the voffice1 lag; the
host-lag investigation and its NEGATIVE result; the rack digest MATCH; that no
gitignored permission rules changed; a transient _fmtprobe.tf; and the PreToolUse
guard firing on PROSE containing a guarded command's name.
GA-R7 MEMORY REVIEW: CLEAN -- zero entries claiming operator policy, priority or
posture. One update: instrument-currency gains its NINTH instance in a new shape
-- a "closed" PATH that still held the binaries it was meant to hide. Generalised
as: when a negative depends on an ABSENCE, prove it rather than arranging it.
LEDGER ROTATED (GA-R4 rule 3): stood at 296 lines and this close would have
breached 300. The two oldest closed-session summaries moved VERBATIM to
docs/archive/session-ledger-rotated-20260801.md; ledger now 282. No orphaned
session, so rule 7 is a no-op. NO STAGE OPENED OR CLOSED -- Stage 5 remains OPEN
and this is a session bookend, not a GA-R6 stage close.
OWNED: my BUG-1 fix was wrong; I scoped the v6 experiment wrong (ceph couples
storage+replication); I stated an agent's input source wrongly; I scoped a research
agent with no repo path so 355 lines landed in /tmp and needed rescuing; and one
commit went red on ASCII-only em-dashes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Next steps: nodes released, D-139 execution list replaced, LP draft written
...
(1) TEST NODES RELEASED. t7ymp6 / fg6gxm -> Ready / owner=None, released
individually and read back. Nothing stranded. G19 was run live before the window
closed, which was the whole reason for holding them.
(2) D-139 GAINS A CORRECTION NOTE REPLACING ITS EXECUTION LIST. Neither ruling is
touched. The original list is preserved verbatim inside the note rather than
silently overwritten -- same reason the struck RFC 6724 rationale was kept: a
later reader must be able to see what was wrong or they will re-derive it.
SILENT FAILURE (why this could not wait): dc-node-v6-carve.py assigns v6 "on
every plane WHERE IT ALREADY CARRIES IPv4", keys the subnet off the same MAAS
vlan as that v4 link, derives the host part from the v4 octet, and its loop body
is `if not v4: continue`. Run after v4 removal it carves FOUR FEWER PLANES PER
NODE AND EXITS CLEAN -- and the original list's "remove v4 LAST" wording
actively invited that ordering.
UNSATISFIABLE: "remove v4 LAST, after each is proven" -- per-plane conversion is
atomic, so no such state exists.
UNSAFE STEP FUSION: apex CREATE and RETIRE in one step, with 52 dependent
ip-addresses (the v6 VIP legs) inside the retiring ULA /64s.
Replacement is 7 steps: CREATE-only apex push; MAAS GUA carve ALONGSIDE the ULA;
node statics re-carved WHILE v4 IS STILL PRESENT; Octavia SAN reissue +
lib-net.sh; lb-mgmt carve; re-home the 52 VIPs THEN retire; and v4 removal as a
SEPARATE experimental step scoped to storage + replication TOGETHER per "B plus
C" (together because ceph-osd's prefer-ipv6 sets ms_bind_ipv4=False globally).
Step 7's prerequisite -- network-get on a v6-only bound space -- is stated unmet.
(3) LP DRAFT WRITTEN, NOT FILED: docs/audit/lp-draft-20260801-ceph-osd-ipv6-
static.md. The "C" half of the ruling. A COMMENT on the EXISTING LP #2061836, not
a new bug -- the ceph-mon task is already Fix Committed and only ceph-osd remains
New. Every code claim quoted from the downloaded rev-953 artifact. Claude does not
post to Launchpad; the operator files it (lp-draft-20260721 precedent).
Two things written INTO the draft's operator notes rather than left implicit: do
NOT cite LP #1590598 (Fix Released, a different defect -- the inverted-citation
class); and this bug does NOT block us locally, since setting ceph-public-network
/ ceph-cluster-network to the v6 CIDRs bypasses the buggy call. Overstating our
dependence on an upstream fix would be a bad-faith way to raise its priority.
OWNED: first push attempt went red on L1 -- I used em-dashes in the changelog and
this repo is ASCII-only. Caught by lint before commit, fixed, re-run 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

G19 BUILT and PASSED LIVE; finds a latent defect in D-139's execution list
...
scripts/dc-node-v6-verify.sh (270 lines, 24-line header) + tests/dc-node-v6-verify
(55 cases); manifest 95 -> 96. All three subcommands exercised LIVE this session:
plan vr1-dc0 PASS exit 0 (9 machines, v6 plane set IDENTICAL across all 9, 6
planes, prefixes DERIVED); node vr1-dc0 <spec> on storage-01 against live peer
storage-02 PASS exit 0 with 12 assertions -- six address-presence, six
peer-reachability, every plane "up, global, DAD complete, sole global on the NIC"
and "replies, neighbour DELAY"; bridges vr1-dc0 reporting 7 bridges. Harness
55/55, gauntlet ALL GREEN (96), three mutations each turning a NAMED test red.
RUN AGAINST A REAL NODE while the boot window was open specifically so it would
not ship fixture-green -- the snap proxy's harness was green for a week before
anything was proven end to end.
NO v6 PREFIX LITERAL EXISTS IN THE FILE, and that was a constraint not a nicety:
D-139 retires the ULA /48 for GUA, so a baked table would be right today and wrong
on carve day. plan derives from live MAAS and emits a SPEC; node consumes it and
knows no prefixes. A harness case drives the same green path with a ULA fixture
AND a GUA fixture, so the property is EXECUTED, not asserted.
>>> LATENT DEFECT IN D-139's OWN EXECUTION LIST <<<
D-139 names dc-node-v6-carve.py to "re-carve 54 node v6 statics". VERIFIED against
that script's source: it assigns v6 "on every plane WHERE IT ALREADY CARRIES
IPv4", picks the v6 subnet "on the SAME MAAS vlan as that v4 link", derives the
host part from "the last octet of the node's own v4 address", and its loop body is
a bare `if not v4: continue`. Under D-139 five planes lose v4 and lb-mgmt never
had a v4 twin, so after v4 removal it would SILENTLY CARVE FOUR FEWER PLANES PER
NODE rather than fail. The v6 carve MUST run BEFORE v4 removal, or the tool must
be rewritten -- compounding the already-recorded fact that D-139's "remove v4
LAST" clause is unsatisfiable. LOGGED, NOT FIXED. My own agent mandate stated this
tool's input source WRONGLY (apex/D-136; it is live MAAS); the agent checked
rather than inheriting the error, which is why the defect surfaced.
THREE MORE FINDINGS: (i) the multicast reading is ESTATE-WIDE -- all six plane
bridges plus the WAN bridge at snooping=1/querier=0 on dc0 AND dc1, and the
jumphost's uplinks too, i.e. the default here, not a chosen setting; the
subcommand tags it SUSPECT and asserts no Linux behaviour, correctly, since
cold-start multicast ND was already measured working across it. (ii) plan's PEER
SELECTION IS DEPLOYMENT-BLIND -- it picked control-01, which is not deployed, so
the derived SPEC cannot pass in a PARTIAL deployment; worked around by
substituting the live peer, logged not fixed. (iii) a harness bug caught by the
instrument-currency rule, NINTH of its kind: absent-ip/absent-virsh REFUSE cases
were driven with PATH=/usr/bin:/bin, which still contains both, so they were red
for the WRONG REASON.
TWO OPEN CHECKS SETTLED on the live nodes: every NIC carries exactly ONE global
(sole-global predicate holds today), and an OVS-internal br-ex does report
LOWER_UP (predicate safe).
GA-R2: the build agent wrote its own changelog; folded VERBATIM into this
session's single changelog and removed. Crossing midnight does not start a new
session. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

IPv6 PROVEN WORKING on the dc0 node planes -- and already load-bearing
...
Live experiment, operator-approved: two storage nodes MAAS-deployed (jammy) to
open a boot window, issued individually. Region asserted first (vr1-dc0-region ->
one rack controller hot-kid; admin -> the three Office1 racks); maas-profile-
assert.sh is MISSING on voffice1, so the property was asserted inline not skipped.
A1 ADDRESSES COME UP -- PASS. Six global v6 addresses live on the NICs, matching
MAAS exactly, 0 tentative / 0 dadfailed, on-link /64 routes, no v6 default
(expected). Asserted on the INTERFACE, never on MAAS -- 54 carved statics were
never evidence any existed on a NIC, and this is the first check on a ROLE NODE.
A2 THE PLANE CARRIES v6 -- PASS on all six planes, 0% loss, every neighbour
REACHABLE. THE RISK I FLAGGED IS REFUTED FOR THIS PATH: every plane bridge is
multicast_snooping=1 / querier=0, a known ND failure mode, and the neighbour
table was COLD, so the first solicitation went out as MULTICAST and resolved on
all six. HONEST RESIDUAL: proves ND from cold, not behaviour across long idle
periods where snooping entries age out.
A3 G17's ROLE-NODE HALF CAPTURED -- PASS. Open since 2026-07-30 (that capture was
taken on the CONTROLLER VM and said so). Mirror real package path: exit 0, HTTP
200, 269219 bytes, all four body fields. AND the 2026-07-31 deploy blocker is
confirmed fixed FROM A REAL NODE -- all four suites 200, jammy-backports included,
where it was 404 and put 22 units into hook failed: "install".
A4 HEADLINE, and it inverts this repo's framing: THE NODE'S TIME SOURCE IS ALREADY
IPv6. ServerName fd50:840e:74e2:220::6, ServerAddress family 10 = AF_INET6,
Stratum 3, root distance 57.327ms, poll backed off to 2min8s, ntp.ubuntu.com
unused as fallback. G17 assertion (2) passes, and separately IPv6 is not a future
state here -- it is a live operational dependency predating all this work.
F7 CLOSED IN PASSING: "still unverified that a redeploy applies the new hostname"
-- it does; both report their ruled names from `hostname` on the running OS.
SCOPE: these are ULA addresses, the PRE-D-139 carve, superseded by design. The
PROPERTY tested is family-agnostic and transfers. Nothing here tests a charm, a
container, network-get or Ceph -- those remain the untested rungs.
Nodes deliberately left Deployed so the G19 gate can run against a LIVE node
rather than ship fixture-green. Release owed. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

RULED "B plus C": D-139 ruling A narrowed to an experiment + upstream fix
...
Operator, exact utterance: "B plus C". Ruling B's GUA carve is unaffected and
proceeds; ruling A's five-plane v6-only scope narrows to a bounded experiment;
metal-internal, data-tenant and lb-mgmt stay DUAL-STACK pending upstream fixes.
That last sentence is recorded as an INTERPRETATION, not as the utterance.
>>> I SCOPED THE EXPERIMENT WRONG AND THIS CORRECTS IT. <<<
MEASURED against shipped ceph-osd rev 953, ceph_hooks.py:544-553: prefer-ipv6 is
ONE switch that sets ms_bind_ipv4 = False GLOBALLY, so it governs Ceph's PUBLIC
network (storage) and CLUSTER network (replication) TOGETHER. There is no state
in which Ceph runs cluster on v6 and public on v4. I recommended "replication
alone, smallest blast radius" -- not achievable. The experiment is storage +
replication as a PAIR, with a larger blast radius than I represented.
A PATH THE EARLIER ANALYSIS MISSED: the buggy bare get_ipv6_addr() (LP #2061836,
dynamic/SLAAC-only, never fixed on ceph-osd, while our nodes hold MAAS statics) is
reached ONLY under `if not public_network:` / `if not cluster_network:`. Setting
ceph-public-network and ceph-cluster-network to the v6 CIDRs means it is never
used for address selection -- the defect becomes AVOIDABLE BY CONFIG. This is a
CODE READING, NOT A MEASUREMENT, and it does not address the get_mon_hosts()/
get_host_ip race (LP #2109798).
REVIEW PASS on netbox/d139-gua-carve.py (independent second agent, per operator
standing instruction). 4 real defects fixed, 4 reported, 1 charter miss upheld.
D1 is the mandate's own named class "a refusal that does not refuse": an
unreachable apex gave a raw traceback and EXIT 1 -- the tool's own header defines
exit 1 as a write error -- on a dry run that wrote nothing. _req caught only
HTTPError so URLError walked past the REFUSE handler. Fixed, REFUSE(2).
D2: a comment asserted ruling B was "FLAGGED" -- already false at commit time.
Replaced with a citation; prose in a script rots against governance state, a
citation cannot. D3/D4: two greps over the tool's OWN SOURCE and two vacuous
negatives replaced with behavioural coverage.
main() decomposition DECLINED on a reusable argument: the `if not a.commit:
return 0` guard is ADJACENT to the write block, and extracting commit_writes()
would split that safety proof in two. Line count is not the criterion.
R1 IS A REAL SECURITY FINDING, MEASURED: urllib carries the Authorization header
across a CROSS-HOST redirect (loopback probe -- the target received the token
verbatim). SANDBOX_HOSTS gates the supplied url, not a redirect from it. LOW
severity, logged not executed.
Harness 65/65 -> 71/71; all 5 original mutations RE-RUN and still standing, plus
3 new; gauntlet ALL GREEN (95); dry-run body diffed BYTE-FOR-BYTE identical to the
committed capture; apex 139 prefixes before and after. QUALIFIED: a dry run
executes zero POSTs, so D1's change to the WRITE path's failure semantics is
UNTESTED and is declared, not implied.
PROCESS FAILURE REPEATED AND OWNED: the review mandate required a docs/audit/
capture and forbade touching CURRENT-STATE.md -- jointly unsatisfiable against
L10. The agent correctly left the tree RED and said so rather than working around
it. Standing fix, now twice learned: an agent mandate must name a repo path for
findings AND leave the committer able to satisfy L10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 ruling A NOT achievable at these charm revisions; apex carve tool built
...
TWO agent results, both recorded; nothing actioned, nothing pushed to NetBox.
>>> RULING A (v6-only on five planes) IS BLOCKED BY UPSTREAM CHARM DEFECTS <<<
RULING B (full GUA) IS ORTHOGONAL AND STANDS -- GUA-vs-ULA is independent of
whether a plane keeps v4, so the carve tool is unaffected and a GUA DUAL-STACK
carve is achievable today. This needs an operator ruling.
RE-VERIFIED INDEPENDENTLY against the shipped ceph-osd rev 953 artifact, because
it decides the ruling. hooks/utils.py:203-217 get_host_ip() returns
get_ipv6_addr()[0] when prefer-ipv6 is set, else socket.inet_aton(hostname) and
on failure dns.resolver.query(hostname, 'A') -- an IPv4-ONLY lookup -- under the
charm's OWN comment "...just let it kill the hook". So ceph-osd fails on a
v6-only plane in BOTH directions: false -> inet_aton raises on a v6 literal then
NXDOMAINs an IN A lookup (LP #2109798, New); true -> bare get_ipv6_addr() (also
ceph_hooks.py:547, WITHOUT dynamic_only=False) returns DYNAMIC/SLAAC only while
every VR1 node holds a MAAS STATIC v6 (LP #2061836 -- Fix Committed on ceph-mon,
New and never fixed on ceph-osd, confirmed absent from rev 953).
Reported from the capture and NOT re-verified here, recorded as such: mysql /
mysql-router bare user:pw@addr URIs invalid for an unbracketed v6 literal;
hacluster 2.4/stable rev 166 (released 2026-06-29) still hard-writes
ip_version: ipv4 though LP #2111852 is Fix Committed -- FIX COMMITTED IS NOT FIX
RELEASED; OVN documenting encap as "The IPv4 address of the encapsulation tunnel
endpoint" with zero v6 encap values in OVN 24.03's test suite.
Counted locally: metal-internal appears 273 times in bundle.yaml across ~44 apps;
rabbitmq-server is DEPLOYED by the bundle but ABSENT from PREFER_IPV6_CHARMS --
the only mismatch across all 33 bundle pairs, where invariant 9a would emit a
factually false message.
APEX CARVE TOOL (netbox/d139-gua-carve.py, 232 lines) + tests/d139-gua-carve
(65 cases, offline stub apex); manifest 94->95. Live dry run: CREATE 22 |
EXISTS 6 untouched | RETIRE-REPORT 18, and it surfaced 52 DEPENDENT ip-addresses
inside the retiring ULA /64s that a later delete would orphan. Proof no write
occurred: apex prefix count 139 before and after. 5 mutations, each turning a
NAMED test red, tool restored sha256-identical.
OWED: (a) the foundational measurement is STILL not taken -- does network-get
return v6 on a v6-only bound space? (b) D-139's "remove v4 LAST, after each is
proven" is UNSATISFIABLE as written; per-plane conversion is atomic.
(c) the rabbitmq-server mismatch.
PROCESS FAILURE, OWNED: the research was written to /tmp and would have been LOST
-- I scoped the agent read-only and gave it no repo path. Rescued, sha256-verified.
STANDING CONSEQUENCE: every agent mandate must NAME A REPO PATH for its findings;
"read-only" constrains what it may CHANGE, not where it may RECORD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

The "juju client blocker" is NOT one -- it is D-138 working correctly
...
CORRECTS this repo's own 2026-07-31 finding 9d, which called it a blocker.
Stage 5 can reach `add-model` TODAY, from the dc0 rack. No key movement needed.
VERIFIED INDEPENDENTLY from the main session, not accepted on the agent's word:
juju 3.6.27-genericlinux-amd64 at /snap/bin/juju on the rack; controller
vr1-dc0-controller* admin/superuser on cloud vr1-maas; the controller model reads
"Last connection: just now"; client credentials list EXACTLY ONE entry,
vr1-maas -> vr1-dc0-cred.
ROOT CAUSE OF THE ssh REFUSAL, MEASURED ON THE MACHINE: the alias is correct in
every part -- hop chain, host, port, per-hop identity, host key. subtle-grouse's
authorized_keys holds exactly two lines, both Juju's, so the office1_svc key the
alias offers is genuinely absent and the refusal is right. The obvious hypothesis
was REFUTED rather than assumed: the machine booted AFTER the 2026-07-30 key
import and still got only Juju keys, because deploy-time cloud-config carries
Juju:juju-client-key alone. `juju ssh -m controller 0` already works.
SEC-026 control (1) DISCHARGED BY MEASUREMENT on both sides -- rack lists one
credential, voffice1 lists two, so the forbidden whole-store copy did not happen.
No new credential residency; no new security-ledger row owed.
ONLY HYGIENE OWED, NOT EXECUTED (hard rule 3): a dangling current-model pointing
at the model destroyed 2026-07-31 makes bare `juju status` error. Measured NOT to
be an add-model precondition.
FOUR GAPS LOGGED NOT FIXED, in consequence order:
G4 the `openstack` CLI is MISSING ON THE RACK -- D-138 definition-of-done gap
blocking phase-03+. SAME item as F1 from the 2026-07-30 sweep, now confirmed
on the D-138 host. It has survived two sweeps.
G3 preflight CANNOT PASS on vcloud by construction: the octavia-pki overlay is
gitignored PKI material absent from the vcloud tree, and
pre-flight-checks.sh:161 hard-fails on it. The host-dependence class again.
G2 the D-138 client host's deploy input has NO PROVENANCE -- ~/repo-stage on the
rack is a hand-staged 9-file copy with no .git (8/9 byte-identical by sha256).
A third copy beside the two clones verified at c58bf95.
G1 phase-4 runbook :379,398 still say "voffice1" for add-model/spaces, stale vs
D-138; voffice1 has the binary but no registered controller, so the runbook as
written fails. DOCFIX owed.
DECLARED UNMEASURED with the reason: whether MAAS user juju-vr1-dc0 has an
imported ssh key -- no usable maas CLI profile exists anywhere (voffice1's
~/.maascli.db is ZERO BYTES). Does not change the fix; the proximate cause was
measured directly. The PreToolUse guard refused a command that would print the
MAAS API key (DOCFIX-016) and it was NOT retried in an altered shape.
Staged by explicit path; agent files still in flight are deliberately excluded.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 ruling B RECONSIDERED and CONFIRMED; refuted rationale STRUCK in place
...
Operator, exact utterance: "Deciding to hold to gua does not cost anything
operationally and the case for ULA over gua is not very strong. Let's stay with
gua". Recorded as RULING NOTE 2026-08-01 -- D-139.
RULED: ruling B STANDS, unchanged in substance. The GUA carve table is confirmed,
the ULA /48 stays RETIRED for VR1, the D-101/D-111 amendments hold, and Phase-2
execution of D-139 is UNBLOCKED.
WHAT CHANGED IS THE RATIONALE OF RECORD. The RFC 6724 precedence argument is
STRUCK IN PLACE in D-139 -- quoted, with the measurement that kills it and a
pointer to the capture, marked DO NOT CITE IT. Struck rather than deleted on
purpose: deleting it would leave a future session free to re-derive an argument
that has already been considered and refuted. The block also warns against the
near-miss rescue -- glibc 2.35 DOES carry fc00::/7 in default_labels[] (label 6),
which drives SOURCE-selection rules 5/6, a different mechanism from destination
precedence; finding it does not resurrect the struck claim.
Ruling B now rests on two project-constraint arguments -- conformance (Willamette
and VR0 DC0 are already full GUA; VR1 was the outlier) and MINIMIZE DELTA TO
ROOSEVELT -- plus the operator's recorded reasoning that GUA costs nothing
operationally. MADE EXPLICIT because it was not visible before: on glibc 2.35 GUA
buys NO address-selection advantage on the two dual-stack planes; ULA would have
won equally at 40. Anyone later reasoning "GUA was chosen so v6 would beat v4" is
reasoning from the struck argument.
NEW STANDING RULE, recorded on D-139 because it generalises: the repo's citation
rule (open it, check STATUS and DATES) was FOLLOWED here and was NOT ENOUGH. The
citation was real, current, correctly quoted and correctly understood, and still
wrong, because nobody checked whether the IMPLEMENTATION follows the standard.
For a STANDARDS citation, add a third check -- confirm the deployed software
implements it, at the deployed version. An RFC is not a description of your system.
Staged by explicit path; an agent's in-flight files (netbox/d139-gua-carve.py,
tests/d139-gua-carve/, docs/audit/d139-carve-dryrun-20260801.txt) are deliberately
NOT included. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 ruling B's deciding reason REFUTED by measurement; ruling FLAGGED
...
Closing the unverified premise D-139 named at ruling time inverted its own
argument. Recorded, flagged for operator reconsideration, NOT amended.
MEASURED, two steps:
1. jammy libc-bin 2.35-0ubuntu3 ships /etc/gai.conf with ZERO active lines --
every line commented. The node image applies NO override, so glibc's
compiled-in table governs.
2. glibc 2.35 default_precedence[], fetched in full from sourceware at tag
glibc-2.35 and read rather than recalled, carries glibc's OWN comment
"See RFC 3484 for the details" and is:
::1/128 50 | 2002::/16 30 | ::/96 20 | ::ffff:0:0/96 10 | ::/0 40
THERE IS NO fc00::/7 ENTRY IN THE PRECEDENCE TABLE.
CONSEQUENCE: ULA falls through to ::/0 = 40, identical to GUA; IPv4-mapped is
10. So on the node image ULA = GUA = 40 and BOTH outrank IPv4. Ruling B was
declared on "ULA loses to IPv4 (3 vs 35), GUA wins (40 vs 35)" -- the RFC 6724
table. RFC 6724 obsoletes RFC 3484 and does define those values, but glibc 2.35
DOES NOT IMPLEMENT IT. Right about the RFC, wrong about the software this cloud
runs. The fc00::/7 entry that does exist is in default_labels[] (label 6), which
drives SOURCE selection rules 5/6 -- a different mechanism, recorded so the two
are not later conflated into "the argument stands after all".
A NEW VARIANT OF THE CITATION CLASS, worth naming: the citation was real,
current and correctly quoted, and still wrong here because nobody checked
whether the IMPLEMENTATION follows the RFC. Checking a source's status and dates
is not sufficient for a standards citation.
SURVIVING REASONS (independent of the refuted one): conformance with Willamette
(a real site) and VR0 DC0, both full GUA; and MINIMIZE DELTA TO ROOSEVELT, whose
2602:f3e2:103::/48 is GUA with no ULA and no plane carve yet. The ruling may well
stand on these -- but under GA-R5 that is the operator's call, since they
answered a question whose stated deciding reason no longer holds.
D-139's text is UNCHANGED. Nothing of D-139 is executed, so nothing is half-built
in either direction. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 ADOPTED: VR1 goes IPv6-only east-west, whole carve moves to GUA
...
TWO GA-R5 rulings, taken in SEPARATE exchanges, recorded separately.
A -- family matrix. Operator: "IPv6 on all planes except for metal-admin and
provider-public which will remain dual stack". metal-internal, data-tenant,
storage, replication and lb-mgmt become IPv6-ONLY. Amends D-101's matrix twice:
metal-internal's "datastore east-west stays v4-bound" is superseded, and lb-mgmt
(ruled v6-only by D-101 but NEVER carved anywhere) becomes a first-class plane.
B -- addressing model. Operator: "Full GUA on every plane (Recommended)". Every
plane carves from its DC's GUA /48 on the :10/:11/:20/:21/:30/:40/:50/:80 octet
map VR0 DC0 and the Willamette site ALREADY use -- conforming to an existing org
standard, not inventing one. The ULA /48 fd50:840e:74e2::/48 is RETIRED for VR1.
Amends D-101 and D-111. Deciding reason is measurable: RFC 6724 ranks IPv4-mapped
at precedence 35 and ULA at 3 while GUA falls under ::/0 at 40, so on a dual-stack
plane a ULA leg LOSES address selection to IPv4 and is decorative. SEC-010, D-052,
D-125 and D-107 unchanged -- containment lives at the forwarding layer.
THE RECORDED ROOT CAUSE OF THE v4-ONLY CONTAINERS WAS WRONG AND THIS REPO CARRIED
IT (GA-R1 C2 -- measurement corrects the document). CURRENT-STATE said MAAS "has
nothing to give for v6". Measured, and re-verified independently: subnet
statistics on fd50:840e:74e2:220::/64 returns available_string "100%",
num_available 18446744069414584320. Zero ipranges rows means zero RESTRICTIONS,
not zero availability. dc-plane-ipam.sh:368-372 has carried the correct behaviour
since 2026-07-27, in the tree, contradicting the authoritative doc the whole time.
The real mechanism is juju-side and there is NO knob: EthernetDeviceForBridge
(tag v3.6.27) takes addrs[0] from an UNSORTED query and derives one CIDR -> one
LinkSubnetArgs. gomaasapi documents "Any number of STATIC links can exist on an
interface", so MAAS would accept both families; the limit is juju's. LP #1723240
is this exact symptom, Triaged/Low since 2017, and the theory this repo recorded
is the one its reporter rebutted in-thread. Consequence: on a container plane
DUAL-STACK IS NOT EXPRESSIBLE while v6-only IS -- ruling A is the achievable
configuration, not merely the desired one.
SNAP PROXY: dc0 INSTALLED, RUNNING, check dc0 PASS exit 0 (16 assertions,
re-verified independently). A REAL SNAP PAYLOAD WAS FETCHED THROUGH IT -- HTTP 206,
first bytes hsqs. No longer fixture-green. BUG-3 confirmed by measurement, no
assertion changed. BUG-4 fixed at BOTH sites (the review named one); T20b/T41b NEW.
OWNED: my BUG-1 fix was WRONG. ip route get 10.12.8.4 -> src 10.12.8.2; .4 is a
SECONDARY and Linux never auto-selects a secondary as a source, so the rule is
still dead. Not fixed now -- editing the generator turns the LIVE gate red until
install re-runs. T23b annotated so nobody "fixes" the test to a wrong value.
Generalisable: any ACL keyed to a service ALIAS (the whole D-134 utility .4 band)
misses that host's own traffic -- same exposure in dc-mirror.sh, dc-cache-proxy.sh.
OWED, unverified: the deployed jammy /etc/gai.conf, which can override the RFC 6724
default table that ruling B rests on. Nothing of D-139 is executed.
repo-lint 0 fail; tests/dc-snap-proxy 55/55; 4 mutations killed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-31 |

SESSION CLOSE 2026-07-31 (GA-R4 bookend): Stage 5 dc0 deploy + teardown
...
Bounded ledger summary (10 lines, cap 15), rotation, sweep, memory review.
DURABILITY: 0 uncommitted / 0 unpushed, 45 commits today. dc0 rack repo-stage
digests all MATCH. HAZARD, not loss: the voffice1 clone is 26 commits behind --
pull it forward before it is used.
GATES: repo-lint 0 fail / 1 standing warn / 641 files. Gauntlet ALL GREEN (94).
ledger-scan: 3 open decisions, SEC 26, D 139 / DOCFIX 207 / BUNDLEFIX 053.
Ledger 296 lines after rotating the oldest summary out to
docs/archive/session-ledger-rotated-20260731.md.
SWEEP: docs/audit/queued-findings-20260731-stage5-deploy.txt -- ELEVEN FIRST
SURFACE items that existed only in the transcript. Highest-consequence is F1:
MAAS's own squid ALREADY CONNECT-proxies api.snapcraft.io and would unblock the
deploy today with nothing built, measured from the failing container -- RULED
AGAINST, with the operator's verbatim utterance and the deciding downsides
recorded, precisely because a future session will rediscover it and reach for
it. Also first surface: containers source from METAL-ADMIN not the address juju
status displays; ports 3128 and 8000 already held by MAAS's squid on both racks;
the four reviewed bugs in dc-snap-proxy.sh; the teardown instrument error; and
that a MAAS hostname rename on a Deployed machine is record-only.
Method note: a phrase-exact grep produced false negatives on items recorded under
different wording. Every candidate was re-grepped on distinctive tokens before
being classified -- the same instrument-error class this sweep documents.
GA-R7 MEMORY REVIEW -- one real violation found and corrected: the
multi-workstation memory asserted 'never additions to allow for mutations', an
operator-posture claim memory may not hold, contradicted by the recorded
2026-07-30 ruling 'Add it to allow'. Re-pointed to an observation with the
contradiction recorded. instrument-currency memory gained this session's two
misreads.
OWNED: a red-lint push (| tail masked the exit code); a git add -A that swept an
agent's files into an unrelated commit; two instrument misreads (a juju models
summary read as progress, stale unit logs read as current); and a wrong
'immutable system_id' claim the operator corrected.
NEXT: apply the dc-snap-proxy review cuts, RE-RUN the mutation pass (S8
invalidates M32-M39/M42-M47/M49-M52 as evidence), confirm BUG-3's dstdomain deny
shape once at apply time, install the proxy, then add-model + spaces + deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Snap proxy tool BUILT and REVIEWED (not yet applied); 4 real bugs logged
...
Ruling 2's repo side. squid 6.14 as a dedicated instance on each DC rack at the
D-134 utility .4:3129, CONNECT-only, restricted to Canonical's documented store
and CDN hosts, consumed via juju snap-http-proxy/snap-https-proxy. Design cited
to vendor sources, not preference: every snap endpoint is :443 so a CONNECT
proxy is mandatory and CACHING IS IMPOSSIBLE (squid cannot read a CONNECT
tunnel), which removes the apt-style caching argument entirely; squid is in
main, tinyproxy is universe with six unevaluated CVEs and defaults to allowing
all CONNECT ports.
MEASUREMENTS THAT CHANGED THE DESIGN:
- the failing apps are LXD containers sourcing from METAL-ADMIN (10.12.8.122),
NOT the 10.12.12.116 juju status displays. An ACL on the displayed address
would have denied every client the proxy exists for.
- ports 3128 AND 8000 are already held wildcard-bound by MAAS's own squid on
both racks -- hence 3129 and a dedicated instance.
- no new D-134 octet needed: .4 is already aliased on both racks.
- LOGGED NOT ADOPTED: MAAS's squid ALREADY CONNECT-proxies api.snapcraft.io,
measured working from the failing container. snap-https-proxy pointed at
10.12.8.6:8000 may unblock the deploy today with nothing built. Rejected as
the shipped design (per-revision path, NO destination restriction, the
hidden-coupling shape D-135 already has a scar from) but it is a real
operator option, not a dismissal.
QUALITY: 53 mutations, 53 killed, 0 survived -- including the dc-mirror.sh
existence-vs-content false-green class reproduced and caught. One survivor was
a MALFORMED mutation that broke bash -n and therefore tested nothing; recorded
as malformed and redone rather than counted. Three of the harness's own cases
were found too weak and were STRENGTHENED, not left green.
INDEPENDENT REVIEW: ruled design HONOURED on all four counts -- utility band, no
direct node egress (in fact stronger: destination-allowlisted, and that scope is
flagged as a build-time choice NOT a ruling), site-keyed both DCs with the dc1
limit stated honestly, and it OWNS its own network prerequisites rather than
borrowing another service's -- with the coupling anti-pattern now under test
(M09/M10) rather than only under comment.
FOUR REAL BUGS LOGGED, NOT FIXED (hard rule 1 -- the file was frozen for review):
BUG-1 dead snap_probe ACL; the probes are actually admitted by snap_clients,
and the comment misdescribes its own mechanism
BUG-2 | grep -q under pipefail -- INHERITED from dc-mirror.sh and
dc-cache-proxy.sh alike; a three-script sweep, not a local fix
BUG-3 the deny probe's 403 shape was measured from a PORT deny, never a
DSTDOMAIN deny. If squid answers differently the gate REFUSES
PERMANENTLY -- and a gate that cannot PASS blocks a GA-R6 close as hard
as one that cannot fail. Requires a one-time apply-time confirmation.
BUG-4 dpkg -s reports a deinstall-ok-config-files package as present
SIZE: 636 lines / 320 code vs dc-mirror 385/227 and dc-cache-proxy 396/200. The
reviewer's landing zone is ~535/~250 and it RECOMMENDS AGAINST forcing 385 --
two extra generators (75 lines) are measured-forced because the packaged squid
cannot be repointed, and further cuts would delete mutation-proven assertions.
~50 lines of the excess is header prose, which carries zero bug risk.
TWO BLOCKERS BEFORE THE PLAN IS APPLIED: the S8 consolidation invalidates
M32-M39/M42-M47/M49-M52 as evidence, so the mutation pass MUST be re-run; and
BUG-3 must be confirmed once against the built proxy before check is cited to
close anything.
NOTHING IS PROVEN END TO END -- no proxy installed, no snap fetched through one.
The harness green is FIXTURE green and says so.
repo-lint 0 fail; gauntlet ALL GREEN (94).
OWNED: my earlier teardown commit e57ad09 used 'git add -A' while the agent was
writing and swept its audit capture and CURRENT-STATE paragraph into a commit
about the model teardown. This commit is staged EXPLICITLY by path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Teardown: stalled on stopped agents, forced clear, no nodes stranded
...
Plain destroy-model STALLED and would not self-resolve -- 'attempt 30 ... model
not empty, found 26 machines, 37 applications', flat ~19 min with the app set
byte-identical across a 12-minute name-level diff.
MECHANISM MEASURED, and it makes the stall terminal rather than slow: ALL 26
machine/container agents were 'stopped', so NO hook could execute. Units already
in error from the snap failures could never run their teardown hooks.
MAAS was NOT the bottleneck -- the six nodes juju released went to Ready/owner
None in minutes; juju never issued a release for the other three.
destroy-model --force --no-wait cleared it (18 -> 5 -> 2 machines, then 'Model
destroyed.'), and NO NODES WERE STRANDED: all nine read back Ready/owner=None,
so no maas machine release was needed or run. subtle-grouse correctly stays
Deployed -- it is the D-104 controller VM in the controller model.
INSTRUMENT ERROR, OWNED: I reported '0 machines / 0 units' from the juju models
SUMMARY COLUMNS while juju status -m read 26 machines / 37 applications. The
summary zeroes during 'destroying' and is not a progress signal. The real tell --
three control nodes stuck Deployed -- was visible and I explained it away.
Standing lesson: during a teardown, juju status -m <model> is the instrument;
juju models counts are not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Set apt-mirror as a controller model-default; record the durability gap
...
Operator: 'Yes, set the defaults'.
The teardown demonstrated the gap: destroy-model takes the model config with it,
so Step 3.5's apt-mirror vanished and NOTHING in the repo would have caught it --
the next deploy would have failed on package fetches and read like a mirror fault
rather than a missing model setting. That is the 'prose cannot close a stage'
class: Step 3.5 is runbook prose with no gate.
MEASURED: juju model-defaults carries apt-mirror, snap-https-proxy,
snap-store-proxy and snap-store-proxy-url, all previously unset at Controller
level. Set and read back: apt-mirror -> Controller http://10.12.8.4/ubuntu.
Defaults are inherited by NEW models only, which suits the pending add-model.
Per-DC by construction: D-104 gives each DC its own Juju controller, and these
values are per-DC, so per-controller defaults map onto per-DC values with no
cross-DC coupling.
THE SNAP KEY IS DELIBERATELY NOT SET YET -- the proxy does not exist, and
pointing snap-https-proxy at a dead address would make snap installs fail WORSE
than they do now. It lands with the proxy.
OWED: a site-keyed dc-model-defaults.sh with a check, so the values are VERIFIED
at every DC standup rather than remembered. Roosevelt shape: fold into D-136's
per-DC render pipeline. DC1 lesson: set model-defaults at controller bootstrap,
before its first add-model.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Node-naming canary: a MAAS rename alone is cosmetic-only
...
Operator asked for the juju status Inst id values to become role-based.
CORRECTION to an earlier claim this session: that column shows the MAAS
HOSTNAME, not the system_id. My first reading came from the JSON instance-id
field (677cta) and was wrong -- the displayed value is the hostname and it IS
renameable.
Canary: civil-bug resolved BY PINNED BOOT MAC (52:54:00:2b:ed:ab) to the ruled
vr1-dc0-storage-04, then renamed.
MEASURED: MAAS ACCEPTED the rename on a Deployed machine (fqdn
vr1-dc0-storage-04.maas), but juju status still shows civil-bug AND the running
OS still answers civil-bug (hostname and hostnamectl --static). MAAS's record
moves and nothing else does -- juju captured the name at provisioning, and the
OS hostname is applied by cloud-init at DEPLOY time.
CONSEQUENCE: renaming the other eight buys nothing where the operator is looking
and leaves a three-way divergence. Nothing functional rides on it -- the bundle
places by TAG and every gate resolves by PINNED BOOT MAC.
THE CLEAN POINT IS ENLISTMENT: folds into the DC1 standup as a DoD item (set the
ruled hostname BEFORE commissioning so MAAS, the OS and the ruled name agree),
and reaches DC0 at its next node redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Ruling 1 applied live: only the snap class remains
...
Overlay re-staged hash-verified (e3be85e4); the in-DC UCA origin set on all 15
apps and read back on a sample of each kind.
MEASURED AFTER: units in error are EXACTLY the three snap-dependent applications
-- mysql-innodb-cluster (x3), ovn-central (x3), vault -- 7 units. Every ceph and
OpenStack API app has cleared. State: 14 waiting, 5 maintenance, 4 blocked,
3 active, 7 error.
The deploy is now blocked on exactly ONE unbuilt thing: the snap path (ruling 2),
the D-135 items 2-3 gap.
Session progression, all measured: 22 error -> 17 (backports) -> 4 (prefer-ipv6)
-> 7 units in the single snap class. The count rose at the last step because more
units REACHED the snap stage, not because more broke.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Build ruling 1: UCA points at the in-DC mirror for 15 apps (scope derived, not guessed)
...
overlays/vr1-dc0-machines.yaml sets
deb http://10.12.8.4/cloud-archive jammy-updates/caracal main
on 15 apps. SCOPE DERIVED FROM THE CHARM SCHEMAS rather than hand-listed: every
app whose charm accepts an origin key AND resolves to a UCA pocket -- 12
openstack-origin (explicit cloud:jammy-caracal or the charm default 'caracal')
plus 3 ceph 'source'. Everything else defaults to 'distro' (Ubuntu archive only)
and needs nothing, which is why the mysql-router subordinates, ceph-rbd-mirror,
mysql-innodb-cluster, glance-simplestreams-sync and rabbitmq-server are absent.
CORRECTION: the earlier in-session figure of 'six apps' was the count that set it
EXPLICITLY and was never the scope. Measured on the containers, both keystone/0
and ceph-mon/1 carry the upstream UCA source.
Also measured: keystone CLEARED install anyway, so the unreachable UCA is not
universally fatal -- it is fatal where a charm's apt_update uses --error-on=any
(the ceph charms), and a CORRECTNESS problem everywhere else, since a node that
cannot reach the Caracal pocket silently gets jammy's own OpenStack instead.
Per-DC by necessity: the mirror address differs per DC, so this cannot live in
bundle.yaml. dc1 gets no equivalent block -- its apt-cacher-ng forwards the
upstream URL transparently, and that asymmetry is D-135's experiment result.
No |key suffix needed: measured, the UCA signing key is already on the nodes and
debmirror preserved the upstream Release/InRelease signatures.
LOGGED NOT CHANGED: ovn-central's charm default is source: zed, not caracal -- it
points at a pocket the dc0 mirror does not carry, and repointing would change its
RELEASE rather than its URL. Separate question.
Gauntlet ALL GREEN (93); repo-lint 0 fail; provider-bundle-check PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

RULED 2026-07-31 (GA-R5) x2: UCA points in-DC; snaps get an in-DC forward proxy
...
Presented together at operator direction ('Yes, both') and answered SEPARATELY,
so neither is a batch adoption. Both OPS under GA-R3; D-107 UNAMENDED.
RULING 1 -- UCA. Operator utterance: 'Point origin/source at the mirrored UCA,
per-DC overlay (Recommended)'. dc0 gets an explicit deb line at
http://10.12.8.4/cloud-archive in the per-DC hand-maintained overlay; the mirror
address is per-DC so it cannot live in bundle.yaml. dc1 UNCHANGED -- its
apt-cacher-ng forwards the upstream URL transparently, and that asymmetry is
D-135's experiment RESULT, not a defect. Measured precondition: the UCA signing
key is already on the nodes, so a raw deb line verifies with no |key suffix.
RULING 2 -- SNAPS. Operator utterance: 'HTTP(S) forward proxy in the DC utility
band + juju snap-https-proxy (Recommended)'. D-107's core statement stays TRUE --
nodes reach an in-DC proxy, not the internet. Closes the D-135 items 2-3 gap for
BOTH DCs with one mechanism rather than widening the mirror-vs-proxy asymmetry.
Placement is build-time engineering; if it takes its own VM the D-134 octet map
needs a ruled octet first, since that map is a standing cross-DC standard.
Committed and pushed BEFORE the dependent work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Ruling (b) applied live: error 17 -> 4; remaining blocker is snap access
...
Applied live after the repo build: overlay re-staged hash-verified, the seven set
to false and each read back, every erroring unit resolved. MEASURED: units in
error went 17 -> 4 and all seven prefer-ipv6 apps CLEARED.
THE REMAINING BLOCKER IS ONE CLASS -- snap installs from an airgapped node, and
it is the already-recorded D-135 items 2-3 gap hitting for real:
mysql-innodb-cluster snap install mysql-shell (api.snapcraft.io unreachable)
ovn-central snap install prometheus-ovn-exporter
vault snap install core
This document already records the shape: three artifact classes, only ONE local
-- apt to 10.12.8.4 (local), MAAS boot images to images.maas.io (not local),
juju agent stream and snaps to streams.canonical.com / api.snapcraft.io (not
local). It will hit dc1 EQUALLY, since apt-cacher-ng proxies apt and not snaps,
so unlike the UCA finding this is NOT a full-mirror-only asymmetry.
LOGGED NOT FIXED: a D-107 (airgap) / D-135 (items 2-3) decision, not an
engineering choice.
D2 (the upstream UCA) is still unfixed and still owed -- it stopped being the
visible error only because those units now block earlier on snaps.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Fix the red-lint push: D-101 heading collision + note placement
...
OWNED: the previous commit was pushed with repo-lint RED. 'repo-lint | tail -2 &&'
masks the lint exit code with tail's, so the && proceeded on a FAIL -- the same
silenced-pipeline trap already recorded in this repo (| grep -q under pipefail;
git pull -q &&).
Two defects, both fixed here:
- the note used a '### D-NNN --' heading, which L5 reads as a second DEFINITION
of D-101 (collision). Re-titled to the established
'### RULING NOTE <date> -- D-101:' form the other four notes use.
- it was appended after D-136 instead of beside the other D-101 notes. Moved to
sit immediately after note (a).
Also recorded: repo-lint exits 2 on WARN and 0 only when fully clean, so 'exit 0'
is the WRONG success predicate for this repo -- its standing state carries one
warn (the legacy D-001..018 non-ASCII carve-out). Gate on '0 fail'.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

RULED 2026-07-31 (GA-R5): prefer-ipv6 set on NO application until IPv6 is operational
...
Operator utterance, verbatim: 'Set it false on the seven, keep every v6 VIP leg
(Recommended)'. Standing context from the same exchange: 'We have DC1 to stand
up with the IPv6 configuration changes. Lets continue with the IPv4/6 stand up
on DC0. We will fold in all lessons learned from the DC0 stand up into the DC1
stand up.'
Recorded as a D-101 RULING NOTE 2026-07-31 (b). OPS under GA-R3, no D-number;
D-101's matrix is UNAMENDED and every dual-family VIP is retained. It supersedes
the emission half of note (a) and now covers all thirteen apps.
Measured cause: get_relation_ip() returns early with get_ipv6_addr()[0] when the
option is true, and the LXD containers hold ONLY a link-local v6 while the HOSTS
are fully dual-stacked.
Why this is not abandoning v6: the option is a unit ADDRESS-FAMILY switch, not a
listener switch. HAProxy's :::port bind is gated on the kernel sysctl and
pacemaker selects IPv6addr by family detection. Neither consults it.
HONEST RESIDUAL, recorded not glossed: pacemaker will place a v6 VIP on a
container whose eth0 has no global v6, and whether that leg is ROUTABLE is
UNVERIFIED. Not a regression -- true today -- but it is what the v6 completion
work must close.
GATE CONSEQUENCE: invariant 9b coupled the option to the v6 legs for declaring
charms and would now FAIL. It is RE-POINTED to the new invariant and re-proven,
never deleted; when IPv6 becomes operational it returns to its coupling form and
this note records why it left.
Committed and pushed BEFORE the dependent work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

v6 sweep: the IPv6 half is carved as addresses but never made operational
...
Operator: 'The dual stack was only a safety net instead of jumping straight into
a ipv6 only deployment but it appears that more items were not configured with
IPv6 like they should have been.' So IPv6 is the target and v4 the fallback,
matching D-101's own 'v6 wherever possible, v4 only where forced'. A systematic
dc0 sweep was run rather than fixing the container layer alone.
BUILT on v6: node statics (54 links), the six v6 plane subnets, juju's per-plane
host bridges (all six carry global v6), spaces in both families, the dual-family
VIPs, Octavia PKI v6 IP SANs.
ABSENT on v6, all measured:
(i) ALL SIX v6 plane subnets carry ZERO ip ranges, against every v4 plane
holding its D-134 reserved bands and metal-admin also holding dynamic
.201-.254. Uniform, not a metal-admin quirk.
(ii) THE RACK HAS NO GLOBAL v6 AT ALL -- so the mirror, the node-DNS forwarder
and the MAAS rack agent are v4-only by construction. F2 from 2026-07-30,
now measured as the whole rack rather than one plane.
(iii) The mirror does not answer over v6 (000), following from (ii).
(iv) Nodes have no v6 default route -- no RA, no gateway.
CONSEQUENCE: completing IPv6 to the charm layer is a PROJECT, not a fix. A range
alone would give containers v6 addresses with no v6 default route and no
v6-reachable services, so prefer-ipv6 would still not produce a working cloud.
The safety net is doing what it was put there for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D3 root cause: dual-stack is RULED but was never built to the container layer
...
Operator: "The dual stack configuration was supposed to have included charms. We
have had conversations and decisions were made to approval the dual stack
configuration all the way down." CHECKED AGAINST THE RECORD, not accepted: the
operator is CORRECT. D-101's 2026-07-25 note carries "Dual stack deployment for
DC0 and DC1" verbatim and R2 re-confirmed it. A grep for any decision text
covering the LXD CONTAINER layer returns NOTHING.
MEASURED MECHANISM: the host is fully dual-stacked -- juju built a bridge per
plane on machine 0, every one with a global v6. What is missing is anything to
ALLOCATE: 10.12.8.0/22 has a dynamic range .201-.254 plus the D-134 reserved
bands; fd50:840e:74e2:220::/64 has NO IP RANGES AT ALL. juju asks MAAS for a
container address on the bound space, gets v4, and there is nothing for v6 -- so
every container is v4-only on a dual-stacked host.
THIS INVERTS THE FIX DIRECTION AND MY EARLIER FRAMING WAS WRONG. Dropping
prefer-ipv6 from the seven would make the deploy green by ABANDONING a ruled
posture at the charm layer, and would leave the dual-family v6 VIPs on
containers with no v6 leg.
STATED AND UNVERIFIED: whether juju REQUESTS a v6 address for a container when
an allocatable v6 range exists. Adding a range is necessary; it is NOT proven
sufficient, and that must be measured before any range is created.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|