| 2026-08-02 |

D-139 step 6 tool built + adversarially reviewed; 4 defects fixed. Apex NOT yet written.
...
Implements the two 2026-08-02 rulings ("Full step 6 first, then deploy" / "Deprecate both,
delete nothing"): CREATE 26 GUA VIP addresses -> read-back verify -> deprecate 26 ULA
addresses + 9 ULA prefixes. NO delete path anywhere, asserted against the artifact.
Dry run (live apex): CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9. The 26
CREATE targets diff EXACTLY against the 26 GUA VIP legs in overlays/vr1-dc0-vips.yaml --
every address written is one the deploy configures. --commit has NOT been run.
AN ADVERSARIAL REVIEW RETURNED "FIX FIRST" AND WAS RIGHT ON ALL FOUR COUNTS
(docs/audit/d139-step6-tool-review-20260802.txt). Mapping logic was correct; the gaps were
preconditions and coverage.
DEF-1 CRITICAL -- --dc vr1-dc1 would ORPHAN-CREATE. Targets were computed arithmetically
and never checked to exist. Measured: dc1's GUA carve is incomplete (four provider-public
rows under 2602:f3e2:f03::/48, no :20::/64, no :21::/64), so dc1 planned 26 creates into
non-existent prefixes then deprecated dc1's only authoritative rows, rc=0, no warning. dc0
hid it because all sixteen of its targets happen to exist. Reachable via the other valid
value of a required flag. FIXED + verified live: dc1 refuses, dc0 unchanged at 26/26/9.
DEF-2 CRITICAL -- the apex-IDENTITY guard was gone. It lives in d139-gua-carve.py's main()
(:159-163) and importing a module never runs its main(), so subclassing C.NB inherited the
TRANSPORT and left the SAFETY POSTURE behind: netbox.baldurkeep.com (the v1 reference)
would have connected fine and taken writes. FIXED: identity checked before any network call.
DEF-3 HIGH -- silent under-count. One missing ULA /64 row gave CREATE=13/DEPA=13/DEPP=8 at
exit 0. FIXED: any ULA address claimed by no prefix row refuses. My first fix was itself
wrong and RUNNING it caught that -- it scanned the whole retired /48 and flagged dc1's 26
VIPs while planning dc0; the /48 is SHARED (dc0 :22x, dc1 :32x). A /60 parent deliberately
does not count as coverage: the reviewer's scenario was a missing /64 whose /60 survived.
DEF-4 HIGH -- main() had ZERO coverage; the reviewer hoisted the deprecate loops above the
create phase and the suite reported ALL PASS. FIXED: T16-T18 drive main() through a fake
client that records CALL ORDER, proven by re-running that exact mutation on a copy (T18
goes RED).
TWO OF MY ASSERTIONS COULD NOT FAIL and the review killed both. T13 asserted the ABSENCE of
a string, so a traceback satisfied it -- it passed against a tool file that did not parse;
it now requires a positive, well-formed, DIFFERENT target, and new T15 asserts the tool
parses. T14 grepped ONE file, so adding a delete to the IMPORTED d139-gua-carve.py left it
green; it now covers both.
Corrected in the ruling record (GA-R1 C2): the amendment said the GUA records would be
status=active. Measured: the live ULA VIP records are "reserved", and
dc-plane-apex-import.py:186,200 creates addresses reserved. Also corrected my own
docstring overclaim -- the 26+9 deprecations are reversible, the 26 CREATES are not.
OPEN SCOPE QUESTION, MEASURED, not a tool defect: D-139 says retire the ULA rows "in the
apex AND in MAAS"; this tool is apex-only, so step 6 is NOT complete when it finishes. MAAS
on the dc0 region holds five ULA /64s beside six GUA; four are empty but
fd50:840e:74e2:220::/64 still holds 2 allocated entries. MAAS has no deprecated status for
a subnet, so delete-or-leave is a separate operator decision.
Gates: harness 20/20 (was 14, delta = the 6 cases added); gauntlet ALL GREEN (98, manifest
recorded deliberately 97 -> 98); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

sweep F2 ROOT-CAUSED: one bad archive.ubuntu.com backend; "not transient" WITHDRAWN
...
Operator ruling, exact utterance: "Root-cause the curl/debmirror anomaly first" -- taken over
accepting the red gate, over recording error shape/count, and over --ignore-small-errors.
ROOT CAUSE IS UPSTREAM AND BACKEND-SPECIFIC, not a defect in this deployment.
archive.ubuntu.com is round-robin across NINE A records and ONE, 91.189.92.23, HANGS on
dists/jammy-backports/main/dep11/Components-amd64.yml.gz while serving its directory
siblings normally. The control is what makes it a finding rather than "a backend is down":
that same backend returns the .xz sibling 200 / 5776 B / 0.543s, and the other eight return
the .gz in 0.14-0.51s. The resolver ROTATES -- six consecutive lookups, six different
orderings -- so every fetch re-rolls the backend.
MEASURED RATE: 12 sequential fetches -> 11 x 200, 1 timeout (~8%), consistent with exactly
one bad record of nine.
THEREFORE THE PRIOR COMMIT'S "NOT TRANSIENT" IS WITHDRAWN (GA-R1 C2 -- measurement wins over
a document, including one written an hour earlier). The failure is probabilistic; two
consecutive debmirror failures were not the proof of persistence they appeared to be.
A hypothesis was tested and REFUTED, and is kept because it is the one a reader forms first:
debmirror sets keep_alive => 1 and curl does not reuse connections, so a stale persistent
connection looked like the answer. The test inverted it -- keep_alive=1 (debmirror's own
setting) returned 200 in 1s while keep_alive=0 timed out, same library, seconds apart.
ALSO CORRECTED: the "500" was never a server 500. LWP synthesises 500 for CLIENT-side
failures, so it is LWP reporting its own 300s timeout (debmirror:629). I had read it as an
archive-side error, which points the investigation the wrong way.
Two structural facts read from the vendor script, both bearing on options that were on the
table: there is NO dep11 exclusion option (dep11_from_release :1201 and get_dep11_files
:1384 are unconditional), so the mirror's scope cannot be narrowed without patching
debmirror; and --ignore-small-errors is NOT a bounded tolerance -- :2891 DISCARDS the error
count, so it would read clean on a wholly failed mirror. It stays refuted.
RESIDUAL, declared not explained: debmirror failed 2 of 2, which a naive 1-in-9 model
predicts at ~1.2%. The per-IP sweep tested each backend ONCE, so it establishes that
91.189.92.23 was bad, NOT that it is the only bad one.
SECOND, INDEPENDENT FINDING: the local Release expects 6349 B while every backend now serves
6361 B, and the object was republished DURING the investigation. A successful download would
now fail HASH verification against the stale local Release -- a different failure mode from
the timeout, not fixed by fixing the timeout.
scripts/dc-mirror.sh UNTOUCHED. repo-lint 0 fail; ledger-scan unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Queued-findings backlog: sweep F1 + F6 CLOSED, F2 diagnosed, F3/F4/F5 graduated, DOCFIX-207
...
Works the 2026-08-02 close sweep (docs/audit/queued-findings-20260802-stage5-edge-fold.txt)
with three read-only agents in parallel. Cites the SWEEP register (F1-F6); the runbook fold
register has its own F1-F12 and is untouched.
sweep F1 CLOSED -- the dc0 rack's staged deploy input matches the repo again. One gated scp
of overlays/vr1-dc0-vips.yaml, ed19d989e80da8da -> 80d861560a6b3c52. A single-file copy was
provably sufficient because the WHOLE staging dir was enumerated first: 14 files, exactly 1
diverged, 0 missing from the Step-4 deploy closure (policies/overrides.zip present at the
repo digest), 0 orphans. All 14 re-verified against CURRENT HEAD after the copy. The 0600
octavia PKI overlay is untouched (same digest/mode/mtime) and was hashed, never read.
Repo-side correctness MEASURED not inherited: 0 ULA legs, 39 GUA, render-drift 4/4 naming
the file with a proof-of-teeth case.
sweep F6 CLOSED, PASS -- the dc0 MAAS region DB is proven uncorrupted. pg_dump read every
page of every table in maasdb: 23,878,796 bytes / 37,199 lines / exit 0 / empty stderr,
completion marker asserted separately. F6's own diagnosis was wrong: snap confinement does
not reproduce as ubuntu. The discriminators are ROLE (maas, not ubuntu) and TRANSPORT (over
TCP the role is password-challenged, over the unix socket it needs no credential). Both
identity values now measured -- maasdb had been prose. Dump streamed, nothing persisted.
sweep F2 DIAGNOSED, not fixed -- reading R1 is true, R2 refuted. debmirror prints "All done."
then exits non-zero; confirmed at vendor source and re-verified independently here
(debmirror 1:2.39ubuntu2, 1615 say("All done."), 1620 exit 1 if !$ignore_small_errors).
Cause: one 500 read timeout on a jammy-backports dep11 index. TWO corrections to the sweep:
its "the log says it succeeded" quotes all came from the PASSING UCA leg; and "stays RED"
overstates it -- measured 16 runs, 7 finished, 9 failed, with four fail-then-succeed pairs
hours apart. No remedy applied: debmirror's exit code conflates "nothing mirrored" with
"mirrored minus N transient files", so any tolerance change alters what the gate attests.
DOCFIX-207 -- preflight P6 quoted "50 apps / 97 relations"; measured is 56 / 108. The same
figures were corrected in phase-01's own gate on 2026-07-10 and this copy was missed.
sweep F3/F4/F5 graduated from the audit capture to durable homes: systemctl show fabricating
Result=success for a non-existent unit -> platform-traps 5c; assert the harness case count
moved, and two scripts probing one endpoint must share the probe definition -> script-authoring.
Logged NOT fixed (hard rule 1): preflight P2 validates a merged input including
vr1-dc0-machines.yaml while phase4:527-528 says the file does not exist and Step 4 does not
pass it -- a gate grading a different artifact than the deploy consumes, on the Step-4 path.
And security-ledger SEC-029(3) calls ~/repo-stage a nine-file copy; it is fourteen.
session-ledger machine-derived block re-seeded (was the 2026-07-27 seed: 21 SEC / D-138;
now 28 SEC / D-141 / DOCFIX-208, each added SEC row verified against the register).
Gates: repo-lint 0 fail / 1 warn (legacy carve-out); run-tests-all ALL GREEN (97 harnesses,
count unchanged, so nothing moved silently); tests/preflight 43/43; tests/render-drift 4/4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

SESSION CLOSE 2026-08-02 (GA-R4 bookend): dc0 edge rebuilt, D-139 steps 1-3, fold opened
...
Durability triad caught a live deploy hazard: the dc0 rack's ~/repo-stage VIP overlay is
STALE (ed19d989 vs repo 80d86156) because this session re-rendered it onto GUA and nothing
propagates to a directory with no git. Under D-138 that rack IS the deploy client, so a
deploy from it would place 26 VIP legs on the ULA prefixes D-139 retires. The 08-01 sweep
recorded these digests matching, so that note now misleads. Logged, not fixed -- first
item next session.
Gates: repo-lint 0 fail / 1 warn (654 files); gauntlet ALL GREEN (97); ledger-scan 3 open
decisions, 28 open SEC, next-free D-141 / DOCFIX-207 / BUNDLEFIX-053. Reconciled against
what the session claimed: D 140->141, SEC 26->28, manifest 96->97.
Mirror sync re-ran and the content IS current (both Release files rewritten today, UCA now
succeeds so egress is genuinely fixed) but records FAIL: the journal says "Everything OK...
All done." then systemd status=1/FAILURE. dc-mirror.sh check stays red on an exit code that
disagrees with the sync's own verdict. Not relaxed to clear it.
Sweep: docs/audit/queued-findings-20260802-stage5-edge-fold.txt, 6 FIRST SURFACE, each
grep-proven absent first. Memory review clean; instrument-currency gains instances ten
through twelve. Ledger rotated 283 -> 266, now 286; no orphaned session. No stage opened
or closed -- Stage 5 remains OPEN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc0 edge REBUILT, egress restored (8/8); SEC-032; two dc-egress-check defects fixed
...
Agent-executed rebuild. tofu -replace asserted on `tofu show -json`: exactly 2 non-no-op
changes, both delete,create, both naming vr1_dc0_opnsense, 0 not naming it -- and the
POSITIVE half asserted too, because a plan replacing only the DOMAIN would have reattached
the corrupt qcow2 and passed a "nothing extra" check. Artifact verified, not the log.
Convergence re-plan clean. pfctl -s nat explicitly verified to show real outbound NAT --
the exact defect dc1 is sitting in.
MY BRIEF CARRIED A WRONG BASE-IMAGE PATH. I cited the module default; d124-inner.auto.tfvars
overrides it to a path on voffice1. The apply destroys the volume before recreating, and
neither edge disk has a backingStore, so a wrong path would have left no disk and no
rollback. The agent verified before mutating. Same guess-instead-of-look failure the
operator called out earlier today, in a brief written to prevent exactly that.
dc-egress-check: A4's snap probe omitted `Snap-Device-Series: 16` (400 without it through
the proxy AND direct; 200 with) -- a false FAIL on a healthy proxy. A3 now names the
forwards-without-translating signature when A2 passed. Harness 14 -> 16; T15 asserts on
the CODE after a first cut passed on the comment explaining the header.
Three findings needing attention pre-Stage-5: the rebuilt edge runs dnsmasq on udp4 *:67
over vtnet0 with a stale 192.168.1.x range on the segment carrying juju .5 and MAAS .6
(measured; inertness reasoned, not measured); SEC-021(a) answered -- the off-jumphost copy
exists; and a rebuilt edge invalidates the rack's known_hosts, which no runbook mentions.
Gauntlet ALL GREEN (97); repo-lint 0 fail / 1 warn. Open SEC 28.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc1 edge: forwards but does not TRANSLATE; config intact; REPAIR not rebuild (SEC-031)
...
Read-only agent assessment, docs/audit/dc1-edge-assessment-20260802.txt (643 lines).
Decisive measurement: simultaneous tcpdump on BOTH edge taps while the rack pinged
1.1.1.1 -- the same packet appears on the LAN tap and 0.8 ms later on the WAN tap with
source STILL 10.12.64.2. vcloud's virbr1 masquerades only 172.30.3.0/24, so it is never
translated and no reply returns. Routing works, pf is not blocking, there is no outbound
NAT. My "config lost" hypothesis was half right and the half that mattered was wrong.
Config INTACT on positive evidence: fsck names every inode it deletes, all 50 enumerated,
/conf/config.xml not among them; the edge booted onto its ruled as-built addresses, not
the factory ones. Qualifier: 92 further casualties are inode-only, so strong not absolute.
What fsck destroyed is the FreeBSD base-system user DB -- /etc/master.passwd and
/etc/group, plus BOTH /var/backups copies. Hence "Configuring firewall.....failed." on
every boot since, no pf ruleset, and SEC-031: the edge is an open router serving its GUI
to the simulated ISP. Confirmed against a control (office1 edge: WAN ICMP 100% loss, GUI
000; dc1: 0% loss, GUI 200 in 0.014s). Bounded by lab topology; D-125 unaffected.
Repair not rebuild -- rebuilding discards a good config and re-incurs D-112(c)+D-113.
Blocked on there being no working credential path (sshd down, authed API hangs, console
login likely broken by the same passwd DB); repair needs boot -s, a mutation, so the
read-only agent stopped.
Two findings that outlive this: dc-egress-check's A2 is structurally blind to
forwards-but-does-not-translate and needs a positive "the edge TRANSLATES" assertion --
a defect in a gate built earlier the same day. And NEITHER edge had ever been
reboot-tested; 2026-08-01 was the first time dc1's as-built config ever booted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

CORRECTION: the edge break was UFS damage from an unclean power cut, not an update
...
My earlier "partially-applied OPNsense update" diagnosis was an inference from the shape
of the symptom and is refuted by the log: ZERO pkg-static / opnsense-update / firmware
lines in the entire 477 KB serial history.
What precedes the first missing-library error is an fsck salvage of a badly damaged UFS
root -- UNREF FILE x2533, UNEXPECTED SOFT UPDATE INCONSISTENCY x785, SALVAGE? yes x257,
then immediately ld-elf.so.1: libcrypto.so.17 not found. The libraries were fsck
casualties.
Causal chain: the 08-01 memory resize was an IN-PLACE tofu update, which this document
already records BOUNCES the guest (confirmed on dc1 as canary that day). A bounce of a
containment VM is a hard power cut to every inner VM. Under D-127 the DC edges are the
only inner VMs with autostart=true, so they were the only ones hard-cut and auto-
restarted onto damaged filesystems.
Confirmed by the dc1 control -- same event, different severity: dc0 2533/785 with ld-elf
errors, dc1 76/181 with zero. dc1's "healthy edge that does not forward", logged earlier
as a separate thread, is NOT separate -- same incident, lesser damage.
Durable finding is a procedure gap: the resize plan was asserted on content and passed a
capacity gate, and none of it covered the inner guests. "In-place" describes the tofu
resource, not the blast radius. Any future in-place change to a containment VM must shut
down its inner guests first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc0 edge root cause: partially-applied OPNsense update, exposed (not caused) by the reboot
...
Console read: the edge sits at a single-user prompt with libcrypto.so.17 and
libpython3.13.so.1.0 missing, 10-configd / 15-templates / 90-carp all failing. The
running system had survived on already-mapped libraries; the 05:49 boot is where it fell
over. Symptom appears nowhere in appendix-A -- a new class.
Scope measured, not assumed: dc0 only. dc1's edge log has zero matches and booted clean
to a login prompt with its WAN addressed. Upstream healthy -- vcloud and voffice1 both
reach the internet, both uplink nets are active NAT, both racks hold default routes.
Recovery material: opnsense-26.7-nano.qcow2 is in the same pool, but the broken disk has
an empty backingStore -- standalone, no CoW base to roll back to.
Logged not conflated: dc1's rack also lacks egress despite a healthy edge; a dc1
forwarding/NAT question, off the deploy path, not diagnosed here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Step 3.5 done; BLOCKER: the dc0 edge has been down since the 08-01 05:48 reboot
...
Step 3.5 complete: model vr1-dc0 created (credential vr1-dc0-cred), spaces gate PASS
0 fatal, apt-mirror verified at http://10.12.8.4/ubuntu. Spaces script run from the
07-30 staged copy on the rack, all three files sha256-verified against the repo first.
Blocker found by dc-mirror.sh check dc0 -- the content-assertion fix from 2026-07-27
earning its keep. The rack cannot reach archive.ubuntu.com, streams.canonical.com,
api.snapcraft.io via the snap proxy, 1.1.1.1, or its own default gateway 10.12.4.1.
ip neigh shows 10.12.4.1 FAILED while .5 and .6 on the same segment answer. The edge
VM is running with both NICs attached, qga not connected, qemu log last written at the
reboot. Last successful mirror sync predates it. Step 4 cannot proceed: the deploy
needs the juju agent stream and snap upstream, both dead.
Two gaps exposed: no gate anywhere tests DC egress (dc-rack-net check passes with it
dead), and this session's own post-reboot verification cited five true facts none of
which tested egress -- a listener is not a path, and a 200 from a local service says
nothing about upstream.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 steps 2 and 3 EXECUTED for vr1-dc0: the nodes are on GUA
...
Step 2: dc-plane-ipam.sh carve-v6 --commit, applied=5, each GUA /64 on the same vlan as
its v4 twin and read back. 2 errors are correct refusals (lbaas-mgmt and oob have no v4
plane to pair with). Independent read: 11 v6 subnets, 6 GUA + 5 ULA, every GUA paired.
Step 3: dc-node-v6-carve.py replace --v6-family gua --commit, applied=45 skipped=9
errors=0, READ-BACK 45/45. Independent query: PRE 54 v6 links (GUA 9 / ULA 45) -> POST 54
(GUA 54 / ULA 0), and ZERO NICs carry more than one global v6, so G19's sole-global
predicate holds. v4 was not touched -- the ordering step 3 exists to enforce.
Gates after: dc-node-v6-carve check --v6-family gua PASS (54 correct, 0 missing, 0
errors); dc-plane-ipam check 29 pass / 2 fail, both expected absences and neither new
(lbaas-mgmt is step 5, oob has no MAAS plane). Was 7 fail before step 2.
Reversible: replace --v6-family ula --commit. Nothing was deleted; both families remain
in MAAS until step 6 retires the ULA rows. vr1-dc1 untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-08-01 |
D-139 step 1 EXECUTED for vr1-dc0: apex GUA carve pushed, 139 -> 152 prefixes
...
netbox/d139-gua-carve.py --dc vr1-dc0 --commit, exit 0. CREATE 13 | EXISTS 3 |
RETIRE-REPORT 9; READ-BACK 13/13; errors=0.
Verified by an independent API query rather than the tool's own read-back: apex count
139 -> 152 exact, 17 GUA rows scoped vr1-dc0 (4 pre-existing + 13 new) with correct
roles including both oob rows. Nothing deleted or orphaned -- the 9 ULA prefixes remain
and all 26 ULA VIP legs still resolve; retirement is step 6.
Prefixes only: no MAAS subnet, node address or VIP was touched. vr1-dc1 not pushed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Conformance: dc0 deploy inputs vs the D-139 specs -- the VIP overlay FAILS
...
Every ruled row of D-139 ruling A, ruling B, the "B plus C" narrowing and the OOB
amendment compared to the ARTIFACT (live MAAS, apex, deploy overlays), not to another
document. Capture: docs/audit/d139-conformance-dc0-20260801.txt.
BLOCKER: overlays/vr1-dc0-vips.yaml carries an R2 sextet per service, and 26 of its 39
v6 VIP legs are ULA (13 metal-admin :220::, 13 metal-internal :221::). Those 26 are
exactly the 26 dependents the step-1 dry run reports inside the retiring ULA /64s --
the same objects, confirmed by description. Deploying after steps 1-3 alone would place
26 VIP legs on prefixes ruling B retires, while their GUA replacements exist in apex and
MAAS but appear nowhere in the deploy input. Overlay must be re-rendered before Step 4.
Also measured: lib-net.sh contains ZERO IPv6, so step 4's "update the v6 arm" has no arm
to update; and no gate compares D-139's tables to any artifact, so this drift class has
no detector. Provider-public conforms in both families; all nine nodes match MAAS (G19).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Capture: Stage-5 preflight dc0 2026-08-01, post-P8-fix (11 FAIL = the ruled-accepted P5 set)
...
Named-check capture for the Stage-5 entry gate, run on voffice1 at HEAD db15666 with
the region profile. P8 now WARNs correctly off-substrate; the 12th FAIL is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

SESSION CLOSE 2026-08-01 (GA-R4 bookend): snap proxy LIVE, D-139, IPv6 proven
...
DURABILITY TRIAD: vcloud 0 uncommitted / 0 unpushed at 844b2e4. voffice1 was 36
commits BEHIND and has been brought current. The dc0 rack's ~/repo-stage (D-138
client input, NO git) was DIGEST-COMPARED not assumed: bundle.yaml and both
non-PKI overlays sha256-MATCH the repo, octavia-pki correctly absent (gitignored).
The provenance gap is real; the drift is not.
GATES: repo-lint 0 fail / 1 warn (legacy carve-out), 649 files; gauntlet ALL GREEN
(96 harnesses); ledger-scan 3 open decisions, 26 open SEC (none opened this
session), next-free D-140 / DOCFIX-207 / BUNDLEFIX-053. RECONCILED: D moved
139->140 matching the one D-number assigned; SEC unchanged matching zero opened.
SWEEP: docs/audit/queued-findings-20260801-stage5-ipv6-d139.txt -- SEVEN FIRST
SURFACE items, each grep-proven absent from every repo surface before writing.
F1 is highest-consequence: the capacity headroom the next deploy needs is HELD BY
TWO IDLE VMs (vvr1-dc0 434 GB RSS, vvr1-dc1 371 GB, ~805 of 1007 GB) with dc0's
nodes powered off and dc1 carrying no model -- and this document's own capacity
record was computed against ALLOCATION, not residency. Also: the voffice1 lag; the
host-lag investigation and its NEGATIVE result; the rack digest MATCH; that no
gitignored permission rules changed; a transient _fmtprobe.tf; and the PreToolUse
guard firing on PROSE containing a guarded command's name.
GA-R7 MEMORY REVIEW: CLEAN -- zero entries claiming operator policy, priority or
posture. One update: instrument-currency gains its NINTH instance in a new shape
-- a "closed" PATH that still held the binaries it was meant to hide. Generalised
as: when a negative depends on an ABSENCE, prove it rather than arranging it.
LEDGER ROTATED (GA-R4 rule 3): stood at 296 lines and this close would have
breached 300. The two oldest closed-session summaries moved VERBATIM to
docs/archive/session-ledger-rotated-20260801.md; ledger now 282. No orphaned
session, so rule 7 is a no-op. NO STAGE OPENED OR CLOSED -- Stage 5 remains OPEN
and this is a session bookend, not a GA-R6 stage close.
OWNED: my BUG-1 fix was wrong; I scoped the v6 experiment wrong (ceph couples
storage+replication); I stated an agent's input source wrongly; I scoped a research
agent with no repo path so 355 lines landed in /tmp and needed rescuing; and one
commit went red on ASCII-only em-dashes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Next steps: nodes released, D-139 execution list replaced, LP draft written
...
(1) TEST NODES RELEASED. t7ymp6 / fg6gxm -> Ready / owner=None, released
individually and read back. Nothing stranded. G19 was run live before the window
closed, which was the whole reason for holding them.
(2) D-139 GAINS A CORRECTION NOTE REPLACING ITS EXECUTION LIST. Neither ruling is
touched. The original list is preserved verbatim inside the note rather than
silently overwritten -- same reason the struck RFC 6724 rationale was kept: a
later reader must be able to see what was wrong or they will re-derive it.
SILENT FAILURE (why this could not wait): dc-node-v6-carve.py assigns v6 "on
every plane WHERE IT ALREADY CARRIES IPv4", keys the subnet off the same MAAS
vlan as that v4 link, derives the host part from the v4 octet, and its loop body
is `if not v4: continue`. Run after v4 removal it carves FOUR FEWER PLANES PER
NODE AND EXITS CLEAN -- and the original list's "remove v4 LAST" wording
actively invited that ordering.
UNSATISFIABLE: "remove v4 LAST, after each is proven" -- per-plane conversion is
atomic, so no such state exists.
UNSAFE STEP FUSION: apex CREATE and RETIRE in one step, with 52 dependent
ip-addresses (the v6 VIP legs) inside the retiring ULA /64s.
Replacement is 7 steps: CREATE-only apex push; MAAS GUA carve ALONGSIDE the ULA;
node statics re-carved WHILE v4 IS STILL PRESENT; Octavia SAN reissue +
lib-net.sh; lb-mgmt carve; re-home the 52 VIPs THEN retire; and v4 removal as a
SEPARATE experimental step scoped to storage + replication TOGETHER per "B plus
C" (together because ceph-osd's prefer-ipv6 sets ms_bind_ipv4=False globally).
Step 7's prerequisite -- network-get on a v6-only bound space -- is stated unmet.
(3) LP DRAFT WRITTEN, NOT FILED: docs/audit/lp-draft-20260801-ceph-osd-ipv6-
static.md. The "C" half of the ruling. A COMMENT on the EXISTING LP #2061836, not
a new bug -- the ceph-mon task is already Fix Committed and only ceph-osd remains
New. Every code claim quoted from the downloaded rev-953 artifact. Claude does not
post to Launchpad; the operator files it (lp-draft-20260721 precedent).
Two things written INTO the draft's operator notes rather than left implicit: do
NOT cite LP #1590598 (Fix Released, a different defect -- the inverted-citation
class); and this bug does NOT block us locally, since setting ceph-public-network
/ ceph-cluster-network to the v6 CIDRs bypasses the buggy call. Overstating our
dependence on an upstream fix would be a bad-faith way to raise its priority.
OWNED: first push attempt went red on L1 -- I used em-dashes in the changelog and
this repo is ASCII-only. Caught by lint before commit, fixed, re-run 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

G19 BUILT and PASSED LIVE; finds a latent defect in D-139's execution list
...
scripts/dc-node-v6-verify.sh (270 lines, 24-line header) + tests/dc-node-v6-verify
(55 cases); manifest 95 -> 96. All three subcommands exercised LIVE this session:
plan vr1-dc0 PASS exit 0 (9 machines, v6 plane set IDENTICAL across all 9, 6
planes, prefixes DERIVED); node vr1-dc0 <spec> on storage-01 against live peer
storage-02 PASS exit 0 with 12 assertions -- six address-presence, six
peer-reachability, every plane "up, global, DAD complete, sole global on the NIC"
and "replies, neighbour DELAY"; bridges vr1-dc0 reporting 7 bridges. Harness
55/55, gauntlet ALL GREEN (96), three mutations each turning a NAMED test red.
RUN AGAINST A REAL NODE while the boot window was open specifically so it would
not ship fixture-green -- the snap proxy's harness was green for a week before
anything was proven end to end.
NO v6 PREFIX LITERAL EXISTS IN THE FILE, and that was a constraint not a nicety:
D-139 retires the ULA /48 for GUA, so a baked table would be right today and wrong
on carve day. plan derives from live MAAS and emits a SPEC; node consumes it and
knows no prefixes. A harness case drives the same green path with a ULA fixture
AND a GUA fixture, so the property is EXECUTED, not asserted.
>>> LATENT DEFECT IN D-139's OWN EXECUTION LIST <<<
D-139 names dc-node-v6-carve.py to "re-carve 54 node v6 statics". VERIFIED against
that script's source: it assigns v6 "on every plane WHERE IT ALREADY CARRIES
IPv4", picks the v6 subnet "on the SAME MAAS vlan as that v4 link", derives the
host part from "the last octet of the node's own v4 address", and its loop body is
a bare `if not v4: continue`. Under D-139 five planes lose v4 and lb-mgmt never
had a v4 twin, so after v4 removal it would SILENTLY CARVE FOUR FEWER PLANES PER
NODE rather than fail. The v6 carve MUST run BEFORE v4 removal, or the tool must
be rewritten -- compounding the already-recorded fact that D-139's "remove v4
LAST" clause is unsatisfiable. LOGGED, NOT FIXED. My own agent mandate stated this
tool's input source WRONGLY (apex/D-136; it is live MAAS); the agent checked
rather than inheriting the error, which is why the defect surfaced.
THREE MORE FINDINGS: (i) the multicast reading is ESTATE-WIDE -- all six plane
bridges plus the WAN bridge at snooping=1/querier=0 on dc0 AND dc1, and the
jumphost's uplinks too, i.e. the default here, not a chosen setting; the
subcommand tags it SUSPECT and asserts no Linux behaviour, correctly, since
cold-start multicast ND was already measured working across it. (ii) plan's PEER
SELECTION IS DEPLOYMENT-BLIND -- it picked control-01, which is not deployed, so
the derived SPEC cannot pass in a PARTIAL deployment; worked around by
substituting the live peer, logged not fixed. (iii) a harness bug caught by the
instrument-currency rule, NINTH of its kind: absent-ip/absent-virsh REFUSE cases
were driven with PATH=/usr/bin:/bin, which still contains both, so they were red
for the WRONG REASON.
TWO OPEN CHECKS SETTLED on the live nodes: every NIC carries exactly ONE global
(sole-global predicate holds today), and an OVS-internal br-ex does report
LOWER_UP (predicate safe).
GA-R2: the build agent wrote its own changelog; folded VERBATIM into this
session's single changelog and removed. Crossing midnight does not start a new
session. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

IPv6 PROVEN WORKING on the dc0 node planes -- and already load-bearing
...
Live experiment, operator-approved: two storage nodes MAAS-deployed (jammy) to
open a boot window, issued individually. Region asserted first (vr1-dc0-region ->
one rack controller hot-kid; admin -> the three Office1 racks); maas-profile-
assert.sh is MISSING on voffice1, so the property was asserted inline not skipped.
A1 ADDRESSES COME UP -- PASS. Six global v6 addresses live on the NICs, matching
MAAS exactly, 0 tentative / 0 dadfailed, on-link /64 routes, no v6 default
(expected). Asserted on the INTERFACE, never on MAAS -- 54 carved statics were
never evidence any existed on a NIC, and this is the first check on a ROLE NODE.
A2 THE PLANE CARRIES v6 -- PASS on all six planes, 0% loss, every neighbour
REACHABLE. THE RISK I FLAGGED IS REFUTED FOR THIS PATH: every plane bridge is
multicast_snooping=1 / querier=0, a known ND failure mode, and the neighbour
table was COLD, so the first solicitation went out as MULTICAST and resolved on
all six. HONEST RESIDUAL: proves ND from cold, not behaviour across long idle
periods where snooping entries age out.
A3 G17's ROLE-NODE HALF CAPTURED -- PASS. Open since 2026-07-30 (that capture was
taken on the CONTROLLER VM and said so). Mirror real package path: exit 0, HTTP
200, 269219 bytes, all four body fields. AND the 2026-07-31 deploy blocker is
confirmed fixed FROM A REAL NODE -- all four suites 200, jammy-backports included,
where it was 404 and put 22 units into hook failed: "install".
A4 HEADLINE, and it inverts this repo's framing: THE NODE'S TIME SOURCE IS ALREADY
IPv6. ServerName fd50:840e:74e2:220::6, ServerAddress family 10 = AF_INET6,
Stratum 3, root distance 57.327ms, poll backed off to 2min8s, ntp.ubuntu.com
unused as fallback. G17 assertion (2) passes, and separately IPv6 is not a future
state here -- it is a live operational dependency predating all this work.
F7 CLOSED IN PASSING: "still unverified that a redeploy applies the new hostname"
-- it does; both report their ruled names from `hostname` on the running OS.
SCOPE: these are ULA addresses, the PRE-D-139 carve, superseded by design. The
PROPERTY tested is family-agnostic and transfers. Nothing here tests a charm, a
container, network-get or Ceph -- those remain the untested rungs.
Nodes deliberately left Deployed so the G19 gate can run against a LIVE node
rather than ship fixture-green. Release owed. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

RULED "B plus C": D-139 ruling A narrowed to an experiment + upstream fix
...
Operator, exact utterance: "B plus C". Ruling B's GUA carve is unaffected and
proceeds; ruling A's five-plane v6-only scope narrows to a bounded experiment;
metal-internal, data-tenant and lb-mgmt stay DUAL-STACK pending upstream fixes.
That last sentence is recorded as an INTERPRETATION, not as the utterance.
>>> I SCOPED THE EXPERIMENT WRONG AND THIS CORRECTS IT. <<<
MEASURED against shipped ceph-osd rev 953, ceph_hooks.py:544-553: prefer-ipv6 is
ONE switch that sets ms_bind_ipv4 = False GLOBALLY, so it governs Ceph's PUBLIC
network (storage) and CLUSTER network (replication) TOGETHER. There is no state
in which Ceph runs cluster on v6 and public on v4. I recommended "replication
alone, smallest blast radius" -- not achievable. The experiment is storage +
replication as a PAIR, with a larger blast radius than I represented.
A PATH THE EARLIER ANALYSIS MISSED: the buggy bare get_ipv6_addr() (LP #2061836,
dynamic/SLAAC-only, never fixed on ceph-osd, while our nodes hold MAAS statics) is
reached ONLY under `if not public_network:` / `if not cluster_network:`. Setting
ceph-public-network and ceph-cluster-network to the v6 CIDRs means it is never
used for address selection -- the defect becomes AVOIDABLE BY CONFIG. This is a
CODE READING, NOT A MEASUREMENT, and it does not address the get_mon_hosts()/
get_host_ip race (LP #2109798).
REVIEW PASS on netbox/d139-gua-carve.py (independent second agent, per operator
standing instruction). 4 real defects fixed, 4 reported, 1 charter miss upheld.
D1 is the mandate's own named class "a refusal that does not refuse": an
unreachable apex gave a raw traceback and EXIT 1 -- the tool's own header defines
exit 1 as a write error -- on a dry run that wrote nothing. _req caught only
HTTPError so URLError walked past the REFUSE handler. Fixed, REFUSE(2).
D2: a comment asserted ruling B was "FLAGGED" -- already false at commit time.
Replaced with a citation; prose in a script rots against governance state, a
citation cannot. D3/D4: two greps over the tool's OWN SOURCE and two vacuous
negatives replaced with behavioural coverage.
main() decomposition DECLINED on a reusable argument: the `if not a.commit:
return 0` guard is ADJACENT to the write block, and extracting commit_writes()
would split that safety proof in two. Line count is not the criterion.
R1 IS A REAL SECURITY FINDING, MEASURED: urllib carries the Authorization header
across a CROSS-HOST redirect (loopback probe -- the target received the token
verbatim). SANDBOX_HOSTS gates the supplied url, not a redirect from it. LOW
severity, logged not executed.
Harness 65/65 -> 71/71; all 5 original mutations RE-RUN and still standing, plus
3 new; gauntlet ALL GREEN (95); dry-run body diffed BYTE-FOR-BYTE identical to the
committed capture; apex 139 prefixes before and after. QUALIFIED: a dry run
executes zero POSTs, so D1's change to the WRITE path's failure semantics is
UNTESTED and is declared, not implied.
PROCESS FAILURE REPEATED AND OWNED: the review mandate required a docs/audit/
capture and forbade touching CURRENT-STATE.md -- jointly unsatisfiable against
L10. The agent correctly left the tree RED and said so rather than working around
it. Standing fix, now twice learned: an agent mandate must name a repo path for
findings AND leave the committer able to satisfy L10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 ruling A NOT achievable at these charm revisions; apex carve tool built
...
TWO agent results, both recorded; nothing actioned, nothing pushed to NetBox.
>>> RULING A (v6-only on five planes) IS BLOCKED BY UPSTREAM CHARM DEFECTS <<<
RULING B (full GUA) IS ORTHOGONAL AND STANDS -- GUA-vs-ULA is independent of
whether a plane keeps v4, so the carve tool is unaffected and a GUA DUAL-STACK
carve is achievable today. This needs an operator ruling.
RE-VERIFIED INDEPENDENTLY against the shipped ceph-osd rev 953 artifact, because
it decides the ruling. hooks/utils.py:203-217 get_host_ip() returns
get_ipv6_addr()[0] when prefer-ipv6 is set, else socket.inet_aton(hostname) and
on failure dns.resolver.query(hostname, 'A') -- an IPv4-ONLY lookup -- under the
charm's OWN comment "...just let it kill the hook". So ceph-osd fails on a
v6-only plane in BOTH directions: false -> inet_aton raises on a v6 literal then
NXDOMAINs an IN A lookup (LP #2109798, New); true -> bare get_ipv6_addr() (also
ceph_hooks.py:547, WITHOUT dynamic_only=False) returns DYNAMIC/SLAAC only while
every VR1 node holds a MAAS STATIC v6 (LP #2061836 -- Fix Committed on ceph-mon,
New and never fixed on ceph-osd, confirmed absent from rev 953).
Reported from the capture and NOT re-verified here, recorded as such: mysql /
mysql-router bare user:pw@addr URIs invalid for an unbracketed v6 literal;
hacluster 2.4/stable rev 166 (released 2026-06-29) still hard-writes
ip_version: ipv4 though LP #2111852 is Fix Committed -- FIX COMMITTED IS NOT FIX
RELEASED; OVN documenting encap as "The IPv4 address of the encapsulation tunnel
endpoint" with zero v6 encap values in OVN 24.03's test suite.
Counted locally: metal-internal appears 273 times in bundle.yaml across ~44 apps;
rabbitmq-server is DEPLOYED by the bundle but ABSENT from PREFER_IPV6_CHARMS --
the only mismatch across all 33 bundle pairs, where invariant 9a would emit a
factually false message.
APEX CARVE TOOL (netbox/d139-gua-carve.py, 232 lines) + tests/d139-gua-carve
(65 cases, offline stub apex); manifest 94->95. Live dry run: CREATE 22 |
EXISTS 6 untouched | RETIRE-REPORT 18, and it surfaced 52 DEPENDENT ip-addresses
inside the retiring ULA /64s that a later delete would orphan. Proof no write
occurred: apex prefix count 139 before and after. 5 mutations, each turning a
NAMED test red, tool restored sha256-identical.
OWED: (a) the foundational measurement is STILL not taken -- does network-get
return v6 on a v6-only bound space? (b) D-139's "remove v4 LAST, after each is
proven" is UNSATISFIABLE as written; per-plane conversion is atomic.
(c) the rabbitmq-server mismatch.
PROCESS FAILURE, OWNED: the research was written to /tmp and would have been LOST
-- I scoped the agent read-only and gave it no repo path. Rescued, sha256-verified.
STANDING CONSEQUENCE: every agent mandate must NAME A REPO PATH for its findings;
"read-only" constrains what it may CHANGE, not where it may RECORD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

The "juju client blocker" is NOT one -- it is D-138 working correctly
...
CORRECTS this repo's own 2026-07-31 finding 9d, which called it a blocker.
Stage 5 can reach `add-model` TODAY, from the dc0 rack. No key movement needed.
VERIFIED INDEPENDENTLY from the main session, not accepted on the agent's word:
juju 3.6.27-genericlinux-amd64 at /snap/bin/juju on the rack; controller
vr1-dc0-controller* admin/superuser on cloud vr1-maas; the controller model reads
"Last connection: just now"; client credentials list EXACTLY ONE entry,
vr1-maas -> vr1-dc0-cred.
ROOT CAUSE OF THE ssh REFUSAL, MEASURED ON THE MACHINE: the alias is correct in
every part -- hop chain, host, port, per-hop identity, host key. subtle-grouse's
authorized_keys holds exactly two lines, both Juju's, so the office1_svc key the
alias offers is genuinely absent and the refusal is right. The obvious hypothesis
was REFUTED rather than assumed: the machine booted AFTER the 2026-07-30 key
import and still got only Juju keys, because deploy-time cloud-config carries
Juju:juju-client-key alone. `juju ssh -m controller 0` already works.
SEC-026 control (1) DISCHARGED BY MEASUREMENT on both sides -- rack lists one
credential, voffice1 lists two, so the forbidden whole-store copy did not happen.
No new credential residency; no new security-ledger row owed.
ONLY HYGIENE OWED, NOT EXECUTED (hard rule 3): a dangling current-model pointing
at the model destroyed 2026-07-31 makes bare `juju status` error. Measured NOT to
be an add-model precondition.
FOUR GAPS LOGGED NOT FIXED, in consequence order:
G4 the `openstack` CLI is MISSING ON THE RACK -- D-138 definition-of-done gap
blocking phase-03+. SAME item as F1 from the 2026-07-30 sweep, now confirmed
on the D-138 host. It has survived two sweeps.
G3 preflight CANNOT PASS on vcloud by construction: the octavia-pki overlay is
gitignored PKI material absent from the vcloud tree, and
pre-flight-checks.sh:161 hard-fails on it. The host-dependence class again.
G2 the D-138 client host's deploy input has NO PROVENANCE -- ~/repo-stage on the
rack is a hand-staged 9-file copy with no .git (8/9 byte-identical by sha256).
A third copy beside the two clones verified at c58bf95.
G1 phase-4 runbook :379,398 still say "voffice1" for add-model/spaces, stale vs
D-138; voffice1 has the binary but no registered controller, so the runbook as
written fails. DOCFIX owed.
DECLARED UNMEASURED with the reason: whether MAAS user juju-vr1-dc0 has an
imported ssh key -- no usable maas CLI profile exists anywhere (voffice1's
~/.maascli.db is ZERO BYTES). Does not change the fix; the proximate cause was
measured directly. The PreToolUse guard refused a command that would print the
MAAS API key (DOCFIX-016) and it was NOT retried in an altered shape.
Staged by explicit path; agent files still in flight are deliberately excluded.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 ruling B's deciding reason REFUTED by measurement; ruling FLAGGED
...
Closing the unverified premise D-139 named at ruling time inverted its own
argument. Recorded, flagged for operator reconsideration, NOT amended.
MEASURED, two steps:
1. jammy libc-bin 2.35-0ubuntu3 ships /etc/gai.conf with ZERO active lines --
every line commented. The node image applies NO override, so glibc's
compiled-in table governs.
2. glibc 2.35 default_precedence[], fetched in full from sourceware at tag
glibc-2.35 and read rather than recalled, carries glibc's OWN comment
"See RFC 3484 for the details" and is:
::1/128 50 | 2002::/16 30 | ::/96 20 | ::ffff:0:0/96 10 | ::/0 40
THERE IS NO fc00::/7 ENTRY IN THE PRECEDENCE TABLE.
CONSEQUENCE: ULA falls through to ::/0 = 40, identical to GUA; IPv4-mapped is
10. So on the node image ULA = GUA = 40 and BOTH outrank IPv4. Ruling B was
declared on "ULA loses to IPv4 (3 vs 35), GUA wins (40 vs 35)" -- the RFC 6724
table. RFC 6724 obsoletes RFC 3484 and does define those values, but glibc 2.35
DOES NOT IMPLEMENT IT. Right about the RFC, wrong about the software this cloud
runs. The fc00::/7 entry that does exist is in default_labels[] (label 6), which
drives SOURCE selection rules 5/6 -- a different mechanism, recorded so the two
are not later conflated into "the argument stands after all".
A NEW VARIANT OF THE CITATION CLASS, worth naming: the citation was real,
current and correctly quoted, and still wrong here because nobody checked
whether the IMPLEMENTATION follows the RFC. Checking a source's status and dates
is not sufficient for a standards citation.
SURVIVING REASONS (independent of the refuted one): conformance with Willamette
(a real site) and VR0 DC0, both full GUA; and MINIMIZE DELTA TO ROOSEVELT, whose
2602:f3e2:103::/48 is GUA with no ULA and no plane carve yet. The ruling may well
stand on these -- but under GA-R5 that is the operator's call, since they
answered a question whose stated deciding reason no longer holds.
D-139's text is UNCHANGED. Nothing of D-139 is executed, so nothing is half-built
in either direction. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

D-139 ADOPTED: VR1 goes IPv6-only east-west, whole carve moves to GUA
...
TWO GA-R5 rulings, taken in SEPARATE exchanges, recorded separately.
A -- family matrix. Operator: "IPv6 on all planes except for metal-admin and
provider-public which will remain dual stack". metal-internal, data-tenant,
storage, replication and lb-mgmt become IPv6-ONLY. Amends D-101's matrix twice:
metal-internal's "datastore east-west stays v4-bound" is superseded, and lb-mgmt
(ruled v6-only by D-101 but NEVER carved anywhere) becomes a first-class plane.
B -- addressing model. Operator: "Full GUA on every plane (Recommended)". Every
plane carves from its DC's GUA /48 on the :10/:11/:20/:21/:30/:40/:50/:80 octet
map VR0 DC0 and the Willamette site ALREADY use -- conforming to an existing org
standard, not inventing one. The ULA /48 fd50:840e:74e2::/48 is RETIRED for VR1.
Amends D-101 and D-111. Deciding reason is measurable: RFC 6724 ranks IPv4-mapped
at precedence 35 and ULA at 3 while GUA falls under ::/0 at 40, so on a dual-stack
plane a ULA leg LOSES address selection to IPv4 and is decorative. SEC-010, D-052,
D-125 and D-107 unchanged -- containment lives at the forwarding layer.
THE RECORDED ROOT CAUSE OF THE v4-ONLY CONTAINERS WAS WRONG AND THIS REPO CARRIED
IT (GA-R1 C2 -- measurement corrects the document). CURRENT-STATE said MAAS "has
nothing to give for v6". Measured, and re-verified independently: subnet
statistics on fd50:840e:74e2:220::/64 returns available_string "100%",
num_available 18446744069414584320. Zero ipranges rows means zero RESTRICTIONS,
not zero availability. dc-plane-ipam.sh:368-372 has carried the correct behaviour
since 2026-07-27, in the tree, contradicting the authoritative doc the whole time.
The real mechanism is juju-side and there is NO knob: EthernetDeviceForBridge
(tag v3.6.27) takes addrs[0] from an UNSORTED query and derives one CIDR -> one
LinkSubnetArgs. gomaasapi documents "Any number of STATIC links can exist on an
interface", so MAAS would accept both families; the limit is juju's. LP #1723240
is this exact symptom, Triaged/Low since 2017, and the theory this repo recorded
is the one its reporter rebutted in-thread. Consequence: on a container plane
DUAL-STACK IS NOT EXPRESSIBLE while v6-only IS -- ruling A is the achievable
configuration, not merely the desired one.
SNAP PROXY: dc0 INSTALLED, RUNNING, check dc0 PASS exit 0 (16 assertions,
re-verified independently). A REAL SNAP PAYLOAD WAS FETCHED THROUGH IT -- HTTP 206,
first bytes hsqs. No longer fixture-green. BUG-3 confirmed by measurement, no
assertion changed. BUG-4 fixed at BOTH sites (the review named one); T20b/T41b NEW.
OWNED: my BUG-1 fix was WRONG. ip route get 10.12.8.4 -> src 10.12.8.2; .4 is a
SECONDARY and Linux never auto-selects a secondary as a source, so the rule is
still dead. Not fixed now -- editing the generator turns the LIVE gate red until
install re-runs. T23b annotated so nobody "fixes" the test to a wrong value.
Generalisable: any ACL keyed to a service ALIAS (the whole D-134 utility .4 band)
misses that host's own traffic -- same exposure in dc-mirror.sh, dc-cache-proxy.sh.
OWED, unverified: the deployed jammy /etc/gai.conf, which can override the RFC 6724
default table that ruling B rests on. Nothing of D-139 is executed.
repo-lint 0 fail; tests/dc-snap-proxy 55/55; 4 mutations killed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-31 |

SESSION CLOSE 2026-07-31 (GA-R4 bookend): Stage 5 dc0 deploy + teardown
...
Bounded ledger summary (10 lines, cap 15), rotation, sweep, memory review.
DURABILITY: 0 uncommitted / 0 unpushed, 45 commits today. dc0 rack repo-stage
digests all MATCH. HAZARD, not loss: the voffice1 clone is 26 commits behind --
pull it forward before it is used.
GATES: repo-lint 0 fail / 1 standing warn / 641 files. Gauntlet ALL GREEN (94).
ledger-scan: 3 open decisions, SEC 26, D 139 / DOCFIX 207 / BUNDLEFIX 053.
Ledger 296 lines after rotating the oldest summary out to
docs/archive/session-ledger-rotated-20260731.md.
SWEEP: docs/audit/queued-findings-20260731-stage5-deploy.txt -- ELEVEN FIRST
SURFACE items that existed only in the transcript. Highest-consequence is F1:
MAAS's own squid ALREADY CONNECT-proxies api.snapcraft.io and would unblock the
deploy today with nothing built, measured from the failing container -- RULED
AGAINST, with the operator's verbatim utterance and the deciding downsides
recorded, precisely because a future session will rediscover it and reach for
it. Also first surface: containers source from METAL-ADMIN not the address juju
status displays; ports 3128 and 8000 already held by MAAS's squid on both racks;
the four reviewed bugs in dc-snap-proxy.sh; the teardown instrument error; and
that a MAAS hostname rename on a Deployed machine is record-only.
Method note: a phrase-exact grep produced false negatives on items recorded under
different wording. Every candidate was re-grepped on distinctive tokens before
being classified -- the same instrument-error class this sweep documents.
GA-R7 MEMORY REVIEW -- one real violation found and corrected: the
multi-workstation memory asserted 'never additions to allow for mutations', an
operator-posture claim memory may not hold, contradicted by the recorded
2026-07-30 ruling 'Add it to allow'. Re-pointed to an observation with the
contradiction recorded. instrument-currency memory gained this session's two
misreads.
OWNED: a red-lint push (| tail masked the exit code); a git add -A that swept an
agent's files into an unrelated commit; two instrument misreads (a juju models
summary read as progress, stale unit logs read as current); and a wrong
'immutable system_id' claim the operator corrected.
NEXT: apply the dc-snap-proxy review cuts, RE-RUN the mutation pass (S8
invalidates M32-M39/M42-M47/M49-M52 as evidence), confirm BUG-3's dstdomain deny
shape once at apply time, install the proxy, then add-model + spaces + deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Snap proxy tool BUILT and REVIEWED (not yet applied); 4 real bugs logged
...
Ruling 2's repo side. squid 6.14 as a dedicated instance on each DC rack at the
D-134 utility .4:3129, CONNECT-only, restricted to Canonical's documented store
and CDN hosts, consumed via juju snap-http-proxy/snap-https-proxy. Design cited
to vendor sources, not preference: every snap endpoint is :443 so a CONNECT
proxy is mandatory and CACHING IS IMPOSSIBLE (squid cannot read a CONNECT
tunnel), which removes the apt-style caching argument entirely; squid is in
main, tinyproxy is universe with six unevaluated CVEs and defaults to allowing
all CONNECT ports.
MEASUREMENTS THAT CHANGED THE DESIGN:
- the failing apps are LXD containers sourcing from METAL-ADMIN (10.12.8.122),
NOT the 10.12.12.116 juju status displays. An ACL on the displayed address
would have denied every client the proxy exists for.
- ports 3128 AND 8000 are already held wildcard-bound by MAAS's own squid on
both racks -- hence 3129 and a dedicated instance.
- no new D-134 octet needed: .4 is already aliased on both racks.
- LOGGED NOT ADOPTED: MAAS's squid ALREADY CONNECT-proxies api.snapcraft.io,
measured working from the failing container. snap-https-proxy pointed at
10.12.8.6:8000 may unblock the deploy today with nothing built. Rejected as
the shipped design (per-revision path, NO destination restriction, the
hidden-coupling shape D-135 already has a scar from) but it is a real
operator option, not a dismissal.
QUALITY: 53 mutations, 53 killed, 0 survived -- including the dc-mirror.sh
existence-vs-content false-green class reproduced and caught. One survivor was
a MALFORMED mutation that broke bash -n and therefore tested nothing; recorded
as malformed and redone rather than counted. Three of the harness's own cases
were found too weak and were STRENGTHENED, not left green.
INDEPENDENT REVIEW: ruled design HONOURED on all four counts -- utility band, no
direct node egress (in fact stronger: destination-allowlisted, and that scope is
flagged as a build-time choice NOT a ruling), site-keyed both DCs with the dc1
limit stated honestly, and it OWNS its own network prerequisites rather than
borrowing another service's -- with the coupling anti-pattern now under test
(M09/M10) rather than only under comment.
FOUR REAL BUGS LOGGED, NOT FIXED (hard rule 1 -- the file was frozen for review):
BUG-1 dead snap_probe ACL; the probes are actually admitted by snap_clients,
and the comment misdescribes its own mechanism
BUG-2 | grep -q under pipefail -- INHERITED from dc-mirror.sh and
dc-cache-proxy.sh alike; a three-script sweep, not a local fix
BUG-3 the deny probe's 403 shape was measured from a PORT deny, never a
DSTDOMAIN deny. If squid answers differently the gate REFUSES
PERMANENTLY -- and a gate that cannot PASS blocks a GA-R6 close as hard
as one that cannot fail. Requires a one-time apply-time confirmation.
BUG-4 dpkg -s reports a deinstall-ok-config-files package as present
SIZE: 636 lines / 320 code vs dc-mirror 385/227 and dc-cache-proxy 396/200. The
reviewer's landing zone is ~535/~250 and it RECOMMENDS AGAINST forcing 385 --
two extra generators (75 lines) are measured-forced because the packaged squid
cannot be repointed, and further cuts would delete mutation-proven assertions.
~50 lines of the excess is header prose, which carries zero bug risk.
TWO BLOCKERS BEFORE THE PLAN IS APPLIED: the S8 consolidation invalidates
M32-M39/M42-M47/M49-M52 as evidence, so the mutation pass MUST be re-run; and
BUG-3 must be confirmed once against the built proxy before check is cited to
close anything.
NOTHING IS PROVEN END TO END -- no proxy installed, no snap fetched through one.
The harness green is FIXTURE green and says so.
repo-lint 0 fail; gauntlet ALL GREEN (94).
OWNED: my earlier teardown commit e57ad09 used 'git add -A' while the agent was
writing and swept its audit capture and CURRENT-STATE paragraph into a commit
about the model teardown. This commit is staged EXPLICITLY by path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Teardown: stalled on stopped agents, forced clear, no nodes stranded
...
Plain destroy-model STALLED and would not self-resolve -- 'attempt 30 ... model
not empty, found 26 machines, 37 applications', flat ~19 min with the app set
byte-identical across a 12-minute name-level diff.
MECHANISM MEASURED, and it makes the stall terminal rather than slow: ALL 26
machine/container agents were 'stopped', so NO hook could execute. Units already
in error from the snap failures could never run their teardown hooks.
MAAS was NOT the bottleneck -- the six nodes juju released went to Ready/owner
None in minutes; juju never issued a release for the other three.
destroy-model --force --no-wait cleared it (18 -> 5 -> 2 machines, then 'Model
destroyed.'), and NO NODES WERE STRANDED: all nine read back Ready/owner=None,
so no maas machine release was needed or run. subtle-grouse correctly stays
Deployed -- it is the D-104 controller VM in the controller model.
INSTRUMENT ERROR, OWNED: I reported '0 machines / 0 units' from the juju models
SUMMARY COLUMNS while juju status -m read 26 machines / 37 applications. The
summary zeroes during 'destroying' and is not a progress signal. The real tell --
three control nodes stuck Deployed -- was visible and I explained it away.
Standing lesson: during a teardown, juju status -m <model> is the instrument;
juju models counts are not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Three post-deploy defects; memcached scaled; ceph-rbd-mirror PINNED (item 22)
...
D1 jammy-backports: RULED + FIXED. Verified on CONTENT (Release fields + a real
.deb fetch), mirror 951G -> 952G. Units moved 22 error -> 17 with 15 apps past
install (was 1), so the fix works and auto-retry is running.
D2 the UPSTREAM UCA is unreachable from nodes -- LOGGED NOT FIXED. Measured FROM
A NODE, not the rack: upstream 000, mirrored 200 with real content. SIX apps set
it across TWO option names (openstack-origin on barbican/magnum/octavia, source
on ceph-mon/ceph-osd/ceph-radosgw); every other charm defaults to
openstack-origin: caracal, same upstream pocket. The UCA signing key is already
on the node, so a raw deb line verifies with no |key suffix. Genuine D-135
experiment result: the full-mirror DC must rewrite every non-Ubuntu source; the
proxy DC needs none. dc1 will not hit this.
D3 prefer-ipv6: true is FATAL on the seven charms that declare it, and is now
the blocker. This is the exact risk flagged earlier today as "never observed
live" and deliberately left OUT of the D-101 ruling note. keystone/0:
"Interface 'eth0' does not have a scope global non-temporary ipv6 address" --
get_relation_ip() returns early with get_ipv6_addr() when the option is true.
Measured on the container: eth0 holds only fe80::/64. The NODES are dual-stacked;
the LXD CONTAINERS are not. All seven true-valued apps are erroring. NOT TAKEN --
R2/D-101 are ruled surfaces and this needs its own GA-R5 exchange. One question
is stated and NOT answered: whether a v6 VIP on a container with no global v6 is
routable.
memcached scaled 1 -> 3 in dc-ha-scaleup (operator-directed); its exclusion
comment re-pointed rather than left stale. ceph-rbd-mirror PINNED as gap
register item 22 with four options and a recommendation -- (d) add the missing
liveness detection now, (b) scale to 2 active/standby at Roosevelt -- and the
overlay now forbids a silent scale without a D-108 amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

>>> BUNDLE DEPLOY SUCCEEDED: "Deploy of bundle completed." exit 0 <<<
...
Option D half 2. The 23 application definitions were removed INDIVIDUALLY,
never batched (hard rule 3), each read back, with the precondition re-verified
immediately before the first removal and the model read back empty afterwards.
Step-3.5 state survived the clear -- apt-mirror still set, all six spaces still
bound with both address families, checked not assumed.
MEASURED IMMEDIATELY AFTER: juju 56 applications / 9 machines / 33 units,
machines pending+allocating; MAAS 10 machines -- 9 Deploying, 1 Deployed (the
controller).
Three defects closed in the order they were hit: attempt 1's unknown option
prefer-ipv6 (D-101 ruling note); attempt 2a's missing policies/overrides.zip on
the client host; attempt 2b's barbican 265-vs-261 downgrade refusal (explicit
base on all 56 AND a clean model).
Operational note, because the first removal read as a failure: juju
remove-application PROMPTS by default and aborts on non-interactive stdin.
--no-prompt is required.
WHAT IS NOT CLAIMED: this means juju ACCEPTED and QUEUED the bundle. It does
NOT mean the cloud is up. 9 machines are allocating, 33 units pending; the
settle takes hours and nothing here asserts unit health or any service verdict.
G17's dc0 half is now genuinely armable -- the existing capture was taken on
the CONTROLLER VM and says so. The nine role nodes are booting for the first
time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Deploy attempt 2: prefer-ipv6 fixed, two new defects, model partially populated
...
The prefer-ipv6 failure did NOT recur. Two new ones did, and for the first time
the model is not empty: MEASURED 23 applications, 0 machines, 0 units.
Attempt 1 aborted during VALIDATION; this one reached EXECUTION, which is why
it left residue. NOTHING IS PROVISIONED -- no MAAS machine left Ready, no disk
written.
DEFECT 1: the rack's ~/repo-stage is a PARTIAL COPY of the repo with no
policies/, and bundle.yaml:214 is the only local-file reference in the deploy
input. juju resolves it relative to the bundle, so it must exist ON THE HOST
THAT DEPLOYS. Nothing caught it: --dry-run does not upload resources (the
SECOND --dry-run blind spot in two days), attempt 1 aborted before resource
upload, and preflight runs on voffice1 where policies/ DOES exist -- the same
wrong-host instrument class already recorded twice, and the second instance
this session. Fixed forward, both files sha256-verified.
DEFECT 2: a bundle relying on default-base is deployable exactly ONCE. The
re-run failed on barbican 265-vs-261. Measured: 265 is the ubuntu@22.04
revision, 261 the ubuntu@24.04 one. The 23 existing apps are all 22.04, so run
2a honoured bundle.yaml:85 -- but on a re-run, for an app that ALREADY EXISTS,
juju resolves without that default. A model-level default-base has NO EFFECT
(tested, then RESET). An explicit per-application base WORKS (tested read-only
on a throwaway; the error moved to the next app).
OWNED: two of those read-only tests were staged under /tmp and failed with
"no charm was found" -- the juju snap's PRIVATE /tmp, a trap this repo has
already recorded in CURRENT-STATE and the phase-4 runbook.
The rollback decision tree is written for tofu apply and defaults to
fix-forward, which is measured not to work here. A partial juju deploy is a
failure mode the repo does not cover. Recovery is an operator decision;
options are enumerated in the capture and the model residue is untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Pre-deploy: rack input refreshed + hash-verified, preflight captured, dry-run green
...
(a) The dc0 rack's ~/repo-stage is a COPY, not a clone -- nothing updates it, and
it still held the pre-ruling vips overlay. Exactly one of four files differed;
copied, then ALL FOUR hash-compared against repo HEAD. The gitignored
octavia-pki overlay was deliberately not touched (SEC-029) and verifies
byte-identical at 5fc117f1, still 0600. A sweep is evidence about the deploy
only if the swept bytes ARE the deployed bytes.
(b) Preflight captured (242 lines, exit 1), run with
MAAS_PROFILE=vr1-dc0-region -- without it preflight is REGION-BLIND and emits 19
false negatives. Instrument proven current first: maas-profile-assert.sh
vr1-dc0-region hot-kid exits 0. P1-P4 and P7 PASS, incl. aligned VIPs 13 OK / 0
bad against the RE-RENDERED overlay. P5 FAIL, 11 findings.
(c) The P5 delta is ENUMERATED. The 2026-07-30 acceptance covered six findings
and says it covers "these six, enumerated, and nothing else". All six are still
present; FIVE NEW are all vr1-dc1 S2 rows (SEC-027/-028) -- the D-137 forward
register correctly reporting that dc1's region is not built. Diff appended to
the capture. Needs its own GA-R5 exchange before the deploy.
(d) Step 4.2 --dry-run against the fixed input: exit 0, 56 apps / 108 relations
/ 33 unit placements. The ceph-osd tags=openstack exposure is now MEASURED, not
reasoned: the plan places ceph-osd/0..3 on machines 5-8 by explicit id, so the
absent tag never has to match. The residual (a later UNPLACED add-unit) stands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Sweep both deploy inputs for option NAMES and VALUE TYPES: clean, and proven able to fail
...
The two extensions owed after the ruling, now run:
dc0 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
dc1 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
Both EXIT 0. dc1 introduces no new charm@channel pair -- checked, not assumed.
PROOF OF TEETH, because a clean sweep across two inputs is exactly the
implausibly-uniform result this repo has been burned by. Three defects planted
in a THROWAWAY copy, all three caught, EXIT 1:
[FAIL name] barbican does not declare option 'prefer-ipv6' (attempt-1 defect)
[FAIL type] keystone 'prefer-ipv6': declared boolean, got str ('true')
[note] glance 'vip': declared string, value is unquoted float
Type judgement is deliberately conservative: a mismatch is reported only where
juju's own coercion cannot save it. A scalar where a STRING is declared is a
NOTE, not a failure -- but it is still surfaced, because an unquoted value is
how a version string silently becomes a float. An UNRECOGNISED declared type
REFUSES rather than passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|