queued-findings-20260801-stage5-ipv6-d139.txt
=============================================
Close sweep for the 2026-07-31/08-01 session (snap proxy apply + D-139 IPv6).
Method (ruled 2026-07-31): read back over the whole session, enumerate every
finding/decision/measurement/mistake, then GREP each candidate against repo
surfaces. An item with a hit is ALREADY ON SURFACE and says where. An item with
NO hit is FIRST SURFACE and would have been lost on a context clear.
Session body: docs/changelog-20260731-snap-proxy-apply-ipv6.md (20 items).
Status claims: docs/CURRENT-STATE.md ONLY. Nothing here is a status claim.
--------------------------------------------------------------------------------------
FIRST SURFACE -- these existed ONLY in the transcript. Listed first, by consequence.
--------------------------------------------------------------------------------------
F1. >>> THE CAPACITY HEADROOM THE NEXT DEPLOY NEEDS IS CURRENTLY HELD BY TWO IDLE
VMs, AND NOTHING SAYS SO. <<< greps: "434 GB" 0 hits, "idle VMs" 0, "balloon"
0, "free-page" 0, "host memory" 0.
MEASURED on vcloud 2026-08-01: vvr1-dc0 RSS 434 GB, vvr1-dc1 RSS 371 GB --
~805 GB of the host's 1007 GB -- while dc0's nine nodes were POWERED OFF and
dc1 has no deployed model at all. qemu does not return guest-freed pages to the
host without free-page-reporting or a balloon deflate. Swap is 6.8 GB of 8 GB
used and 6.887 GB of that is ONE process, vvr1-dc1; si/so measured 0, so it is
parked, not thrashing.
WHY IT MATTERS AND WHY IT IS NOT MERELY TRIVIA: CURRENT-STATE:411 records the
capacity gate passing at "RAM 870/1024 = 85%, FIT, 154 GiB headroom". THAT
HEADROOM IS THE THING CURRENTLY OCCUPIED. A future session re-running
dc-dc-whole-host-budget.py before a deploy will read a pass that was computed
against allocation, not against resident usage.
LOGGED, NOT EXECUTED (hard rule 1): reclaiming it is a live mutation on running
DC VMs and is an operator call.
F2. voffice1's CLONE IS 36 COMMITS BEHIND UPSTREAM. greps: "36 commits behind" 0
hits, "behind upstream" 0 hits.
MEASURED at close: voffice1 HEAD = fbe7b31 ("Build the 2026-07-31 ruling:
renderer + invariant 9 replaced, overlays re-rendered"); upstream is 36 ahead.
Surfaced during the session as a symptom rather than a cause -- scripts/maas-
profile-assert.sh was ABSENT there, so the repo's own wrong-region gate could
not be run and the property had to be asserted inline by rack-controller
identity instead. That inline assertion PASSED, so nothing was done against the
wrong region; but the gate being unavailable on the Plane-2 host is the point.
This is the 2026-07-27 stale-clone hazard, live, on the host D-128 designates
for MAAS/NetBox/inner-tofu work. NOT A LOSS -- the remote holds everything --
but a hazard, and the fix is one `git pull` the next session must not skip.
F3. THE HOST-LAG INVESTIGATION AND ITS NEGATIVE RESULT. grep: "tmux" 0 hits.
Operator reported the tmux session lagging badly and asked whether anything
local caused it. MEASURED: load 1.74 on a 108-vCPU host; vmstat us/sy 0/0, id
99-100%, wa 0, si/so 0/0 across a live 5s sample; 198 GB available + 205 GB
cache; disk 19% used with 7.7 TB free; tmux panes ~3 MB against a 50000-line
limit and 0.0% CPU; all Claude processes ~1.7 GB RSS combined.
VERDICT: nothing local was causing it -- the host was essentially idle, and the
lag is on the path or the client. RECORDED BECAUSE A NEGATIVE RESULT IS STILL A
RESULT: without this, the next person to see lag re-runs the same investigation.
STATED LIMIT AT THE TIME AND REPEATED HERE: the sample was 5 seconds, so an
INTERMITTENT burst would look exactly like this. If the lag recurs, sample long.
F4. THE RACK'S STAGED DEPLOY INPUT IS DIGEST-IDENTICAL TO THE REPO. (Partial --
"repo-stage" itself has 9 hits, but the MATCH is a new measurement.)
G2 (2026-08-01) recorded that ~/repo-stage on the dc0 rack is a hand-staged
copy with NO .git, i.e. no provenance. MEASURED AT CLOSE, which is the half
that was missing: bundle.yaml sha256 4c8a7852... MATCHES the repo exactly, and
so do overlays/vr1-dc0-machines.yaml (e3be85e4...) and overlays/vr1-dc0-vips
.yaml (ed19d989...). vr1-dc0-octavia-pki.yaml is absent from the repo, which is
CORRECT -- it is gitignored PKI material.
So the D-138 client host's deploy input is CURRENT, not drifted. The provenance
gap remains real; the drift did not happen. Both halves belong on the record --
G2 alone reads worse than the measurement warrants.
F5. NO GITIGNORED PERMISSION RULES WERE ADDED THIS SESSION.
MEASURED at close: .claude/settings.local.json holds allow=287, ask=11, deny=0
-- byte-for-byte the same counts read at session OPEN. The verbatim record made
by the 2026-07-30 sweep (F1 there) and carried by the 2026-07-31 sweep therefore
STANDS UNCHANGED and is still the recovery copy for this gitignored file.
Recorded because "no change" is only knowable if someone checks; an unchecked
session leaves the next one unable to tell.
F6. A TRANSIENT UNTRACKED FILE APPEARS DURING repo-lint. grep: "_fmtprobe" 0 hits.
Observed mid-session: opentofu/modules/base-image/_fmtprobe.tf appeared in
`git status` and had vanished by the time it was inspected -- almost certainly
a `tofu fmt` probe repo-lint writes and removes. Harmless in itself. RECORDED
because a stray untracked .tf appearing in a window where a concurrent session
might run `git add -A` is exactly the shape of the e57ad09 contamination
incident. LOW consequence; noted, not chased.
F7. THE PreToolUse GUARD FIRES ON PROSE, NOT ONLY ON COMMANDS.
At close, a `python3 - <<'PY'` heredoc whose TEXT contained the literal string
"maas list" -- inside a memory-file paragraph being appended, not as a command
-- was BLOCKED by .claude/hooks/guard-destructive.py with "prints the MAAS API
key (DOCFIX-016); use 'maas admin ...' directly". The guard matched the shell
invocation's full text, which included the quoted prose.
NOT A DEFECT IN THE GUARD'S INTENT -- matching broadly is the safe direction,
and the guard has fired correctly twice this session. Recorded because the
RESPONSE matters and will recur: the correct move was to write the file with
the Write/Edit tools, changing the CONTENT and avoiding the shell, rather than
re-issuing the same shell command in a disguised shape. Re-shaping a command to
slip past a guard is the behaviour the rule against retrying refused commands
exists to prevent, and "the guard was wrong" is exactly the reasoning that makes
it feel justified.
CONSEQUENCE FOR FUTURE SESSIONS: writing documentation ABOUT a guarded command
can be blocked by the guard for that command. Use the file tools for prose.
--------------------------------------------------------------------------------------
ALREADY ON SURFACE (grep-verified; recorded for completeness)
--------------------------------------------------------------------------------------
A1. Snap proxy installed, check dc0 PASS exit 0, real snap payload (hsqs) fetched
-- CURRENT-STATE + changelog items 6/7, docs/audit/dc0-snap-proxy-install-*.
A2. BUG-4 fixed at BOTH sites; BUG-3 confirmed; BUG-1's fix WRONG (a secondary
IPv4 alias is never the kernel's chosen source) -- items 2/3/7/8, T23b in-file.
A3. D-139 ADOPTED (rulings A and B, separate exchanges, exact utterances) --
design-decisions.md, CURRENT-STATE, item 15.
A4. RFC 6724 refutation + ruling B reconsidered and CONFIRMED; struck rationale
kept in place -- design-decisions.md strike block + RULING NOTE 2026-08-01,
docs/audit/gai-conf-rfc6724-verification-20260801.txt, items 10/11.
A5. Juju/container single-address mechanism, no knob, LP #1723240 -- CURRENT-STATE,
docs/audit/v6-only-charm-viability-20260801.txt.
A6. D-139 ruling A not achievable at current charm revisions (ceph-osd both
directions, mysql URIs, hacluster ip_version, OVN encap) -- item 14 + capture.
A7. ceph couples storage+replication via ms_bind_ipv4=False; ceph-*-network CIDRs
bypass the buggy call -- item 16, CURRENT-STATE.
A8. IPv6 PROVEN on the node planes; NTP already running over IPv6; jammy-backports
serving; F7 (redeploy applies hostname) closed -- item 18, g19-ipv6-node-plane-
verify-20260801.txt.
A9. G19 built and PASSED LIVE; dc-node-v6-carve.py's v4 pivot; plan's peer
selection deployment-blind; the ninth instrument-currency harness bug --
item 19, D-139 CORRECTION NOTE, g19-ipv6-plane-gate-build-20260801.txt.
A10. multicast_snooping=1/querier=0 is ESTATE-WIDE (both racks, WAN bridge, jumphost
uplinks) and cold-start multicast ND works across it anyway -- CURRENT-STATE,
both g19 captures.
A11. Apex IPv6 plan pulled live; zero v6 ip-ranges; v6 records are VIP-only; VR1
diverges from VR0/Willamette -- docs/audit/apex-ipv6-plan-20260731.txt.
A12. The carve tool + its independent review (D1 "a refusal that does not refuse",
R1 Authorization header across a cross-host redirect) -- items 13/17, d139-
carve-review-20260801.txt.
A13. The juju "blocker" is D-138 working correctly; G1-G4 -- item 12 + capture.
A14. Nodes released to Ready/owner=None; D-139 execution list replaced; LP draft
written not filed -- item 20, lp-draft-20260801-ceph-osd-ipv6-static.md.
A15. Operator approval of the agent's rack permissions, with its scope limits --
item 5.
--------------------------------------------------------------------------------------
CONTRADICTION DETECTOR -- measured this session vs standing documents
--------------------------------------------------------------------------------------
C1. CURRENT-STATE's recorded root cause of the v4-only containers ("MAAS has
nothing to give for v6", "MAAS answers v4 from the dynamic range") is REFUTED by
measurement (100% available; the v4 container address sat OUTSIDE the dynamic
range). CORRECTED in the same document per GA-R1 C2. RESOLVED.
C2. D-139 ruling B's stated deciding reason (RFC 6724 precedence) is REFUTED for
glibc 2.35, which implements RFC 3484. Ruling reconsidered and CONFIRMED on its
other grounds; rationale STRUCK IN PLACE. RESOLVED.
C3. D-139's own EXECUTION list contained a silent-failure ordering. REPLACED by a
correction note. RESOLVED.
C4. scripts/dc-plane-ipam.sh:368-372 has carried the CORRECT MAAS reserved-range
behaviour since 2026-07-27 while CURRENT-STATE carried the wrong one. The doc is
now fixed; the script was right all along. RESOLVED, recorded as a reminder that
a script comment can outrank the status document.
C5. STILL OPEN, unchanged from the 2026-07-31 sweep: preflight P6 quotes "50 apps /
97 relations" against a MEASURED 56/108; G17's text names `chronyc`, absent from
the MAAS jammy image (RE-CONFIRMED on a live node this session); the phase-4
runbook omits the machines overlay from the dc0 deploy block and still labels
add-model/spaces "voffice1" against D-138. All DOCFIX material, none numbered.
--------------------------------------------------------------------------------------
DELIBERATELY NOT DONE
--------------------------------------------------------------------------------------
N1. The dc-snap-proxy simplification (S-plan) was NOT re-derived -- the plan was lost
at a session boundary and re-deriving it is the GA-F06 record-churn case.
Consequence: the prior 53/53 mutation evidence STANDS for untouched assertions.
N2. BUG-1's real fix (probe ACL -> the rack's PRIMARY metal-admin leg) NOT applied:
editing gen_squid_conf turns the LIVE dc0 gate red until install re-runs. To land
WITH the dc1 install so both DCs move together. T23b is annotated so nobody
"fixes" the test to a wrong value.
N3. BUG-2 (`| grep -q` under pipefail) NOT fixed -- a three-script sweep, out of scope.
N4. The DOCFIX backlog (C5 above) NOT swept -- handing an agent a record-cleanup pass
mid-stage is the exact GA-F06 shape. Should ride the next surface touched for a
real reason.
N5. dc1's snap proxy NOT installed; the operator's rack-permission approval was scoped
to dc0 and does not cover the dc1 rack.
N6. The metal-admin container-family DETECTION gate (the only available mitigation for
juju's non-deterministic single-address behaviour on a dual-stack container plane)
is OWED, NOT BUILT.
N7. R1 (Authorization header across a cross-host redirect) logged not fixed -- a
redirect handler is untested code on a control path.
N8. The apex push has NOT been run. The tool is dry-run-only and nothing has been
written to NetBox.
N9. `network-get` on a v6-only bound space -- the prerequisite for D-139's step 7 --
is STILL NOT MEASURED. It needs a deployed unit and must ride the next deploy.
