QUEUED FINDINGS -- session 2026-08-08 (part 3): amphora build FIXED + geneve-over-v6 ROOT-CAUSED
Sweep method: model read the whole session, grepped each candidate against the repo.
Status authority is docs/CURRENT-STATE.md; this is a sweep record, not status.
Session body (what/why/revert per item): docs/changelog-20260808-amphora-geneve.md (Items 1-5).
================================================================================
FIRST SURFACE (existed ONLY in transcript -- would be lost on a clear)
================================================================================
F1. LIVE CLOUD MUTATIONS LEFT IN PLACE (host-local / not in git) -- rebuild-relevant, need revert:
(a) LOOP-DEVICE PASS-THROUGH into LXD container juju-d4f45a-1-lxd-1 on machine 1
(vr1-dc0-control-02). VERBATIM (run on machine 1 as root):
lxc config device add juju-d4f45a-1-lxd-1 loop-control unix-char path=/dev/loop-control
lxc config device add juju-d4f45a-1-lxd-1 loop0 unix-block path=/dev/loop0 (... loop1..loop7)
Host-local LXD state -- NOT in git/juju/bundle; LOST on container recreation; loop0-4 are
the host's live snap mounts. REVERT: lxc config device remove juju-d4f45a-1-lxd-1
loop-control loop0 loop1 loop2 loop3 loop4 loop5 loop6 loop7. (Now also in changelog Item 2a.)
(b) juju config octavia-diskimage-retrofit ubuntu-mirror=<dc0 mirror http://10.12.8.4/ubuntu
jammy{,-updates,-security} main universe>. REVERT: juju config ... --reset ubuntu-mirror.
(changelog Item 2b.)
F2. TEST LB + AMPHORA + NET LEFT LIVE (operator chose savegame mid-test) -- TEARDOWN OWED next
session (consuming a compute instance): LB 24bd326f-611c-4a8c-a348-e9c67f5d156c (PENDING_CREATE),
amphora nova 66542b9f-..., lb-smoke-net 917899c9-..., lb-smoke-subnet 45cfc927-... .
Teardown: openstack loadbalancer delete 24bd326f... --cascade; openstack subnet delete
45cfc927...; openstack network delete 917899c9... . (changelog Item 4.)
F3. RETROFIT MANUAL-DIAGNOSIS TRAPS (rebuild-relevant; no repo tool exists for dib triage):
- the callable is the snap app `octavia-diskimage-retrofit` (a wrapper for retrofit.sh), NOT
`retrofit.sh` on PATH; under sudo use /snap/bin/octavia-diskimage-retrofit (sudo drops /snap/bin).
- retrofit REQUIRES input AND output paths to reside under
/var/snap/octavia-diskimage-retrofit/common (else exit 65 "must reside within ...").
- the tool WIPES common/tmp on start -- a diag log/input placed there is unlinked mid-run
(writes go to a deleted inode; wc -l reads 0). Stage under common/ ROOT, or pipe to tail live.
- flags are single-dash getopts: -d (verbose/xtrace), -O <fmt>, -r (no-resize), -u/-c/-m/-p.
F4. F4-CORRECTION to part-2 sweep: python3-octaviaclient (+ designateclient) installed on the rack
came from archive.ubuntu.com noble/main DIRECT -- the OUTER host vvr1-dc0 (172.31.0.2) HAS
egress (noble, juju client). This is NOT dc0-mirror (D-135) evidence; the mirror requirement is
specific to the ISOLATED DC NODES (where the retrofit dib build runs), which is exactly why the
build failed and the rack install did not.
F5. ENV FACT: 172.31.0.2 = vvr1-dc0, the OUTER libvirt/substrate host, runs the juju client for
vr1-dc0 AND the openstack CLI (reaches cloud VIPs over the internal net) AND has internet
egress. The DC role/compute NODES (e.g. machine 1 = vr1-dc0-control-02) are isolated and use
the in-DC mirror/proxy. Do not conflate "the rack" (egress) with "a node" (isolated).
F6. bfd_status EMPTY on all geneve tunnels is AMBIGUOUS evidence (OVN geneve may not populate BFD);
the decisive geneve-fail evidence is the encap FAMILY split + 100% ICMPv6 loss. compute<->compute
(v6<->v6) data path is UNTESTED (no second instance existed to ping). (In the audit file.)
================================================================================
ALREADY ON SURFACE (verified present)
================================================================================
- Amphora build FIXED (two-layer: loop-devices + apt-unreachable-mirror) -> changelog Item 2 +
CURRENT-STATE part-3 progress block. Image 775ebeba ACTIVE+tagged (measured).
- G18 OWED#3 o-hm0 MTU PASS -> changelog Item 3 + CURRENT-STATE.
- geneve-over-v6 wrap-gate FAIL, root-caused -> docs/audit/geneve-over-v6-rootcause-20260808.md +
CURRENT-STATE block + commits cd1f9cd/f5daadd.
- DELIVERY (pushed f85daa7..a6340e6; bookend 8e2b18b; voffice1 synced f5daadd) -> changelog Item 1.
================================================================================
ALWAYS-SWEEP FIVE
================================================================================
1. GITIGNORED STATE: NO new .claude/settings.local.json permission rules added this session -- a
broad `Bash(ssh *)` rule (pre-existing) covered every command. (341 allow rules total.)
Credential material: none read into context this session.
2. DANGLING REFS: commits 8e2b18b/cd1f9cd/f5daadd cite geneve-over-v6-rootcause-20260808.md,
changelog-20260808-amphora-geneve.md (this session's body), the part-2 residue files -- all resolve.
3. RULING FIDELITY: two operator AskUserQuestion selections this session -- "Fix-forward on dc0
(gated)" (amphora) and "Investigate the overlay now" + "Push 4 + commit bookend residue" +
"Pause -- savegame now". No GA-R5 D-ruling made (geneve D-number PROPOSED, not minted).
4. AS-EXECUTED LOG GAP: run-logged.sh NOT opened (structurally unusable for tool-driven Bash; Nth
consecutive). The changelog + this sweep + the transcript ARE the as-executed record for this
session's live mutations (loop pass-through, ubuntu-mirror config, amphora build, LB/net create).
5. CONTRADICTION DETECTOR: the geneve-over-v6 verify gate result (FAIL) is the measured live
observation D-101/D-134 owed; recorded in CURRENT-STATE + the audit file. No measurement left
contradicting a standing doc unrecorded.
================================================================================
OWNED (own-mistakes; all caught + corrected this session)
================================================================================
- Nearly reported the yoga-vs-caracal amphora-agent as the LB-stuck cause; the octavia HM log
(ConnectTimeout, not ssl/cert) + ping6 100% loss REFUTED it -> measured before concluding. The
advisor had predicted the yoga path as the likely LB failure, which made the wrong cause tempting.
- Three diagnostic-capture false-starts (retrofit.sh not the binary; path-residency exit 65;
self-wiping tmp unlinking my log) -- my capture harness bugs, not cloud faults; caught by iterating.
- LB-poll label swap (openstack -c column order) -- cosmetic; caught (amphora list was authoritative).
- ubuntu-mirror config file-through-double-ssh failed (temp file didn't resolve) -> switched to stdin.
- First loop-device fix was necessary but NOT sufficient -- treated it as the fix before measuring
the actual dib stderr; the real blocker (apt-unreachable) surfaced only on the -d capture.
- run-logged NOT opened (Nth). F4 initially miscast as D-135 mirror evidence (corrected, F4 above).