Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260808-amphora-geneve.md
@JANeumatrix JANeumatrix 4 hours ago 6 KB savegame 2026-08-08 (part 3): amphora build FIXED + geneve-over-v6 wrap-gate ROOT-CAUSED
Raw Blame History

changelog 2026-08-08 (part 3) -- amphora build FIXED + geneve-over-v6 wrap-gate ROOT-CAUSED

Session body (GA-R2, one per session). Status lives ONLY in docs/CURRENT-STATE.md. Branch dc-dc-stage5-preconditions. dc0 activation checkpoint (throwaway; 10.13 rebuild follows). All cloud reads/mutations run FROM the dc0 rack 172.31.0.2 (host vvr1-dc0) via voffice1 (D-138). Live-mutation items carry a REVERT; dc0 is throwaway so most reverts are "torn down with the cloud".

Item 1 -- DELIVERY (operator-authorized: "Push 4 + commit bookend residue")

Pushed the prior session's 4 commits f85daa7..a6340e6 (origin was at a25ed99). Then landed the prior part-2 GA-R4 bookend residue + this session's dc0-activation progress in 8e2b18b (session-ledger part-2 close, 2026-08-08 ledger rotation to archive/, the part-2 sweep file, CURRENT-STATE progress touch clearing repo-lint L10). geneve root cause in cd1f9cd + correction f5daadd. voffice1 pulled to f5daadd at close. Revert: git revert <sha> per commit (records only; the live mutations are Items 2/4).

Item 2 -- Amphora image build FIXED (D-021 pipeline complete on dc0)

Part-2 left the amphora build as an INCIDENT (octavia-diskimage-retrofit exit 1). Root-caused this session as TWO layers, both MEASURED:

  • (a) No loop devices in the LXD container. octavia-diskimage-retrofit/0 is a subordinate on 1/lxd/1 (an LXD container on vr1-dc0-control-02). dib needs loop devices; the container had none (losetup -f -> "No such device"). FIX (live, gated): hot-plugged the host's loop devices into the container on machine 1: sudo lxc config device add juju-d4f45a-1-lxd-1 loop-control unix-char path=/dev/loop-control and loop0..loop7 as unix-block. losetup roundtrip then works (no privileged escalation needed). REVERT: sudo lxc config device remove juju-d4f45a-1-lxd-1 loop-control loop0 loop1 loop2 loop3 loop4 loop5 loop6 loop7 (run on machine 1). HAZARD (rebuild finding): this is host-local LXD state, NOT in git/juju/bundle -- it does NOT survive container recreation, and loop0-4 are the host's live snap mounts (pass-through gives the container access to them; a rebuild should narrow to loop5-7 or place the retrofit's principal on metal).
  • (b) dib build's apt targeted UNREACHABLE public archives. With loop fixed, dib got into the libguestfs appliance and failed at install-packages (returncode 100): E: Unable to locate package amphora-agent + universe pkgs; measured cause = apt hitting http://archive.ubuntu.com / security.ubuntu.com -> "Network is unreachable" from the isolated node. The build does NOT inherit the model apt-http-proxy=http://10.12.8.4:3142, and the retrofit charm's ubuntu-mirror/uca-mirror/retrofit-uca-pocket were all empty/unset. FIX (live, gated): juju config octavia-diskimage-retrofit ubuntu-mirror=<3 deb lines at http://10.12.8.4/ubuntu jammy{,-updates,-security} main universe> (the dc0 D-135 full mirror, reachable 200, carries main+universe incl. amphora-agent). retrofit-uca-pocket left UNSET deliberately -> the build takes the jammy-universe (yoga-era) amphora-agent; caracal-matched agent is a rebuild refinement (NOT the LB blocker -- see Item 4). REVERT: juju config octavia-diskimage-retrofit --reset ubuntu-mirror.
  • RESULT: rebuilt via the tested phase-05-amphora-pipeline.sh (sha256-verified staged copy; MODEL=vr1-dc0). Image 775ebeba-32af-4d99-a304-d7b7795a4740 ACTIVE, disk_format=raw, tags [octavia-diskimage-retrofit, octavia-amphora] (two-sourced: script confirm + openstack image list --tag octavia-amphora). Leftover glance base 04c982c2... harmless. REVERT: openstack image delete 775ebeba... (torn down with the cloud).
  • REBUILD FINDING: the retrofit ubuntu-mirror (and, for caracal, uca-mirror/ retrofit-uca-pocket) must be wired PER-DC in the bundle so the isolated-DC amphora build uses the in-DC mirror. Not currently in bundle.yaml. Tied to D-021 + D-135. NO tool exists for retrofit/dib diagnosis (appendix-A has no symptom) -> appendix-A entry + optional tool owed.

Item 3 -- G18 OWED#3 (o-hm0 MTU, LP#2018998): PASS (read-only)

lb-mgmt-net MTU = 1442; all three octavia units' o-hm0 = 1442. No mismatch. Record-only, no fix. (G18 OWED#1 prefix + #2 router-isolation were done in part 2.) G18 obligations now all discharged.

Item 4 -- 1 test LB smoke test: amphora BOOTS but blocked by the geneve-over-v6 overlay

Created a throwaway VIP net/subnet + LB (installed python3-octaviaclient 3.7.0 on the rack -- NOTE: from archive.ubuntu.com noble/main direct, i.e. the OUTER host vvr1-dc0 has egress; this is NOT dc0-mirror evidence, correcting part-2 sweep F4). LIVE OBJECTS, LEFT UP per operator (savegame-now): lb-smoke-net 917899c9-ea33-43b3-bc0d-20dd925d2c26, lb-smoke-subnet 45cfc927-... (198.51.100.0/24), LB 24bd326f-611c-4a8c-a348-e9c67f5d156c (VIP 198.51.100.21, stuck PENDING_CREATE), amphora nova instance 66542b9f-... (ACTIVE -> image validated). TEARDOWN (owed next session): openstack loadbalancer delete 24bd326f... --cascade; openstack subnet delete 45cfc927...; openstack network delete 917899c9.... The amphora nova-boots fine (image good) but Octavia can't complete the agent handshake -> Item 5.

Item 5 -- geneve-over-v6 wrap-gate: FAIL, ROOT-CAUSED (full record in audit file)

octavia HM log = ConnectTimeout to [fc00:...:9473]:9443 (not ssl/cert -> not a version/agent issue); SG correctly allows v6 tcp/9443 + icmp; ping6 o-hm0 -> amphora (same /64) = 100% loss. Root cause (measured): OVN geneve encap IPs SPLIT ACROSS FAMILIES -- containerized control-node chassis (octavia LXD) have v4-only data-tenant addresses (10.12.16.x; D-134 auto-picked) -> IPv4 encap; carved compute METAL uses IPv6 (2602:f3e2:f02:30::x). Same dual-stack data-tenant plane (lib-net.sh PLANE_CIDRS), so it's an address-family SELECTION defect, not a mis-binding. Cross-family geneve tunnels don't pass traffic (verified on compute-01, the amphora's host) -> 100% cross-node loss -> amphora unreachable. Roosevelt-delta: v6 builds must give containerized OVN chassis a v6 data-tenant address; warrants a D-number (assigned at ruling). Fix NOT applied (substantial + rebuild-relevant; operator chose savegame over a live dc0 workaround). Full record: docs/audit/geneve-over-v6-rootcause-20260808.md. Revert: N/A (record only; no fix applied).

Not done (deliberate)

Designate real-Stage-7 4a depth measurement, the wrap gates (cloud-assert BOM, controller backup, verify-live Ceph), and the re-IP live-free checks -- all deferred to next session (operator chose savegame). The geneve fix + the LB teardown are owed next session.