diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index edce64d..112cdef 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -98,7 +98,8 @@ > rebuild (consolidated executable **REBUILD RECIPE** in the root-cause record): container v6 carve; > persistent unbracket override (no fixed ovn-chassis rev pinned by search 2026-08-09); neutron > `overlay_ip_version=6`; D-139's metal-admin-leg-IPv4 gate (specced, build at rebuild). Changelog: -> `docs/changelog-20260809-geneve-v6-rootcause-gate.md`. +> `docs/changelog-20260809-geneve-v6-rootcause-gate.md`; sweep: +> `docs/audit/queued-findings-20260809-geneve-v6.txt`. > ## >>> STANDING OPERATOR DIRECTIVE, 2026-07-30: THE NEXT SESSION PROCEEDS TO THE JUJU DEPLOYMENT (STAGE 5). NO MATTER WHAT. <<< > diff --git a/docs/audit/queued-findings-20260809-geneve-v6.txt b/docs/audit/queued-findings-20260809-geneve-v6.txt new file mode 100644 index 0000000..3bfd166 --- /dev/null +++ b/docs/audit/queued-findings-20260809-geneve-v6.txt @@ -0,0 +1,96 @@ +SWEEP -- session 2026-08-09 (geneve-over-v6 root cause + gate + D-139 amendment) +============================================================================== +Method (GA-R4 / ruled 2026-07-31): enumerate every finding/decision/measurement/ +trap/mistake from the session, grep each against repo surfaces, mark ALREADY-ON- +SURFACE (with location) or FIRST SURFACE (would have been lost). FIRST SURFACE led. + +Commits this session (branch dc-dc-stage5-preconditions, UNPUSHED at write time): + ee48f2a geneve root-cause + gate + phase-04 fix + 96eb98f D-139 AMENDMENT 2026-08-09 (GA-R5 ruling) + 9837332 geneve REBUILD RECIPE (item 3) + +------------------------------------------------------------------------------ +FIRST SURFACE (existed only in the transcript before this file) -- all are +Claude-harness / live-test MECHANICS, useful to the NEXT session doing live +cloud testing; none is a cloud/repo defect. +------------------------------------------------------------------------------ + +FS1. KEYLESS VM CONNECTIVITY TEST METHOD (reusable). To prove overlay/datapath + connectivity between two tenant instances WITHOUT an SSH keypair (the + destructive-guard blocks handling a private-key file) and WITHOUT interactive + console: boot an instance with cloud-init user-data that pings the target and + `tee`s the result to /dev/console, then read `openstack console log show `. + Worked 2026-08-09 (geneve-proof-c -> geneve-proof-b, 8/8 0% loss). Timing: the + ping runs at cloud-init `modules:final` (~120s uptime) + any user `sleep`; + the serial console log lags ~30s. Grep: 0 durable surface (only a passing + "via the console log" mention in geneve-over-v6-rootcause-20260808.md). + +FS2. CLASSIFIER blocks a remote HEREDOC that writes+executes a script + (`cat > /tmp/x.sh < /tmp/x` PASSED. Prefer printf/ + single-line for remote script/user-data creation. Grep: 0 surface. + +FS3. DESTRUCTIVE-GUARD blocks `openstack keypair create > ~/*.pem` + reading it + (`wc -l < key.pem`) as secret-adjacent -- correct behavior; do NOT work around + it. Use FS1 (keyless) instead. Grep: 0 durable surface. + +FS4. `scripts/run-tests-all.sh` BUFFERS all output to the end -- an interrupted/ + killed run leaves a 0-byte log (uninformative). Run it fully DETACHED + (run_in_background) to completion, or `stdbuf -oL` for incremental output. + A `timeout 400` is too short (103 harnesses). Grep: 0 specific surface. + +FS5. dc0-rack live-op auth: `source ~/admin-openrc` on the rack (172.31.0.2 / + vvr1-dc0, via voffice1, D-138) provides the openstack client creds. (The rack + ACCESS shape is on surface in changelog-20260808-amphora-geneve.md:5; the + admin-openrc path specifically was not.) Minor. + +------------------------------------------------------------------------------ +ALREADY ON SURFACE (verified by grep; recorded here for completeness) +------------------------------------------------------------------------------ + +ON1. geneve-over-v6 ROOT CAUSE (family split + bracketed ovn-encap-ip) + LIVE + CONFIRM (VM->VM 8/8) + the RETRACTED localport mis-test. + -> docs/audit/geneve-over-v6-rootcause-20260808.md (localport=6, unbracket=8 hits). + +ON2. D-139 AMENDMENT 2026-08-09 (GA-R5, operator verbatim "D-139 amendment"): + v6-only data-tenant viable; containerized chassis need carved v6 + unbracketed + encap; v4-forced off the table. Ruling fidelity OK (exact utterance recorded). + -> docs/design-decisions.md, docs/CURRENT-STATE.md, changelog (3 hits). + +ON3. GATE scripts/geneve-encap-assert.sh (+harness 16/16, manifest pinned) + phase-04 + Step 12.2/Step 6 corrections (family-only -> family+tunnel-ofport, ULA->GUA). + -> scripts/, tests/, runbooks/dc-dc-phase4-*.md, changelog (7 hits). + +ON4. REBUILD RECIPE (container v6 carve + unbracket override + overlay_ip_version=6 + + gate; no fixed ovn-chassis rev pinned by search; metal-admin-leg gate specced- + not-built). -> geneve-over-v6-rootcause-20260808.md, CURRENT-STATE section 1. + +ON5. GUA-carve completion PROPOSAL (DC1 GUA gap: 7 planes/13 rows; lb-mgmt :80 + active-should-be-reserved; v6 utility records=0; 78 v4 VIPs reserved; .4 apt- + cacher no apex row; matrix; scan). DRAFT, not executed (operator: "ignore DC1"). + -> docs/audit/gua-carve-completion-proposal-20260809.md; apex dump netbox/draft/ + vr1-office1-current-20260809.json. + +ON6. Versions measured on compute (authoritative): OVS 3.3.0, OVN 24.03.2, kernel + 5.15.0-186, geneve.ko 0.6; kernel datapath supports v6 geneve UNbracketed. + -> geneve-over-v6-rootcause-20260808.md, D-139 amendment. + +------------------------------------------------------------------------------ +OWNED (own-mistakes, corrected in-session) +------------------------------------------------------------------------------ + +OW1. Reported "geneve-over-v6 PROVEN NON-FUNCTIONAL -> v4-forced" into the durable + audit record from an INVALID test (pings sourced from an OVN localport/ovnmeta + namespace, which never tunnels by design). Caught by the fable advisor, not by + self-review. Retracted + corrected same session; the real cause (bracketed + encap) then found + live-confirmed. Instrument-currency #23 sibling: the test + did not bypass a layer (localport drop) that independently produces the same + negative. Recorded in the geneve doc RETRACTION section. + +OW2. Early over-confident geneve fix framing ("make data-tenant v6-only fixes it") + glossed the repo's own Item 14 (OVN encap-doc / test-coverage). Fable review + flagged; folded into the proposal + then superseded by the live confirm. + +DELIBERATELY NOT DONE: full Octavia LB end-to-end on dc0 (container-v6 half not +applied; dc0 throwaway); DC1 GUA carve execution (operator "ignore DC1"); metal- +admin-leg-IPv4 gate build (needs live cloud); OOB v4 (rides re-IP). All on surface. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 4660e1e..e8e9e84 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -281,3 +281,14 @@ - LEFTOVERS (owed next session): TEAR DOWN test LB 24bd326f / net lb-smoke-* / amphora (left LIVE per operator savegame-now); live mutations (loop passthrough + ubuntu-mirror config) carry reverts in the changelog. - Gates: repo-lint 0-fail (1 legacy warn); gauntlet ALL GREEN (102); ledger-scan reconciled; ledger rotated (08-06 x2 -> archive/session-ledger-rotated-20260809.md). - NEXT: tear down the test LB; Designate real-Stage-7 4a depth (read-only); wrap gates (cloud-assert BOM, controller backup, verify-live Ceph); re-IP 3 live-free checks -> present re-IP GA-R5 ruling. Rebuild items: wire retrofit ubuntu-mirror per-DC in bundle + give containerized OVN chassis a v6 data-tenant address. Sweep: docs/audit/queued-findings-20260808-amphora-geneve.txt. Body: docs/changelog-20260808-amphora-geneve.md. Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-09 -- geneve-over-v6 ROOT-CAUSED (bracket bug) + LIVE-CONFIRMED; D-139 amendment + executable gate (bounded, GA-R4) + +- Branch dc-dc-stage5-preconditions; 3 commits this session: ee48f2a (root-cause+gate+phase-04), 96eb98f (D-139 AMENDMENT ruling), 9837332 (rebuild recipe). Scan: 4 open decisions (D-068/131/132/142), SEC 28, next-free D-143/DOCFIX-214/BUNDLEFIX-059 (D-139 AMENDED, no new number). +- Opened by checking the geneve finding vs the Office1 NetBox (fresh read-only apex dump 20260809): DC1 GUA carve INCOMPLETE + gaps -> gua-carve-completion-proposal (DRAFT; operator "ignore DC1", NOT executed). +- GENEVE (operator directed a live confirm on throwaway dc0): TWO defects -- (a) encap family split (D-134 auto-pick), (b) ovn-chassis 24.03 sets the v6 ovn-encap-ip BRACKETED -> OVS rejects (ofport -1). Unbracketed -> real VM->VM 8/8, 0% loss. geneve-over-v6 WORKS (OVS3.3/OVN24.03/kernel5.15); v4-forced OFF the table. +- RULED (GA-R5, operator verbatim "D-139 amendment"): D-139 AMENDMENT 2026-08-09. +- DELIVERED: scripts/geneve-encap-assert.sh (+harness 16/16, manifest 103); phase-04 Step 12.2/Step 6 corrected; consolidated rebuild recipe. Test scaffold + owed lb-smoke LB TORN DOWN (dc0 clean). +- OWNED (#24): a wrong "v6 broken -> v4-forced" NEGATIVE rode into the audit record from a localport-sourced test (never tunnels by design); caught by the FABLE ADVISOR, retracted, corrected. +- Gates: repo-lint 0-fail (1 legacy warn); gauntlet ALL GREEN (103); ledger-scan reconciled. +- NEXT (rebuild-time): confirm fixed ovn-chassis rev or wire the unbracket override; container v6 carve; build the metal-admin-leg-IPv4 gate; DC1 GUA carve + OOB v4 (ride re-IP); present the re-IP GA-R5 ruling. Sweep: docs/audit/queued-findings-20260809-geneve-v6.txt. Body: docs/changelog-20260809-geneve-v6-rootcause-gate.md. Status ONLY in CURRENT-STATE.md.