diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index b848976..f34d80b 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -72,7 +72,13 @@ > must give containerized OVN chassis a v6 data-tenant address (warrants a D-number, PROPOSED -- > number assigned at ruling). > Fix is substantial + rebuild-relevant; NOT applied. Root-cause record: -> `docs/audit/geneve-over-v6-rootcause-20260808.md`. Full changelog at session close. +> `docs/audit/geneve-over-v6-rootcause-20260808.md`. **SESSION CLOSED (savegame) 2026-08-08 part 3;** +> body `docs/changelog-20260808-amphora-geneve.md`, sweep +> `docs/audit/queued-findings-20260808-amphora-geneve.txt`. OWED next session: TEAR DOWN the live +> test LB `24bd326f`/net `lb-smoke-*`/amphora; Designate real-Stage-7 `4a` depth; wrap gates +> (cloud-assert BOM, controller backup, verify-live Ceph); re-IP 3 live-free checks -> present the +> re-IP GA-R5 ruling. Rebuild items: wire retrofit `ubuntu-mirror` per-DC in the bundle + give +> containerized OVN chassis a v6 data-tenant address. > ## >>> STANDING OPERATOR DIRECTIVE, 2026-07-30: THE NEXT SESSION PROCEEDS TO THE JUJU DEPLOYMENT (STAGE 5). NO MATTER WHAT. <<< > diff --git a/docs/archive/session-ledger-rotated-20260809.md b/docs/archive/session-ledger-rotated-20260809.md new file mode 100644 index 0000000..ee77be0 --- /dev/null +++ b/docs/archive/session-ledger-rotated-20260809.md @@ -0,0 +1,31 @@ +# Session ledger -- rotated summaries (2026-08-09 pass, GA-R4 rule 3) + +The two oldest closed-session summaries below were moved VERBATIM out of +`docs/session-ledger.md` to restore the 300-line cap at the 2026-08-08 (part 3) +close (amphora build FIXED + geneve-over-v6 root-caused). Oldest-first. + +--- + +## SESSION CLOSE 2026-08-06 (part 3) -- phase-03 core-verify: F-CV1/F-CV2/F-CV3 RESOLVED + binding conformance (bounded, GA-R4) + +- Branch dc-dc-stage5-preconditions; 9 commits (b8d2b25..edba699 pushed; cd13baa ec8a1d3 ace0e16 UNPUSHED). Scan: 29 SEC, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (used 056/057/058). +- STEP 7 (phase-03 core-verify, adapted vr1-dc0, run from the dc0 rack): CORE-API VERIFIED (falsifiable settle walk, haproxy sweep, admin-openrc scoped token, IP-only endpoints, two-sourced VIP). F-CV2 RESOLVED: openstack CLI installed on the dc0 rack (D-138; was voffice1-only). +- F-CV1 (designate _admin backend DOWN) RESOLVED (BUNDLEFIX-056): bundle omitted public+internal -> orphaned VIP triple; fixed + live-rebound. Governing = D-052 + D-020 amdt, NOT D-141. +- F-CV3 (dashboard VIP plaintext) RESOLVED via D-072 AMENDMENT (VR1) ratified GA-R5 "Ratified, land the config-of-record" + BUNDLEFIX-057: VR1 split-metal INVERTS D-072 (charm serves metal-internal, no admin/internal binding/os-*-network lever) -> cluster->metal-internal. Proven LIVE before ratifying (operator prove-then-rule method). +- BUNDLEFIX-058: designate-stack conformance (designate amqp+cluster, designate-bind cluster -> metal-internal). Binding conformance now CLEAN cloud-wide (no other active deviations). +- D-134 Roosevelt-delta annotation (LXD container addrs auto-picked, not carved); gap-21 access-model context (tailnet -> metal-admin dashboards). +- OWNED: assert-before-check recurred (F-CV1 "CONFIRMED"+D-141 before the governing check; "lone exception" premise) + filter/WebFetch false-negatives -- all caught by sweep/governing-check-first + prove-then-rule. +- DURABILITY: this host 0 uncommitted / 3 UNPUSHED; voffice1 edba699 (=origin, syncs after push); dc0 rack ~/repo-stage/bundle.yaml STALE (O10, re-stage before any redeploy). repo-lint 0-fail; gauntlet ALL GREEN (99) at ace0e16. +- NEXT: Step 3.4 (keystone domain-manager policy PO:+G3) is the LAST phase-03 exit-gate item; then Steps 8-12 -> Stage-5 exit. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt (O11 MAAS-query method FIRST SURFACE). Body: docs/changelog-20260806-phase03-coreverify.md. Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-06/07 -- phase-03 Step 3.4 G3 DONE + per-DC Tailscale rulings + .7 VMs stood up both DCs (bounded, GA-R4) + +- Branch dc-dc-stage5-preconditions; pushed afbaed2..9e74f43 (7 commits); the node-vm harness reconcile is UNCOMMITTED in this bookend. voffice1 synced 9e74f43. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- all D-129/D-134 AMENDMENTS). +- STEP 3.4 (phase-03) DONE: scripts/g3-domain-manager-probe.sh (+harness 12/12) as the GA-R6 named check; live G3 PASS on dc0 (7 ok/0 fail, teardown verified clean); stage-1 PO: verified per-UNIT (app-aggregate hid it). +- DECISION C (Horizon): reconciled to VR1 -- both dashboard VIPs serve HTTPS login 200+csrftoken; D-044 + the VR0 nginx-repoint are plain-HTTP-leg artifacts that would only WEAKEN the cookie. phase-03 does NOT close; Step 3.3 SPLITS to its own gate (tailnet-access-gated). +- PER-DC TAILSCALE (D-129(iii) AMENDMENT, rulings a-d, BOTH DCs): (a) dedicated .7 VM; (b) STAR operator->DC; (c) SINGLE, HA scale-up PINNED; (d) SNAT on. D-134 octet map ->.7; D-107 citation DOCFIX-in-amendment. site-tailscale.sh (+harness 15/15). +- SUBSTRATE: .7 VMs APPLIED (tofu, gated) + MACs pinned + tofu No-changes on BOTH DCs -- dc0 tailscale; dc1 region VM + tailscale (operator "dc1 full"). Capacity re-gated FIT 874/1024=85%. +- DEFERRED: Headscale-side join (tagged authkey / autoApprovers / star ACL / Office1-untagged-fix) BLOCKED on control-plane access (operator lacks it); per-VM MAAS commission/deploy/carve/install; dc1 region SETUP workstream. +- OWNED: shipped the substrate commits without re-running the gauntlet -> node-vm exact-count 11/66->12/72 went red, caught only at the savegame gauntlet (the EXACT 2026-07-30 lesson this harness's own comment records, repeated); v6-posture mis-frame (operator corrected before it biased Decision B); app-aggregate PO: near-miss (caught per-unit); jget + g3-harness bugs (caught by fixtures); ADVISOR caught a would-be D-143 mint. +- Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 AFTER the node-vm reconcile. Sweep: docs/audit/queued-findings-20260807-tailscale-substrate.txt (O1-O4 FIRST SURFACE). Body: docs/changelog-20260806-step34-g3-probe.md. +- NEXT: when Headscale access -> the join + Office1-untagged fix; else MAAS commission/deploy/carve/install the 3 VMs + the dc1 region setup; then Horizon-over-tailnet confirm closes Step 3.3. Status ONLY in CURRENT-STATE.md. diff --git a/docs/audit/queued-findings-20260808-amphora-geneve.txt b/docs/audit/queued-findings-20260808-amphora-geneve.txt new file mode 100644 index 0000000..0c53427 --- /dev/null +++ b/docs/audit/queued-findings-20260808-amphora-geneve.txt @@ -0,0 +1,92 @@ +QUEUED FINDINGS -- session 2026-08-08 (part 3): amphora build FIXED + geneve-over-v6 ROOT-CAUSED +Sweep method: model read the whole session, grepped each candidate against the repo. +Status authority is docs/CURRENT-STATE.md; this is a sweep record, not status. +Session body (what/why/revert per item): docs/changelog-20260808-amphora-geneve.md (Items 1-5). + +================================================================================ +FIRST SURFACE (existed ONLY in transcript -- would be lost on a clear) +================================================================================ + +F1. LIVE CLOUD MUTATIONS LEFT IN PLACE (host-local / not in git) -- rebuild-relevant, need revert: + (a) LOOP-DEVICE PASS-THROUGH into LXD container juju-d4f45a-1-lxd-1 on machine 1 + (vr1-dc0-control-02). VERBATIM (run on machine 1 as root): + lxc config device add juju-d4f45a-1-lxd-1 loop-control unix-char path=/dev/loop-control + lxc config device add juju-d4f45a-1-lxd-1 loop0 unix-block path=/dev/loop0 (... loop1..loop7) + Host-local LXD state -- NOT in git/juju/bundle; LOST on container recreation; loop0-4 are + the host's live snap mounts. REVERT: lxc config device remove juju-d4f45a-1-lxd-1 + loop-control loop0 loop1 loop2 loop3 loop4 loop5 loop6 loop7. (Now also in changelog Item 2a.) + (b) juju config octavia-diskimage-retrofit ubuntu-mirror=. REVERT: juju config ... --reset ubuntu-mirror. + (changelog Item 2b.) + +F2. TEST LB + AMPHORA + NET LEFT LIVE (operator chose savegame mid-test) -- TEARDOWN OWED next + session (consuming a compute instance): LB 24bd326f-611c-4a8c-a348-e9c67f5d156c (PENDING_CREATE), + amphora nova 66542b9f-..., lb-smoke-net 917899c9-..., lb-smoke-subnet 45cfc927-... . + Teardown: openstack loadbalancer delete 24bd326f... --cascade; openstack subnet delete + 45cfc927...; openstack network delete 917899c9... . (changelog Item 4.) + +F3. RETROFIT MANUAL-DIAGNOSIS TRAPS (rebuild-relevant; no repo tool exists for dib triage): + - the callable is the snap app `octavia-diskimage-retrofit` (a wrapper for retrofit.sh), NOT + `retrofit.sh` on PATH; under sudo use /snap/bin/octavia-diskimage-retrofit (sudo drops /snap/bin). + - retrofit REQUIRES input AND output paths to reside under + /var/snap/octavia-diskimage-retrofit/common (else exit 65 "must reside within ..."). + - the tool WIPES common/tmp on start -- a diag log/input placed there is unlinked mid-run + (writes go to a deleted inode; wc -l reads 0). Stage under common/ ROOT, or pipe to tail live. + - flags are single-dash getopts: -d (verbose/xtrace), -O , -r (no-resize), -u/-c/-m/-p. + +F4. F4-CORRECTION to part-2 sweep: python3-octaviaclient (+ designateclient) installed on the rack + came from archive.ubuntu.com noble/main DIRECT -- the OUTER host vvr1-dc0 (172.31.0.2) HAS + egress (noble, juju client). This is NOT dc0-mirror (D-135) evidence; the mirror requirement is + specific to the ISOLATED DC NODES (where the retrofit dib build runs), which is exactly why the + build failed and the rack install did not. + +F5. ENV FACT: 172.31.0.2 = vvr1-dc0, the OUTER libvirt/substrate host, runs the juju client for + vr1-dc0 AND the openstack CLI (reaches cloud VIPs over the internal net) AND has internet + egress. The DC role/compute NODES (e.g. machine 1 = vr1-dc0-control-02) are isolated and use + the in-DC mirror/proxy. Do not conflate "the rack" (egress) with "a node" (isolated). + +F6. bfd_status EMPTY on all geneve tunnels is AMBIGUOUS evidence (OVN geneve may not populate BFD); + the decisive geneve-fail evidence is the encap FAMILY split + 100% ICMPv6 loss. compute<->compute + (v6<->v6) data path is UNTESTED (no second instance existed to ping). (In the audit file.) + +================================================================================ +ALREADY ON SURFACE (verified present) +================================================================================ +- Amphora build FIXED (two-layer: loop-devices + apt-unreachable-mirror) -> changelog Item 2 + + CURRENT-STATE part-3 progress block. Image 775ebeba ACTIVE+tagged (measured). +- G18 OWED#3 o-hm0 MTU PASS -> changelog Item 3 + CURRENT-STATE. +- geneve-over-v6 wrap-gate FAIL, root-caused -> docs/audit/geneve-over-v6-rootcause-20260808.md + + CURRENT-STATE block + commits cd1f9cd/f5daadd. +- DELIVERY (pushed f85daa7..a6340e6; bookend 8e2b18b; voffice1 synced f5daadd) -> changelog Item 1. + +================================================================================ +ALWAYS-SWEEP FIVE +================================================================================ +1. GITIGNORED STATE: NO new .claude/settings.local.json permission rules added this session -- a + broad `Bash(ssh *)` rule (pre-existing) covered every command. (341 allow rules total.) + Credential material: none read into context this session. +2. DANGLING REFS: commits 8e2b18b/cd1f9cd/f5daadd cite geneve-over-v6-rootcause-20260808.md, + changelog-20260808-amphora-geneve.md (this session's body), the part-2 residue files -- all resolve. +3. RULING FIDELITY: two operator AskUserQuestion selections this session -- "Fix-forward on dc0 + (gated)" (amphora) and "Investigate the overlay now" + "Push 4 + commit bookend residue" + + "Pause -- savegame now". No GA-R5 D-ruling made (geneve D-number PROPOSED, not minted). +4. AS-EXECUTED LOG GAP: run-logged.sh NOT opened (structurally unusable for tool-driven Bash; Nth + consecutive). The changelog + this sweep + the transcript ARE the as-executed record for this + session's live mutations (loop pass-through, ubuntu-mirror config, amphora build, LB/net create). +5. CONTRADICTION DETECTOR: the geneve-over-v6 verify gate result (FAIL) is the measured live + observation D-101/D-134 owed; recorded in CURRENT-STATE + the audit file. No measurement left + contradicting a standing doc unrecorded. + +================================================================================ +OWNED (own-mistakes; all caught + corrected this session) +================================================================================ +- Nearly reported the yoga-vs-caracal amphora-agent as the LB-stuck cause; the octavia HM log + (ConnectTimeout, not ssl/cert) + ping6 100% loss REFUTED it -> measured before concluding. The + advisor had predicted the yoga path as the likely LB failure, which made the wrong cause tempting. +- Three diagnostic-capture false-starts (retrofit.sh not the binary; path-residency exit 65; + self-wiping tmp unlinking my log) -- my capture harness bugs, not cloud faults; caught by iterating. +- LB-poll label swap (openstack -c column order) -- cosmetic; caught (amphora list was authoritative). +- ubuntu-mirror config file-through-double-ssh failed (temp file didn't resolve) -> switched to stdin. +- First loop-device fix was necessary but NOT sufficient -- treated it as the fix before measuring + the actual dib stderr; the real blocker (apt-unreachable) surfaced only on the -d capture. +- run-logged NOT opened (Nth). F4 initially miscast as D-135 mirror evidence (corrected, F4 above). diff --git a/docs/changelog-20260808-amphora-geneve.md b/docs/changelog-20260808-amphora-geneve.md new file mode 100644 index 0000000..45baac8 --- /dev/null +++ b/docs/changelog-20260808-amphora-geneve.md @@ -0,0 +1,84 @@ +# changelog 2026-08-08 (part 3) -- amphora build FIXED + geneve-over-v6 wrap-gate ROOT-CAUSED + +Session body (GA-R2, one per session). Status lives ONLY in `docs/CURRENT-STATE.md`. +Branch `dc-dc-stage5-preconditions`. dc0 activation checkpoint (throwaway; 10.13 rebuild follows). +All cloud reads/mutations run FROM the dc0 rack `172.31.0.2` (host `vvr1-dc0`) via `voffice1` +(D-138). Live-mutation items carry a REVERT; dc0 is throwaway so most reverts are "torn down +with the cloud". + +## Item 1 -- DELIVERY (operator-authorized: "Push 4 + commit bookend residue") +Pushed the prior session's 4 commits `f85daa7..a6340e6` (origin was at `a25ed99`). Then landed +the prior part-2 GA-R4 bookend residue + this session's dc0-activation progress in `8e2b18b` +(session-ledger part-2 close, 2026-08-08 ledger rotation to archive/, the part-2 sweep file, +CURRENT-STATE progress touch clearing repo-lint L10). geneve root cause in `cd1f9cd` + +correction `f5daadd`. voffice1 pulled to `f5daadd` at close. +**Revert:** `git revert ` per commit (records only; the live mutations are Items 2/4). + +## Item 2 -- Amphora image build FIXED (D-021 pipeline complete on dc0) +Part-2 left the amphora build as an INCIDENT (octavia-diskimage-retrofit exit 1). Root-caused +this session as TWO layers, both MEASURED: + - **(a) No loop devices in the LXD container.** octavia-diskimage-retrofit/0 is a subordinate on + `1/lxd/1` (an LXD container on vr1-dc0-control-02). dib needs loop devices; the container had + none (`losetup -f` -> "No such device"). **FIX (live, gated):** hot-plugged the host's loop + devices into the container on machine 1: + `sudo lxc config device add juju-d4f45a-1-lxd-1 loop-control unix-char path=/dev/loop-control` + and `loop0..loop7` as `unix-block`. losetup roundtrip then works (no privileged escalation + needed). **REVERT:** `sudo lxc config device remove juju-d4f45a-1-lxd-1 loop-control loop0 loop1 + loop2 loop3 loop4 loop5 loop6 loop7` (run on machine 1). **HAZARD (rebuild finding):** this is + host-local LXD state, NOT in git/juju/bundle -- it does NOT survive container recreation, and + loop0-4 are the host's live snap mounts (pass-through gives the container access to them; a + rebuild should narrow to loop5-7 or place the retrofit's principal on metal). + - **(b) dib build's apt targeted UNREACHABLE public archives.** With loop fixed, dib got into the + libguestfs appliance and failed at install-packages (returncode 100): `E: Unable to locate + package amphora-agent` + universe pkgs; measured cause = apt hitting + `http://archive.ubuntu.com` / `security.ubuntu.com` -> "Network is unreachable" from the + isolated node. The build does NOT inherit the model `apt-http-proxy=http://10.12.8.4:3142`, and + the retrofit charm's `ubuntu-mirror`/`uca-mirror`/`retrofit-uca-pocket` were all empty/unset. + **FIX (live, gated):** `juju config octavia-diskimage-retrofit ubuntu-mirror=<3 deb lines at + http://10.12.8.4/ubuntu jammy{,-updates,-security} main universe>` (the dc0 D-135 full mirror, + reachable 200, carries main+universe incl. amphora-agent). `retrofit-uca-pocket` left UNSET + deliberately -> the build takes the jammy-universe (yoga-era) amphora-agent; caracal-matched + agent is a rebuild refinement (NOT the LB blocker -- see Item 4). **REVERT:** + `juju config octavia-diskimage-retrofit --reset ubuntu-mirror`. + - **RESULT:** rebuilt via the tested `phase-05-amphora-pipeline.sh` (sha256-verified staged copy; + `MODEL=vr1-dc0`). Image `775ebeba-32af-4d99-a304-d7b7795a4740` ACTIVE, `disk_format=raw`, tags + `[octavia-diskimage-retrofit, octavia-amphora]` (two-sourced: script confirm + `openstack image + list --tag octavia-amphora`). Leftover glance base `04c982c2...` harmless. **REVERT:** + `openstack image delete 775ebeba...` (torn down with the cloud). + - **REBUILD FINDING:** the retrofit `ubuntu-mirror` (and, for caracal, `uca-mirror`/ + `retrofit-uca-pocket`) must be wired PER-DC in the bundle so the isolated-DC amphora build uses + the in-DC mirror. Not currently in `bundle.yaml`. Tied to D-021 + D-135. NO tool exists for + retrofit/dib diagnosis (appendix-A has no symptom) -> appendix-A entry + optional tool owed. + +## Item 3 -- G18 OWED#3 (o-hm0 MTU, LP#2018998): PASS (read-only) +lb-mgmt-net MTU = 1442; all three octavia units' `o-hm0` = 1442. No mismatch. Record-only, no fix. +(G18 OWED#1 prefix + #2 router-isolation were done in part 2.) G18 obligations now all discharged. + +## Item 4 -- 1 test LB smoke test: amphora BOOTS but blocked by the geneve-over-v6 overlay +Created a throwaway VIP net/subnet + LB (installed `python3-octaviaclient 3.7.0` on the rack -- +NOTE: from `archive.ubuntu.com noble/main` direct, i.e. the OUTER host `vvr1-dc0` has egress; this +is NOT dc0-mirror evidence, correcting part-2 sweep F4). **LIVE OBJECTS, LEFT UP per operator +(savegame-now):** `lb-smoke-net` `917899c9-ea33-43b3-bc0d-20dd925d2c26`, `lb-smoke-subnet` +`45cfc927-...` (198.51.100.0/24), LB `24bd326f-611c-4a8c-a348-e9c67f5d156c` (VIP 198.51.100.21, +stuck PENDING_CREATE), amphora nova instance `66542b9f-...` (ACTIVE -> image validated). +**TEARDOWN (owed next session):** `openstack loadbalancer delete 24bd326f... --cascade; openstack +subnet delete 45cfc927...; openstack network delete 917899c9...`. +The amphora nova-boots fine (image good) but Octavia can't complete the agent handshake -> Item 5. + +## Item 5 -- geneve-over-v6 wrap-gate: FAIL, ROOT-CAUSED (full record in audit file) +octavia HM log = `ConnectTimeout` to `[fc00:...:9473]:9443` (not ssl/cert -> not a version/agent +issue); SG correctly allows v6 tcp/9443 + icmp; ping6 o-hm0 -> amphora (same /64) = 100% loss. +**Root cause (measured):** OVN geneve encap IPs SPLIT ACROSS FAMILIES -- containerized control-node +chassis (octavia LXD) have v4-only data-tenant addresses (`10.12.16.x`; D-134 auto-picked) -> +IPv4 encap; carved compute METAL uses IPv6 (`2602:f3e2:f02:30::x`). Same dual-stack `data-tenant` +plane (`lib-net.sh` PLANE_CIDRS), so it's an address-family SELECTION defect, not a mis-binding. +Cross-family geneve tunnels don't pass traffic (verified on compute-01, the amphora's host) -> +100% cross-node loss -> amphora unreachable. **Roosevelt-delta:** v6 builds must give containerized +OVN chassis a v6 data-tenant address; warrants a D-number (assigned at ruling). **Fix NOT applied** +(substantial + rebuild-relevant; operator chose savegame over a live dc0 workaround). Full record: +`docs/audit/geneve-over-v6-rootcause-20260808.md`. **Revert:** N/A (record only; no fix applied). + +## Not done (deliberate) +Designate real-Stage-7 `4a` depth measurement, the wrap gates (cloud-assert BOM, controller +backup, verify-live Ceph), and the re-IP live-free checks -- all deferred to next session +(operator chose savegame). The geneve fix + the LB teardown are owed next session. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 958c4f2..4660e1e 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -30,11 +30,11 @@ ## Machine-derived (re-seed from `scripts/ledger-scan.sh`; do not hand-edit) -_RE-SEEDED 2026-08-05 from `bash scripts/ledger-scan.sh` (this commit). The prior seed was -2026-08-02; it had gone STALE by one decision (D-142), one SEC row (SEC-033), and the D / -DOCFIX next-free (D 141 -> 143, DOCFIX 208 -> 210) -- the per-session summaries below carried -the current figures while this block did not, which is the one thing this block exists to -prevent. Re-run the scan to refresh._ +_RE-SEEDED 2026-08-08 (part 3 close) from `bash scripts/ledger-scan.sh`. The prior seed was +2026-08-05; it had gone STALE on the SEC count (29 -> 28, SEC-031 CLOSED 2026-08-07), the DOCFIX +next-free (210 -> 214) and the BUNDLEFIX next-free (053 -> 059) -- exactly the drift this block +exists to prevent (the per-session summaries carried the current figures while this block did +not). Re-run the scan to refresh._ - **PROPOSED / OPEN decisions (4):** D-068 (Vault substrate hardening, Roosevelt -- sole remainder is Q2 path selection at Roosevelt Vault design time), D-131 (node-facing DNS for @@ -43,19 +43,19 @@ (vault-init workflow QoL sweep -- APPROVED-IN-PRINCIPLE / IMPL-DEFERRED, R2 off-host transport unresolved). D-136/D-137 dropped off the scan when ADOPTED (correct). Status lines in `docs/design-decisions.md` are the only ruling authority. -- **OPEN security rows:** **29** per `bash scripts/ledger-scan.sh` (was 28 at the last - seed). The one added since, verified against the register: SEC-033 2026-08-04 (the - tls-certificates relation databag exposes vault's global-client private key to any juju - model reader; interface-level, mitigate via juju RBAC). **SEC-031 remains the live-exposure - one**: the dc1 edge lost its FreeBSD user database to fsck and runs with NO pf ruleset, i.e. - an open router serving its GUI. The SEC register of record is `docs/security-ledger.md`; +- **OPEN security rows:** **28** per `bash scripts/ledger-scan.sh` (was 29 at the 2026-08-05 + seed). **SEC-031 CLOSED 2026-08-07** -- the dc1 edge was REBUILT via the proven dc0 procedure + (`dc-egress-check dc1` 8/8), so the open-router live-exposure is resolved. Still open and + notable: SEC-033 2026-08-04 (tls-certificates relation databag exposes vault's global-client + private key to any juju model reader; interface-level, mitigate via juju RBAC) and SEC-020 + (per-DC MAAS/creds consolidation). The SEC register of record is `docs/security-ledger.md`; row-level dispositions live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows. -- **Next-free numbers:** D = **143**, DOCFIX = **210**, BUNDLEFIX = **053**. D moved - 141 -> 143 as D-141 (IPAM allocations dual-stack, status-distinguished) and D-142 - (vault-init QoL, PROPOSED) were assigned. DOCFIX moved 208 -> 210: **DOCFIX-208** (phase-4 - dc0 machines-overlay omission, 2026-08-02) and **DOCFIX-209** (this session's SKILL.md - close-bookend correction) are both assigned. +- **Next-free numbers:** D = **143**, DOCFIX = **214**, BUNDLEFIX = **059** (scan highest + assigned: D-142 / DOCFIX-213 / BUNDLEFIX-058). DOCFIX moved 210 -> 214 since the last seed + (through DOCFIX-213, phase-04 network scripts MAAS_PROFILE-aware, 2026-08-08); BUNDLEFIX moved + 053 -> 059 (BUNDLEFIX-056/057/058, designate/dashboard binding conformance, 2026-08-06). No + new numbers assigned 2026-08-08 part 3 (the geneve-over-v6 D-number is PROPOSED, not minted). - **Gates:** G17 OPENED 2026-07-27 by operator ruling (node-side artifact reachability split out of Stage 4; the gate table in `docs/CURRENT-STATE.md` is the authority). - **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) @@ -221,29 +221,12 @@ `docs/archive/session-ledger-rotated-20260808.md`. Rotated at the 2026-08-08 dc0-activation close to keep the live ledger under the 300-line cap (GA-R4 rule 3). -## SESSION CLOSE 2026-08-06 (part 3) -- phase-03 core-verify: F-CV1/F-CV2/F-CV3 RESOLVED + binding conformance (bounded, GA-R4) +## ROTATED 2026-08-09 (GA-R4 rule 3 -- oldest-first, cap restored at the 2026-08-08 part-3 close) -- Branch dc-dc-stage5-preconditions; 9 commits (b8d2b25..edba699 pushed; cd13baa ec8a1d3 ace0e16 UNPUSHED). Scan: 29 SEC, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (used 056/057/058). -- STEP 7 (phase-03 core-verify, adapted vr1-dc0, run from the dc0 rack): CORE-API VERIFIED (falsifiable settle walk, haproxy sweep, admin-openrc scoped token, IP-only endpoints, two-sourced VIP). F-CV2 RESOLVED: openstack CLI installed on the dc0 rack (D-138; was voffice1-only). -- F-CV1 (designate _admin backend DOWN) RESOLVED (BUNDLEFIX-056): bundle omitted public+internal -> orphaned VIP triple; fixed + live-rebound. Governing = D-052 + D-020 amdt, NOT D-141. -- F-CV3 (dashboard VIP plaintext) RESOLVED via D-072 AMENDMENT (VR1) ratified GA-R5 "Ratified, land the config-of-record" + BUNDLEFIX-057: VR1 split-metal INVERTS D-072 (charm serves metal-internal, no admin/internal binding/os-*-network lever) -> cluster->metal-internal. Proven LIVE before ratifying (operator prove-then-rule method). -- BUNDLEFIX-058: designate-stack conformance (designate amqp+cluster, designate-bind cluster -> metal-internal). Binding conformance now CLEAN cloud-wide (no other active deviations). -- D-134 Roosevelt-delta annotation (LXD container addrs auto-picked, not carved); gap-21 access-model context (tailnet -> metal-admin dashboards). -- OWNED: assert-before-check recurred (F-CV1 "CONFIRMED"+D-141 before the governing check; "lone exception" premise) + filter/WebFetch false-negatives -- all caught by sweep/governing-check-first + prove-then-rule. -- DURABILITY: this host 0 uncommitted / 3 UNPUSHED; voffice1 edba699 (=origin, syncs after push); dc0 rack ~/repo-stage/bundle.yaml STALE (O10, re-stage before any redeploy). repo-lint 0-fail; gauntlet ALL GREEN (99) at ace0e16. -- NEXT: Step 3.4 (keystone domain-manager policy PO:+G3) is the LAST phase-03 exit-gate item; then Steps 8-12 -> Stage-5 exit. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt (O11 MAAS-query method FIRST SURFACE). Body: docs/changelog-20260806-phase03-coreverify.md. Status ONLY in CURRENT-STATE.md. - -## SESSION CLOSE 2026-08-06/07 -- phase-03 Step 3.4 G3 DONE + per-DC Tailscale rulings + .7 VMs stood up both DCs (bounded, GA-R4) - -- Branch dc-dc-stage5-preconditions; pushed afbaed2..9e74f43 (7 commits); the node-vm harness reconcile is UNCOMMITTED in this bookend. voffice1 synced 9e74f43. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- all D-129/D-134 AMENDMENTS). -- STEP 3.4 (phase-03) DONE: scripts/g3-domain-manager-probe.sh (+harness 12/12) as the GA-R6 named check; live G3 PASS on dc0 (7 ok/0 fail, teardown verified clean); stage-1 PO: verified per-UNIT (app-aggregate hid it). -- DECISION C (Horizon): reconciled to VR1 -- both dashboard VIPs serve HTTPS login 200+csrftoken; D-044 + the VR0 nginx-repoint are plain-HTTP-leg artifacts that would only WEAKEN the cookie. phase-03 does NOT close; Step 3.3 SPLITS to its own gate (tailnet-access-gated). -- PER-DC TAILSCALE (D-129(iii) AMENDMENT, rulings a-d, BOTH DCs): (a) dedicated .7 VM; (b) STAR operator->DC; (c) SINGLE, HA scale-up PINNED; (d) SNAT on. D-134 octet map ->.7; D-107 citation DOCFIX-in-amendment. site-tailscale.sh (+harness 15/15). -- SUBSTRATE: .7 VMs APPLIED (tofu, gated) + MACs pinned + tofu No-changes on BOTH DCs -- dc0 tailscale; dc1 region VM + tailscale (operator "dc1 full"). Capacity re-gated FIT 874/1024=85%. -- DEFERRED: Headscale-side join (tagged authkey / autoApprovers / star ACL / Office1-untagged-fix) BLOCKED on control-plane access (operator lacks it); per-VM MAAS commission/deploy/carve/install; dc1 region SETUP workstream. -- OWNED: shipped the substrate commits without re-running the gauntlet -> node-vm exact-count 11/66->12/72 went red, caught only at the savegame gauntlet (the EXACT 2026-07-30 lesson this harness's own comment records, repeated); v6-posture mis-frame (operator corrected before it biased Decision B); app-aggregate PO: near-miss (caught per-unit); jget + g3-harness bugs (caught by fixtures); ADVISOR caught a would-be D-143 mint. -- Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 AFTER the node-vm reconcile. Sweep: docs/audit/queued-findings-20260807-tailscale-substrate.txt (O1-O4 FIRST SURFACE). Body: docs/changelog-20260806-step34-g3-probe.md. -- NEXT: when Headscale access -> the join + Office1-untagged fix; else MAAS commission/deploy/carve/install the 3 VMs + the dc1 region setup; then Horizon-over-tailnet confirm closes Step 3.3. Status ONLY in CURRENT-STATE.md. +The two oldest closed-session summaries (2026-08-06 part 3 -- phase-03 core-verify F-CV1/2/3; +AND 2026-08-06/07 -- Step 3.4 G3 + per-DC tailscale rulings + .7 VMs stood up) moved VERBATIM to +`docs/archive/session-ledger-rotated-20260809.md`. Rotated at the 2026-08-08 (part 3) close +(amphora build FIXED + geneve-over-v6 root-caused) to keep the live ledger under the 300-line cap. ## SESSION CLOSE 2026-08-07 (dc0 tailscale + dc1 region) -- dc0 .7 carved-and-ready; naming convention; vr1-dc1-region LIVE (bounded, GA-R4) @@ -287,3 +270,14 @@ - **OWNED:** grep -viE oneline mis-filter (caught, re-queried); my own echo string tripped the DOCFIX-016 maas-list guard (caught); amphora script MODEL defaulted openstack -> needs vr1-dc0 (caught, fast-fail, no partial state); "configure-resources decides G18 by execution" reframed after prep showed R8 already ruled the substance. - Gates: repo-lint 0 fail / 1 legacy warn; gauntlet ALL GREEN (102); ledger-scan reconciled (DOCFIX 213 used). Ledger rotated (08-06 x2 -> archive/session-ledger-rotated-20260808.md). - **NEXT:** diagnose retrofit incident (sudo on octavia-diskimage-retrofit/0) -> unblock the LB test; Designate real-Stage-7 (confirm os-public-hostname depth); wrap gates + G18 OWED#3 o-hm0 MTU; re-IP live-free checks -> present the re-IP ruling. Operator PUSH the 4 commits -> voffice1 pull. Sweep: docs/audit/queued-findings-20260808-dc0-activation-checkpoint.txt. Body: docs/changelog-20260808-dc0-activation.md. Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-08 (part 3) -- amphora build FIXED + geneve-over-v6 wrap-gate ROOT-CAUSED (bounded, GA-R4) + +- Branch dc-dc-stage5-preconditions; pushed this session: prior 4 (f85daa7..a6340e6) + bookend 8e2b18b + geneve records cd1f9cd/f5daadd. voffice1 pulled to f5daadd. Scan: 4 open decisions, SEC 28, next-free D-143/DOCFIX-214/BUNDLEFIX-059 (NO new numbers -- geneve D-number PROPOSED, not minted). +- AMPHORA BUILD FIXED (was part-2 retrofit exit-1 INCIDENT): two-layer root cause MEASURED -- (a) no loop devices in the retrofit LXD container; (b) dib build's apt hit UNREACHABLE public archives (build doesn't inherit model apt-proxy; charm ubuntu-mirror empty). FIX (live, gated): loop-device passthrough into the container (host-local, rebuild item) + juju config octavia-diskimage-retrofit ubuntu-mirror=. Image 775ebeba ACTIVE+tagged (two-sourced). +- G18 OWED#3 (o-hm0 MTU) PASS (1442 match); G18 obligations all discharged. +- 1 TEST LB: amphora BOOTS (nova ACTIVE -> image validated) but BLOCKED by geneve-over-v6 wrap-gate FAIL, ROOT-CAUSED: OVN geneve encap SPLIT across families -- containerized control chassis IPv4-only on data-tenant (10.12.16.x, D-134 auto-picked) vs compute METAL IPv6 (2602:f3e2:f02:30::x) -> cross-family tunnels don't pass -> 100% cross-node loss. Fix NOT applied (rebuild-relevant; warrants a D-number at ruling). Record: docs/audit/geneve-over-v6-rootcause-20260808.md. +- OWNED: nearly reported the yoga-agent as the LB cause (refuted by octavia log ConnectTimeout + ping6 100% loss -> geneve overlay); 3 diagnostic-capture false-starts (all my harness); loop-fix necessary-not-sufficient; run-logged NOT opened (Nth). Instrument-currency #23. +- LEFTOVERS (owed next session): TEAR DOWN test LB 24bd326f / net lb-smoke-* / amphora (left LIVE per operator savegame-now); live mutations (loop passthrough + ubuntu-mirror config) carry reverts in the changelog. +- Gates: repo-lint 0-fail (1 legacy warn); gauntlet ALL GREEN (102); ledger-scan reconciled; ledger rotated (08-06 x2 -> archive/session-ledger-rotated-20260809.md). +- NEXT: tear down the test LB; Designate real-Stage-7 4a depth (read-only); wrap gates (cloud-assert BOM, controller backup, verify-live Ceph); re-IP 3 live-free checks -> present re-IP GA-R5 ruling. Rebuild items: wire retrofit ubuntu-mirror per-DC in bundle + give containerized OVN chassis a v6 data-tenant address. Sweep: docs/audit/queued-findings-20260808-amphora-geneve.txt. Body: docs/changelog-20260808-amphora-geneve.md. Status ONLY in CURRENT-STATE.md.