diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index e600253..c20cb30 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2421,6 +2421,14 @@ `https://10.12.8.58` over the tailnet. **phase-03 does NOT close this session; Step 3.3 travels forward as its own gate.** The cert's IP-SAN already covers 10.12.8.58 (F-CV3), so CA-trust is the only cert residual. D-044/D-075/nginx-repoint are NOT applied (VR0-specific; would regress). + **PER-DC TAILSCALE BUILD STATUS (2026-08-07):** rulings a-d RULED (D-129(iii) amdt); tooling + built (`scripts/site-tailscale.sh` + harness); the `.7` subnet-router VMs are APPLIED via tofu + on BOTH DCs and MACs pinned (`vr1-dc0-tailscale-01`; `vr1-dc1-tailscale-01`; AND `vr1-dc1-maas-01` + the dc1 region VM, applied full per operator "dc1 full") -- all three POWERED OFF (autostart=false), + tofu No-changes. OWED next: start -> MAAS enlist/commission/deploy -> carve `.7` metal-admin + + provider-public -> install tailscale; then the Headscale-side join (tagged authkey / autoApprovers + / star ACL / Office1-untagged fix) which is BLOCKED on control-plane access; then the browser-login + confirm closes Step 3.3. dc1 region VM also owes its full region SETUP workstream. F-CV1: designate _admin backend DOWN -- **RESOLVED 2026-08-06 (BUNDLEFIX-056, operator-approved fix EXECUTED + VERIFIED).** Root cause (governing = D-052 + generic binding rule + the D-020 amendment's ruled .62 triple, NOT D-141): designate's bundle bindings OMITTED public + internal, diff --git a/docs/archive/session-ledger-rotated-20260807.md b/docs/archive/session-ledger-rotated-20260807.md new file mode 100644 index 0000000..1f408a6 --- /dev/null +++ b/docs/archive/session-ledger-rotated-20260807.md @@ -0,0 +1,19 @@ +# Rotated session-ledger summaries -- moved 2026-08-07 (GA-R4 rule 3 / F1) + +Moved VERBATIM from `docs/session-ledger.md` at the 2026-08-06/07 close to keep the live +ledger under the 300-line cap (rule 3). Oldest-first. + +--- + +## SESSION CLOSE 2026-08-03 -- Stage 5 dc0: bundle DEPLOYED, controller rebuilt, vault up; ovn-central cert DEFERRED (bounded, GA-R4) + +- Branch `dc-dc-stage5-preconditions`, ~23 commits pushed. NO stage opened/closed. Scan: 3 decisions, **SEC 28**, **D 142 / DOCFIX 209 / BUNDLEFIX 053** (D-141 + DOCFIX-208 assigned this session). +- **THE dc0 BUNDLE IS DEPLOYED and mostly converged:** 9 machines started, mysql cluster ONLINE, vault init+unseal+root-CA (operator-run), ~25 units active, 0 in error. DOCFIX-208 fixed the machines-overlay omission; UCA signing-key failure root-caused; **D-135 amendment (b)** converged dc0 onto the apt caching proxy; the **v4 VIP revert (D-141)** cleared keystone's `Invalid vips`. +- **CONTROLLER REBUILT (Path C):** a `--force` model destroy orphaned the model (missing status doc) and took the controller API down; rebuilt fresh from MAAS end to end. New runbook **Paths M + C** (juju model + controller teardown/rebuild), plus the measured `kill-controller`-fails-when-API-down + `--no-wait` guidance. +- **RULINGS:** **D-141** (IPAM dual-stack, v4 `active` / v6 `reserved`-until-capable); **D-052 AMENDED** (ovn-central `""` default -> metal-internal, architecturally correct -- OVN NB/SB DB is a metal-internal service). +- **>>> ovn-central x3 DEGRADED, DEFERRED:** charm-ovn-central **LP #2044324** -- cert request carries NO `common_name` -> vault issues no server cert -> OVN NB/SB cluster not formed. Three remedies exhausted (reissue-certificates / rebind / relation bounce). Only OVN/tenant-networking is gated. <<< +- **GATE RED AT CLOSE:** gauntlet **1/98 FAIL (provider-bundle-check)** -- the D-141 v4 revert broke 4 dual-family harness cases; LOGGED, harness owes a reconcile to the v4-only input. repo-lint 0 fail. +- **CONFIRMED (operator Q):** a dual-stack deploy overlay with unpopulated container v6 causes the `Invalid vips` block on all 11 API charms (the charm hard-errors on an unplaceable VIP) -- so v4-only-deploy + v6-reserved-in-apex (D-141) is the correct split until the container-v6 gate clears. +- **OWNED:** twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update. +- **DURABILITY:** vcloud 0/0; voffice1 was **36 behind, PULLED to sync** (new savegame **Step 1b**, operator-directed); dc0 rack `~/repo-stage` matches HEAD. +- **NEXT:** escalate LP #2044324 + decide accept-degraded vs the unverified `os-*-network` avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: `docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt` (**4 FIRST SURFACE**). Status ONLY in CURRENT-STATE.md. diff --git a/docs/audit/queued-findings-20260807-tailscale-substrate.txt b/docs/audit/queued-findings-20260807-tailscale-substrate.txt new file mode 100644 index 0000000..3aa0c4c --- /dev/null +++ b/docs/audit/queued-findings-20260807-tailscale-substrate.txt @@ -0,0 +1,89 @@ +# Queued findings -- session 2026-08-06/07: phase-03 Step 3.4 + Decision C + per-DC Tailscale +# Survives-a-clear sweep (savegame Step 3). Status authority is CURRENT-STATE.md. +# Body: docs/changelog-20260806-step34-g3-probe.md. Each item says where it already lives, +# or that THIS file is its first surface. + +=== FIRST SURFACE (transcript-only until this file) === + +O1 DOCFIX candidate -- octet-map surface LAGS. design-decisions.md:5954 (D-134 AMENDMENT + 2026-08-07, this session) states the STANDING utility octet map as .4 artifact / .5 Juju / + .6 region / .7 Tailscale. The OLDER D-132 addendum at design-decisions.md:7168 still reads + ".4 ... .5 ... .6 MAAS region" with NO .7 -- correct as of 07-30, now lagging. Append-only + register, so it is history; but a reader grepping the map at :7168 misses .7. DOCFIX: annotate + :7168 -> "extended to .7 Tailscale 2026-08-07, see the D-134 amendment". LOGGED, not fixed. + +O2 ENV/EXECUTION TRAIT: an `ssh voffice1 ''` session's default cwd is NOT the repo clone + (/home/jessea123/openstack-caracal-dc-dc); git/tofu commands fail "not a git repository" + unless the remote command LEADS with `cd $REPO`. Cost ~6 retries during this savegame's + Step-1b sync. Prepend the cd in every remote git/tofu invocation. First surface. + +O3 AS-EXECUTED LOG IS PARTIAL for this window (F6 class). `run-logged.sh` was NOT opened this + session; the live mutations (G3 probe, four tofu applies, four MAC-pin applies) ran gated + but unwrapped. Every action is in the changelog + CURRENT-STATE with read-backs, and the G3 + probe has its own capture (docs/audit/g3-dc0-probe-20260806.txt), but the as-executed log + must NOT be read as complete for this window. + +O4 TOFU STATE DOES NOT CARRY AUTO-GENERATED MACs. With `macs = []`, the libvirt provider does + not read the generated MAC addresses back into tofu state (`tofu state show` shows none); + they must be captured live via `virsh domiflist ` over the qemu+ssh provider URI, then + pinned. The region-VM comments already say "pin from virsh domiflist"; the state-blindness + nuance is recorded here so a future session does not look for them in state. + +=== ALREADY ON SURFACE (recorded where noted) === + +R1 Decision C -- phase-03 Horizon reconciled to VR1; Step 3.3 splits to its own gate row. + Operator: "We need to pull the tailscale steps forward so we can close out horizon properly." + -> CURRENT-STATE.md (phase-03 (c) clause) + changelog Item 5. + +R2 Four Tailscale rulings (a-d), EXACT utterances, + the "both DCs" directive: + (a) "Dedicated VM at utility .7 (Recommended)"; (b) "Star: operator->DC only (Recommended)"; + (c) "We will not be creating HA for this now. Pin HA scale up for Headscale/Tailscale."; + (d) "SNAT ON now; pin source-IP preservation (Recommended)"; "Lets plan and push to both DC0 + and DC1 in this step." -> design-decisions.md D-129(iii) AMENDMENT 2026-08-07 + D-134 + AMENDMENT + gap-21 register row + changelog Item 6. + +R3 Substrate apply scope ruling: "dc0 full + dc1 FULL (also the region VM)" -> changelog Item 9. + Build-pace ruling: "Push, build tooling AND stand up the .7 VMs". G3 build: "Build g3-probe.sh + + harness"; "Run G3 probe now". -> changelog Items 1/8/9. + +R4 Measurements: G3 PASS live (7 ok/0 fail, teardown clean); Step 3.4 stage-1 PO: at UNIT level + (app-aggregate hid it); both dashboard VIPs HTTPS 200 + csrftoken Secure; D-044/D-075 NOT + applied; cert IP-SAN covers 10.12.8.58; capacity FIT 874/1024=85%; dc1 plan 4-add (region VM + bundled); 3 VMs applied + MACs pinned + tofu clean. -> CURRENT-STATE + changelog Items 3-9 + + substrate main.tf comments. + +R5 Headscale facts: control plane tailscale.baldurkeep.com (Cloudflare-fronted, server version + UNMEASURED, operator no access this session); Office1 node UNTAGGED (AdvertiseTags null, + 180-day key-expiry defect to fix); star ACL / autoApprovers / tagged authkey are the + control-plane prerequisites. -> D-129(iii) amendment notes 1+4 + site-tailscale.sh header. + +R6 Security note: the .7 tailscale VM attaches all six planes (node-vm module default) but + ADVERTISES only metal-admin; the other five legs stay uncarved/unrouted. -> substrate main.tf + comment (both DCs). + +=== DELIBERATELY NOT DONE (owed, next sessions) === + +N1 Headscale-side build: tagged pre-auth key, autoApprovers (write BEFORE first advertise), + star ACL, the join via site-tailscale.sh install, fix the Office1 untagged node. BLOCKED on + Headscale control-plane access (operator lacks it this session). +N2 Per VM (3): start -> MAAS enlist/commission/deploy Ubuntu -> carve legs (.7 metal-admin + + provider-public gw) -> install tailscale. All VMs currently powered off (autostart=false). +N3 dc1 MAAS-region SETUP workstream (vr1-dc1-maas-01 stood up but not configured): init / + PostgreSQL / image sync / eventual dc1 node migration. +N4 SEC row for per-DC Tailscale key custody -- opens at authkey-mint time in the Headscale build + (D-129(iii) amendment note). +N5 O10 (carried, pre-existing): dc0/dc1 rack ~/repo-stage/bundle.yaml STALE vs repo; re-stage + before any redeploy. This session touched no bundle/overlay, so the staleness is unchanged. + +=== GITIGNORED / THROWAWAY (Step 3d.1) === +- Throwaway tofu saved plans on voffice1 (dc0-tailscale.tfplan, dc0-macpin.tfplan, + dc1-full.tfplan, dc1-macpin.tfplan) -- gitignored, already applied, no durable value. +- No permission-rule (.claude/settings.local.json) changes this session. + +=== STAGE-CLOSE OWED (added 2026-08-07, operator-directed "make sure the v6 posture carries forward") === +O5 Fold the IPv6-PRIMARY posture invariant into the openstack-cloud-ops SKILL Posture section at + the next STAGE close (the skill is the invariant home; swept at stage close). Auto-memory + `ipv6-primary-posture.md` is the always-loaded defense NOW (added this session, pointer to + D-101/D-139/D-141); the skill Posture line is the second surface so a stage-close skill sweep + also carries it. The v6 posture is IPv6-primary (v6 wherever possible, v4/dual-stack only where + forced); D-141's v4-active is the NARROW container-VIP necessity case, not a cloud-wide lean. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 5f15353..64a6646 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -207,18 +207,12 @@ `docs/archive/session-ledger-rotated-20260802.md`. The live ledger stood at 283 lines and this close's summary would have breached the 300-line cap. -## SESSION CLOSE 2026-08-03 -- Stage 5 dc0: bundle DEPLOYED, controller rebuilt, vault up; ovn-central cert DEFERRED (bounded, GA-R4) +## ROTATED 2026-08-07 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close) -- Branch `dc-dc-stage5-preconditions`, ~23 commits pushed. NO stage opened/closed. Scan: 3 decisions, **SEC 28**, **D 142 / DOCFIX 209 / BUNDLEFIX 053** (D-141 + DOCFIX-208 assigned this session). -- **THE dc0 BUNDLE IS DEPLOYED and mostly converged:** 9 machines started, mysql cluster ONLINE, vault init+unseal+root-CA (operator-run), ~25 units active, 0 in error. DOCFIX-208 fixed the machines-overlay omission; UCA signing-key failure root-caused; **D-135 amendment (b)** converged dc0 onto the apt caching proxy; the **v4 VIP revert (D-141)** cleared keystone's `Invalid vips`. -- **CONTROLLER REBUILT (Path C):** a `--force` model destroy orphaned the model (missing status doc) and took the controller API down; rebuilt fresh from MAAS end to end. New runbook **Paths M + C** (juju model + controller teardown/rebuild), plus the measured `kill-controller`-fails-when-API-down + `--no-wait` guidance. -- **RULINGS:** **D-141** (IPAM dual-stack, v4 `active` / v6 `reserved`-until-capable); **D-052 AMENDED** (ovn-central `""` default -> metal-internal, architecturally correct -- OVN NB/SB DB is a metal-internal service). -- **>>> ovn-central x3 DEGRADED, DEFERRED:** charm-ovn-central **LP #2044324** -- cert request carries NO `common_name` -> vault issues no server cert -> OVN NB/SB cluster not formed. Three remedies exhausted (reissue-certificates / rebind / relation bounce). Only OVN/tenant-networking is gated. <<< -- **GATE RED AT CLOSE:** gauntlet **1/98 FAIL (provider-bundle-check)** -- the D-141 v4 revert broke 4 dual-family harness cases; LOGGED, harness owes a reconcile to the v4-only input. repo-lint 0 fail. -- **CONFIRMED (operator Q):** a dual-stack deploy overlay with unpopulated container v6 causes the `Invalid vips` block on all 11 API charms (the charm hard-errors on an unplaceable VIP) -- so v4-only-deploy + v6-reserved-in-apex (D-141) is the correct split until the container-v6 gate clears. -- **OWNED:** twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update. -- **DURABILITY:** vcloud 0/0; voffice1 was **36 behind, PULLED to sync** (new savegame **Step 1b**, operator-directed); dc0 rack `~/repo-stage` matches HEAD. -- **NEXT:** escalate LP #2044324 + decide accept-degraded vs the unverified `os-*-network` avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: `docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt` (**4 FIRST SURFACE**). Status ONLY in CURRENT-STATE.md. +The oldest closed-session summary (2026-08-03 -- Stage 5 dc0 bundle DEPLOYED, controller +rebuilt Path C, ovn-central cert DEFERRED) moved VERBATIM to +`docs/archive/session-ledger-rotated-20260807.md`. The live ledger stood at 292 lines and this +close's summary would have breached the 300-line cap. ## SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4) @@ -290,3 +284,15 @@ - OWNED: assert-before-check recurred (F-CV1 "CONFIRMED"+D-141 before the governing check; "lone exception" premise) + filter/WebFetch false-negatives -- all caught by sweep/governing-check-first + prove-then-rule. - DURABILITY: this host 0 uncommitted / 3 UNPUSHED; voffice1 edba699 (=origin, syncs after push); dc0 rack ~/repo-stage/bundle.yaml STALE (O10, re-stage before any redeploy). repo-lint 0-fail; gauntlet ALL GREEN (99) at ace0e16. - NEXT: Step 3.4 (keystone domain-manager policy PO:+G3) is the LAST phase-03 exit-gate item; then Steps 8-12 -> Stage-5 exit. Sweep: docs/audit/queued-findings-20260806-phase03-coreverify.txt (O11 MAAS-query method FIRST SURFACE). Body: docs/changelog-20260806-phase03-coreverify.md. Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-06/07 -- phase-03 Step 3.4 G3 DONE + per-DC Tailscale rulings + .7 VMs stood up both DCs (bounded, GA-R4) + +- Branch dc-dc-stage5-preconditions; pushed afbaed2..9e74f43 (7 commits); the node-vm harness reconcile is UNCOMMITTED in this bookend. voffice1 synced 9e74f43. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- all D-129/D-134 AMENDMENTS). +- STEP 3.4 (phase-03) DONE: scripts/g3-domain-manager-probe.sh (+harness 12/12) as the GA-R6 named check; live G3 PASS on dc0 (7 ok/0 fail, teardown verified clean); stage-1 PO: verified per-UNIT (app-aggregate hid it). +- DECISION C (Horizon): reconciled to VR1 -- both dashboard VIPs serve HTTPS login 200+csrftoken; D-044 + the VR0 nginx-repoint are plain-HTTP-leg artifacts that would only WEAKEN the cookie. phase-03 does NOT close; Step 3.3 SPLITS to its own gate (tailnet-access-gated). +- PER-DC TAILSCALE (D-129(iii) AMENDMENT, rulings a-d, BOTH DCs): (a) dedicated .7 VM; (b) STAR operator->DC; (c) SINGLE, HA scale-up PINNED; (d) SNAT on. D-134 octet map ->.7; D-107 citation DOCFIX-in-amendment. site-tailscale.sh (+harness 15/15). +- SUBSTRATE: .7 VMs APPLIED (tofu, gated) + MACs pinned + tofu No-changes on BOTH DCs -- dc0 tailscale; dc1 region VM + tailscale (operator "dc1 full"). Capacity re-gated FIT 874/1024=85%. +- DEFERRED: Headscale-side join (tagged authkey / autoApprovers / star ACL / Office1-untagged-fix) BLOCKED on control-plane access (operator lacks it); per-VM MAAS commission/deploy/carve/install; dc1 region SETUP workstream. +- OWNED: shipped the substrate commits without re-running the gauntlet -> node-vm exact-count 11/66->12/72 went red, caught only at the savegame gauntlet (the EXACT 2026-07-30 lesson this harness's own comment records, repeated); v6-posture mis-frame (operator corrected before it biased Decision B); app-aggregate PO: near-miss (caught per-unit); jget + g3-harness bugs (caught by fixtures); ADVISOR caught a would-be D-143 mint. +- Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 AFTER the node-vm reconcile. Sweep: docs/audit/queued-findings-20260807-tailscale-substrate.txt (O1-O4 FIRST SURFACE). Body: docs/changelog-20260806-step34-g3-probe.md. +- NEXT: when Headscale access -> the join + Office1-untagged fix; else MAAS commission/deploy/carve/install the 3 VMs + the dc1 region setup; then Horizon-over-tailnet confirm closes Step 3.3. Status ONLY in CURRENT-STATE.md. diff --git a/tests/node-vm/run-tests.sh b/tests/node-vm/run-tests.sh index 11559f8..acaebfc 100755 --- a/tests/node-vm/run-tests.sh +++ b/tests/node-vm/run-tests.sh @@ -71,16 +71,24 @@ # session's gauntlet. Recorded here rather than silently corrected -- the lesson is that # an exact-count assertion is only as good as the gauntlet run that follows the commit. # - # ASYMMETRY IS EXPECTED RIGHT NOW, and is NOT a defect: dc1's root also carries 11 macs - # lists but only 60 literals, because `vr1-dc1-maas-01` is authored with `macs = []` and - # has NOT been applied -- there is nothing measured to pin yet. When dc1's region VM is - # applied and pinned, dc1 reaches 66 too. This harness reads the dc0 root only. - [ "$MACS_LISTS" -eq 11 ] \ - && ok "T8 inner root: 11 per-node macs lists (9 role + juju controller + MAAS region)" \ - || no "T8 inner root: 11 per-node macs lists (found $MACS_LISTS)" - [ "$MAC_LITERALS" -eq 66 ] \ - && ok "T9 inner root: 66 pinned MAC literals (11 nodes x 6 planes)" \ - || no "T9 inner root: 66 pinned MAC literals (found $MAC_LITERALS)" + # RE-POINTED AGAIN 2026-08-07: now 12 / 72. D-129(iii) AMENDMENT (rulings a-d) adds the + # DEDICATED per-DC TAILSCALE subnet router -tailscale-01 at utility .7, applied at dc0 + # with its six MACs pinned from measurement the same day (same enlist-boundary reasoning as + # the juju controller and region VMs). AND IT WENT RED THE SAME WAY AS 2026-07-30: the + # session that added the VM re-ran the gauntlet after the TOOLING commit (101 green) but not + # after the SUBSTRATE commit, so 11->12 went undetected until the savegame gauntlet -- the + # exact lesson this comment already records. Re-run the gauntlet after EVERY commit that + # touches the inner root, not just the last one. + # + # ASYMMETRY IS GONE: as of 2026-08-07 dc1's root ALSO carries 12 macs lists and 72 literals + # -- `vr1-dc1-maas-01` was applied+pinned and `vr1-dc1-tailscale-01` added the same day. Both + # roots are 12/72. This harness still reads the dc0 root only. + [ "$MACS_LISTS" -eq 12 ] \ + && ok "T8 inner root: 12 per-node macs lists (9 role + juju controller + MAAS region + Tailscale)" \ + || no "T8 inner root: 12 per-node macs lists (found $MACS_LISTS)" + [ "$MAC_LITERALS" -eq 72 ] \ + && ok "T9 inner root: 72 pinned MAC literals (12 nodes x 6 planes)" \ + || no "T9 inner root: 72 pinned MAC literals (found $MAC_LITERALS)" DUPES="$(grep -oE '"([0-9a-f]{2}:){5}[0-9a-f]{2}"' "$INNER" | sort | uniq -d | wc -l)" [ "$DUPES" -eq 0 ] \ && ok "T10 inner root: no duplicate MACs" \