diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index c9989c7..3fe07fa 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2433,8 +2433,10 @@ Ready; (3) NEW aux-carve tooling (`dc-node-carve.sh` `is_tailscale_host`/`is_two_plane_host` + lib-hosts `CARVE_AUX_HOSTS`, harness 58/0, gauntlet ALL GREEN 101 -- `docs/audit/gauntlet-20260807-tailscale-carve.txt`); (4) carve applied + verified `pass=8 fail=0` (metal-admin `10.12.8.7`, provider-public `10.12.4.7` - VLAN 5002, NO br-ex); (5) MAAS-deployed jammy -> **Deployed**, sshd live on `10.12.8.7`. The `.7` - router is NOT juju-managed so it MUST be MAAS-deployed (DOCFIX-200's "stay Ready" is juju-only). + VLAN 5002, NO br-ex); (5) MAAS-deployed jammy -> **Deployed**; BOTH carved legs rendered + live + (from the rack, provider-public leg MEASURED `virbr5 10.12.4.2/22`: ping `10.12.8.7` AND + `10.12.4.7` 0% loss). Only the in-VM default route via `10.12.4.1` is unproven (G17-class, + join-time). The `.7` router is NOT juju-managed so it MUST be MAAS-deployed (DOCFIX-200 juju-only). **TWO JOIN PREREQUISITES REMAIN, both off-session:** (a) a TAGGED pre-auth key + Headscale autoApprovers/star-ACL (operator's key is PLAIN -- ruled 2026-08-07; a plain key cannot authorise `tag:subnet-router`, so the join was deliberately NOT attempted -- an untagged first-advertise is diff --git a/docs/archive/session-ledger-rotated-20260807.md b/docs/archive/session-ledger-rotated-20260807.md index 1f408a6..2737e17 100644 --- a/docs/archive/session-ledger-rotated-20260807.md +++ b/docs/archive/session-ledger-rotated-20260807.md @@ -17,3 +17,16 @@ - **OWNED:** twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update. - **DURABILITY:** vcloud 0/0; voffice1 was **36 behind, PULLED to sync** (new savegame **Step 1b**, operator-directed); dc0 rack `~/repo-stage` matches HEAD. - **NEXT:** escalate LP #2044324 + decide accept-degraded vs the unverified `os-*-network` avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: `docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt` (**4 FIRST SURFACE**). Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4) + +- Branch `dc-dc-stage5-preconditions`, **5 commits** pushed (`bd7a1d0..9b49b84`; the day opened on the provider-bundle-check reconcile to D-141). NO stage opened/closed. Scan: 3 decisions, **SEC 29** (SEC-033 opened), D 142 / DOCFIX 209 / BUNDLEFIX 053 (no new arch numbers -- all OPS). +- **>>> ovn-central "awaiting server certificate data" ROOT-CAUSED (measured). <<<** charm-ovn-central derives its TLS common_name from `get_hostname(its metal-internal address)`; metal-internal is the DELIBERATELY-ISOLATED D-052 plane with no reachable resolver, so the reverse lookup returns None -> empty CN -> vault issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR exist; only the reverse is unreachable from the isolated plane. +- **The prior diagnosis was WRONG on every count, corrected in-record:** LP #2044324 NO MATCH; rdns_mode fix VOID (all planes already 2); dns_servers fix VOID; binding approach REFUTED live (3 configs -- the charm uses the metal-internal address regardless of binding). +- **App STAYS on metal-internal** (D-052-correct for its OVSDB/certificates data type -- operator call). The fix does NOT relocate it. +- **Fix PROVEN end-to-end (controlled single-unit LIVE test):** an /etc/hosts reverse entry -> CN populated -> vault issued `ovn-central_0.server.cert` -> the charm wrote its /etc/ovn cert files; the two control units without the entry stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN. +- **WIRED for the redeploy:** NEW `scripts/dc-node-etchosts.sh` + `tests/dc-node-etchosts` (9/9) render a per-DC `cloudinit-userdata` adding each node's metal-internal address -> hostname to /etc/hosts at provision (CIDR from lib-net; the harness caught a guessed dc1 value). Applied as gated **Step 1.2b** of `phase-01-bundle-deploy.md`. A NEW mechanism borrowing D-008's shape, NOT D-008. +- **F2/F3 resolve in the clean redeploy:** the hacluster stale-IPv6-CIB block (4 units) is gone from the v4-only config-of-record; octavia's error is downstream of ovn-central. Deploy input verified v4-only. +- Gauntlet **ALL GREEN (99)**, repo-lint 0 fail. voffice1 PULLED to sync (`9b49b84`); dc0 rack `~/repo-stage` lacks the new script (redeploy prereq -- sweep F3). Live tests were reversible; model at its captured before-state (residue: certificates:145, /0 holds its proof cert -- sweep F2). +- **OWNED:** called the root cause wrong THREE times before right; explained away a null resolvectl result (advisor caught it); two false-negative binding tests (didn't re-fire the relation); nearly wired the fix without confirming a server cert issues (advisor made that test blocking). +- **NEXT:** we are at **STEP 3** -- Path M model teardown -> preflight -> phase-01 (with Step 1.2b) -> phase-02 vault init -> converge. Sweep: `docs/audit/queued-findings-20260804-ovn-cert-fix.txt` (4 FIRST SURFACE). Body: `docs/changelog-20260804-ovn-central-cert-fix.md`. Status ONLY in CURRENT-STATE.md. diff --git a/docs/audit/queued-findings-20260807-dc0-tailscale-provisioning.txt b/docs/audit/queued-findings-20260807-dc0-tailscale-provisioning.txt new file mode 100644 index 0000000..9b05b7f --- /dev/null +++ b/docs/audit/queued-findings-20260807-dc0-tailscale-provisioning.txt @@ -0,0 +1,97 @@ +# Queued findings -- session 2026-08-07: dc0 Tailscale .7 provisioning to carved-and-ready +# Survives-a-clear sweep (savegame Step 3). Status authority is CURRENT-STATE.md. +# Body: docs/changelog-20260807-dc0-tailscale-provisioning.md. Commits d36d815..c8ddfb6. +# Each item says where it already lives, or that THIS file is its first surface. + +=== FIRST SURFACE (transcript/changelog-only until this file) === + +F1 GAP -- NO RUNBOOK for building a per-DC MAAS REGION in the DC. The 2026-08-07 operator + ruling "No migration. Build region on DC1 correctly." makes dc1's next step the standup of + vr1-dc1-region on vr1-dc1-maas-01 (init / PostgreSQL / image sync / SSH-key import / DHCP / + rack attach), but no runbook covers it. runbooks/dc-dc-phase3's own gap-#1 notes "No + OpenTofu module stands up a per-DC MAAS rack controller VM" -- a DIFFERENT gap (substrate). + The REGION-CONFIG runbook is absent. dc0's region came from the ONE-TIME migration (not a + procedure). Next session building dc1's region has no tested path -- propose authoring one + (or extending phase-00-maas-standup, which currently REFUSES vr1-* per the skill). HIGH + consequence: the ruled dc1 work is blocked on an unwritten runbook. + +F2 FINDING (candidate SEC row / D-137 fork) -- vr1-dc0-region injects the vr1-office1-svc SSH + key into DEPLOYED machines (MEASURED: `maas vr1-dc0-region sshkeys read` = 1 key, + "vr1-office1-svc"). So the .7 VM (and every dc0-region deploy) is reachable ONLY by the + holder of the OFFICE1 service key, not a per-DC key. This is cross-DC key reuse in tension + with SEC-012/SEC-016 (per-DC power/SSH key isolation, "NEVER a cross-DC reuse"). The + vr1-office1-svc key name is already on surface (CURRENT-STATE, changelog-20260730-migration); + the finding that it is the DEPLOY-INJECTED key on dc0-region -- and the SEC tension -- is new. + Pre-existing (region-wide), logged-not-actioned (hard rule 1). Consequence for the join: + `site-tailscale.sh install` must run as the vr1-office1-svc holder over `ssh -J voffice1, + ubuntu@10.12.8.7` (operator has this key; I did NOT fish for it). + +F3 DOCFIX candidate -- runbooks/dc-dc-phase3 Step 4 (DOCFIX-200) reads "nodes STAY Ready; NO + MAAS deploy" as a blanket rule. It is JUJU-NODE-specific (Juju allocates Ready machines and + cannot allocate a Deployed one). A NON-juju aux VM (the Tailscale .7 router) is NOT in the + juju model and MUST be MAAS-deployed to carry an OS -- the runbook carries no exception note, + so a reader could wrongly leave an aux VM in Ready forever. Owed: an aux-VM carve+deploy note + on phase-3 (the carve-while-Ready -> MAAS-deploy order, since MAAS refuses interface changes + on a Deployed machine). + +F4 ENV/CLASSIFIER -- the carve `check` command with a trailing `| tail -12; rc=${PIPESTATUS...}` + was BLOCKED by the auto-mode classifier; the identical plain form (no pipe/PIPESTATUS) passed. + Matches the standing multi-workstation memo (a wrapped form fails to MATCH an ask rule). Use + the plain form for remote carve/maas reads; do not add a bypass. + +=== ALREADY ON SURFACE (recorded where noted) === + +R1 Ruling (GA-R5, verbatim 2026-08-07): dc1 = "No migration. Build region on DC1 correctly." + -> CURRENT-STATE tailscale block + changelog-20260807 header + F1 above. Reaffirms the + standing region-per-DC (D-132 q1); NO new D-number (operational sequencing + reaffirmation). + +R2 Operator statement 2026-08-07: the tagged-join key on hand is PLAIN (not tag-scoped) -> + CURRENT-STATE (join prereq a) + changelog header + Item 5. The join was deliberately NOT + attempted (an untagged first-advertise is the office1 defect; Headscale does not approve + routes retroactively). + +R3 Measured corrections (GA-R1 C2): (i) dc0 .7 already self-enlisted as known-marten (n4parh), + New/power-unset -- prior "powered off" omitted the enlisted part; (ii) vr1-dc1-region NOT + registered, dc1 rack nmpcq4 + 9 nodes in the Office1 admin region; (iii) subtle-grouse is the + juju controller (power_id vr1-dc0-juju-01), MAAS-random name, carved by lib-hosts LOGICAL name + + boot MAC. -> changelog-20260807 "Measured corrections" + CURRENT-STATE. + +R4 dc0 .7 driven to carved-and-ready: power set (10.12.8.2), commissioned Ready, aux-carve + tooling (gauntlet 101), carve pass=8/0, deployed jammy, BOTH legs live (ping 10.12.8.7 + + 10.12.4.7 0% from the rack, provider-public leg virbr5 10.12.4.2/22). -> changelog Items 1-4, + CURRENT-STATE, docs/audit/gauntlet-20260807-tailscale-carve.txt, commits d36d815/c8ddfb6. + +R5 aux-carve tooling (CARVE_AUX_HOSTS, is_tailscale_host/is_two_plane_host, harness 58/0) -> + changelog Item 2; implements D-129(iii) amdt + D-134 octet map, no new D-number. + +=== STRUCTURALLY INVISIBLE (savegame Step 3d) === + +S1 Gitignored state: no NEW gitignored artifact created this session (no settings.local.json + permission change, no new key/overlay). Last session's applied throwaway tfplans unchanged. + +S2 Dangling refs: commits d36d815/c8ddfb6 cite docs/audit/gauntlet-20260807-tailscale-carve.txt + and docs/changelog-20260807-dc0-tailscale-provisioning.md -- both EXIST. + +S3 As-executed log (O3, SECOND CONSECUTIVE session): run-logged.sh was NOT opened (it opens an + interactive script(1) subshell a background agent cannot drive). Every live mutation is gated + + captured in the changelog with read-backs, but the as-executed log MUST NOT be read as + complete for this window. Recurs in queued-findings-20260807-tailscale-substrate (O3) and + earlier -- the recurrence itself is the finding; a durable fix (non-interactive capture path) + is owed. + +S4 Contradiction detector: the known-marten-enlisted vs "powered off" divergence is the only + doc-vs-measurement conflict; corrected in CURRENT-STATE (C2). No unrecorded contradiction. + +=== DELIBERATELY NOT DONE (owed, next sessions) === + +N1 The Tailscale JOIN: tagged pre-auth key + Headscale autoApprovers / star ACL / Office1-untagged + fix, then site-tailscale.sh install, then browser-login https://10.12.8.58 closes Step 3.3. + BLOCKED on a TAGGED key (operator has a plain one) + control-plane policy work. +N2 dc1 MAAS-region build (vr1-dc1-region on vr1-dc1-maas-01) -- ruled this session, no runbook (F1). +N3 dc1 .7 provisioning (power/commission/carve/deploy) -- AFTER N2. lib-hosts already carries + vr1-dc1-tailscale-01 (octet 7, MAC 52:54:00:f9:d6:df); the aux-carve reuses on dc1. +N4 dc0 .7 package pre-install + in-VM default-route verify -- needs SSH via vr1-office1-svc + (F2), which I do not hold. Deferred to the join session (site-tailscale.sh install exercises + egress + route directly). +N5 SEC row for per-DC Tailscale key custody -- opens at tagged-key mint (carried from last + session's N4 / D-129(iii) amdt note). diff --git a/docs/changelog-20260807-dc0-tailscale-provisioning.md b/docs/changelog-20260807-dc0-tailscale-provisioning.md index 23d70a7..b76e28d 100644 --- a/docs/changelog-20260807-dc0-tailscale-provisioning.md +++ b/docs/changelog-20260807-dc0-tailscale-provisioning.md @@ -111,9 +111,12 @@ jammy chosen = the established DC image (role nodes) and confirmed deployable (`boot-resources read`: ubuntu/jammy + ubuntu/noble). Set distro_series explicitly (hard rule 2). -READ-BACK: the VM's sshd answers on metal-admin `10.12.8.7` (a ProxyJump via the rack reached -`Permission denied (publickey)` = the leg is live + sshd up). In-VM route/egress NOT verified -- -see Item 5. +READ-BACK: the VM's sshd answers on metal-admin `10.12.8.7`. BOTH carved legs rendered and are +live -- from the dc0 rack (provider-public leg MEASURED `virbr5 10.12.4.2/22`, not inferred): +ping `10.12.8.7` 0% loss AND ping `10.12.4.7` 0% loss. So netplan rendered both legs; the +provider-public egress leg (the reason the leg exists) is up. The in-VM DEFAULT ROUTE via +`10.12.4.1` is the only unproven piece (needs in-VM access; a G17-class first-boot fact, +exercised at join time by `site-tailscale.sh install`). See Item 5 for the access gap. REVERT: `maas vr1-dc0-region machine release n4parh` -> back to Ready (the carve persists). diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 64a6646..86815d9 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -209,23 +209,10 @@ ## ROTATED 2026-08-07 (GA-R4 rule 3 / F1 -- oldest-first, cap restored at this close) -The oldest closed-session summary (2026-08-03 -- Stage 5 dc0 bundle DEPLOYED, controller -rebuilt Path C, ovn-central cert DEFERRED) moved VERBATIM to -`docs/archive/session-ledger-rotated-20260807.md`. The live ledger stood at 292 lines and this -close's summary would have breached the 300-line cap. - -## SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4) - -- Branch `dc-dc-stage5-preconditions`, **5 commits** pushed (`bd7a1d0..9b49b84`; the day opened on the provider-bundle-check reconcile to D-141). NO stage opened/closed. Scan: 3 decisions, **SEC 29** (SEC-033 opened), D 142 / DOCFIX 209 / BUNDLEFIX 053 (no new arch numbers -- all OPS). -- **>>> ovn-central "awaiting server certificate data" ROOT-CAUSED (measured). <<<** charm-ovn-central derives its TLS common_name from `get_hostname(its metal-internal address)`; metal-internal is the DELIBERATELY-ISOLATED D-052 plane with no reachable resolver, so the reverse lookup returns None -> empty CN -> vault issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR exist; only the reverse is unreachable from the isolated plane. -- **The prior diagnosis was WRONG on every count, corrected in-record:** LP #2044324 NO MATCH; rdns_mode fix VOID (all planes already 2); dns_servers fix VOID; binding approach REFUTED live (3 configs -- the charm uses the metal-internal address regardless of binding). -- **App STAYS on metal-internal** (D-052-correct for its OVSDB/certificates data type -- operator call). The fix does NOT relocate it. -- **Fix PROVEN end-to-end (controlled single-unit LIVE test):** an /etc/hosts reverse entry -> CN populated -> vault issued `ovn-central_0.server.cert` -> the charm wrote its /etc/ovn cert files; the two control units without the entry stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN. -- **WIRED for the redeploy:** NEW `scripts/dc-node-etchosts.sh` + `tests/dc-node-etchosts` (9/9) render a per-DC `cloudinit-userdata` adding each node's metal-internal address -> hostname to /etc/hosts at provision (CIDR from lib-net; the harness caught a guessed dc1 value). Applied as gated **Step 1.2b** of `phase-01-bundle-deploy.md`. A NEW mechanism borrowing D-008's shape, NOT D-008. -- **F2/F3 resolve in the clean redeploy:** the hacluster stale-IPv6-CIB block (4 units) is gone from the v4-only config-of-record; octavia's error is downstream of ovn-central. Deploy input verified v4-only. -- Gauntlet **ALL GREEN (99)**, repo-lint 0 fail. voffice1 PULLED to sync (`9b49b84`); dc0 rack `~/repo-stage` lacks the new script (redeploy prereq -- sweep F3). Live tests were reversible; model at its captured before-state (residue: certificates:145, /0 holds its proof cert -- sweep F2). -- **OWNED:** called the root cause wrong THREE times before right; explained away a null resolvectl result (advisor caught it); two false-negative binding tests (didn't re-fire the relation); nearly wired the fix without confirming a server cert issues (advisor made that test blocking). -- **NEXT:** we are at **STEP 3** -- Path M model teardown -> preflight -> phase-01 (with Step 1.2b) -> phase-02 vault init -> converge. Sweep: `docs/audit/queued-findings-20260804-ovn-cert-fix.txt` (4 FIRST SURFACE). Body: `docs/changelog-20260804-ovn-central-cert-fix.md`. Status ONLY in CURRENT-STATE.md. +The oldest closed-session summaries (2026-08-03 -- Stage 5 dc0 bundle DEPLOYED, controller +rebuilt Path C; AND 2026-08-04 -- ovn-central cert root-caused + wired for redeploy) moved +VERBATIM to `docs/archive/session-ledger-rotated-20260807.md`. Rotated oldest-first across the +2026-08-07 closes to keep the live ledger under the 300-line cap (GA-R4 rule 3). ## SESSION CLOSE 2026-08-05 -- vault init DONE + ovn-central RESOLVED; D-142 vault-init QoL saved (bounded, GA-R4) @@ -296,3 +283,14 @@ - OWNED: shipped the substrate commits without re-running the gauntlet -> node-vm exact-count 11/66->12/72 went red, caught only at the savegame gauntlet (the EXACT 2026-07-30 lesson this harness's own comment records, repeated); v6-posture mis-frame (operator corrected before it biased Decision B); app-aggregate PO: near-miss (caught per-unit); jget + g3-harness bugs (caught by fixtures); ADVISOR caught a would-be D-143 mint. - Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 AFTER the node-vm reconcile. Sweep: docs/audit/queued-findings-20260807-tailscale-substrate.txt (O1-O4 FIRST SURFACE). Body: docs/changelog-20260806-step34-g3-probe.md. - NEXT: when Headscale access -> the join + Office1-untagged fix; else MAAS commission/deploy/carve/install the 3 VMs + the dc1 region setup; then Horizon-over-tailnet confirm closes Step 3.3. Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-07 (dc0 tailscale) -- dc0 Tailscale .7 driven to CARVED-AND-READY + aux-carve tooling (bounded, GA-R4) + +- Branch dc-dc-stage5-preconditions; pushed `d36d815..c8ddfb6` (2 commits) + this bookend. voffice1 synced. Scan: 4 open decisions, SEC 29, next-free D-143 / DOCFIX-213 / BUNDLEFIX-059 (NO new numbers -- implements the D-129(iii) amendment). +- dc0 .7 subnet-router DRIVEN TO CARVED-AND-READY (all gated, read back): power set (`10.12.8.2`) -> commissioned Ready -> carve pass=8/0 (metal-admin `10.12.8.7` + provider-public `10.12.4.7`, no br-ex) -> MAAS-deployed jammy; BOTH legs live (ping 0% from the rack). MEASURED correction: the .7 had ALREADY self-enlisted as `known-marten` (prior close's "powered off" omitted the enlisted part). +- NEW aux-carve tooling: `dc-node-carve.sh` `is_tailscale_host`/`is_two_plane_host` + lib-hosts `CARVE_AUX_HOSTS` (isolates the .7 from every HOSTS consumer); harness 58/0; gauntlet ALL GREEN 101. +- RULINGS (verbatim): dc1 = "No migration. Build region on DC1 correctly." (dc1 .7 GATED behind building `vr1-dc1-region` -- MEASURED not registered); the join key on hand is PLAIN (not tag-scoped) -> the Tailscale JOIN stays blocked. +- TWO JOIN PREREQS remain (off-session): a TAGGED pre-auth key + Headscale autoApprovers/ACL; SSH access via `vr1-office1-svc` (region injects only that key -- operator holds it). +- OWNED: nearly read the prior close's "powered off" as "not enlisted" (measured `known-marten` first); the advisor caught the unverified provider-public leg (then measured both legs live) and the missing bookend; `run-logged.sh` NOT opened (O3, 2nd consecutive -- a background agent cannot drive its interactive subshell). +- Gates: repo-lint 0 fail (1 legacy warn); gauntlet ALL GREEN 101 (docs-only edits since). Sweep: `docs/audit/queued-findings-20260807-dc0-tailscale-provisioning.txt` (F1-F4 FIRST SURFACE: no per-DC MAAS-region-build runbook; the vr1-office1-svc inject vs SEC-012/016; phase-3 aux-deploy DOCFIX). Body: `docs/changelog-20260807-dc0-tailscale-provisioning.md`. +- NEXT: build `vr1-dc1-region` (no runbook -- F1) for dc1; and once a TAGGED key + Headscale access -> `site-tailscale.sh install` on dc0 .7 -> browser login `https://10.12.8.58` closes Step 3.3. Status ONLY in CURRENT-STATE.md.