diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f8a86b6..fba52aa 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -54,6 +54,13 @@ > only item gating the end goal; R7 teardown-credential-revocation + R16 10.13 naming-DOCFIX are > the two re-IP-coupled gaps to build. `docs/audit/open-items-review-20260809.md`; sweep: > `docs/audit/queued-findings-20260809-open-items-review.txt`. +> **ROOSEVELT/FUTURE-HELD DECISIONS REVIEW, 2026-08-09** (two-pass self + fable advisor; NOT a +> ruling): reevaluated all ~24 future-deployment-held decisions against current state. Key frame: +> "next deployment" = Roosevelt bare-metal, NOT the re-IP redeploy. Honest yield = ~4 build-process +> pull-forwards the redeploy makes actionable (D-137 revocation, D-142 vault-QoL, D-136 v6-octet, +> D-069 custodian) + ~3 analysis-now (D-131 sub-4, D-140 provider-read, D-068 probes) + 1 access- +> gated reconsideration (D-129(iii) tailnet ACL); the majority (18 rows) genuinely stay bare-metal/ +> scale-bound. `docs/audit/roosevelt-held-decisions-review-20260809.md`. > > **dc0 checkpoint scope (operator 2026-08-08): "activate + smoke-test"** -- networks + Octavia (1 test > LB) + Designate (1 test zone) + wrap gates (cloud-assert BOM, controller backup, verify-live diff --git a/docs/audit/roosevelt-held-decisions-review-20260809.md b/docs/audit/roosevelt-held-decisions-review-20260809.md new file mode 100644 index 0000000..1c6d4f1 --- /dev/null +++ b/docs/audit/roosevelt-held-decisions-review-20260809.md @@ -0,0 +1,251 @@ +# Roosevelt / future-deployment HELD-decisions review -- reevaluated against current project state + +**ANALYSIS + RECOMMENDATION PACKAGE, NOT A RULING.** Mutates no authoritative surface. +Every "pull forward" is a candidate for a gated operator exchange (GA-R5 to un-defer a +ruling; presented-and-approved to change implementation timing). Companion to +`docs/audit/open-items-review-20260809.md`. + +- Author: this session, 2026-08-09. Read-only. +- Inventory source: an exhaustive sweep of `docs/design-decisions.md` (every D-block + + amendment) for decisions/sub-decisions whose RULING or IMPLEMENTATION is held to + Roosevelt / "the next deployment" / end-of-deployment review. Full raw enumeration is + reproduced in Section 5. + +--- + +## 0. The evaluative frame -- what changed, and the one disambiguation that governs it all + +The operator's premise: enough has changed to reevaluate what is still correctly held for +the future. The material changes since these deferrals were set: + +1. **The re-IP redeploy on `10.13.0.0/16` (D-143, ruled today) is a near-term SECOND full + build.** dc0 finishes as a checkpoint, then the substrate is torn down and redeployed. +2. **geneve-over-v6 is LIVE-CONFIRMED working** (D-139 amendment) -- v6-only east-west is viable. +3. **The deploy method is now proven** -- dc0 is a live 66-machine / 162-unit cloud. +4. **The tailnet-collision incident** -- the shared, unpoliced, allow-all Headscale is what + caused the entire re-IP pivot. That converted a *theoretical* isolation exposure into a + *demonstrated* one. +5. **Precedent: Roosevelt items get pulled forward when the need arises** -- D-132 q1 (per-DC + MAAS region) and D-138 (client-in-DC) were both pulled forward from Roosevelt into VR1. + +**>>> THE DISAMBIGUATION THAT PREVENTS THE OBVIOUS ERROR. <<<** In these decisions, +**"next deployment" means Roosevelt -- the next DISTINCT, bare-metal build with real hardware +specs -- NOT the intra-VR1 re-IP redeploy.** The 10.13 redeploy is a re-build of the *same* +VR1 rehearsal on the *same* nested VMs; it has no real switch/NIC inventory, no IPMI/BMC, no +bare-metal capacity. So it CANNOT exercise anything hardware- or scale-bound (bonds/trunks, +NIC budgets, HA at scale, hardware edge plugins). Treating the redeploy as "the next +deployment" would wrongly pull those forward. + +**But the redeploy IS a fresh-build opportunity for anything that (a) is exercised at ANY +full build and (b) does not need bare metal:** the credential lifecycle, the Vault init, the +v6 re-carve. Those are the genuine pull-forwards. The recommendation groups below split +exactly on that line. + +--- + +## 1. GROUP 1 -- PULL FORWARD to the 10.13 redeploy (fresh-build items, not bare-metal-bound) + +These retire naturally at the redeploy because the redeploy re-runs the exact build step they +govern. No new ruling needed for most -- this is implementation timing. + +- **D-137 -- "real credential hardening (rotation, revocation, non-shared tokens)" deferred + to post-teardown cycles.** The re-IP TEARDOWN *is* the post-teardown cycle D-137 named. + This is the same finding as open-items-review **R7** (the teardown runbook has no + credential-revocation checklist). **Recommendation:** build the revocation/rotation step + into the teardown, so the redeploy re-mints on a clean slate. Actionable now (build R7). +- **D-142 -- vault-init QoL sweep, "implement + test on the next opportunity."** The next + opportunity is literally the redeploy's Vault standup. Same as open-items-review **R6**. + **Recommendation:** fix R1's `-m` model-target DEFECT before the next Vault init, put R2's + off-host transport question to the operator, build + test the QoL there. +- **D-136 (octet-convention half only) -- "IPv6 host-part octet convention: review at the end + of the project."** The re-IP re-carves the entire VR1 v6 apex under the new B2 role. That + re-carve is the natural, cheaper moment to settle the octet convention than a separate + end-of-project pass. **Recommendation:** fold the octet-convention review into the D-143 B2 + v6 re-carve. NOTE: D-136's *other* held half -- the automation/CI placement (F2/F3, the + Jenkins job on per-DC utility-node agents) -- does NOT move; it is coupled to D-132 and is + in Group 4. +- **D-069 / SEC-003 -- unseal-key custodian ASSIGNMENT (+ second-person unseal rehearsal).** + This is *operator input*, not a hardware/scale blocker -- decidable any time, and the + redeploy re-inits Vault, which is the natural rehearsal window for a second-person unseal. + **COUNTERWEIGHT (state it, do not gloss):** this is a STANDING operator choice, not an + oversight -- SEC-003 was operator-DEFERRED 2026-07-06 and reaffirmed, and its own row + records that the deferral "keeps d011-06 MANUAL, gates D-011 full close" (bus-factor open + since 2026-07-03). **Recommendation:** SURFACE it for reconsideration at the redeploy Vault + init -- the re-init is a rare low-cost window to assign custodians and rehearse, and D-011's + full close depends on it -- but it is the operator's standing call to un-defer (a GA-R5 + ruling). NOTE: D-069's *other* held item, auto-unseal (transit/cloud KMS), is a SEPARATE + future decision and is NOT decidable here -- it is in Group 4 (version-bound, D-068-coupled). + +## 2. GROUP 2 -- PULL FORWARD the ANALYSIS/PREP only (the RULING stays future) + +Research is cheap; these are reviews that can be *conducted* now without the future +deployment, feeding a future ruling. Doing them now de-risks the eventual ruling. + +- **D-131 sub-4 -- the PINNED node-DNS architectural review** (best-practice conformance; + security of the metal-admin forwarder; vendor-documented utility-node DC-to-DC config). + dc0's forwarder is live and proven, so the review has a real subject NOW. **Recommendation:** + conduct the (a)/(b)/(c) review this deployment; the sub-4 *ruling* stays Roosevelt (still + gated on the upstream MAAS-agent-resolver LP outcome). +- **D-140 -- OpenTofu-manages-Juju, the owed "provider capabilities must be READ, not + recalled."** The ruling stays at end-of-deployment (and per its sequencing lands *after* + the 10.13 redeploy, since the "hardened + tested" deployment is the redeployed one). But + the precondition it named (a proven deploy method) is now largely met, and the + provider-capability read is prep that can be done anytime. **Recommendation:** do the + `references/opentofu-provider-docs.md` provider-capability read now so the end-of-deployment + ruling is ready; keep the ruling itself deferred. +- **D-068 item 1 -- the V1-V5 migration re-verify probes** (the ruling requires them RE-RUN + at Roosevelt, "not cited stale"; last run 2026-07-23). **Recommendation:** re-baseline the + probes opportunistically (they inform the D-071 monthly review too); the Q2 PATH ruling + (2a/2b/2c) genuinely stays Roosevelt Vault design time. + +## 3. GROUP 3 -- RECONSIDER given a specific project change (surface for operator decision) + +- **D-129(iii) -- TAGGED identity + autoApprovers + the STAR ACL, "deferred to bare-metal" + (operator, 2026-08-07: "We can pin those for the bare metal install").** This is the one + the collision incident directly bears on. **The exposure D-129(iii)(b) describes -- "an + unpoliced Headscale is allow-all" with no enforced DC-isolation boundary -- is exactly the + class of failure that produced the 10.12/10.13 collision.** The re-IP re-stands the `.7` + routers on 10.13, which is the natural moment to bring them up TAGGED with the star ACL, + and doing so would also retire the note-1 key-expiry outage risk the untagged nodes carry. + **HONEST COUNTERWEIGHT:** the deferral has a HARD blocker that has not changed -- the + Headscale control plane is Cloudflare-fronted and **the operator has no access to it** + (note 4, measured). Without that access the tagged/ACL work cannot be built regardless of + the risk calculus. **Recommendation:** SURFACED for reconsideration -- the collision moved + this from theoretical to demonstrated, so it is worth pulling forward to the redeploy *iff* + Headscale control-plane access is obtainable. If access stays blocked, it correctly remains + deferred, and the star boundary stays a KNOWN, accepted VR1 exposure (as D-129(iii)(b) + already records). + +## 4. GROUP 4 -- STAYS FUTURE, correctly held (bare-metal / scale-bound; redeploy is nested VMs) + +Each of these needs real hardware, real scale, or an external spec the redeploy does not +supply. The reevaluation CONFIRMS the deferral. No action. + +| Decision | What is held | Why it genuinely stays | +|---|---|---| +| **D-052(a)/(b)** | dedicated corosync heartbeat ring; dedicated live-migration plane + QEMU-native TLS | need dedicated planes/hardware; hardening, not blocking (QEMU-TLS is VM-testable if ever wanted, low value) | +| **D-059** | four-NIC collapse | gated on Roosevelt hardware sheets (NIC budget) | +| **D-100** | netem inter-DC link profile | awaiting the external Roosevelt link spec | +| **D-104** | 3-unit Juju controller HA | "when bare-metal resources allow"; VR1 VM capacity is ~85% FIT | +| **D-108** | rbd-mirror daemon HA | Roosevelt scale-up | +| **D-121** | Vault HA backend Raft-vs-etcd | Raft is a 1.16-charm feature, incompatible now; coupled to the D-068 vault-version path | +| **D-129 os-frr** | dynamic routing at the edge | Roosevelt inter-DC design (needs the D-100 link spec + D-132 topology) | +| **D-129 metal-edge profile** | os-smart / os-nut / os-cpu-microcode / os-lldpd | hardware-specific (SMART disks, UPS, CPU microcode) -- inert on VMs | +| **D-129 security hardening** | Suricata / crowdsec / Zenarmor / SIEM | deferred by the closed-test posture to post-teardown/Roosevelt | +| **D-129(iii)(c)/(d)** | two-router Tailscale HA; per-operator source-IP preservation | HA scale-up class (D-121-coupled) | +| **D-132 Q1/Q2/Q3** | HA-region sub-questions; rack-top rack controllers; cross-site backup custody | multi-rack / HA / cross-site scale -- the redeploy has one rack per DC | +| **D-133 part 2** | hardware-faithful NIC topology (bonds/trunks) | needs real switch/NIC inventory; the redeploy is still 6-flat-NIC VMs | +| **D-139 VPN (:e0); external v6 routing** | operator-access VPN; external v6 transit | Roosevelt edge design; the VR1 edge has no v6 transport | +| **D-029** | Keystone SSO (`k8s-keystone-auth`) for workload clusters | can't be exercised -- Magnum/CAPI is deferred to the redeploy (no workload clusters to run SSO on), and D-029's dependency is the Roosevelt cloud-internal-DNS + trusted-cert foundation. D-106's Designate reactivation is a *partial* move toward that foundation but does NOT discharge D-029 | +| **D-068 item 2 (listener TLS)** | Vault listener-TLS at the region | a "Roosevelt build requirement -- lands at Roosevelt Vault standup with its own runbook step"; VR1 stays cleartext by ruling | +| **D-068 item 3 (AppRole lifecycle)** | TTL audit, renewable/auto-renew TTLs, per-consumer auth probe | "all three legs are Roosevelt build requirements" (the per-consumer probe already ships into `cloud-assert.sh` for VR1; the rest is Roosevelt) | +| **D-069 auto-unseal** | evaluate transit/cloud-KMS auto-unseal | a SEPARATE future decision, and version-bound -- transit/KMS auto-unseal capability tracks the Vault version the D-068 item-1 Q2 path hasn't ruled, so the redeploy cannot settle it | +| **D-136 automation/CI half** | F2/F3 (CI runner / event delivery / status-back); Jenkins job placement | "the Roosevelt shape once D-132 ratifies" -- gated on D-132 (which itself stays, Group 4) | + +## 5. GROUP 5 -- RECORD HYGIENE (defers that project state has already superseded/moved) + +- **D-019 (Designate deferred to v2/Roosevelt) is SUPERSEDED FOR VR1 by D-106** -- Designate + DEPLOYS at Stage 5 in VR1 (the checkpoint's owed "test zone" is exactly this). The VR0/v1 + record holds, but the "held for Roosevelt" framing is stale for the current mission. + **Recommendation:** no action needed (D-106 already governs); noted so it is not mistaken + for a live hold. +- **D-042 is DISCHARGED, not held** -- the released-tag driver pin (`magnum-capi-helm==1.4.0`) + shipped the capability; the interim dev-commit path is retired. Listed only to close the loop. +- **D-010 NetBox merge-back to `netbox.baldurkeep.com` ("a maybe" at end-of-deployment)** -- + the re-IP re-carves the working apex on 10.13 under the new B2 role, which changes *what* + would merge back. **Recommendation:** revisit the merge-back plan when the re-IP re-carve + is done; still a "maybe," now with a different payload. + +--- + +## 6. Prioritized recommendation summary + +| # | Decision(s) | Recommendation | When | +|---|---|---|---| +| 1 | **D-137 / R7** | Build the teardown credential-revocation checklist | at the re-IP teardown | +| 2 | **D-142 / R6** | Fix the `-m` model defect; build+test the vault-init QoL; put R2 to operator | before the 10.13 Vault init | +| 3 | **D-136** | Fold the IPv6 octet-convention review into the B2 v6 re-carve | with the D-143 re-carve | +| 4 | **D-069 / SEC-003** | Decide unseal custodians + rehearse second-person unseal (needs a ruling to un-defer) | at the redeploy Vault init | +| 5 | **D-131 sub-4** | Conduct the pinned node-DNS architectural review now (ruling stays Roosevelt) | this deployment | +| 6 | **D-140** | Do the provider-capability READ now (ruling stays end-of-deployment) | anytime before the review | +| 7 | **D-129(iii) ACL** | Reconsider tagged-identity + star ACL at the redeploy IFF Headscale access is obtainable | redeploy, access-gated | +| 8 | **D-068 item 1** | Re-baseline the V1-V5 probes opportunistically (Q2 path stays Roosevelt) | low priority | +| -- | **Group 4 (18 rows)** | Deferral CONFIRMED -- bare-metal / scale / version / Magnum-bound; no action | Roosevelt | +| -- | **D-019 / D-042 / D-010** | Record hygiene only | -- | + +**Bottom line:** the operator's instinct is right for a specific, bounded subset. The +**re-IP redeploy is the lever** -- it makes the *build-process* deferrals (D-137 revocation, +D-142 vault-QoL, D-136 v6 octet, D-069 custodian) actionable now, and it makes the +D-129(iii) tailnet-ACL worth reconsidering because the collision proved the exposure. The +**analysis-only** items (D-131 sub-4, D-140 provider read, D-068 probes) can be done now +regardless. But the **majority (Group 4, 18 rows) genuinely stay** -- they are bare-metal / +scale / version / Magnum-bound and the redeploy is still nested VMs, so reevaluation confirms +rather than overturns them. The reevaluation's honest yield is **~4 pull-forwards +(D-137/D-142/D-136-octet/D-069-custodian) + ~3 analysis-now (D-131-sub4/D-140/D-068-probes) + +1 access-gated reconsideration (D-129(iii) ACL)**, not a wholesale un-deferral. + +--- + +## 7. Full raw inventory (from the exhaustive sweep) + +*Section A = held to Roosevelt/next-deployment/end-of-deployment; Section B = held to a +specific VR1 step (near-term, not future-deployment); Section C = checked, NOT held.* + +**Section A (future-deployment held):** D-010 (NetBox merge-back, end-of-deploy, :204); +D-019 (Designate, v2/Roosevelt, :384 -- superseded for VR1 by D-106); D-029 (Keystone SSO, +Roosevelt, :535); D-052 (corosync ring :899; live-migration plane + QEMU TLS :901); D-059 +(four-NIC collapse, Roosevelt sheets, :1090/:1104); D-068 (item1 Q2 path :5813-5819; item2 +listener TLS :1660; item3 AppRole :1665); D-069 (custodian assignment/SEC-003 :1769; +auto-unseal :1786); D-100 (netem profile :2239); D-104 (Juju HA :2808); D-108 (rbd-mirror HA +:3049); D-121 (Vault Raft-vs-etcd :4542/:4628); D-129 (os-frr :5411; metal-edge profile +:5439; security hardening :5443; two-router HA :5548; source-IP :5553; tagged/ACL :5579-5611); +D-131 (sub-4 + arch review :5677-5683); D-132 (Q1 HA sub-qs / Q2 / Q3 :7137-7172); D-133 +(part 2 hardware NIC :5842-5847); D-136 (automation F2/F3; octet convention; Jenkins +placement :6421-6426/:6709); D-137 (rotation/revocation post-teardown :7008); D-139 (VPN :e0 +:7649; external v6 routing :7403); D-140 (OpenTofu-Juju, end-of-deploy :7783); D-142 +(vault-init QoL, next opportunity :8034). + +**Section B (VR1-step held, near-term -- NOT future deployment):** D-129 edge profiles +(phase-2 :5434); os-node_exporter (phase-6 Step 10 :5441); D-135 item 2 images/simplestreams +(Stage 5 :6267); D-135 item 3 charmhub/snaps (measure-at-Stage-5 :6270); D-134 v6 bands +(after the GUA carve :6014). + +**Section C (checked, NOT held):** D-042 (discharged -- 1.4.0 shipped, :665); D-134 (applies +now; VIP headroom is a Roosevelt-analog only). Excluded per scope: D-105 and the many +"[ARCH] a Roosevelt build session greps this" A1-test lines (ADOPTED-and-apply-now). + +--- + +## 8. Fable advisor review + consensus (corrections visible) + +**What the advisor caught on the drafted review, and what changed:** +1. **Unplaced inventory items (the same completeness class as last review's "sums to 28").** + D-029 (Keystone SSO), D-068 item 2 (listener TLS), and D-068 item 3 (AppRole) were in the + Section 7 inventory but in NO recommendation group. Fixed: all three added to Group 4 with + their actual gating reasons (D-029 is Magnum-deferred + needs the Roosevelt cloud-internal- + DNS/cert foundation, only partially advanced by D-106; D-068 items 2/3 are named Roosevelt + Vault-standup requirements). +2. **D-069 conflation.** The draft merged the custodian ASSIGNMENT (operator input, decidable + now) with AUTO-UNSEAL (a separate, version-bound future decision) in one Group-1 item. + Recommending them together implied the redeploy could settle auto-unseal, which it can't. + Split: custodian -> Group 1; auto-unseal -> Group 4 (coupled to the D-068 Q2 vault-version + path, since transit/KMS auto-unseal capability is version-bound). +3. **D-069 deferral history missing as a counterweight.** Added: SEC-003 was operator-deferred + 2026-07-06 and reaffirmed, and gates D-011's full close -- so the recommendation reads as a + standing choice to revisit, not an oversight to fix. +4. **D-136 half-move.** Only the octet-convention half is a Group-1 pull-forward; the + automation/CI (Jenkins) half is coupled to D-132 (which stays) and belongs in Group 4. + Fixed both places. + +**CONSENSUS -- the agreed final position (both reviewers):** +- The **"next deployment != the re-IP redeploy"** disambiguation is what makes this a + reevaluation rather than a wholesale un-deferral, and it holds. Group 4's confirmations are + correctly bare-metal/scale/version/Magnum-bound and are NOT softened. +- The honest yield is a **bounded** set: ~4 build-process pull-forwards the redeploy makes + actionable, ~3 analysis-now items, and 1 access-gated reconsideration (D-129(iii) ACL, where + the collision changed the risk calculus but the Headscale-access blocker did not). +- **The model to reuse (D-129(iii)):** when a project change bears on a held decision, state + the changed risk calculus AND the unchanged blocker, and SURFACE for operator decision + rather than recommending an un-defer the blocker won't allow. +- No new D-numbers are warranted by this review; the pull-forwards that need a ruling to + un-defer (D-069 custodian; D-129(iii) ACL) are operator-gated GA-R5 exchanges.