Newer
Older
openstack-caracal-dc-dc / docs / audit / roosevelt-held-decisions-review-20260809.md

Roosevelt / future-deployment HELD-decisions review -- reevaluated against current project state

ANALYSIS + RECOMMENDATION PACKAGE, NOT A RULING. Mutates no authoritative surface. Every "pull forward" is a candidate for a gated operator exchange (GA-R5 to un-defer a ruling; presented-and-approved to change implementation timing). Companion to docs/audit/open-items-review-20260809.md.

  • Author: this session, 2026-08-09. Read-only.
  • Inventory source: an exhaustive sweep of docs/design-decisions.md (every D-block + amendment) for decisions/sub-decisions whose RULING or IMPLEMENTATION is held to Roosevelt / "the next deployment" / end-of-deployment review. Full raw enumeration is reproduced in Section 5.

0. The evaluative frame -- what changed, and the one disambiguation that governs it all

The operator's premise: enough has changed to reevaluate what is still correctly held for the future. The material changes since these deferrals were set:

  1. The re-IP redeploy on 10.13.0.0/16 (D-143, ruled today) is a near-term SECOND full build. dc0 finishes as a checkpoint, then the substrate is torn down and redeployed.
  2. geneve-over-v6 is LIVE-CONFIRMED working (D-139 amendment) -- v6-only east-west is viable.
  3. The deploy method is now proven -- dc0 is a live 66-machine / 162-unit cloud.
  4. The tailnet-collision incident -- the shared, unpoliced, allow-all Headscale is what caused the entire re-IP pivot. That converted a theoretical isolation exposure into a demonstrated one.
  5. Precedent: Roosevelt items get pulled forward when the need arises -- D-132 q1 (per-DC MAAS region) and D-138 (client-in-DC) were both pulled forward from Roosevelt into VR1.

>>> THE DISAMBIGUATION THAT PREVENTS THE OBVIOUS ERROR. <<< In these decisions, "next deployment" means Roosevelt -- the next DISTINCT, bare-metal build with real hardware specs -- NOT the intra-VR1 re-IP redeploy. The 10.13 redeploy is a re-build of the same VR1 rehearsal on the same nested VMs; it has no real switch/NIC inventory, no IPMI/BMC, no bare-metal capacity. So it CANNOT exercise anything hardware- or scale-bound (bonds/trunks, NIC budgets, HA at scale, hardware edge plugins). Treating the redeploy as "the next deployment" would wrongly pull those forward.

But the redeploy IS a fresh-build opportunity for anything that (a) is exercised at ANY full build and (b) does not need bare metal: the credential lifecycle, the Vault init, the v6 re-carve. Those are the genuine pull-forwards. The recommendation groups below split exactly on that line.


1. GROUP 1 -- PULL FORWARD to the 10.13 redeploy (fresh-build items, not bare-metal-bound)

These retire naturally at the redeploy because the redeploy re-runs the exact build step they govern. No new ruling needed for most -- this is implementation timing.

  • D-137 -- "real credential hardening (rotation, revocation, non-shared tokens)" deferred to post-teardown cycles. The re-IP TEARDOWN is the post-teardown cycle D-137 named. This is the same finding as open-items-review R7 (the teardown runbook has no credential-revocation checklist). Recommendation: build the revocation/rotation step into the teardown, so the redeploy re-mints on a clean slate. Actionable now (build R7).
  • D-142 -- vault-init QoL sweep, "implement + test on the next opportunity." The next opportunity is literally the redeploy's Vault standup. Same as open-items-review R6. Recommendation: fix R1's -m model-target DEFECT before the next Vault init, put R2's off-host transport question to the operator, build + test the QoL there.
  • D-136 (octet-convention half only) -- "IPv6 host-part octet convention: review at the end of the project." The re-IP re-carves the entire VR1 v6 apex under the new B2 role. That re-carve is the natural, cheaper moment to settle the octet convention than a separate end-of-project pass. Recommendation: fold the octet-convention review into the D-143 B2 v6 re-carve. NOTE: D-136's other held half -- the automation/CI placement (F2/F3, the Jenkins job on per-DC utility-node agents) -- does NOT move; it is coupled to D-132 and is in Group 4.
  • D-069 / SEC-003 -- unseal-key custodian ASSIGNMENT (+ second-person unseal rehearsal). This is operator input, not a hardware/scale blocker -- decidable any time, and the redeploy re-inits Vault, which is the natural rehearsal window for a second-person unseal. COUNTERWEIGHT (state it, do not gloss): this is a STANDING operator choice, not an oversight -- SEC-003 was operator-DEFERRED 2026-07-06 and reaffirmed, and its own row records that the deferral "keeps d011-06 MANUAL, gates D-011 full close" (bus-factor open since 2026-07-03). Recommendation: SURFACE it for reconsideration at the redeploy Vault init -- the re-init is a rare low-cost window to assign custodians and rehearse, and D-011's full close depends on it -- but it is the operator's standing call to un-defer (a GA-R5 ruling). NOTE: D-069's other held item, auto-unseal (transit/cloud KMS), is a SEPARATE future decision and is NOT decidable here -- it is in Group 4 (version-bound, D-068-coupled).

2. GROUP 2 -- PULL FORWARD the ANALYSIS/PREP only (the RULING stays future)

Research is cheap; these are reviews that can be conducted now without the future deployment, feeding a future ruling. Doing them now de-risks the eventual ruling.

  • D-131 sub-4 -- the PINNED node-DNS architectural review (best-practice conformance; security of the metal-admin forwarder; vendor-documented utility-node DC-to-DC config). dc0's forwarder is live and proven, so the review has a real subject NOW. Recommendation: conduct the (a)/(b)/(c) review this deployment; the sub-4 ruling stays Roosevelt (still gated on the upstream MAAS-agent-resolver LP outcome).
  • D-140 -- OpenTofu-manages-Juju, the owed "provider capabilities must be READ, not recalled." The ruling stays at end-of-deployment (and per its sequencing lands after the 10.13 redeploy, since the "hardened + tested" deployment is the redeployed one). But the precondition it named (a proven deploy method) is now largely met, and the provider-capability read is prep that can be done anytime. Recommendation: do the references/opentofu-provider-docs.md provider-capability read now so the end-of-deployment ruling is ready; keep the ruling itself deferred.
  • D-068 item 1 -- the V1-V5 migration re-verify probes (the ruling requires them RE-RUN at Roosevelt, "not cited stale"; last run 2026-07-23). Recommendation: re-baseline the probes opportunistically (they inform the D-071 monthly review too); the Q2 PATH ruling (2a/2b/2c) genuinely stays Roosevelt Vault design time.

3. GROUP 3 -- RECONSIDER given a specific project change (surface for operator decision)

  • D-129(iii) -- TAGGED identity + autoApprovers + the STAR ACL, "deferred to bare-metal" (operator, 2026-08-07: "We can pin those for the bare metal install"). This is the one the collision incident directly bears on. The exposure D-129(iii)(b) describes -- "an unpoliced Headscale is allow-all" with no enforced DC-isolation boundary -- is exactly the class of failure that produced the 10.12/10.13 collision. The re-IP re-stands the .7 routers on 10.13, which is the natural moment to bring them up TAGGED with the star ACL, and doing so would also retire the note-1 key-expiry outage risk the untagged nodes carry. HONEST COUNTERWEIGHT: the deferral has a HARD blocker that has not changed -- the Headscale control plane is Cloudflare-fronted and the operator has no access to it (note 4, measured). Without that access the tagged/ACL work cannot be built regardless of the risk calculus. Recommendation: SURFACED for reconsideration -- the collision moved this from theoretical to demonstrated, so it is worth pulling forward to the redeploy iff Headscale control-plane access is obtainable. If access stays blocked, it correctly remains deferred, and the star boundary stays a KNOWN, accepted VR1 exposure (as D-129(iii)(b) already records).

4. GROUP 4 -- STAYS FUTURE, correctly held (bare-metal / scale-bound; redeploy is nested VMs)

Each of these needs real hardware, real scale, or an external spec the redeploy does not supply. The reevaluation CONFIRMS the deferral. No action.

Decision What is held Why it genuinely stays
D-052(a)/(b) dedicated corosync heartbeat ring; dedicated live-migration plane + QEMU-native TLS need dedicated planes/hardware; hardening, not blocking (QEMU-TLS is VM-testable if ever wanted, low value)
D-059 four-NIC collapse gated on Roosevelt hardware sheets (NIC budget)
D-100 netem inter-DC link profile awaiting the external Roosevelt link spec
D-104 3-unit Juju controller HA "when bare-metal resources allow"; VR1 VM capacity is ~85% FIT
D-108 rbd-mirror daemon HA Roosevelt scale-up
D-121 Vault HA backend Raft-vs-etcd Raft is a 1.16-charm feature, incompatible now; coupled to the D-068 vault-version path
D-129 os-frr dynamic routing at the edge Roosevelt inter-DC design (needs the D-100 link spec + D-132 topology)
D-129 metal-edge profile os-smart / os-nut / os-cpu-microcode / os-lldpd hardware-specific (SMART disks, UPS, CPU microcode) -- inert on VMs
D-129 security hardening Suricata / crowdsec / Zenarmor / SIEM deferred by the closed-test posture to post-teardown/Roosevelt
D-129(iii)(c)/(d) two-router Tailscale HA; per-operator source-IP preservation HA scale-up class (D-121-coupled)
D-132 Q1/Q2/Q3 HA-region sub-questions; rack-top rack controllers; cross-site backup custody multi-rack / HA / cross-site scale -- the redeploy has one rack per DC
D-133 part 2 hardware-faithful NIC topology (bonds/trunks) needs real switch/NIC inventory; the redeploy is still 6-flat-NIC VMs
D-139 VPN (:e0); external v6 routing operator-access VPN; external v6 transit Roosevelt edge design; the VR1 edge has no v6 transport
D-029 Keystone SSO (k8s-keystone-auth) for workload clusters can't be exercised -- Magnum/CAPI is deferred to the redeploy (no workload clusters to run SSO on), and D-029's dependency is the Roosevelt cloud-internal-DNS + trusted-cert foundation. D-106's Designate reactivation is a partial move toward that foundation but does NOT discharge D-029
D-068 item 2 (listener TLS) Vault listener-TLS at the region a "Roosevelt build requirement -- lands at Roosevelt Vault standup with its own runbook step"; VR1 stays cleartext by ruling
D-068 item 3 (AppRole lifecycle) TTL audit, renewable/auto-renew TTLs, per-consumer auth probe "all three legs are Roosevelt build requirements" (the per-consumer probe already ships into cloud-assert.sh for VR1; the rest is Roosevelt)
D-069 auto-unseal evaluate transit/cloud-KMS auto-unseal a SEPARATE future decision, and version-bound -- transit/KMS auto-unseal capability tracks the Vault version the D-068 item-1 Q2 path hasn't ruled, so the redeploy cannot settle it
D-136 automation/CI half F2/F3 (CI runner / event delivery / status-back); Jenkins job placement "the Roosevelt shape once D-132 ratifies" -- gated on D-132 (which itself stays, Group 4)

5. GROUP 5 -- RECORD HYGIENE (defers that project state has already superseded/moved)

  • D-019 (Designate deferred to v2/Roosevelt) is SUPERSEDED FOR VR1 by D-106 -- Designate DEPLOYS at Stage 5 in VR1 (the checkpoint's owed "test zone" is exactly this). The VR0/v1 record holds, but the "held for Roosevelt" framing is stale for the current mission. Recommendation: no action needed (D-106 already governs); noted so it is not mistaken for a live hold.
  • D-042 is DISCHARGED, not held -- the released-tag driver pin (magnum-capi-helm==1.4.0) shipped the capability; the interim dev-commit path is retired. Listed only to close the loop.
  • D-010 NetBox merge-back to netbox.baldurkeep.com ("a maybe" at end-of-deployment) -- the re-IP re-carves the working apex on 10.13 under the new B2 role, which changes what would merge back. Recommendation: revisit the merge-back plan when the re-IP re-carve is done; still a "maybe," now with a different payload.

6. Prioritized recommendation summary

# Decision(s) Recommendation When
1 D-137 / R7 Build the teardown credential-revocation checklist at the re-IP teardown
2 D-142 / R6 Fix the -m model defect; build+test the vault-init QoL; put R2 to operator before the 10.13 Vault init
3 D-136 Fold the IPv6 octet-convention review into the B2 v6 re-carve with the D-143 re-carve
4 D-069 / SEC-003 Decide unseal custodians + rehearse second-person unseal (needs a ruling to un-defer) at the redeploy Vault init
5 D-131 sub-4 Conduct the pinned node-DNS architectural review now (ruling stays Roosevelt) this deployment
6 D-140 Do the provider-capability READ now (ruling stays end-of-deployment) anytime before the review
7 D-129(iii) ACL Reconsider tagged-identity + star ACL at the redeploy IFF Headscale access is obtainable redeploy, access-gated
8 D-068 item 1 Re-baseline the V1-V5 probes opportunistically (Q2 path stays Roosevelt) low priority
-- Group 4 (18 rows) Deferral CONFIRMED -- bare-metal / scale / version / Magnum-bound; no action Roosevelt
-- D-019 / D-042 / D-010 Record hygiene only --

Bottom line: the operator's instinct is right for a specific, bounded subset. The re-IP redeploy is the lever -- it makes the build-process deferrals (D-137 revocation, D-142 vault-QoL, D-136 v6 octet, D-069 custodian) actionable now, and it makes the D-129(iii) tailnet-ACL worth reconsidering because the collision proved the exposure. The analysis-only items (D-131 sub-4, D-140 provider read, D-068 probes) can be done now regardless. But the majority (Group 4, 18 rows) genuinely stay -- they are bare-metal / scale / version / Magnum-bound and the redeploy is still nested VMs, so reevaluation confirms rather than overturns them. The reevaluation's honest yield is ~4 pull-forwards (D-137/D-142/D-136-octet/D-069-custodian) + ~3 analysis-now (D-131-sub4/D-140/D-068-probes) + 1 access-gated reconsideration (D-129(iii) ACL), not a wholesale un-deferral.


7. Full raw inventory (from the exhaustive sweep)

Section A = held to Roosevelt/next-deployment/end-of-deployment; Section B = held to a specific VR1 step (near-term, not future-deployment); Section C = checked, NOT held.

Section A (future-deployment held): D-010 (NetBox merge-back, end-of-deploy, :204); D-019 (Designate, v2/Roosevelt, :384 -- superseded for VR1 by D-106); D-029 (Keystone SSO, Roosevelt, :535); D-052 (corosync ring :899; live-migration plane + QEMU TLS :901); D-059 (four-NIC collapse, Roosevelt sheets, :1090/:1104); D-068 (item1 Q2 path :5813-5819; item2 listener TLS :1660; item3 AppRole :1665); D-069 (custodian assignment/SEC-003 :1769; auto-unseal :1786); D-100 (netem profile :2239); D-104 (Juju HA :2808); D-108 (rbd-mirror HA :3049); D-121 (Vault Raft-vs-etcd :4542/:4628); D-129 (os-frr :5411; metal-edge profile :5439; security hardening :5443; two-router HA :5548; source-IP :5553; tagged/ACL :5579-5611); D-131 (sub-4 + arch review :5677-5683); D-132 (Q1 HA sub-qs / Q2 / Q3 :7137-7172); D-133 (part 2 hardware NIC :5842-5847); D-136 (automation F2/F3; octet convention; Jenkins placement :6421-6426/:6709); D-137 (rotation/revocation post-teardown :7008); D-139 (VPN :e0 :7649; external v6 routing :7403); D-140 (OpenTofu-Juju, end-of-deploy :7783); D-142 (vault-init QoL, next opportunity :8034).

Section B (VR1-step held, near-term -- NOT future deployment): D-129 edge profiles (phase-2 :5434); os-node_exporter (phase-6 Step 10 :5441); D-135 item 2 images/simplestreams (Stage 5 :6267); D-135 item 3 charmhub/snaps (measure-at-Stage-5 :6270); D-134 v6 bands (after the GUA carve :6014).

Section C (checked, NOT held): D-042 (discharged -- 1.4.0 shipped, :665); D-134 (applies now; VIP headroom is a Roosevelt-analog only). Excluded per scope: D-105 and the many "[ARCH] a Roosevelt build session greps this" A1-test lines (ADOPTED-and-apply-now).


8. Fable advisor review + consensus (corrections visible)

What the advisor caught on the drafted review, and what changed:

  1. Unplaced inventory items (the same completeness class as last review's "sums to 28"). D-029 (Keystone SSO), D-068 item 2 (listener TLS), and D-068 item 3 (AppRole) were in the Section 7 inventory but in NO recommendation group. Fixed: all three added to Group 4 with their actual gating reasons (D-029 is Magnum-deferred + needs the Roosevelt cloud-internal- DNS/cert foundation, only partially advanced by D-106; D-068 items 2/3 are named Roosevelt Vault-standup requirements).
  2. D-069 conflation. The draft merged the custodian ASSIGNMENT (operator input, decidable now) with AUTO-UNSEAL (a separate, version-bound future decision) in one Group-1 item. Recommending them together implied the redeploy could settle auto-unseal, which it can't. Split: custodian -> Group 1; auto-unseal -> Group 4 (coupled to the D-068 Q2 vault-version path, since transit/KMS auto-unseal capability is version-bound).
  3. D-069 deferral history missing as a counterweight. Added: SEC-003 was operator-deferred 2026-07-06 and reaffirmed, and gates D-011's full close -- so the recommendation reads as a standing choice to revisit, not an oversight to fix.
  4. D-136 half-move. Only the octet-convention half is a Group-1 pull-forward; the automation/CI (Jenkins) half is coupled to D-132 (which stays) and belongs in Group 4. Fixed both places.

CONSENSUS -- the agreed final position (both reviewers):

  • The "next deployment != the re-IP redeploy" disambiguation is what makes this a reevaluation rather than a wholesale un-deferral, and it holds. Group 4's confirmations are correctly bare-metal/scale/version/Magnum-bound and are NOT softened.
  • The honest yield is a bounded set: ~4 build-process pull-forwards the redeploy makes actionable, ~3 analysis-now items, and 1 access-gated reconsideration (D-129(iii) ACL, where the collision changed the risk calculus but the Headscale-access blocker did not).
  • The model to reuse (D-129(iii)): when a project change bears on a held decision, state the changed risk calculus AND the unchanged blocker, and SURFACE for operator decision rather than recommending an un-defer the blocker won't allow.
  • No new D-numbers are warranted by this review; the pull-forwards that need a ruling to un-defer (D-069 custodian; D-129(iii) ACL) are operator-gated GA-R5 exchanges.