diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 37e76f2..9638525 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -489,7 +489,7 @@ | G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | CLOSED 2026-07-23 (operator-ruled "Merge to main + full close"; commissioning 9/9 READY, merge commit on `main`, branch retired) -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED + WIRED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). **COMMISSIONING 9/9 READY 2026-07-23** (`docs/audit/dc1-commissioning-verify-20260723.txt`): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, `power_type=virsh` set + verified by real query-power-state (SEC-016 path proven), commissioned to **ALL 9 READY in ~3.5 min** (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. **G12 [V] leg (the dc1 build) is COMPLETE.** NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, **operator-gated merge of `dc-dc-g12-dc1-substrate` -> `main`** (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close). | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | CLOSED 2026-07-23 (operator-approved full maintenance bundle, logged window ops-sec010-reassert): qga channel retrofitted via outer tofu saved-plan apply 0/1/0 exact (`docs/audit/outer-plan-20260723-office1-qga.txt`; the apply's edge bounce = the ruled "next scheduled restart"; MACs were pinned 07-22 so the in-place-update trap class was closed); edge updated 26.7 -> 26.7.1 via REST (no reboot required; os-iperf had been REFUSED on 26.7 pending exactly this update); both plugins installed=1 by firmware-info read-back, `guest-ping` -> `{"return":{}}`, agent reports both legs, egress 0% loss, outer plan re-converged ZERO DIFF (`docs/audit/outer-plan-20260723-postqga-converged.txt`). Named close capture: `docs/audit/g13-close-20260723.txt` | | G14 | 12 OPEN SEC rows (SEC-001, -003..-008, SEC-012, -013, -014, plus SEC-015 + SEC-016 opened 2026-07-23 for dc1 credentials; SEC-010/-011 CLOSED) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012/-016 carry the same libvirt-group SCOPE hardening question; SEC-016 also a snap-refresh re-assert (queued to DC standup DoD) | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-23 (12 open) | -| G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 remains PROPOSED/OPEN (items 2-3 + the item-1 re-scoped migration plan) | +| G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 items 2-3 RULED 2026-07-21; item 1 re-scoped plan DRAFTED 2026-07-23 (`docs/D-068-vault-migration-plan-draft.md`) -- item 1 OPEN/presentable, rulings pending (Q1/Q2/Q3, one per exchange) | | G16 | office1 edge `channels = []` state reconcile (the D-129 module-schema residual) | [R] operator rules the mechanism; then [V] the converged re-plan capture | operator + session | CLOSED 2026-07-21: RULED "State surgery (Recommended)" (GA-R5, session changelog item 16); executed per G6 precedent -- channels null -> [] injected, serial 29 -> 30, backup kept, guests untouched (office1-opnsense Id 2 running throughout); convergence = ZERO DIFF (`docs/audit/outer-plan-20260721-postG16-converged.txt`); section 5 re-recorded | ## 7. Version pins (measured; the authority for every pin) @@ -521,12 +521,13 @@ open question remains; the item is AWAITING EXTERNAL INPUT (the Roosevelt inter-DC link spec), trigger defined: re-tune via a gated apply when the spec exists, not before. -3. D-068 -- Vault substrate hardening (PROPOSED/OPEN, - `docs/design-decisions.md:1457`). QUESTION: rule items 2 (Vault - listener TLS) and 3 (AppRole secret_id lifecycle) for Roosevelt; item 1 - (Vault version) needs a re-scoped migration plan since the 1.16 - forward-pin was proven NOT viable (amendment, :1650). Evidence in - `docs/D-068-vault-1.8-vs-1.16-analysis.md`. +3. D-068 -- Vault substrate hardening. Items 2-3 RULED 2026-07-21; item 1 + (Vault version) re-scoped plan DRAFTED 2026-07-23 + (`docs/D-068-vault-migration-plan-draft.md`; D-068 amendment of the same + date) -- item 1 is now PRESENTABLE as three separate GA-R5 questions (Q1 + interim rehearsal posture; Q2 Roosevelt path, options 2a-2d; Q3 + triggers/review), with verify items V1-V5 owed before any Q2 ruling. + Evidence base: `docs/D-068-vault-1.8-vs-1.16-analysis.md`. 4. D-071 -- ADOPTED 2026-07-21: all four policy points ruled (monthly review trigger; patch-only controller jumps; standing order; in-channel-only refreshes), each its own GA-R5 exchange -- status diff --git a/docs/D-068-vault-migration-plan-draft.md b/docs/D-068-vault-migration-plan-draft.md new file mode 100644 index 0000000..c3b8754 --- /dev/null +++ b/docs/D-068-vault-migration-plan-draft.md @@ -0,0 +1,175 @@ +# D-068 item 1 -- re-scoped Vault migration plan (DRAFT for operator ruling) + +**Backs:** D-068 item 1 ("Vault version"), re-scoped per the 2026-07-05 amendment +("get off EOL 1.8" stays OPEN; 1.16 ruled OUT as the path). +**Type:** options paper -- NOTHING here is adopted; the operator rules per GA-R5, +one sub-question per exchange. **Date:** 2026-07-23. **Authored by:** session +agent at operator direction ("Draft now", queue-pass session). +**Evidence base:** `docs/D-068-vault-1.8-vs-1.16-analysis.md` (2026-07-05, all of +whose findings this draft treats as still-current UNLESS re-verified below) + +two dated web probes (section 5). + +## 1. The re-scoped question + +The original item 1 question ("pin 1.16 and rehearse the upgrade") is DEAD -- +the 2026-07-05 analysis proved the new operator charm incompatible with this +reactive cloud on five measured axes (certificates V0->V1 showstopper, +Raft-only storage, no upgrade path, Ceph breakage reports, BUSL licensing). + +What remains is really THREE questions, and this draft proposes ruling them +separately (a bundled ruling would violate the one-decision-per-exchange rule +and they have different deadlines): + +- **Q1 (interim posture):** what risk posture covers EOL Vault 1.8.8 for the + REHEARSAL phases (VR0 testcloud, VR1 when Stage 5 brings Juju/Vault)? + Deadline: now-ish -- it is the standing state. +- **Q2 (Roosevelt path):** what is the PRODUCTION plan to get off 1.8? + Deadline: Roosevelt Vault design time (alongside D-068 item 2's listener-TLS + build requirement and D-132's MAAS topology, which are pinned to the same + design window). +- **Q3 (trigger/review):** what re-checks, on what cadence, keep Q2's inputs + fresh so the design-time ruling is made on live facts, not this draft's? + +## 2. Q1 options -- interim EOL posture (rehearsal phases) + +**(1a) Explicit risk-acceptance, bounded to rehearsal.** Record: Vault 1.8.8 +is EOL and receives no security patches; accepted for VR0/VR1 because (i) no +production tenant data or external exposure exists in the rehearsals, (ii) +vault listens only on metal-internal, (iii) the D-069 unseal flow and SEC +register already treat vault material as operator-only. Production (Roosevelt) +is explicitly NOT covered -- Q2 owns that. Cost: a recorded acceptance + +nothing else. This is the 2026-07-05 amendment's candidate (c), scoped to +rehearsal only. + +**(1b) Risk-acceptance + compensating monitoring.** As (1a), plus a standing +task: monitor published CVEs against Vault 1.8.x (no patches will exist; the +monitoring informs WORKAROUNDS or an emergency re-rule, not upgrades). Cost: +a recurring review item -- natural vehicle is D-071's ADOPTED monthly review +trigger, one added checklist line. No new tooling. + +**(1c) Do nothing / leave implicit.** Rejected framing (listed for +completeness): the current state is already de-facto acceptance, but this +project's discipline records postures explicitly; an unrecorded acceptance is +exactly the record-vs-reality gap the grounding audit existed to kill. + +Draft lean (for debate, not a decision): **(1b)** -- the delta over (1a) is one +line in an already-adopted monthly review. + +## 3. Q2 options -- the Roosevelt production path + +All four candidates carry the same hard constraint from the analysis: the 18 +`vault:certificates` V0 relations are the coupling that matters; the Vault +API itself is the EASY half. + +**(2a) Wait-and-re-assess: reactive charms gain tls-certificates V1.** The +2026-07-05 candidate (a). If the Caracal/successor service charms become V1 +requirers, the new operator vault becomes viable and the migration is the +documented backup/restore CA-replacement project (analysis section "migration +cost", 6 workstreams). AS OF 2026-07-23 THERE IS NO SIGNAL OF THIS (section +5) -- so today this is a monitoring stance, not a plan. Risk: indefinite +timeline owned by upstream; Roosevelt could reach design time with nothing to +adopt. BUSL licensing review still owed if this path lands. + +**(2b) OpenBao payload under the EXISTING reactive charm (fork).** OpenBao is +the MPL fork, actively released (section 5). NO Juju charm exists for it +(section 5) -- but the coupling insight cuts the other way: the reactive +`charm-vault` drives its payload over the Vault HTTP API, and OpenBao's API +is compatible at the fork-point feature set this cloud uses (KV v2 + PKI + +AppRole -- all pre-1.14 features; UNVERIFIED for our exact call surface, +verify item V3). A fork of `charm-vault` swapping the payload snap/deb to +OpenBao would keep ALL V0 relations, the MySQL storage backend, and the D-069 +unseal flow intact -- because the CHARM is unchanged, only the daemon binary. +Cost: we own a charm fork (build, test, security-patch tracking of OpenBao +releases); a payload-swap migration on live state (storage format at the +fork point is compatible BY DESIGN CLAIM -- verify item V4); this is real +engineering with a permanent maintenance tail, but it is the only path that +gets MAINTAINED, MPL-licensed software without waiting on upstream charm work. +Roosevelt-delta: smallest of the modernizing options (cloud shape unchanged). + +**(2c) Shrink Vault's job first, then modernize the remnant.** Re-architect +so vault is no longer the cloud-wide CA (move TLS issuance to another V0 +`certificates` provider, or to an offline/enterprise CA workflow the charms +already support via ssl_* config options -- the charm-guide documents both), +leaving vault only as barbican's KV backend. The remaining single relation +(`barbican-vault:secrets-storage`) is a far smaller compatibility surface to +migrate to ANY maintained backend (new vault charm, OpenBao, or barbican's +other backends). Cost: a TLS re-architecture across 18 services -- the +highest-touch option operationally, and it trades a solved problem (vault CA +works) for design work; but it removes the V0/V1 deadlock permanently and +decouples the cloud's trust chain from the vault charm's fate. This option +was NOT in the 2026-07-05 candidate list -- it is new in this draft and needs +the most scrutiny. + +**(2d) Enter Roosevelt on 1.8 with a production risk-acceptance + remediation +deadline.** The 2026-07-05 candidate (c) at production scope: deploy Roosevelt +on the proven 1.8 stack, record a dated remediation obligation, and execute +whichever of (2a)/(2b)/(2c) has matured by the deadline. This is honest about +the possibility that nothing matures; it converts an unbounded wait into a +bounded one. Requires item 2's listener TLS (already RULED: Roosevelt build +requirement) as a mitigating layer. Commercial note: a multi-tenant cloud +running an EOL secret store is a hard sell in any security review -- this +option needs the operator's commercial risk judgment, not just technical. + +Draft lean (for debate): **(2d) as the Roosevelt BASELINE with (2b) as the +funded remediation track** -- baseline keeps Roosevelt deployable on proven +components; the OpenBao-payload fork is the only remediation whose timeline +WE own. (2a) stays a free monitoring line either way. (2c) is the strategic +long-term shape but should not gate Roosevelt's initial deploy. + +## 4. Q3 -- triggers and review (proposed mechanics, not a decision) + +- Fold TWO lines into the D-071 monthly review checklist: (i) Vault 1.8.x CVE + scan (Q1 posture 1b, if ruled); (ii) re-run the section-5 probes (V1 + support signal; OpenBao charm ecosystem; OpenBao release cadence). +- Hard trigger: at Roosevelt Vault DESIGN time, this draft's section 5 + verifications are RE-RUN and the Q2 ruling is made on the fresh results -- + this draft's probes will be stale by then and MUST NOT be cited as current. +- If (2b) is ruled as remediation track: its verify items V3/V4 become the + first funded work package, BEFORE any charm fork begins. + +## 5. What was verified for this draft, and what was NOT + +Probes run 2026-07-23 (web search from the jumphost; both INCONCLUSIVE-BY- +ABSENCE, which is evidence of no-signal, not proof of impossibility): + +- **V1 adoption:** no result indicates the reactive OpenStack service charms + have gained tls-certificates V1 requirer support; current charm-guide TLS + docs still describe the V0 vault workflow. +- **OpenBao charm:** no Juju charm for OpenBao found; OpenBao itself shows an + active release stream (openbao/openbao releases; third-party PKI writeups + dated 2026-03). + +NOT verified here (owed before any Q2 ruling -- the verify-live list): +- **V1:** `juju info vault` channel map + charm-vault upstream activity (is + the reactive charm itself still maintained?). +- **V2:** current Vault 1.8.8 CVE exposure list (informs Q1 urgency). +- **V3:** OpenBao API compatibility against OUR exact call surface (KV v2 + paths, PKI endpoints the charm drives, AppRole flows, `sys/` calls in + `charm-vault`'s handlers). +- **V4:** OpenBao's storage-format compatibility claim for a payload-swap on + existing MySQL-backed state, at our data's fork-point version. +- **V5:** BUSL vs MPL licensing review with counsel (only if a path involving + Vault >=1.15 payload survives). + +## 6. Proposed ruling sequence (each its own GA-R5 exchange) + +1. Q1 interim posture: (1a) vs (1b). +2. Q2 Roosevelt path: baseline + remediation-track structure, or a single + path -- options (2a)-(2d). +3. Q3 mechanics: adopt the D-071 checklist lines + the design-time re-verify + trigger as written (or amended). + +No dependent work starts before its ruling. This document changes NOTHING by +itself. + +## Sources + +- Repo: `docs/D-068-vault-1.8-vs-1.16-analysis.md` (+ its cited Canonical/ + Charmhub/OpenStack sources); `docs/design-decisions.md` D-068 + 2026-07-05 + amendment, D-069, D-071, D-132. +- Web (2026-07-23 probes): OpenStack charm-guide "Managing TLS certificates" + (https://docs.openstack.org/charm-guide/latest/admin/security/tls.html); + openstack/charm-vault (https://github.com/openstack/charm-vault); + openbao/openbao releases (https://github.com/openbao/openbao/releases); + canonical/self-signed-certificates-operator (V1-world provider example; + https://github.com/canonical/self-signed-certificates-operator). diff --git a/docs/changelog-20260723-queue-pass.md b/docs/changelog-20260723-queue-pass.md index d0b2b4d..f99247a 100644 --- a/docs/changelog-20260723-queue-pass.md +++ b/docs/changelog-20260723-queue-pass.md @@ -202,3 +202,17 @@ gate (script-wrapped live commands, ssh/virsh, tofu, git write ops) left untouched -- the ask/deny walls were not modified. - REVERT: remove the two lines. + +## Item 12 -- D-068 item-1 re-scoped migration plan DRAFTED (operator: "Draft now") + +- WHAT: `docs/D-068-vault-migration-plan-draft.md` -- options paper splitting + item 1 into Q1 (interim EOL posture for rehearsals), Q2 (Roosevelt path: + wait-for-V1 / OpenBao-payload-under-reactive-charm fork / shrink-vault's- + job-first / enter-on-1.8-with-deadline) and Q3 (triggers via the D-071 + monthly review), each its own GA-R5 exchange. Two dated web probes + (2026-07-23, both no-signal): no reactive-charm tls-certificates V1 + adoption; no OpenBao Juju charm. Verify list V1-V5 owed before any Q2 + ruling. D-068 amendment appended (item 1 now PRESENTABLE); CURRENT-STATE + section 8 item 3 + G15 row updated (same commit). NOTHING adopted. +- REVERT: delete the draft doc + the amendment block; restore the two + CURRENT-STATE passages. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 0371c8d..f2ba830 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4398,3 +4398,21 @@ (D-131 sub-1) delivered via scripts/dc-rack-net.sh. The A1 test passes: any Roosevelt build session must grep this before laying out MAAS -- topology decides commissioning order, DNS/DHCP shape per rack, and backup custody across DCs. + +--- + +## D-068 -- AMENDMENT (2026-07-23): item-1 re-scoped migration plan DRAFTED; item presentable + +The 2026-07-05 amendment left item 1 ("get off EOL Vault 1.8") OPEN pending "a re-scoped +migration plan". That plan is now DRAFTED at operator direction ("Draft now", 2026-07-23 +queue-pass session): `docs/D-068-vault-migration-plan-draft.md` -- an options paper that +splits the item into three separately-ruled questions (Q1 interim rehearsal posture; Q2 +Roosevelt production path, four options incl. a NEW OpenBao-payload-under-reactive-charm +candidate and a NEW shrink-vault's-job candidate; Q3 triggers/review via the D-071 monthly +vehicle) with a verify-live list (V1-V5) that MUST be re-run before any Q2 ruling. Two +dated web probes (2026-07-23): no tls-certificates V1 adoption signal in the reactive +service charms; no OpenBao Juju charm exists. Nothing is adopted by the draft; item 1 +remains OPEN, now PRESENTABLE -- rulings proceed one per exchange per GA-R5. + +**Status:** item 1 OPEN / presentable (draft exists). Items 2-3 unchanged (RULED +2026-07-21). **Related:** D-069, D-071 (review vehicle), D-132 (same design window).