diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 35af84c..8835383 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -489,7 +489,7 @@ | G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | CLOSED 2026-07-23 (operator-ruled "Merge to main + full close"; commissioning 9/9 READY, merge commit on `main`, branch retired) -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED + WIRED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). **COMMISSIONING 9/9 READY 2026-07-23** (`docs/audit/dc1-commissioning-verify-20260723.txt`): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, `power_type=virsh` set + verified by real query-power-state (SEC-016 path proven), commissioned to **ALL 9 READY in ~3.5 min** (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. **G12 [V] leg (the dc1 build) is COMPLETE.** NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, **operator-gated merge of `dc-dc-g12-dc1-substrate` -> `main`** (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close). | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | CLOSED 2026-07-23 (operator-approved full maintenance bundle, logged window ops-sec010-reassert): qga channel retrofitted via outer tofu saved-plan apply 0/1/0 exact (`docs/audit/outer-plan-20260723-office1-qga.txt`; the apply's edge bounce = the ruled "next scheduled restart"; MACs were pinned 07-22 so the in-place-update trap class was closed); edge updated 26.7 -> 26.7.1 via REST (no reboot required; os-iperf had been REFUSED on 26.7 pending exactly this update); both plugins installed=1 by firmware-info read-back, `guest-ping` -> `{"return":{}}`, agent reports both legs, egress 0% loss, outer plan re-converged ZERO DIFF (`docs/audit/outer-plan-20260723-postqga-converged.txt`). Named close capture: `docs/audit/g13-close-20260723.txt` | | G14 | 12 OPEN SEC rows (SEC-001, -003..-008, SEC-012, -013, -014, plus SEC-015 + SEC-016 opened 2026-07-23 for dc1 credentials; SEC-010/-011 CLOSED) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012/-016 carry the same libvirt-group SCOPE hardening question; SEC-016 also a snap-refresh re-assert (queued to DC standup DoD) | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3); count re-verified vs `bash scripts/ledger-scan.sh` 2026-07-23 (12 open) | -| G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 items 2-3 RULED 2026-07-21; item 1 re-scoped plan DRAFTED 2026-07-23 (`docs/D-068-vault-migration-plan-draft.md`) -- item 1 OPEN/presentable, rulings pending (Q1/Q2/Q3, one per exchange) | +| G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 items 2-3 RULED 2026-07-21; item 1: plan DRAFTED + Q1/Q2-structure/Q3 ALL RULED 2026-07-23 (three amendments, utterances quoted; monthly-review lines delivered). Sole D-068 remainder: Q2 path selection at Roosevelt Vault design time -- G15 is otherwise decision-complete | | G16 | office1 edge `channels = []` state reconcile (the D-129 module-schema residual) | [R] operator rules the mechanism; then [V] the converged re-plan capture | operator + session | CLOSED 2026-07-21: RULED "State surgery (Recommended)" (GA-R5, session changelog item 16); executed per G6 precedent -- channels null -> [] injected, serial 29 -> 30, backup kept, guests untouched (office1-opnsense Id 2 running throughout); convergence = ZERO DIFF (`docs/audit/outer-plan-20260721-postG16-converged.txt`); section 5 re-recorded | ## 7. Version pins (measured; the authority for every pin) @@ -528,8 +528,10 @@ design-decisions.md is the authority). Q2 structural assumption RULED 2026-07-23 (Roosevelt baselines on 1.8/stable + a FUNDED remediation track; path 2a/2b/2c + deadline stay OPEN to Roosevelt design time on - re-verified V1-V5). OPEN: Q2 path selection (design time) + Q3 (review - mechanics, its own exchange). + re-verified V1-V5). Q3 RULED 2026-07-23 (monthly-review lines delivered + into ops-update-procedure 0c; design-time re-verify trigger). SOLE + remainder on item 1: Q2 path selection (2a/2b/2c) + deadline, at + Roosevelt Vault design time. 4. D-071 -- ADOPTED 2026-07-21: all four policy points ruled (monthly review trigger; patch-only controller jumps; standing order; in-channel-only refreshes), each its own GA-R5 exchange -- status diff --git a/docs/changelog-20260723-queue-pass.md b/docs/changelog-20260723-queue-pass.md index 5a76f28..482a55b 100644 --- a/docs/changelog-20260723-queue-pass.md +++ b/docs/changelog-20260723-queue-pass.md @@ -235,3 +235,18 @@ authority). Rules the STRUCTURE only -- path (2a/2b/2c) + deadline stay open to Roosevelt design time. CURRENT-STATE section 8 updated same commit. - REVERT: remove the amendment block + restore the section-8 passage. + +## Item 15 -- D-068 Q3 RULED + delivered (monthly-review lines; runbook re-grounded) + +- RULING (GA-R5): individually confirmed (see Item 13 note); question + exact + utterances in the design-decisions amendment (the authority). +- DELIVERED into `runbooks/ops-update-procedure.md`: new section 0c (the two + standing monthly checks -- Vault 1.8.x CVE scan, D-068 migration-signal + probes; read-only, log-always). ALSO fixed two FLAGGED stale claims found + in the same vehicle: the section-0 CAUTION still said bundle pins vault + 1.16 (reverted 2026-07-05; re-grounded to the ruled Q1/Q2 posture) and the + tail still called D-071 PROPOSED (ADOPTED 2026-07-21). +- CURRENT-STATE section 8 + G15 updated (same commit): D-068's sole + remainder is now the Q2 path selection at Roosevelt design time. +- REVERT: remove the amendment + section 0c; restore the CAUTION/tail + passages + the two CURRENT-STATE passages. diff --git a/docs/design-decisions.md b/docs/design-decisions.md index 91e667e..53b5464 100644 --- a/docs/design-decisions.md +++ b/docs/design-decisions.md @@ -4453,3 +4453,22 @@ **Status:** Q2 structural assumption RULED 2026-07-23; path + deadline OPEN to Roosevelt design time. **Related:** D-068 item 2 (listener TLS), D-132 (same design window), the draft (`docs/D-068-vault-migration-plan-draft.md`). + +## D-068 -- AMENDMENT (2026-07-23): item 1 / Q3 RULED -- monthly-review lines + design-time re-verify trigger + +Question as presented (migration-plan draft section 4, confirmed on exact recording text): +adopt the review mechanics. Operator answer, exact utterances: "Q3 Yes, add to the monthly +review." then "Confirmed (Recommended)". + +RULED: (i) TWO lines join the D-071 monthly review checklist (vehicle: +`runbooks/ops-update-procedure.md`, per D-071's adopted policy) -- a Vault 1.8.x CVE scan +(Q1 posture 1b) and a re-run of the migration-draft probes (reactive-charm +tls-certificates V1 adoption signal; OpenBao charm ecosystem + release cadence). +(ii) HARD TRIGGER: at Roosevelt Vault design time, verify items V1-V5 are RE-RUN and the +Q2 path ruling is made only on those fresh results (the 2026-07-23 probes must not be +cited as current then). (iii) If 2b (OpenBao payload fork) is later ruled the remediation +track, V3/V4 are its first funded work package, BEFORE any charm fork begins. + +**Status:** Q3 RULED 2026-07-23. D-068 item 1 now stands: Q1 ruled, Q2 structure ruled, +Q3 ruled -- the ONLY remainder is the Q2 PATH selection (2a/2b/2c) + deadline at Roosevelt +Vault design time. **Related:** D-071 (vehicle), the two amendments above. diff --git a/runbooks/ops-update-procedure.md b/runbooks/ops-update-procedure.md index 916c661..d4160ea 100644 --- a/runbooks/ops-update-procedure.md +++ b/runbooks/ops-update-procedure.md @@ -28,12 +28,15 @@ - **VAULT IS OUT OF SCOPE.** Live vault stays on `1.8/stable`. Do not refresh `vault` or `vault-mysql-router` in this procedure. -> CAUTION: `bundle.yaml` pins vault `1.16/stable` (D-068 / BUNDLEFIX-007) -> while live runs `1.8/stable`. A naive "sync live to the bundle" or a -> blanket refresh sweep would attempt a multi-minor major Vault upgrade -- -> exactly what D-068 (PROPOSED) says is NOT a casual `juju refresh` (unseal -> keys in hand, storage-format compatibility, rehearsal first). Until D-068 -> is ruled and rehearsed, vault is untouchable here. +> CAUTION (re-grounded 2026-07-23): `bundle.yaml` pins vault `1.8/stable` +> (the 1.16 forward-pin was REVERTED -- D-068 amendment 2026-07-05 / +> BUNDLEFIX-010: the new operator charm is INCOMPATIBLE with this reactive +> cloud, not merely a hard upgrade). Bundle and live now AGREE at 1.8. +> Vault stays untouchable here anyway: 1.8.8 is EOL under an explicit +> rehearsal-scoped risk-acceptance (D-068 Q1, RULED 2026-07-23), and +> modernization is a Roosevelt design-time decision (D-068 Q2) -- never an +> update-window action. Any vault refresh attempt from this procedure is a +> STOP. - **No channel changes.** D-002 pins channels; this procedure only moves revisions WITHIN a pinned channel. A desired channel change is a @@ -57,6 +60,21 @@ | Vault flips to sealed during this window | NOT expected -- nothing here restarts vault. That is an incident: stop, appendix-A. | | An EXCLUDED app's `can-upgrade-to` target changes mid-window | Observed 2026-07-05 (magnum's target moved during the window -- Charmhub republish in flight). Confirms the rule: targets are re-measured per app at refresh time; excluded apps stay excluded until the next window's pre-flight. | +## 0c. Monthly-review standing checks (D-071 vehicle; D-068 Q3, RULED 2026-07-23) + +Run these once per monthly review window (with or without an update window; +both are READ-ONLY -- findings are logged, never acted on mid-window): + +- **Vault 1.8.x CVE scan (D-068 Q1 posture 1b).** Check published CVEs + against Vault 1.8.8. No patches will ever exist (EOL) -- a hit informs + WORKAROUNDS or an emergency re-rule of the D-068 posture, never a refresh. + Log the scan date + outcome in the window's changelog even when clean. +- **D-068 migration-signal probes.** Re-run the migration-draft probes + (`docs/D-068-vault-migration-plan-draft.md` section 5): (a) any sign the + reactive OpenStack service charms gained tls-certificates V1 support; + (b) OpenBao charm ecosystem + release cadence. A signal change pulls the + D-068 Q2 path discussion forward; silence is also logged. + ## 1. Pre-flight and baseline ### 1.1 Session bootstrap @@ -420,5 +438,7 @@ upgrade, no full API blackout) -- revalidate gates on bare metal. - Controller backup story on bare metal: evaluate supported juju-db dump tooling as part of the Roosevelt controller design, not improvised here. -- Cadence policy and window sizing: D-071 (PROPOSED) -- rule before - Roosevelt operations begin. +- Cadence policy and window sizing: D-071 ADOPTED 2026-07-21 (monthly + review trigger, patch-only controller jumps, standing order, + in-channel-only refreshes) -- this runbook is the policy vehicle; the + Roosevelt-specific window sizing still revalidates on bare metal.