diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 112cdef..8a9783e 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -37,6 +37,16 @@ > Sweep: `docs/audit/queued-findings-20260808-dc0-tailscale-incident-reip-pivot.txt`. > Ruling-prep package (drafted GA-R5 question + D-101/115/124/134 reconciliation + measured blast > radius + the 3 OWED live-free checks): `docs/audit/reip-1013-ga-r5-ruling-prep-20260808.md` (2026-08-08). +> **UPDATE 2026-08-09: 2 of the 3 owed live-free checks now PASS with fresh measurement** -- +> live NetBox apex (10.13 = 0 objects; 10.12 = 149 control) and the Headscale/tailnet route +> table (no 10.13 overlap; the exact 10.12 collision reproduced as the positive control). +> Check #3 (VR0 cloud internals) is an operator accept-or-run choice (VR0 is tailnet-only, +> unreachable from the authorized vantage). Captured in the open-items review below. +> **OPEN-ITEMS REVIEW (D/SEC/DOCFIX/BUNDLEFIX vs the hardened end goal), 2026-08-09:** two-pass +> (self + fable advisor) decision package -- NOT a ruling, mutates no status. Conclusion: the +> backlog is overwhelmingly correctly-deferred; the re-IP GA-R5 ruling (would be D-143) is the +> only item gating the end goal; R7 teardown-credential-revocation + R16 10.13 naming-DOCFIX are +> the two re-IP-coupled gaps to build. `docs/audit/open-items-review-20260809.md`. > > **dc0 checkpoint scope (operator 2026-08-08): "activate + smoke-test"** -- networks + Octavia (1 test > LB) + Designate (1 test zone) + wrap gates (cloud-assert BOM, controller backup, verify-live diff --git a/docs/audit/open-items-review-20260809.md b/docs/audit/open-items-review-20260809.md new file mode 100644 index 0000000..aa07fa6 --- /dev/null +++ b/docs/audit/open-items-review-20260809.md @@ -0,0 +1,532 @@ +# Open-items review -- D / SEC / DOCFIX / BUNDLEFIX, against the hardened end goal + +**ANALYSIS + RECOMMENDATION PACKAGE, NOT A RULING.** This file mutates no +authoritative surface (no `design-decisions.md`, no `security-ledger.md`, no +`CURRENT-STATE.md`, no live infra). It exists so the operator can see the entire +open-item backlog in one place, judged against the project's current state and the +hardened end goal, and rule/schedule from there. Every recommendation is a candidate +for a gated exchange (GA-R5 for rulings, presented-and-approved for record edits). + +- Author: this session (open-items review task), 2026-08-09. READ-ONLY. +- Inputs read this session: `docs/CURRENT-STATE.md` (status authority, top section in + full); `docs/security-ledger.md` (all 28 open rows in full); `docs/design-decisions.md` + D-068/131/132/139/140/141/142 blocks + the D-134 amendment index; `scripts/ledger-scan.sh` + (method); `runbooks/dc-dc-teardown-rollback.md` (credential-gate section C.5); + `docs/audit/reip-1013-ga-r5-ruling-prep-20260808.md` (Section A). +- Machine baseline (ledger-scan, this session): **4 PROPOSED/OPEN decisions** + (D-068, D-131, D-132, D-142); **28 open SEC rows**; next-free **D-143 / DOCFIX-214 / + BUNDLEFIX-059**. + +--- + +## 0. The frame -- what "superseded" and "recommendation" are measured against + +Two things define the yardstick for this review; nothing below is judged in the +abstract. + +1. **The hardened end goal (from CURRENT-STATE section 1, the standing pivot).** + - dc0 is driven to a FULL deployment as a **checkpoint** (activate + smoke-test: + networks + Octavia 1 test LB + Designate 1 test zone + wrap gates; Magnum/CAPI + deferred), then the DC substrate is **torn down and redeployed on `10.13.0.0/16`** + because `10.12.0.0/16` collides with the live IPv4 cloud (vr0-dc0). + - The cloud is **IPv6-primary** (D-101 "v6 wherever possible"; D-139 east-west + v6-only on GUA -- and geneve-over-v6 is now LIVE-CONFIRMED WORKING, 2026-08-09). + v4 survives only where a charm forces it (D-141 container-VIP necessity case). + - **Minimize delta to Roosevelt** governs everything: the runbooks/scripts are + deliverables, so most open items are deliberately deferred to a Roosevelt design + gate rather than solved in the rehearsal. + +2. **The three horizons the open items actually retire against.** Almost none of the + backlog is "do it now." Reading every open row, each retires at one of: + - **RE-IP REDEPLOY** (near-term, DC-substrate teardown+redeploy on 10.13). This is a + *rebuild* event, and it is the single most decision-relevant fact in this review: + many SEC rows carry the clause "rotate at v1 close, **or immediately if the host is + rebuilt or shared**", and the redeploy fires that clause for DC-resident material. + - **v1 CLOSE** (end of the whole VR1 rehearsal). The credential rotation/revocation + cascade for material that lives on the *persistent* hosts (Office1 headend, + vcloud jumphost) -- these are NOT torn down by the DC redeploy. + - **ROOSEVELT DESIGN TIME** (the next, bare-metal deployment). The pinned + architectural questions. + +**Bottom line up front:** the open-item backlog is overwhelmingly *correctly deferred*. +The review's job is to confirm those deferrals still hold under the re-IP end goal, and +to surface the small number of items that (a) genuinely gate the end goal, (b) become +actionable *because of* the re-IP, or (c) are stale/superseded records that should be +reconciled. **Exactly one open item actually blocks the end goal: the re-IP GA-R5 +ruling (Section 1).** Everything else is schedule, hygiene, or Roosevelt. + +--- + +## 1. THE ONE ACTUAL BLOCKER -- the owed re-IP GA-R5 ruling (would be D-143) + +**This is not on the D/SEC/DOCFIX/BUNDLEFIX lists yet -- it is an owed ruling with no +number -- but it is the item that gates the hardened end goal, so it leads.** + +- **What:** move VR1 from `10.12.0.0/16` to `10.13.0.0/16`. It is a **D-115 + supersession** (D-115 rejected 10.13 and set the Cloud role to 10.12) and it + **terminates D-101's "vr1-dc0 inherits vr0-dc0 v4 unchanged" clause**. Ruling-prep + package is complete and read-only: `docs/audit/reip-1013-ga-r5-ruling-prep-20260808.md` + (decision reconciliation D-101/115/124/134, measured blast radius, the 3 owed + live-free checks). Draft mapping: octet-preserving `10.12.a.b -> 10.13.a.b`. +- **Status:** NOT YET RULED (CURRENT-STATE section 1). Owed before it can be presented: + the **3 live-free checks** (confirm 10.13 is free on the existing cloud + the tailnet). +- **End-goal interaction:** everything downstream of the checkpoint -- the re-carve, + the redeploy, the D-101/115/124/134 reconciliation, and the D-140 OpenTofu review + (which is pinned to *after* a hardened+tested deployment) -- waits on this. It is also + the natural forcing event for a chunk of the SEC backlog (Section 3b). +- **RECOMMENDATION R1 (HIGH -- the gating action):** complete the 3 live-free checks, + then present the GA-R5 re-IP ruling as **D-143** (grep next-free again at ruling time + per the prep doc). Sequence it as: dc0 checkpoint wrap-gates complete -> re-IP ruling + -> teardown (with the revocation checklist of R7) -> redeploy on 10.13. + +- **LIVE STATUS OF THE 3 OWED CHECKS (measured this session, operator authorized live + discovery -- full detail in the Appendix):** + - **Check #2, live NetBox apex (office1-netbox, read-only dump): PASS.** 10.13.0.0/16 + has **0** objects across aggregates/prefixes/ranges/IPs; positive control 10.12 = 149 + objects (so the query is real, not empty). 10.13 nests cleanly under the existing + `10.0.0.0/8` aggregate. + - **Check #1, Headscale/tailnet routes (from the live `office1-tailscale` node): PASS.** + No approved subnet route overlaps 10.13.0.0/16 (only generic exit-node `0.0.0.0/0`). + Positive control is decisive: the run shows the **exact 10.12 collision that caused + the pivot** -- `vopenstack-jesse-tailscale` (the live VR0 cloud) advertising + `10.12.4.0/22` + `10.12.8.0/22`. 10.13 is also clear of Roosevelt (`10.17.x`), + willamette (`10.1/10.16`), and Office1 (`10.10`) on the shared tailnet. *Blind spot: + this is the approved (AllowedIPs) view from one node; an advertised-but-pending view + needs the Headscale server itself.* + - **Check #3, live VR0 cloud internals: NOT COMPLETABLE from the authorized vantage.** + The VR0 KVM host (10.12.64.1) is unreachable from vcloud and there is no VR0 + openrc/juju here; VR0 (`vopenstack`, logxen's cloud) is a separate trust domain + reachable only over the tailnet, which vcloud is deliberately not joined to (the + STOPPED tailscale workstream -- installing it would be a mutation and is out of + scope). Best available evidence: VR0 advertises **nothing** in 10.13 on the tailnet. + - **NET EFFECT ON R1:** 2 of 3 checks now PASS with live measurement; the collision + *mechanism* that caused the pivot (the shared tailnet) is authoritatively clear of + 10.13. The ruling can be presented once the operator EITHER runs the VR0-internal + MAAS/neutron sweep from a tailnet-joined workstation (D-107) OR accepts the + tailnet+apex evidence as sufficient (defensible: the tailnet is the surface that + actually broke 10.12). **This is the concrete unblock for the end goal.** + +--- + +## 2. Open design decisions (D-NNN) -- status, end-goal interaction, recommendation + +All five open decisions are **deferred by design**; none is a defect and none blocks the +checkpoint. Summary judgement per decision: + +### D-068 -- Vault substrate hardening (Roosevelt) -- PROPOSED (parent), item 1 mostly ruled +- **State:** Q1 (EOL risk) RULED rehearsal-scoped acceptance, posture 1b; Q2 *structure* + RULED (Roosevelt baselines on 1.8 + funded remediation track); Q3 RULED (monthly-review + lines + design-time re-verify trigger via D-071). **Sole remainder OPEN:** the Q2 *path* + selection (2a in-place upgrade / 2b OpenBao payload fork / 2c stay-and-fund) + deadline, + made at **Roosevelt Vault design time** on FRESH probes (the 2026-07-23 V1-V5 probes must + be re-run, not cited). +- **End-goal interaction:** none for VR1/re-IP. The rehearsal runs on Vault 1.8 by ruling. +- **RECOMMENDATION R2 (DEFER -- confirm, don't touch):** correctly parked at Roosevelt. + No action this deployment. The one live obligation is that D-071's monthly review + actually carries the two D-068 lines -- worth a one-line verification at v1 close, not + now. + +### D-131 -- node-facing DNS for rack-only controllers -- PARTIALLY RULED +- **State:** sub-1 (standing per-DC forwarder pattern), sub-2 (metal-admin only), sub-3 + (stale-read, no defect) all RULED/RESOLVED and BUILT (`dc-rack-net.sh`). **sub-4 OPEN:** + Roosevelt shape (same forwarder / region-DNS-over-WAN+firewall exception / wait for the + upstream MAAS fix + minimum version), gated on the Launchpad outcome; carries a PINNED + architectural review (best-practice + security + vendor-config) for next-deployment + design time. +- **End-goal interaction:** the *built* half rides the re-IP unchanged (the forwarder is + site-keyed; it re-applies on 10.13 as part of standup DoD). sub-4 is Roosevelt. +- **RECOMMENDATION R3 (DEFER):** correctly parked. At re-IP redeploy, verify + `dc-rack-net.sh install` re-runs as standup DoD on the 10.13 addresses (it is already in + the standup definition-of-done, so this is a checklist confirmation, not new work). + +### D-132 -- per-DC MAAS topology -- PROPOSED, q1 ruled for VR1 +- **State:** q1 (HA region per DC) RULED *for VR1* as a per-DC region controller (built: + dc0 region VM at utility `.6`). q2 (rack-top controllers deferring to the site region) + and q3 (cross-site backup custody) OPEN, pinned to Roosevelt. +- **End-goal interaction:** the built per-DC region is re-stood-up on 10.13 (SEC-027 + material re-minted). q2/q3 are Roosevelt. +- **RECOMMENDATION R4 (DEFER):** correctly parked. No VR1 action. + +### D-140 -- OpenTofu manages the Juju layer -- PINNED to end-of-deployment review +- **State:** sequencing RULED (do NOT convert now; hardened+tested dc0 deploy becomes the + INPUT to a later translation, reviewed at end-of-deployment). D-139 step-6 ordering + sub-ruling folded in. +- **End-goal interaction (IMPORTANT sequencing note):** "end of deployment" now means + *after the 10.13 redeploy*, because the checkpoint is explicitly not the final cloud. + The D-140 review therefore lands after redeploy + hardening, not after the checkpoint. +- **RECOMMENDATION R5 (DEFER, with a sequencing note):** confirm the D-140 review is + scheduled after the 10.13 redeploy (the hardened+tested deployment the ruling names is + the redeployed one, not the checkpoint). No action now. + +### D-142 -- Vault-init workflow QoL sweep -- PROPOSED, approved-in-principle, deferred +- **State:** R1-R5 approved in principle; implementation + test deferred to "next + opportunity"; **R2 (off-host transport of init.txt) is an UNRESOLVED operator + question**; R1 is also a real DEFECT fix (`-m openstack` targets a non-existent VR1 + model; filed as F13). +- **End-goal interaction:** the per-DC Vault standup recurs at the 10.13 redeploy, so the + QoL/defect fix has a concrete near-term consumer -- this is the "next opportunity." +- **RECOMMENDATION R6 (LOW, but time it to the redeploy):** the R1 model-target defect + should be fixed before the next Vault init runs (i.e. before the 10.13 Vault standup), + and R2's transport question should be put to the operator so R3's gate can be built. + This is the one "deferred" D-item with a near-term forcing event; the rest are + Roosevelt/v1-close. Still not urgent for the *checkpoint* (dc0 Vault is already up). + +**D-items NOT open but worth a note re: the end goal (no manufactured contradictions):** +- **D-141 (dual-stack, v4-active/v6-reserved) is NOT superseded by the geneve-over-v6 + win.** D-141 is the *container-VIP* necessity case (juju LP #1723240 -- API charms in + containers get no v6 address); the 2026-08-09 result is about *east-west geneve + tunnels* (metal chassis), an orthogonal plane. The IPv6-primary posture and D-141 + coexist by design (see the `ipv6-primary-posture` standing note). No change. +- **D-139** is ADOPTED and its 2026-08-09 amendment is ruled; its rebuild recipe (container + v6 carve + unbracketed encap + `overlay_ip_version=6`) is an *input to the 10.13 + redeploy*, tracked in the root-cause record -- not an open decision. + +--- + +## 3. Security ledger -- 28 open rows, categorized by the horizon they retire against + +The rows are not equally urgent; grouping them by *what event closes them* is the useful +lens. The buckets below partition all 28: **3a=10, 3b=8, 3b'=3, 3c=3, 3d=3, 3e-watch=1 +(SEC-024) = 28.** (This partition was corrected after an advisor catch -- the first draft +asserted "sums to 28" while only accounting for 24; SEC-003/015/016/024 were unplaced.) + +### 3a. Persistent-host rotation cascade -> retires at v1 CLOSE (not touched by the re-IP) +These live on the **Office1 headend / vcloud jumphost**, which the DC redeploy does NOT +tear down. They stay on the v1-close schedule and the re-IP changes nothing about them. +(10 rows: SEC-001, 003, 004, 005, 006, 007, 008, 013, 020, 025.) + +- **SEC-003** Vault unseal-key custody is single-operator (bus factor) -- operator-DEFERRED; + per its own row it "keeps d011-06 MANUAL, gates D-011 full close." Arguably the most + *consequential* row in 3a (a bus-factor on the unseal keys), even though it is correctly + deferred; the D-011 close coupling is the reason to keep it visible. +- **SEC-004** repo public->private at v1 close (global). +- **SEC-005** GitBucket PAT (plaintext on jumphost) -- revoke+rotate at v1 close. +- **SEC-006** NetBox API token BURNED (in a transcript) -- revoke+reissue, operator-deferred + to deployment completion. *This one is the weakest deferral in the set* -- a burned token + is live-exposed until revoked (see R8). +- **SEC-007** Office1 edge creds (root pw + hash + SSH key + OPNsense API) -- rotation obligation. +- **SEC-008** Tailscale authkey on vcloud -- rotate/reissue at v1 close. +- **SEC-013** Office1 MAAS API key -- surface narrowed to the operator's CLI profile only. +- **SEC-020** Office1 MAAS region superusers (`admin` + `operator`) -- rotate both; stale-copy + hazard between `voffice1:/root/maas-secrets/admin.pass` and the vcloud copy. +- **SEC-025** NetBox web-GUI admin password -- rotate; stale-copy hazard (D-137 tier-3 V1 now + detects divergence). +- **SEC-001** libvirt SSH credential printed during reenroll -- rotate at v1 close. +- **RECOMMENDATION R8 (MEDIUM, one decision):** re-affirm-or-act on **SEC-006** specifically. + A burned credential deferred to "completion" has been live-exposed since 2026-07-12; the + re-IP redeploy is a natural, earlier checkpoint to revoke+reissue it (NetBox on the + headend survives the redeploy, so it can be done any time). Present: revoke SEC-006 now + vs keep the completion deferral. The rest of 3a stays deferred (correct). + +### 3b. dc0-substrate-resident credentials -> the re-IP redeploy FIRES their rebuild clause +These live on the **dc0 rack / dc0 region VM / dc0 edge / dc0 node power path** -- exactly +the substrate being torn down and rebuilt on 10.13 (residency MEASURED live this session, +Appendix item 5). Each is re-minted fresh at redeploy; the OLD material must be **revoked as +part of teardown**. This is the review's key "superseded by project state" finding for SEC. +(8 rows: SEC-012, 014, 018, 026, 027, 028, 029, 032 -- all dc0.) + +- **SEC-012** dc0 MAAS->libvirt power key. +- **SEC-014** dc0 rack MAAS cluster secret (exposed to session context; re-enrollment rotates it). +- **SEC-018** dc0 Juju->MAAS admin API key (Office1-region-scoped; per-DC regions under + D-132 q1 reduce its role, but the key still exists and needs revocation). +- **SEC-026** MAAS admin key resident on a DC host (regional blast radius) -- residency ends + with the host at teardown. +- **SEC-027** per-DC MAAS *region* credentials (db pw + region admin pw + region API key). +- **SEC-028** per-DC Juju *service* credential (minted in the DC region; rack-resident). +- **SEC-029** Octavia PKI overlay resident on the rack (cert + private key). +- **SEC-032** dc0 edge creds rack-resident (private key + dead `opnsense-api.txt`). +- **THE GAP:** `runbooks/dc-dc-teardown-rollback.md` has a credential *gate* (C.5, + region-scoped) that ensures creds *work* for a rebuild -- it does **not** carry a + *revocation* checklist for the material being abandoned at teardown. So today's teardown + path would tear down the substrate and leave the old DC-resident keys/tokens live. +- **RECOMMENDATION R7 (HIGH -- becomes actionable at the re-IP, and it is a real gap):** + add a **credential-revocation checklist to the teardown runbook** (revoke/shred the + DC-substrate-resident material enumerated above as a definition-of-done of teardown), + so the 10.13 redeploy re-mints on a clean slate and the old keys do not linger. This + simultaneously *discharges* the "rebuild clause" of SEC-012/014/018/019/026/027/028/ + 029/032 for the DC side -- i.e. the re-IP is their natural close, IF teardown revokes. + Roosevelt-relevant (every DC rebuild needs this). Build it as part of the re-IP work. + +### 3b'. dc1-labeled material on PERSISTENT hosts -> v1 CLOSE, NOT the re-IP (dc1 never built) +Important scope boundary the advisor flagged: these are dc1 credentials, but **dc1 was +never built** (0 machines in vr1-dc1-region, held) -- so there is no dc1 substrate to tear +down, and R7's teardown-revocation does NOT cover them. They sit on persistent hosts +(vcloud / voffice1 -- SEC-016's headend key was MEASURED at +`voffice1:~/vr1-dc1-creds/vr1-dc1_svc_ed25519` this session) and retire at **v1 close**. +(3 rows: SEC-015, 016, 019.) +- **SEC-015** dc1 edge SSH key -- transient rack exposure already remediated; standing + surface is the vcloud copy + the minted edge API key. +- **SEC-016** dedicated dc1 MAAS->libvirt power key (headend-resident, measured). +- **SEC-019** dc1 Juju->MAAS admin API key (minted on vcloud; the dc1 twin of SEC-018). +- **Note:** SEC-027/028's *dc1* rows are NOT here -- they are NOT YET MINTED (forward + register only; Section 3f), so there is no material to revoke, only rows that self-clear + when dc1's region is built at the redeploy. + +### 3c. Accepted-by-ruling postures -> not remediations, standing watches +- **SEC-017** caveman plugin DISABLED; standing re-verify duty on any re-enable/update. +- **SEC-030** local permission rules bypass the ask-gating -- ACCEPTED ("leave them as they + are"); compensating control is the presentation discipline; NOT promoted to team policy. +- **SEC-033** tls-certificates databag exposes vault's global-client key -- interface-level, + mitigated by juju model RBAC; accept-or-mitigate is an operator call. +- **RECOMMENDATION R9 (NO ACTION -- confirm):** these are ruled postures, not open work. + The only live duty is SEC-017's re-verify-on-re-enable, which is conditional. Leave as-is. + +### 3d. Detection/tooling scope-gaps -> coupled to D-137 forks (unbuilt), interim by-hand +- **SEC-021(b)** per-DC power-key naming/custody asymmetry (dc0 vs dc1 shape differ). *Note + SEC-021(a) -- the API-access gap -- is already ANSWERED by SEC-032 (off-jumphost copy + found + fresh key consolidated), see Section 3e.* +- **SEC-022** two UNAUDITED shadow `*-creds/` stores on voffice1, structurally invisible to + `creds-audit` (no remote capability) -- coupled to the D-137 `--remote` fork (unruled). +- **SEC-023** sprawl-glob blind spots + the PREDICTED Stage-5 exposure + (`~/admin-openrc` from `phase-03-admin-openrc.sh`, matches no sprawl glob) -- becomes + live when Stage-5 phase-03 runs. +- **RECOMMENDATION R10 (MEDIUM -- one is a near-term trap):** SEC-023's predicted + `~/admin-openrc` exposure fires the moment phase-03 runs on the checkpoint (Stage-5 is + open now). Two cheap, D-137-independent fixes: widen the sprawl globs to cover + `admin-openrc`/`*.apikey`/`*.pem`/`*_ed25519`, and consolidate `~/admin-openrc` by hand + at mint time. Do the by-hand consolidation whenever phase-03 runs; schedule the glob + widening + the D-137 `--remote` fork (SEC-022) as post-checkpoint hardening, not now. + +### 3e. Superseded / stale RECORD notes -> reconcile (record hygiene, operator-gated) +These are places where the *record* lags *project state* -- exactly the "superseded by +project state" the operator asked for. + +- **SEC-021(a) -- access gap CLOSED, but the P5 finding is STILL RED (corrected by live + measurement this session).** SEC-032 found the off-jumphost copy and closed the *access* + gap: `opnsense-api.txt` now exists on both the jumphost and the dc0 rack (measured this + session). BUT the live `creds-matrix.py` run shows **`S2 vr1-dc0 dc0-edge-api + EXPECTED-BUT-ABSENT` is STILL a FAIL** -- the derived manifest (`creds-manifests/ + vr1-dc0.manifest`) was never regenerated to declare it. (My first draft overstated this + as "E1 now CLEAN / effectively resolved"; the measurement corrects it, and SEC-032 + item (2) already flagged the manifest-regen as owed.) **RECOMMENDATION R11 (revised):** + regenerate the dc0 manifest (`creds-matrix.py --render`) to declare the now-present + `opnsense-api.txt`, which clears the S2 finding; and refresh SEC-021's text so (a) reads + "access closed by SEC-032; manifest declaration owed," (b) still open. +- **`creds-matrix.tsv:76` note `n-dc0-edge-api-absent` + `creds-matrix-notes.md:123` are + factually STALE** -- they assert the dc0 edge API cred is absent; it now exists (SEC-032). + SEC-032 itself flags this. **RECOMMENDATION R12:** replace the note wording with the new + invariant (never delete the row to go green -- the note says so). +- **The SEC-ledger's repeated "CURRENT-STATE G14 reads 12, reconciles at Stage 4 close" + count-notes are STALE.** CURRENT-STATE now carries **28 open SEC** (reconciled; e.g. + `:3984`, `:4251-4253`), and Stage 4 is long closed. The "reconciles at Stage 4 close" + deferral in SEC-017/020/023 never happened as written and is now moot. + **RECOMMENDATION R13:** drop the stale G14 count-note clauses from those SEC rows (the + count now lives correctly in CURRENT-STATE per GA-R1). Pure hygiene. +- **SEC-024 (the standing-WATCH row that completes the partition).** tfstate `.backup` at + 0664; the mode defect is fixed and retention was RULED "Keep both" (2026-07-27), so this + is a WATCH, not an open remediation -- the umask *cause* was out of ruled scope, so a + future `tofu apply` may rewrite 0664 and P5 is the detection. **End-goal note:** the + 10.13 redeploy rebuilds the OpenTofu state from scratch, so the specific 0664 files retire + then -- but the umask cause rides along unless fixed, so it will recur on 10.13. **No + action recommended now** (ruled watch); worth a one-line umask check when the 10.13 tofu + roots are first applied. + +### 3f. P5 grew from 6 to 11 findings BY DESIGN (authoritative voffice1 reading, LIVE) +**Framing corrected after an advisor catch + re-measurement on the host that counts.** +The 2026-07-30 acceptance capture (`stage5-preflight-dc0-20260730.txt`, on **voffice1**) = +**101 rows / 19 clean groups / 6 findings**. The matrix then **grew 101 -> 121 rows by +design** on the same day via the SEC-027/028/029 mint commits (`c664f4f`, `4a78a28`, +`d870d76`), which registered the new per-DC region/juju/PKI credentials. Re-running P5 +exactly as preflight does (`creds-matrix.py --tier2`) **on voffice1** this session (clone +verified at `6aae475` == HEAD, not stale): **FAIL, 121 rows, 19 clean groups, 11 findings.** +The delta vs the accepted 6 is **+5, and all 5 are dc1 forward-register S2 rows** +(SEC-027 region-admin/api-key/db + SEC-028 juju-apikey/user). They FAIL **by design** -- +dc1's region does not exist (dc1 held) -- and SEC-028 already records they are "NOT covered +by the operator's 2026-07-30 P5 acceptance" and "must NOT be deleted to go green." They +self-clear when dc1's region is built at the 10.13 redeploy. So the correct statement is +**"the matrix grew as designed; the gate expanded by 5 by-design forward-register rows,"** +NOT "the gate silently grew and you now face 12." +- **SEPARATE, off-gate finding (do not conflate):** my *vcloud* run (vcloud = the jumphost) + surfaced an **UNDECLARED secret `opnsense-api-rebuild-20260807.txt`** (dc1 edge API token, + 2026-08-07 rebuild) sitting in no matrix row. voffice1's P5 does NOT see it -- it correctly + refuses to probe jumphost-local paths (`E0 ... NOT PROBED`), so this is **not** a P5-gate + finding; it is a real jumphost-local hygiene item of exactly the class SEC-022/023 warn + about (an undeclared secret the gate structurally cannot see without `--remote`). +- **RECOMMENDATION R15 (LOW/MEDIUM -- mostly acknowledgement, one real item):** (i) when P5 + is re-cited as accepted, record that it is now **11 by-design findings** (6 accepted + 5 + dc1 forward-register), self-clearing at the dc1 redeploy -- a note, not a re-audit; (ii) + the one genuinely actionable item is the undeclared `opnsense-api-rebuild-20260807.txt` on + the jumphost -- **DECLARE it (one matrix/manifest row). Do NOT shred it** without first + proving it superseded. The filename and SEC-031's closure note (the 2026-08-07 dc1 edge + rebuild sequence was "console bootstrap -> API mint -> addressing") indicate this is very + likely the **live** dc1 edge API credential -- unlike SEC-032's dc0 case, where the + rebuild destroyed the config.xml holding the old hash and made that file provably dead. + If the operator wants it gone, the disambiguator is a live auth-test against the dc1 edge + (status-only, secret never printed); shred only on a proven-dead result. Never weaken a + row to green the gate (standing rule). None of this blocks the checkpoint. + +--- + +## 4. DOCFIX / BUNDLEFIX / queued-findings (F-register) + +- **Nature (verified via `ledger-scan.sh`):** DOCFIX and BUNDLEFIX are *applied-at-creation* + operational items (GA-R3 [OPS]: a runbook edit / session-changelog line / as-built row). + There is **no persistent open DOCFIX/BUNDLEFIX register** -- the scan only computes + next-free (DOCFIX-214, BUNDLEFIX-059). A DOCFIX/BUNDLEFIX is closed when written. +- **Where deferred operational work actually lives:** the per-session + `docs/audit/queued-findings-*.txt` **F-registers**. These carry items explicitly held + for later (e.g. F13 = the D-142 R1 model-target defect; the phase-03/06 F-CV items were + RESOLVED 2026-08-06 via BUNDLEFIX-056/057/058). Recent BUNDLEFIX activity + (053-058) is all bundle binding/HA-chain corrections, applied and harness-covered. +- **RECOMMENDATION R14 (LOW -- sweep at re-IP):** before the 10.13 teardown, do a single + pass over the open `queued-findings-*.txt` F-items to confirm none is a silent + precondition for the redeploy (F13/D-142-R1 is the known live one; it is captured in + Section 2 R6). No standalone DOCFIX/BUNDLEFIX item is open or superseded in a way that + needs action now. +- **10.13 NAMING-COLLISION DOCFIX CANDIDATES (surfaced by the ruling-prep, confirmed live + present this session) -- these ARE "superseded by project state" and are re-IP-coupled:** + - `netbox/README.md:49` -- stale importer default `VR1_DC1_V4_SUPERNET=10.13.0.0/19` + (pre-D-115; D-115 moved dc1 to `10.12.64.0/19`). Confirmed present. + - `tests/dc-rack-mgmt-import/test_logic.py:265` -- a LIVE test literal `10.13.0.5` + (`--rack-ip`), mirrored in `docs/archive/stage3-review-base.patch:3667`. Confirmed + present. The 10.13 subnetting draft MISSED this one. + - **RECOMMENDATION R16 (LOW, but MUST ride R1):** adopting `10.13.0.0/16` as the Cloud /16 + gives 10.13 a THIRD in-repo meaning and puts a rack-IP test fixture INSIDE the new Cloud + space. Reconcile both literals as part of the re-IP change (a DOCFIX folded into the + D-143 work), so the three meanings do not confuse a future reader or a test. + +--- + +## 5. Cross-cutting: what is genuinely SUPERSEDED by project state / the end goal + +Consolidated answer to the operator's explicit ask: + +1. **D-101's v4-inheritance clause is superseded** by the re-IP (terminated -- see the + ruling-prep doc). This is a *design* supersession that needs the D-143 ruling to make + formal (R1). +2. **The DC-substrate SEC rotation obligations (3b) are effectively superseded/discharged + by the re-IP redeploy** -- but ONLY IF teardown revokes the old material (R7). Without + the revocation checklist, the redeploy leaves them live, so this is a conditional + supersession, not automatic. +3. **SEC-021(a) is superseded (resolved) by SEC-032** (R11); the `n-dc0-edge-api-absent` + matrix note is stale (R12); the SEC "G14 reads 12" count-notes are stale (R13). +4. **D-140's "end of deployment" review point moved** -- it now lands after the 10.13 + redeploy, not the checkpoint (R5). +5. **NOT superseded, guard against the manufactured contradiction:** D-141 (dual-stack VIP + necessity) is NOT overturned by the geneve-over-v6 win -- different plane; D-068/131/132 + Roosevelt deferrals are NOT made actionable by the re-IP (they are Roosevelt, not VR1). + +--- + +## 6. Prioritized recommendation list (my review -- pre-advisor) + +| # | Priority | Item | Action | Horizon | +|---|---|---|---|---| +| R1 | **HIGH (blocker)** | re-IP ruling | 2/3 live-free checks now PASS; run VR0-internal or accept -> present GA-R5 as D-143 | before teardown | +| R7 | **HIGH** | teardown revocation gap | add DC-cred revocation checklist (residency measured this session) to teardown runbook | with re-IP | +| R15 | LOW/MED | P5 now 11 not 6 (LIVE, voffice1) | note the +5 as by-design (self-clear at dc1 redeploy); declare/shred the off-gate undeclared jumphost token | before citing P5 accepted | +| R16 | LOW | 10.13 naming collisions | fold `netbox/README.md:49` + test literal `10.13.0.5` DOCFIX into the re-IP | rides R1 | +| R8 | MEDIUM | SEC-006 burned token | present revoke-now vs keep completion-deferral | any time | +| R10 | MEDIUM | SEC-023 `~/admin-openrc` | by-hand consolidate at phase-03; widen sprawl globs | phase-03 run | +| R6 | LOW | D-142 R1 defect + R2 Q | fix `-m` model-target before next Vault init; put R2 to operator | before 10.13 Vault | +| R11 | LOW (hygiene) | SEC-021(a) | regen dc0 manifest (S2 still RED -- measured); refresh SEC-021 text | batchable | +| R12 | LOW (hygiene) | stale matrix note | replace `n-dc0-edge-api-absent` wording | batchable | +| R13 | LOW (hygiene) | stale G14 count-notes | drop from SEC-017/020/023 | batchable | +| R14 | LOW | F-register sweep | one pass before teardown | before teardown | +| R2/R3/R4/R5/R9 | DEFER | D-068/131/132/140 + accepted SEC postures | confirm deferral, no action | Roosevelt / v1-close | + +**Honest framing for the operator:** the backlog looks large (5 open D + 28 SEC) but is +almost entirely correctly deferred. The real content of this review is four things: the +**re-IP ruling (R1)** is the only thing gating the end goal; the **teardown revocation +checklist (R7)** is a genuine gap the re-IP makes actionable; a handful of **near-term +credential traps (R8, R10)**; and a small batch of **stale-record hygiene (R11-R13)**. +Everything else is "confirm the deferral holds." + +--- + +## 7. Appendix -- LIVE confirmations (measured this session, operator-authorized discovery) + +All read-only. No live mutation. Credential dirs listed with `ls` only (contents never +read -- the guard-hook rule). Captures in `$CLAUDE_JOB_DIR/tmp/` (apex dump, tailnet json). + +1. **Live NetBox apex (office1-netbox 10.10.1.10:8000), tested tool `office1-record-dump.py`, + from vcloud, record env sourced (token never printed):** 152 prefixes / 5 aggregates / 27 + ranges / 194 IPs dumped. **10.13.0.0/16 = 0 objects; 10.12.0.0/16 = 149 (positive + control).** Aggregates present: `10.0.0.0/8`, `172.16.0.0/12`, `23.157.124.0/24`, + `2602:f3e2::/36`, `fd50:840e:74e2::/48` -- 10.13 nests under 10.0.0.0/8 with no collision. +2. **Tailnet routes via `office1-tailscale` (`tailscale status --json`, v1.98.9):** 17 + advertised/approved subnet routes. Overlapping 10.13.0.0/16: only 4x `0.0.0.0/0` + exit-node adverts (not a subnet collision). Overlapping 10.12.0.0/16 (control): those + 4 PLUS `10.12.4.0/22` + `10.12.8.0/22` from `vopenstack-jesse-tailscale` -- **the live + VR0 collision, reproduced.** Other projects on the tailnet: Roosevelt `10.17.4.0/22`/ + `10.17.8.0/22`/`10.0.0.0/24`, willamette `10.1.0.0/24`/`10.16.0.0/24`, Office1 + `10.10.0.0/22`. None touch 10.13. +3. **VR0 reach:** `10.12.64.1:22` unreachable from vcloud; no VR0 openrc/juju/clouds.yaml + on vcloud; `juju`/`tailscale` not installed on vcloud. VR0 is tailnet-only, separate + trust domain -> check #3 deferred to an operator workstation run. +4. **creds-matrix P5, AUTHORITATIVE reading on voffice1** (clone `6aae475` == HEAD; run as + preflight does, `creds-matrix.py --tier2`): **FAIL, 121 rows, 19 clean groups, 11 + findings** (6 accepted + 5 dc1 forward-register, all by-design; detail Section 3f). The + 07-30 baseline capture was 101/19/6; the matrix grew 101->121 by design (SEC-027/028/029 + commits). A separate *vcloud* run (`--all --tier2`) additionally surfaced the off-gate + undeclared `opnsense-api-rebuild-20260807.txt` (jumphost-local; voffice1 does not probe + it). Also corrects the draft's SEC-021(a) claim (Section 3e): S2 dc0-edge-api is still + RED (host-independent matrix-vs-manifest check -- manifest regen owed). +5. **Credential residency (ls-only):** + - voffice1 `~/vr1-dc0-creds/`: `maas-virsh_ed25519`, `vr1-dc0_svc_ed25519{,.pub}`; + `~/vr1-dc1-creds/`: `vr1-dc1_svc_ed25519{,.pub}` (SEC-022 shadow stores, confirmed). + - dc0 rack `~/vr1-dc0-creds/`: `maas-juju-api-key.txt`, `maas-virsh_ed25519{,.pub}`, + `opnsense-api.txt`, `vr1-dc0-edge_ed25519`; `~/.local/share/juju/` full client store + (credentials.yaml, accounts.yaml, ...); `~/repo-stage/overlays/vr1-dc0-octavia-pki.yaml` + (+ machines, vips). This is the DC-substrate material R7 must revoke at teardown. + - dc0 region VM (.6) `/var/snap/maas/current/root/.ssh/`: `id_dc0_power`, `config` + (snap-side power key, SEC-016-class); `~/vr1-dc0-creds/` empty (region secrets were + consolidated to the jumphost/rack, not kept on .6). +6. **dc0 checkpoint state (`juju` from the dc0 rack, D-138):** controller + `vr1-dc0-controller` (3.6.27), model `vr1-dc0` = **66 machines / 104 cores / 162 units, + available** -- matches CURRENT-STATE. The end-goal "checkpoint" frame is live-accurate. + +--- + +## 8. Fable advisor review + consensus (two-pass, corrections visible) + +The review ran two adversarial advisor passes over the same items. The value of the +structure is that the corrections are RECORDED, not that the final list looks clean. + +**Pass 1 -- what the advisor caught, and what changed:** +1. **P5 instrument mismatch (the big one).** My draft said "P5 is 12 findings, not 6 -- you + now face 12," from a `creds-matrix.py --all --tier2` run **on vcloud**. The accepted-6 + baseline was measured **on voffice1** (the D-128 host whose P5 reading counts) and + without `--remote`. Re-measured on voffice1 (clone `6aae475` == HEAD): **11 findings**, + and the matrix **grew 101->121 rows BY DESIGN** (SEC-027/028/029 mint commits). The +5 + are all dc1 forward-register rows that fail S2 by design. **The framing inverted** -- + from "the gate silently grew" (which manufactures a decision, this project's named + failure mode) to "the gate expanded by 5 by-design rows." The 12th item (the undeclared + `opnsense-api-rebuild` token) is a vcloud-local, off-gate hygiene item, separated out. +2. **Missing DOCFIX material.** The two 10.13 naming-collision literals (`netbox/README.md:49`, + `tests/dc-rack-mgmt-import/test_logic.py:265`) were in the ruling-prep output but not + carried forward -> added as **R16**, tied to R1. +3. **VR0 check #3** -> sharpened into a named accept-or-run operator option in R1. + +**Pass 2 -- what the advisor caught on the corrected draft, and what changed:** +4. **A false completeness claim.** "Counts sum to the 28 open rows" actually summed to 24 + (SEC-003/015/016/024 unplaced) -- the exact "checker that cannot fail" class. Fixed: the + 28 now partition **3a=10, 3b=8, 3b'=3, 3c=3, 3d=3, 3e-watch=1**, re-derived from the + buckets. A new **bucket 3b'** was added for the key distinction the advisor drew: + **dc1-labeled credentials on persistent hosts retire at v1 close, NOT at the re-IP** -- + dc1 was never built, so R7's teardown-revocation does not reach them. +5. **A destructive-option hazard.** R15 offered "declare **or** shred" for a token that is + very likely the LIVE dc1 edge credential (2026-08-07 rebuild). Reframed to **declare; + shred only on a proven-dead live auth-test.** + +**CONSENSUS -- the agreed final position (both reviewers):** +- **The backlog is overwhelmingly correctly deferred.** 5 open D-decisions are all parked at + Roosevelt / end-of-deployment / v1-close by ruling; the 28 SEC rows are rotation + obligations and accepted postures, not defects. No open item blocks the dc0 checkpoint. +- **Exactly one item gates the hardened end goal: the re-IP GA-R5 ruling (R1).** Two of its + three owed live-free checks now PASS with fresh measurement (tailnet + apex, with working + positive controls); the third (VR0 internals) is an operator accept-or-run choice. This is + the concrete unblock. +- **Two genuine, re-IP-coupled gaps deserve building now-ish:** the teardown credential + REVOCATION checklist (**R7**, residency measured live) and the 10.13 naming-collision + DOCFIX (**R16**). +- **A short list of near-term credential traps + record hygiene** (R8 SEC-006, R10 SEC-023, + R11-R13, R15's undeclared token) -- cheap, batchable, none blocking. +- **Everything else: confirm the deferral holds.** No new D-numbers are warranted by this + review beyond the already-owed D-143 (re-IP). + +**Standing-discipline note:** every recommendation here is a candidate for a gated exchange +(GA-R5 for the re-IP ruling and any SEC-row edit; presented-and-approved for the DOCFIX and +manifest regenerations). This document rules nothing and mutates no authoritative surface; +it is the decision package for the operator. It does not reopen the "proceed to deploy" +directive -- it front-runs the re-IP that the checkpoint's own plan requires.