diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f2d67ba..1efe158 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -700,7 +700,7 @@ | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | | G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | CLOSED 2026-07-23 (operator-ruled "Merge to main + full close"; commissioning 9/9 READY, merge commit on `main`, branch retired) -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED + WIRED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). **COMMISSIONING 9/9 READY 2026-07-23** (`docs/audit/dc1-commissioning-verify-20260723.txt`): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, `power_type=virsh` set + verified by real query-power-state (SEC-016 path proven), commissioned to **ALL 9 READY in ~3.5 min** (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. **G12 [V] leg (the dc1 build) is COMPLETE.** NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, **operator-gated merge of `dc-dc-g12-dc1-substrate` -> `main`** (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close). | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | CLOSED 2026-07-23 (operator-approved full maintenance bundle, logged window ops-sec010-reassert): qga channel retrofitted via outer tofu saved-plan apply 0/1/0 exact (`docs/audit/outer-plan-20260723-office1-qga.txt`; the apply's edge bounce = the ruled "next scheduled restart"; MACs were pinned 07-22 so the in-place-update trap class was closed); edge updated 26.7 -> 26.7.1 via REST (no reboot required; os-iperf had been REFUSED on 26.7 pending exactly this update); both plugins installed=1 by firmware-info read-back, `guest-ping` -> `{"return":{}}`, agent reports both legs, egress 0% loss, outer plan re-converged ZERO DIFF (`docs/audit/outer-plan-20260723-postqga-converged.txt`). Named close capture: `docs/audit/g13-close-20260723.txt` | -| G14 | 12 OPEN SEC rows (SEC-001, -003..-008, SEC-012, -013, -014, plus SEC-015 + SEC-016 opened 2026-07-23 for dc1 credentials; SEC-010/-011 CLOSED) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012/-016 carry the same libvirt-group SCOPE hardening question; SEC-016 also a snap-refresh re-assert (queued to DC standup DoD) | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3). **COUNT RECONCILED 2026-07-25: 19 open**, measured `bash scripts/ledger-scan.sh` (SEC-001, -003..-008, -012..-023). The row's own title text ("12 OPEN SEC rows") is the 2026-07-23 figure and is SUPERSEDED by this cell -- the gate is the ledger, not the count. Since 07-23: SEC-017 (caveman supply-chain), -018/-019 (per-DC MAAS API keys), -020 (MAAS region superuser passwords), and **-021/-022/-023 opened 2026-07-25** from the D-137 credential research -- dc0 custody defects (a consolidated credential ABSENT from its recorded location + per-DC power-key divergence), UNAUDITED shadow `*-creds/` stores on voffice1 (a scope gap in `creds-audit` itself, which has no remote capability), and sprawl-glob blind spots incl. a PREDICTED Stage-5 `~/admin-openrc` exposure. All three are logged-not-actioned (hard rule 1); remediation is coupled to the unruled D-137 forks | +| G14 | 12 OPEN SEC rows (SEC-001, -003..-008, SEC-012, -013, -014, plus SEC-015 + SEC-016 opened 2026-07-23 for dc1 credentials; SEC-010/-011 CLOSED) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012/-016 carry the same libvirt-group SCOPE hardening question; SEC-016 also a snap-refresh re-assert (queued to DC standup DoD) | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3). **COUNT RECONCILED 2026-07-25: 19 open**, measured `bash scripts/ledger-scan.sh` (SEC-001, -003..-008, -012..-023). The row's own title text ("12 OPEN SEC rows") is the 2026-07-23 figure and is SUPERSEDED by this cell -- the gate is the ledger, not the count. Since 07-23: SEC-017 (caveman supply-chain), -018/-019 (per-DC MAAS API keys), -020 (MAAS region superuser passwords), and **-021/-022/-023 opened 2026-07-25** from the D-137 credential research -- dc0 custody defects (a consolidated credential ABSENT from its recorded location + per-DC power-key divergence), UNAUDITED shadow `*-creds/` stores on voffice1 (a scope gap in `creds-audit` itself, which has no remote capability), and sprawl-glob blind spots incl. a PREDICTED Stage-5 `~/admin-openrc` exposure. All three are logged-not-actioned (hard rule 1); remediation is coupled to the unruled D-137 forks. Creation-point research capture: `docs/audit/creds-creation-points-20260725.md` -- 55 MINT sites inventoried, and **12 declared secrets have NO mint command anywhere in the repo** (`ssh-keygen` returns ZERO hits repo-wide; six SSH keypairs + the OPNsense root password/hash are operator-terminal mints recorded only in a manifest comment, i.e. NOT reproducible if the jumphost is rebuilt -- a Roosevelt-transfer defect, not just hygiene). It also names three credential DIRECTORIES outside the SEC-009 `*-creds/` convention and outside `creds-audit` entirely: `~/vault-init/` (Vault 5 unseal shares + root token), `~/octavia-pki/` (8 PKI artifacts incl. CA private keys), `~/tenant-/`; plus `overlays/octavia-pki.yaml`, which lands a CA key + plaintext passphrase INSIDE the repo clone (gitignored -- and SEC-004 says the repo is still PUBLIC) | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 items 2-3 RULED 2026-07-21; item 1: plan DRAFTED + Q1/Q2-structure/Q3 ALL RULED 2026-07-23 (three amendments, utterances quoted; monthly-review lines delivered). Sole D-068 remainder: Q2 path selection at Roosevelt Vault design time -- G15 is otherwise decision-complete | | G16 | office1 edge `channels = []` state reconcile (the D-129 module-schema residual) | [R] operator rules the mechanism; then [V] the converged re-plan capture | operator + session | CLOSED 2026-07-21: RULED "State surgery (Recommended)" (GA-R5, session changelog item 16); executed per G6 precedent -- channels null -> [] injected, serial 29 -> 30, backup kept, guests untouched (office1-opnsense Id 2 running throughout); convergence = ZERO DIFF (`docs/audit/outer-plan-20260721-postG16-converged.txt`); section 5 re-recorded | diff --git a/docs/audit/creds-creation-points-20260725.md b/docs/audit/creds-creation-points-20260725.md new file mode 100644 index 0000000..78d7a3f --- /dev/null +++ b/docs/audit/creds-creation-points-20260725.md @@ -0,0 +1,115 @@ +# Credential creation-point inventory -- 2026-07-25 + +Read-only research capture for the D-137 credential-matrix design. No credential file +was opened; every row below is a file:line reference to CODE or RUNBOOK PROSE that +mints, materializes, or uploads credential material. + +Classification used: **MINT** (new secret material comes into existence), +**MATERIALIZE** (an existing secret is fetched/copied/written somewhere new), +**UPLOAD** (public key registered -- no secret created). Only MINT rows are creation +points; the other two classes were checked and excluded deliberately. + +Operator-ruled scope for the matrix (this session): the **materialization test** -- +in-matrix if a credential is ever written to a file, presented by a human, or consumed +out-of-band, regardless of who generated it. + +## Counts + +| Class | Count | Where | +|---|---|---| +| Scripted MINT (repo code generates the secret) | 25 rows | `scripts/` | +| Runbook/prose MINT (operator runs by hand, no script owns it) | 30 rows | `runbooks/`, `clientdocs/` | +| **Declared secrets with NO creation point anywhere in the repo** | **12** | see below | +| Charm/platform-minted at bundle apply | summarised | `bundle.yaml` deploy | +| MATERIALIZE / UPLOAD (excluded, verified non-creating) | 20+ | see source agent record | + +## FINDING 1 -- twelve declared secrets have NO mint command anywhere + +This is the most consequential result and it **qualifies the "creation points are already +known" premise**: for twelve standing credentials, they are not. + +`grep -rnI "ssh-keygen" .` returns **ZERO hits repo-wide** (searched everything except +`.git`/`.terraform`/tfstate/tfplan, including `tests/`, `docs/archive/`, `clientdocs/`, +`.claude/`). There is likewise no `mkpasswd`, `htpasswd`, `password_hash`, `pwgen`, +`uuidgen`, nor any Terraform `random_password` / `tls_private_key`. Yet the manifests +declare: + +| Credential | Declared at | Only provenance record | +|---|---|---| +| `office1_svc_ed25519{,.pub}` | `vr1-office1.manifest:30-31` | `vr1-office1-as-built.md:126` (SEC-007) | +| `vr1-dc0_svc_ed25519{,.pub}` | `vr1-dc0.manifest:16-17` | a manifest COMMENT (`:10-12`) | +| `vr1-dc1_svc_ed25519{,.pub}` | `vr1-dc1.manifest:16-17` | a manifest COMMENT (`:10-12`) | +| `vr1-dc0-edge_ed25519{,.pub}` | `vr1-dc0.manifest:22-23` | manifest comment (backfilled 2026-07-25) | +| `vr1-dc1-edge_ed25519{,.pub}` | `vr1-dc1.manifest:21-22` | manifest comment (`:18-20`) | +| `vr1-dc1-maas-power_ed25519{,.pub}` | `vr1-dc1.manifest:29-30` | manifest comment (`:26-28`, SEC-016) | +| `opnsense-root-password` | `vr1-office1.manifest:34` | `vr1-office1-as-built.md:128`, SEC-007 -- **HUMAN GUI login** | +| `opnsense-root-hash` | `vr1-office1.manifest:33` | same (bcrypt of the above, for `config.xml`) | +| `tailscale-authkey.txt` | `vr1-office1.manifest:36` | SEC-008 -- operator-supplied, third-party control server (correctly has no repo mint) | +| `vr1-netbox.env` | `vr1-office1.manifest:37` | minted on the upstream apex; SEC-006 (BURNED) | +| `vr1-office1.env` / `vr1-stage1.env` | `.manifest:39-40` | `vr1-office1-as-built.md:132-133` | + +Six SSH keypairs and the OPNsense root password/hash are pure operator-terminal mints +whose only record is a manifest comment. **Consequence: they are not reproducible.** If +the jumphost is rebuilt, no repo artifact states how to recreate them -- which is a +Roosevelt-transfer defect, not merely a hygiene one. The tailscale key is genuinely +third-party and correctly has no repo mint. + +## FINDING 2 -- credential directories OUTSIDE the `*-creds/` convention + +SEC-009 says all site secrets live in `~/-creds/`. The inventory shows at least +three other jumphost directories holding credential material, none of which +`creds-audit` covers (it checks `~/-creds/` folders plus a loose-FILE sprawl glob +in `$HOME` -- it never inspects these DIRECTORIES): + +- `~/vault-init/init.txt` -- Vault **5 unseal shares + root token**, the cloud's entire + root of trust (`runbooks/phase-02-vault-bringup.md:71`, irreversible one-shot). +- `~/octavia-pki/` -- **8 artifacts**: 2 CA passphrases, 2 CA private keys, 2 CA certs, + the controller key, the signed bundle (`runbooks/phase-01-bundle-deploy.md:299-410`). +- `~/tenant-/` -- per-tenant domain-admin / cluster / svc passwords, an + application credential, and a Nova private key (`scripts/tenant-onboard.sh:70-71, + 94-95, 141, 167`). + +All three are named in CLAUDE.md as secret locations, so they are KNOWN -- they are +simply outside the control. This materially extends SEC-023. + +Related: `runbooks/phase-01-bundle-deploy.md:391-410` writes +`overlays/octavia-pki.yaml` containing base64 CA key blobs **plus the issuing-CA +passphrase in plaintext, inside the repo clone** (gitignored). Given SEC-004 (repo +currently PUBLIC), that gitignore is the only thing standing between a CA private key +and publication. + +## FINDING 3 -- highest-concentration mint sites (matrix seeding order) + +1. `scripts/site-headend-install.sh` -- **7 mints** (`:418` primitive, `:430` DB pass, + `:445` rack-enrollment secret, `:450`/`:452` admin password, `:453` API key, `:485` + LXD trust). Every Office1/DC-region MAAS + LXD secret originates here. +2. `scripts/tenant-onboard.sh` -- 7 mints, the entire tenant identity set. +3. `runbooks/phase-01-bundle-deploy.md:299-410` -- 8 Octavia PKI artifacts. +4. `runbooks/tenant-onboarding-v2-DRAFT.md:162-320` -- 6 mints DUPLICATING #2 (two + creation locations, one credential set -- do not double-count). +5. `runbooks/phase-02-vault-bringup.md:71-144` -- 4 mints incl. the root of trust. +6. `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:119-151` -- 4 mints PER DC. + +## FINDING 4 -- corroborations of already-recorded defects + +- `dc-dc-phase4-juju-bundle-per-dc.md:128` mints `juju-` via `maas createadmin` + with **no** `--password`, so MAAS prompts and the value is stored nowhere -- + independently confirming the SEC-020 finding about those two superusers. +- `site-headend-install.sh:452` uses `--password` in **argv**, the exposure SEC-020 + flagged and which this session's `operator` mint avoided via stdin. +- `scripts/phase-03-admin-openrc.sh:46-52` is classified MATERIALIZE (charm-generated, + operator-materialized) -- exactly the case the ruled materialization test captures and + a generation-based test would have missed (SEC-023). + +## What this means for the matrix + +The matrix must carry a `mint-ref` column that can hold **three** provenance kinds, not +one: `script:line`, `runbook:step`, and `operator-terminal` (undocumented). Finding 1 +means a meaningful fraction of today's estate is the third kind, so a checker that +requires every row to name executable code would fail 12 rows on day one. Those rows +are the backlog the matrix exists to surface -- they should be admitted as +`operator-terminal` and tracked as a debt to convert, not treated as parse errors. + +Source: three read-only Explore agents, 2026-07-25. Full per-row tables (55 MINT rows +with file:line, host, destination, stage, and human/service classification) are in the +session transcript; the structural findings and all counts are reproduced above.