diff --git a/creds-manifests/vr1-dc0.manifest b/creds-manifests/vr1-dc0.manifest index edc3762..63f7ad5 100644 --- a/creds-manifests/vr1-dc0.manifest +++ b/creds-manifests/vr1-dc0.manifest @@ -27,3 +27,24 @@ # A runtime copy also lives in Juju's client credential store on voffice1 # (off-manifest -- that store is not this folder). Declared 2026-07-25. maas-api-key.txt 600 local +# Dedicated MAAS->libvirt power key (SEC-012, per-DC isolation; minted 2026-07-20 +# at the step-D power wiring). CONSOLIDATED 2026-07-27 -- the SEC-021(b) remediation: +# "dc0's key should gain a jumphost copy + manifest row". PROVENANCE, stated here +# because the machine field cannot carry it: dc0's key had NO jumphost copy at all, +# so this pair was NOT generated on vcloud. The private half was copied from the +# voffice1 shadow store (~/vr1-dc0-creds/maas-virsh_ed25519, the SEC-022 class), +# sha256-verified byte-identical at both ends; the public half NEVER EXISTED +# anywhere and was DERIVED here with `ssh-keygen -y`, its fingerprint verified +# equal to the key installed in the region MAAS snap (SHA256:Dt/YXTXSF4nXGj9cz8f0 +# owL+qreVpMYBVm/igW10FXY). Provenance reads "local" (held on the jumphost) to stay +# consistent with `creds-matrix.py --render`; the sentences above are the record. +# NAMED for its purpose, per SEC-021(b)'s "reconciliation to the dc1 shape". The +# key's own INTERNAL comment still reads "maas-virsh-pod (D-123 amendment)" -- it +# was minted for the qemu+ssh POD attempt that was REFUTED the same day, and the +# per-machine `power_type=virsh` ruling correctly retained the dedicated key. The +# stale comment is cosmetic and was NOT rewritten (that would alter key material); +# the file name is now authoritative. This copy is also what SEC-012/-016's snap +# fragility needs: /var/snap/maas/current/ is per-revision, and dc0 previously had +# no jumphost copy to re-assert from after a snap refresh. +vr1-dc0-maas-power_ed25519 600 local +vr1-dc0-maas-power_ed25519.pub 644 local diff --git a/creds-manifests/vr1-office1.manifest b/creds-manifests/vr1-office1.manifest index 0beaaf5..4e6447d 100644 --- a/creds-manifests/vr1-office1.manifest +++ b/creds-manifests/vr1-office1.manifest @@ -27,6 +27,16 @@ # db.pass + lxd-trust.pass are region infrastructure, out of scope of this task. maas-admin-password 600 voffice1:/root/maas-secrets/admin.pass maas-operator-password 600 local +# NetBox web-GUI `admin` password (SEC-025). CONSOLIDATED 2026-07-27: it was minted +# on the NetBox VM at stage 2 and had NEVER been pulled to the jumphost -- the SAME +# miss class as the api.token below (the founding SEC-009 defect) and as SEC-020's +# admin.pass, i.e. a VM-minted HUMAN GUI login with no forcing function to reach the +# folder. Found by the 2026-07-27 consolidation audit, which asked which accounts +# actually exist rather than which the register already knew about. Byte-identical +# copy, sha256-verified at both ends, 28 bytes; the VM copy REMAINS source-of-record, +# so any future rotation must update BOTH copies or this one becomes a stale trap +# (the caveat SEC-020 records for maas-admin-password applies identically here). +netbox-admin-password 600 office1-netbox:/root/netbox-secrets/admin.pass office1_svc_ed25519 600 local office1_svc_ed25519.pub 644 local opnsense-api.txt 600 local diff --git a/creds-matrix-notes.md b/creds-matrix-notes.md index e2caa96..5f58ac8 100644 --- a/creds-matrix-notes.md +++ b/creds-matrix-notes.md @@ -86,6 +86,18 @@ off netbox-docker's PUBLIC default. The admin password is a human GUI login and is a separate identity from the service SECRET_KEY -- do not merge these ids. +`netbox-admin` CONSOLIDATED 2026-07-27 (SEC-025): the VM copy is now `source-of-record` and +`~/vr1-office1-creds/netbox-admin-password` is a `verbatim-copy`, so tier-3 V1 asserts the two +stay byte-identical. It had never left the VM it was minted on -- the THIRD instance of the +miss class that SEC-009 (`api.token`) founded and SEC-020 (`admin.pass`) repeated: a +VM-minted HUMAN login with no forcing function to reach the operator's folder. It was +invisible to every control because `off-manifest-known` asserts nothing about the jumphost, +so no check could report it missing. Found only by ENUMERATING the accounts that actually +exist (`maas admin users read` for MAAS, this row's account by inspection) rather than +auditing the ones the register already knew about -- the register cannot report an absence +it has no row for. `netbox-secret-key` is deliberately NOT consolidated: it is a service +secret consumed only by the container, with no human who needs to read it. + ## n-tofu-apikey Office1 MAAS API key consumed by OpenTofu as `TF_VAR_maas_api_key`. diff --git a/creds-matrix.tsv b/creds-matrix.tsv index b4a0100..dec3ea1 100644 --- a/creds-matrix.tsv +++ b/creds-matrix.tsv @@ -59,7 +59,8 @@ netbox-sandbox-token singleton vr1-office1 netbox api.token api service source-of-record stage2 runbook:runbooks/dc-dc-phase1-office1-standup.md:643 SEC-009 n-netbox-sandbox netbox-sandbox-token singleton vr1-office1 jumphost vr1-netbox-sandbox.env api service consolidated stage2 runbook:runbooks/dc-dc-phase1-office1-standup.md:643 SEC-009 n-netbox-sandbox netbox-secret-key singleton vr1-office1 netbox secret_key none service off-manifest-known stage2 operator-terminal - n-netbox-vm-secrets -netbox-admin singleton vr1-office1 netbox admin.pass gui human off-manifest-known stage2 operator-terminal - n-netbox-vm-secrets +netbox-admin singleton vr1-office1 netbox admin.pass gui human source-of-record stage2 operator-terminal SEC-025 n-netbox-vm-secrets +netbox-admin singleton vr1-office1 jumphost netbox-admin-password gui human verbatim-copy stage2 operator-terminal SEC-025 n-netbox-vm-secrets office1-tofu-maas-apikey singleton vr1-office1 jumphost vr1-stage1.env api service consolidated stage2 runbook:runbooks/dc-dc-phase1-office1-standup.md:464 SEC-009 n-tofu-apikey office1-tofu-maas-apikey singleton vr1-office1 jumphost terraform.tfstate api service off-manifest-known stage2 runbook:runbooks/dc-dc-phase1-office1-standup.md:464 - n-tfstate-plaintext office1-env singleton vr1-office1 jumphost vr1-office1.env none service consolidated stage2 operator-terminal SEC-009 n-office1-env diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 6b88533..fcb64f1 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -700,7 +700,7 @@ | G11 | Operator signs THIS document | [R] read top-to-bottom; discrepancies resolved in the document | operator | CLOSED: RE-SIGNED 2026-07-19 at audit exit, section 11 (replaces the 2026-07-18 signature) | | G12 | `vr1-dc1` build | [R] operator rules dc1 transit/rack addressing; then vars + substrate authored | operator + session | CLOSED 2026-07-23 (operator-ruled "Merge to main + full close"; commissioning 9/9 READY, merge commit on `main`, branch retired) -- [R] leg CLOSED 2026-07-21: addressing RATIFIED (D-124 amendment 2026-07-21, utterance quoted). [V] leg IN PROGRESS (branch `dc-dc-g12-dc1-substrate`): apex confirm-free DONE 2026-07-21 -- planes/uplink already assigned+consistent, transit 172.31.0.4/30 + rack 10.12.68.2 FREE (`docs/audit/dc1-apex-confirm-20260721.txt`); importer per-site dc1 support shipped (harness 117/117) with live dry-run preflight PASS (`docs/audit/dc1-rack-import-dryrun-20260721.txt`). vars + substrate root + lib-net dc1 arm COMMITTED 2026-07-22 (successor session landed the disconnected item 3 + the harness reconcile as changelog item 4): six harnesses reconciled to the ratified dc1 arm, phase-00 PLANES parity guard added, rbd-mirror/radosgw cross-DC reminder fixed; gauntlet **ALL GREEN (76)** (`docs/audit/gauntlet-20260722-g12-reconcile.txt`), repo-lint 0-fail. Apex `--commit` EXECUTED 2026-07-22 (operator-gated): 172.31.0.4/30 + 10.12.68.2/22 CREATED, post-commit read-back idempotent (`docs/audit/dc1-rack-import-commit-20260722.txt`). dc1 svc key minted (creds-audit CLEAN), tfvars authored (local), **outer step-A apply DONE 2026-07-22**: saved plan 5/0/0 exact, converged ZERO DIFF (section 5), vvr1-dc1 RUNNING, prior guests untouched (as-executed log dc1-deploy; changelog-20260722-g12-dc1-build.md). **Step B COMPLETE 2026-07-22**: cloudinit-vm interface_macs port + voffice1 dc1-transit NIC (0/2/0 exact, MACs pinned both domains, post-bounce battery ALL PASS, converged zero diff -- `docs/audit/outer-plan-20260722-voffice1-dc1nic.txt`), transit LIVE (voffice1 .5/30 <-> rack .6/30, dc1-key ssh proven), rack ENROLLED (region lists vvr1-dc1 `nmpcq4`), SEC-010 applied+verified BOTH ends, OPNsense 26.7 base staged via hash-verified copy of dc0's proven artifact; named gate EXIT 0 `docs/audit/dc1-stepB-check-20260722-final.txt` (changelog-20260722 items 5-8, three queued findings). **Step C COMPLETE 2026-07-22**: inner apply FROM voffice1 -- plan 28/0/0 exact (54 pinned MACs verified in-capture), one fix-forward (serial-log staging dir absent on dc1; queued to standup DoD), resume 10/0/0 exit 0; 28/28 in state, convergence ZERO DIFF (`docs/audit/inner-converge-20260722-dc1-stepC.txt`), **10/10 domains RUNNING inside vvr1-dc1**, edge at the 26.7 FreeBSD login prompt (D-112 datapoint #2); dc1 inner tfstate ON voffice1 (site backup set). **D-125 egress gate PASS 2026-07-22** (two identical runs, dc0 criteria exact, isolation confirmed -- `docs/audit/d125-egress-gate-20260722-dc1.txt`). **Edge bootstrap + v4 addressing COMPLETE 2026-07-23** (changelog-20260723-g12-dc1-edge.md): D-112(c) console bootstrap done (SSH + dc1 edge key materialized; payload needed `util.inc`/`shell_safe()` -- dc0 lesson iv the `.b64` artifact lacked), key-only SSH VERIFIED (`15.1-RELEASE-p1`); D-113(a2) API key MINTED via the vendor model + smoke test `GET core/firmware/status` exit 0 `product_abi 26.7` (second 26.7 datapoint); edge ADDRESSED -- WAN `172.30.3.2/24` gw `172.30.3.1` (egress 1.1.1.1 0% loss), LAN `192.168.1.1` -> `10.12.64.1/22` (ruled provider-public gw), API answers at the new LAN; interim reach leg removed, rack provider-public leg `10.12.64.2/22` LIVE on virbr4. Creds consolidated to `~/vr1-dc1-creds/opnsense-api.txt` (creds-audit CLEAN, 5 entries); rack edge-key copy shredded (**SEC-015** transient, remediated). Two queued findings: bootstrap `.b64` missing `util.inc`; `opnsense-bootstrap-apikey.sh` scp had a transient post-restart-sshd failure (readiness-wait/retry candidate). **Rack standup + region MAAS config DONE 2026-07-23** (changelog-20260723 items 7-11): dc-rack-net.sh dc1 arm shipped (harness 18/18, gauntlet 76 GREEN) + INSTALLED on the rack (check 10/10, forwarder answers authoritative maas-internal SOA -- D-131 fix; `docs/audit/dc1-rack-net-install-20260723.txt`); region MAAS on metal-admin subnet 11 -- D-120 range 10.12.68.100-.200, D-131 dns_servers=10.12.68.3 allow_dns=false, DHCP dhcp_on=true primary_rack=nmpcq4 (dhcpd verified RUNNING on virbr6, no Temporal incident); **dc1 enlistment PROVEN** via canary (machines 11->12 in ~2 min). **SEC-016 RULED + WIRED 2026-07-23** (operator: "Mint a dedicated dc1 power key" -- per-DC isolation; dedicated key authorized on the rack + installed in the region MAAS snap with per-host ssh config, dc0's SEC-012 key untouched). **COMMISSIONING 9/9 READY 2026-07-23** (`docs/audit/dc1-commissioning-verify-20260723.txt`): all 9 nodes PXE-enlisted by pinned 52:54:01:d1 MACs, `power_type=virsh` set + verified by real query-power-state (SEC-016 path proven), commissioned to **ALL 9 READY in ~3.5 min** (no timeout, no SERVFAIL), shapes EXACT to D-121 Option C (3x16cpu/64GiB + 2x12cpu/48GiB + 4x8cpu/24GiB). dc0's two stacked faults pre-empted by pinned MACs + the dc-rack-net forwarder. **G12 [V] leg (the dc1 build) is COMPLETE.** NEXT: G12 close-out only -- consolidate this session's changelogs (GA-R2), final gauntlet + repo-lint, GA-R7 memory review, skill sweep, **operator-gated merge of `dc-dc-g12-dc1-substrate` -> `main`** (merge commit), branch retirement; then G12 CLOSES. NOTE open SEC rows now include SEC-014/-015/-016 (G14 row count stale -- reconcile in the close). | | G13 | D-129 residuals | [R] operator-gated live plugin install on office1-opnsense; qga channel retrofit at that edge's next scheduled restart. All 4 sub-decisions RULED 2026-07-21 (D-129 Status line) -- only the two execution items remain | operator | CLOSED 2026-07-23 (operator-approved full maintenance bundle, logged window ops-sec010-reassert): qga channel retrofitted via outer tofu saved-plan apply 0/1/0 exact (`docs/audit/outer-plan-20260723-office1-qga.txt`; the apply's edge bounce = the ruled "next scheduled restart"; MACs were pinned 07-22 so the in-place-update trap class was closed); edge updated 26.7 -> 26.7.1 via REST (no reboot required; os-iperf had been REFUSED on 26.7 pending exactly this update); both plugins installed=1 by firmware-info read-back, `guest-ping` -> `{"return":{}}`, agent reports both legs, egress 0% loss, outer plan re-converged ZERO DIFF (`docs/audit/outer-plan-20260723-postqga-converged.txt`). Named close capture: `docs/audit/g13-close-20260723.txt` | -| G14 | 12 OPEN SEC rows (SEC-001, -003..-008, SEC-012, -013, -014, plus SEC-015 + SEC-016 opened 2026-07-23 for dc1 credentials; SEC-010/-011 CLOSED) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012/-016 carry the same libvirt-group SCOPE hardening question; SEC-016 also a snap-refresh re-assert (queued to DC standup DoD) | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3). **COUNT RECONCILED 2026-07-25: 19 open**, measured `bash scripts/ledger-scan.sh` (SEC-001, -003..-008, -012..-023). The row's own title text ("12 OPEN SEC rows") is the 2026-07-23 figure and is SUPERSEDED by this cell -- the gate is the ledger, not the count. Since 07-23: SEC-017 (caveman supply-chain), -018/-019 (per-DC MAAS API keys), -020 (MAAS region superuser passwords), and **-021/-022/-023 opened 2026-07-25** from the D-137 credential research -- dc0 custody defects (a consolidated credential ABSENT from its recorded location + per-DC power-key divergence), UNAUDITED shadow `*-creds/` stores on voffice1 (a scope gap in `creds-audit` itself, which has no remote capability), and sprawl-glob blind spots incl. a PREDICTED Stage-5 `~/admin-openrc` exposure. All three are logged-not-actioned (hard rule 1); remediation is coupled to the unruled D-137 forks. Creation-point research capture: `docs/audit/creds-creation-points-20260725.md` -- 55 MINT sites inventoried, and **12 declared secrets have NO mint command anywhere in the repo** (`ssh-keygen` returns ZERO hits repo-wide; six SSH keypairs + the OPNsense root password/hash are operator-terminal mints recorded only in a manifest comment, i.e. NOT reproducible if the jumphost is rebuilt -- a Roosevelt-transfer defect, not just hygiene). It also names three credential DIRECTORIES outside the SEC-009 `*-creds/` convention and outside `creds-audit` entirely: `~/vault-init/` (Vault 5 unseal shares + root token), `~/octavia-pki/` (8 PKI artifacts incl. CA private keys), `~/tenant-/`; plus `overlays/octavia-pki.yaml`, which lands a CA key + plaintext passphrase INSIDE the repo clone (gitignored -- and SEC-004 says the repo is still PUBLIC) | +| G14 | 12 OPEN SEC rows (SEC-001, -003..-008, SEC-012, -013, -014, plus SEC-015 + SEC-016 opened 2026-07-23 for dc1 credentials; SEC-010/-011 CLOSED) | [R] per-row: rotations/flips at v1 close (external to VR1 track); SEC-012/-016 carry the same libvirt-group SCOPE hardening question; SEC-016 also a snap-refresh re-assert (queued to DC standup DoD) | operator / external | `docs/security-ledger.md` (register of record, GA-R4/F3). **COUNT RECONCILED 2026-07-27: 21 open**, measured `bash scripts/ledger-scan.sh` (SEC-001, -003..-008, -012..-025; SEC-024 opened 2026-07-26, SEC-025 opened 2026-07-27 for the consolidated NetBox GUI admin password). Earlier figures in this cell (19 at 2026-07-25) are history. The row's own title text ("12 OPEN SEC rows") is the 2026-07-23 figure and is SUPERSEDED by this cell -- the gate is the ledger, not the count. Since 07-23: SEC-017 (caveman supply-chain), -018/-019 (per-DC MAAS API keys), -020 (MAAS region superuser passwords), and **-021/-022/-023 opened 2026-07-25** from the D-137 credential research -- dc0 custody defects (a consolidated credential ABSENT from its recorded location + per-DC power-key divergence), UNAUDITED shadow `*-creds/` stores on voffice1 (a scope gap in `creds-audit` itself, which has no remote capability), and sprawl-glob blind spots incl. a PREDICTED Stage-5 `~/admin-openrc` exposure. All three are logged-not-actioned (hard rule 1); remediation is coupled to the unruled D-137 forks. Creation-point research capture: `docs/audit/creds-creation-points-20260725.md` -- 55 MINT sites inventoried, and **12 declared secrets have NO mint command anywhere in the repo** (`ssh-keygen` returns ZERO hits repo-wide; six SSH keypairs + the OPNsense root password/hash are operator-terminal mints recorded only in a manifest comment, i.e. NOT reproducible if the jumphost is rebuilt -- a Roosevelt-transfer defect, not just hygiene). It also names three credential DIRECTORIES outside the SEC-009 `*-creds/` convention and outside `creds-audit` entirely: `~/vault-init/` (Vault 5 unseal shares + root token), `~/octavia-pki/` (8 PKI artifacts incl. CA private keys), `~/tenant-/`; plus `overlays/octavia-pki.yaml`, which lands a CA key + plaintext passphrase INSIDE the repo clone (gitignored -- and SEC-004 says the repo is still PUBLIC) | | G15 | D-068 / D-071 rulings | [R] operator rules (section 8); neither blocks the VR1 substrate | operator | D-071 ADOPTED 2026-07-21 (all four points); D-068 items 2-3 RULED 2026-07-21; item 1: plan DRAFTED + Q1/Q2-structure/Q3 ALL RULED 2026-07-23 (three amendments, utterances quoted; monthly-review lines delivered). Sole D-068 remainder: Q2 path selection at Roosevelt Vault design time -- G15 is otherwise decision-complete | | G16 | office1 edge `channels = []` state reconcile (the D-129 module-schema residual) | [R] operator rules the mechanism; then [V] the converged re-plan capture | operator + session | CLOSED 2026-07-21: RULED "State surgery (Recommended)" (GA-R5, session changelog item 16); executed per G6 precedent -- channels null -> [] injected, serial 29 -> 30, backup kept, guests untouched (office1-opnsense Id 2 running throughout); convergence = ZERO DIFF (`docs/audit/outer-plan-20260721-postG16-converged.txt`); section 5 re-recorded | @@ -984,6 +984,42 @@ (A dangling fragment here, left by an earlier edit in this same session, was repaired at session close -- noted rather than silently fixed, since CURRENT-STATE is the status authority and its defects are worth seeing.) + **CONSOLIDATION BATCH EXECUTED 2026-07-27** (operator question: is there a consolidated set + of login creds on vcloud for every account that exists; operator direction after the audit: + "clear the whole consolidation batch first"). Capture + `docs/audit/creds-consolidation-audit-20260727.txt`; detail + `docs/changelog-20260727-creds-consolidation.md`. Superseding figures: matrix **82 rows**, + creds-matrix harness **60/60** (was 56; V2 had shipped with ZERO cases), creds-audit + **CLEAN on all three sites**, gauntlet **ALL GREEN (81)**, repo-lint 0-fail, findings + **13 -> 7**. **VERIFIED POSITIVE, both previously only asserted:** the MAAS account set is + COMPLETE -- all 6 accounts enumerated live (`maas admin users read`) are accounted for + (`admin` + `operator` passwords on vcloud, `juju-vr1-dc0/dc1` random+unstored BY RULING + with their API keys present, `MAAS`/`maas-init-node` MAAS-internal) -- and tier-3 V1 now + MEASURES `maas-admin-password` byte-identical to the headend source-of-record, so the + stale-trap risk SEC-020 records is clear as of this date. DONE: dc0's SEC-012 power key + consolidated to vcloud + `.pub` DERIVED (SEC-021(b) as written; measured first -- + the headend `maas-virsh_ed25519` and the snap's `id_ed25519` are the SAME key, it IS + dedicated (distinct from the dc0 service key), and dc0 using the snap default identity is + SEC-016's ruled design, so NO re-mint and no live power path touched); dc1 svc `.pub` + backfilled to the headend; NetBox web-GUI `admin` password consolidated (**SEC-025 OPENED** + for the at-rest exposure the copy creates -- open rows **20 -> 21**); V2 taught the + ruled-deferral state so SEC-006's standing "revoke at completion of this deployment" ruling + is ACKNOWLEDGED (still naming the credential as live and exposed) instead of failing every + run -- reissuing it would have CONTRAVENED that ruling. Tier 3 is not in preflight P5, so + the blocking gate's behaviour is unchanged. **RESIDUAL 7 findings, expected, NOT green:** + dc0-edge-api x2 (the `opnsense-api.txt` re-mint is a live edge mutation, deliberately + EXCLUDED from the batch -- sole remaining SEC-021(a) item), S5 x3 (RULED by SEC-016, the + register needs a ruled-exception mechanism -- operator decision), S6 conflation x1 (the + SEC-020 defect), E4 uncheckable x2 (Stage-5/6 rows). **NEW FINDINGS LOGGED NOT ACTIONED:** + (i) NO registered root/console credential at EITHER DC edge -- measured absence of row, + manifest entry and SEC row; what those passwords ARE is UNKNOWN and deliberately unprobed + (hard rule 2), vector is the LAN-reachable GUI + serial console, not SSH (key-only, proven); + (ii) two structural blind spots that let (i) hide -- S5 compares only `cardinality=per-DC` + while all six `per-site` rows are office1-only, and `vm-secret-locations` declares no + `rack`/`edge`/`cloud`/`unit`/`client` location though the checker accepts them (SEC-015 was + rack-resident, so the class is real). The lesson generalises D-137's founding argument one + level up: absence of a ROW is invisible to the register, so enumerating what EXISTS is a + distinct control from auditing what is declared. **QUEUED FINDINGS CAPTURED 2026-07-26 at session close: `docs/audit/queued-findings-20260726.txt`** -- an end-of-session sweep for content that existed ONLY in the session transcript. Part A: the three secrets-storage items NOT diff --git a/docs/audit/creds-consolidation-audit-20260727.txt b/docs/audit/creds-consolidation-audit-20260727.txt new file mode 100644 index 0000000..647fe51 --- /dev/null +++ b/docs/audit/creds-consolidation-audit-20260727.txt @@ -0,0 +1,147 @@ +=== creds consolidation audit -- 2026-07-27 (read-only; metadata only, no credential value read) === +repo HEAD at audit: 4d62f66709366b17f21d36d53c976145e03a9470 branch: dc-dc-stage4-phase3-maas-deploy +question (operator, 2026-07-27): confirm a consolidated set of login creds on vcloud for all +accounts currently created; identify creds missing, needing creation, or not moved into the +vcloud folders. Direction after the audit: 'clear the whole consolidation batch first'. +NOTE: sections below are the POST-REMEDIATION state. The pre-remediation run returned 13 +findings; see docs/changelog-20260727-creds-consolidation.md for the per-item delta. + +--- 1. ACCOUNTS THAT ACTUALLY EXIST on the MAAS region (ssh voffice1 'maas admin users read') --- + Enumerated from the live surface, NOT from the register -- the register can only answer + for accounts it has rows for, which is the blind spot this section exists to close. + count=6 + MAAS superuser=True email=maas@localhost + admin superuser=True email=admin@office1.local + juju-vr1-dc0 superuser=True email=juju-vr1-dc0@vr1.local + juju-vr1-dc1 superuser=True email=juju-vr1-dc1@vr1.local + maas-init-node superuser=False email=node-init-user@localhost + operator superuser=True email=jesse.austin@neumatrix.com + +--- 2. creds-audit.sh (SEC-009 declaration-based audit), all three sites --- +=== creds-audit: vr1-office1 (/home/jessea123/vr1-office1-creds) === + [ok] folder mode 0700 + [ok] all 14 manifest entries present, correct mode, non-empty + +creds-audit vr1-office1: CLEAN +=== creds-audit: vr1-dc0 (/home/jessea123/vr1-dc0-creds) === + [ok] folder mode 0700 + [ok] all 7 manifest entries present, correct mode, non-empty + +creds-audit vr1-dc0: CLEAN +=== creds-audit: vr1-dc1 (/home/jessea123/vr1-dc1-creds) === + [ok] folder mode 0700 + [ok] all 8 manifest entries present, correct mode, non-empty + +creds-audit vr1-dc1: CLEAN + +--- 3. jumphost (vcloud) creds folder contents, filenames only --- +$HOME/vr1-office1-creds/: + maas-admin-password + maas-operator-password + netbox-admin-password + office1_svc_ed25519 + office1_svc_ed25519.pub + opnsense-api.txt + opnsense-root-hash + opnsense-root-password + README + tailscale-authkey.txt + vr1-netbox.env + vr1-netbox-sandbox.env + vr1-office1.env + vr1-stage1.env +$HOME/vr1-dc0-creds/: + maas-api-key.txt + vr1-dc0-edge_ed25519 + vr1-dc0-edge_ed25519.pub + vr1-dc0-maas-power_ed25519 + vr1-dc0-maas-power_ed25519.pub + vr1-dc0_svc_ed25519 + vr1-dc0_svc_ed25519.pub +$HOME/vr1-dc1-creds/: + maas-api-key.txt + opnsense-api.txt + vr1-dc1-edge_ed25519 + vr1-dc1-edge_ed25519.pub + vr1-dc1-maas-power_ed25519 + vr1-dc1-maas-power_ed25519.pub + vr1-dc1_svc_ed25519 + vr1-dc1_svc_ed25519.pub + +--- 4. headend (voffice1) shadow stores, filenames only (SEC-022) --- +voffice1:~/vr1-dc0-creds/: + maas-virsh_ed25519 + vr1-dc0_svc_ed25519 + vr1-dc0_svc_ed25519.pub +voffice1:~/vr1-dc1-creds/: + vr1-dc1_svc_ed25519 + vr1-dc1_svc_ed25519.pub + +--- 5. creds-matrix.py --tier2 --tier3 --remote --privileged (D-137 register, ALL tiers) --- +=== creds-matrix: tier 1 (STATIC) === +=== creds-matrix: tier 2 (EXISTENCE) === +=== creds-matrix: tier 3 (VALIDITY) === + [ok] S1 schema: 82 rows, all enums valid, site-keys region-qualified, no duplicate (id,site,host-role,filename) + [ok] S3 render: 3 source field(s) SKIPPED -- rendering them needs the declared path from creds-manifests/vm-secret-locations (ruling 3); the list now EXISTS but the source-field derivation is not wired + [ok] S3 render drift: rendered row fields (mode, source) match checked-in; header prose and non-jumphost rows are OUT OF SCOPE of this compare + [ok] S4 mint-ref: every script:/runbook: reference resolves to a real location + [ok] S4 provenance debt: 30 row(s) are mint-ref=operator-terminal -- NOT reproducible from the repo (research FINDING 1). Admitted by design; converting them is remediation, not a checker fix. + [ok] S7 notes: 34 note key(s) referenced, all resolve, none orphaned + [ok] E0 jumphost location '~/vault-init/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/issuing-ca/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/controller-ca/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 jumphost location '~/octavia-pki/controller/*' does not exist -- SKIPPED (typically a mint stage this deployment has not reached) + [ok] E0 headend location '/root/maas-secrets/*' read via ESCALATION (sudo -n, metadata only) after an unprivileged probe could not open it + [ok] E0 headend location '/var/snap/maas/current/root/.ssh/*' read via ESCALATION (sudo -n, metadata only) after an unprivileged probe could not open it + [ok] E0 netbox location '/root/netbox-secrets/*' read via ESCALATION (sudo -n, metadata only) after an unprivileged probe could not open it + [ok] E1 18 expected artifact(s) deferred as not-yet-minted per creds-manifests/stages-reached + [ok] V2 ACKNOWLEDGED DEFERRAL: id 'netbox-upstream-token' (vr1-netbox.env) is custody=consolidated and SEC-006 declares it BURNED, but remediation is DEFERRED BY OPERATOR RULING -- the credential is still live and still exposed until that ruling is discharged. Ruling text: 'DEFERRED by operator ruling' + [ok] V2 declared state: no credential is registered healthy while its SEC row declares it burned/revoked/compromised + [ok] V1 provenance: all copies byte-identical for 6 group(s): dc0-svc-key; dc0-svc-key; dc1-svc-key; dc1-svc-key; maas-region-admin (declared verbatim copies); netbox-admin (declared verbatim copies) + [FAIL] S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt' (id dc0-edge-api, SEC-021) is expected by the matrix but NOT declared in the manifest -- the credential is either missing or undeclared + [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=id_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S5 ASYMMETRY: vr1-dc0 declares id=dcN-maas-power-key file=maas-virsh_ed25519 on headend (custody=off-manifest-known) with no counterpart in vr1-dc1 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S5 ASYMMETRY: vr1-dc1 declares id=dcN-maas-power-key file=id_dcN_power on headend (custody=off-manifest-known) with no counterpart in vr1-dc0 -- a per-DC credential must exist at BOTH DCs in the same shape + [FAIL] S6 IDENTITY CONFLATION: id 'maas-region-admin' serves 2 principal types (human via gui; service via api, cli-profile) -- ruling 5 requires one identity to serve one principal type + [FAIL] E1 EXPECTED-BUT-ABSENT: dc0-edge-api 'opnsense-api.txt' expected at jumphost/vr1-dc0, not found (mint-stage stage3, SEC-021) + [FAIL] E4 UNCHECKABLE: 2 row(s) have no declared location for their (role, site) and can never be verified -- add a location row or correct the matrix: capi-mgmt-kubeconfig 'config' (cloud/-); rbd-mirror-peer-token 'rbd-mirror-bootstrap-token' (unit/-) + +FAIL: creds-matrix tier 1 -- 82 row(s), 18 check group(s) clean, 7 finding(s) + +--- 6. cardinality distribution + every per-site row (blind-spot evidence, item 3(a)) --- + 28 per-DC + 6 per-site + 5 per-tenant + 43 singleton +per-site rows (all vr1-office1; S5 compares only cardinality=per-DC, creds-matrix.py:380): + office1-svc-key vr1-office1 office1_svc_ed25519 ssh + office1-svc-key vr1-office1 office1_svc_ed25519.pub none + office1-edge-api vr1-office1 opnsense-api.txt api + office1-edge-root vr1-office1 opnsense-root-password gui + office1-edge-root vr1-office1 opnsense-root-hash none + creds-folder-readme vr1-office1 README none + +--- 7. human-principal rows (the login-credential set) --- + maas-region-admin vr1-office1 headend admin.pass gui source-of-record SEC-020 + maas-region-admin vr1-office1 jumphost maas-admin-password gui verbatim-copy SEC-020 + maas-region-operator vr1-office1 jumphost maas-operator-password gui consolidated SEC-020 + office1-edge-root vr1-office1 jumphost opnsense-root-password gui consolidated SEC-007 + office1-edge-root vr1-office1 jumphost opnsense-root-hash none consolidated SEC-007 + netbox-admin vr1-office1 netbox admin.pass gui source-of-record SEC-025 + netbox-admin vr1-office1 jumphost netbox-admin-password gui verbatim-copy SEC-025 + tenant-domain-admin - jumphost -domain-admin-cred.txt gui off-manifest-known SEC-023 + tenant-cluster-user - jumphost -cluster-cred.txt api off-manifest-known SEC-023 + libvirt-power-password - - - console off-manifest-known - + +--- 8. host-role declaration gap (blind-spot evidence, item 3(b)) --- + checker enum: 45:HOST_ROLE = {"jumphost", "headend", "rack", "edge", "netbox", "cloud", "unit", "client", "-"} + roles actually declared in creds-manifests/vm-secret-locations: + headend + jumphost + netbox + +--- 9. key-identity measurements behind the dc0 power-key decision (fingerprints only) --- + dc0 power key, vcloud consolidated copy : 256 SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY maas-virsh-pod (D-123 amendment; MAAS snap -> local libvirt) (ED25519) + dc0 SERVICE key (must DIFFER, SEC-012) : 256 SHA256:DBkeStCyf2qGi3SYuvN6hQ+EsepmhpkN9oulLz09uAs vr1-dc0_svc (D-126 per-env key) (ED25519) + dc1 power key, vcloud (the model shape) : 256 SHA256:BUAdNZEWbbn8Rg+QVSLaY8M6NSvc2nyTaMsQUHQEJhQ vr1-dc1-maas-power (SEC-016 dedicated MAAS->libvirt, per-DC) (ED25519) diff --git a/docs/changelog-20260727-creds-consolidation.md b/docs/changelog-20260727-creds-consolidation.md new file mode 100644 index 0000000..1bd188b --- /dev/null +++ b/docs/changelog-20260727-creds-consolidation.md @@ -0,0 +1,182 @@ +# changelog 2026-07-27 -- credential consolidation batch (SEC-021(b), SEC-025, V2 deferral) + +Session changelog (ONE per session, GA-R2/D1). Status claims live ONLY in +`docs/CURRENT-STATE.md`; this file records WHAT changed, WHY, and HOW TO REVERT. + +Branch `dc-dc-stage4-phase3-maas-deploy`. Stage 4 stays OPEN; no gate closed. + +Operator question that opened the work (2026-07-27): "Check and confirm that we have a +consolidated set of login creds on vcloud for all accounts currently created. Are there any +creds that should be in place that are missing, need to be created, or have not been moved +into their vcloud folders?" Operator direction after the audit: "clear the whole +consolidation batch first" -- i.e. the consolidation items, NOT the dc0 edge-API re-mint, +which was presented as a separate live-mutation item and is deliberately NOT done here. + +--- + +## 1. Audit capture (read-only) + +`docs/audit/creds-consolidation-audit-20260727.txt` -- the full three-site sweep, metadata +only, no credential value read anywhere. Sections: live MAAS account enumeration, +`creds-audit` x3, both jumphost and headend folder listings, the full +`creds-matrix.py --tier2 --tier3 --remote --privileged` run, the cardinality/per-site +blind-spot evidence, the human-principal row set, and the host-role declaration gap. + +**The finding that drove everything else:** `creds-audit` read CLEAN on all three sites +while the matrix returned 13 findings, and the two most consequential gaps were in NEITHER +surface -- they were found by enumerating the accounts that ACTUALLY EXIST. The register +cannot report the absence of a credential it has no row for. `creds-audit` CLEAN remains +not-evidence-of-completeness. + +Revert: delete the capture file (it is a record, nothing depends on it). + +## 2. dc1 svc public key backfilled to the headend store + +`voffice1:~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub` placed (0644), sha256-verified identical +to the vcloud copy. Clears one `E1 EXPECTED-BUT-ABSENT` (SEC-022). Public-key material only. +dc0's shadow store already held both halves; dc1's held only the private -- the mirror image +of the dc0 gap in item 3. + +Revert: `ssh voffice1 'rm ~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub'`. + +## 3. dc0 MAAS->libvirt power key consolidated to vcloud (SEC-021(b)) + +`~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519` (0600) + `.pub` (0644), plus two rows in +`creds-manifests/vr1-dc0.manifest`. This is SEC-021(b) executed as written: "dc0's key +should gain a jumphost copy + manifest row". Clears `S2` x2 and `E1` x2. + +MEASURED before acting, because the naming looked like a defect and was not: + +- The headend's `maas-virsh_ed25519` and the region MAAS snap's `id_ed25519` are the SAME + key (fingerprint `SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY`). +- It IS the dedicated SEC-012 key, not a reuse: the dc0 SERVICE key is a different key + (`SHA256:DBkeStCyf2qGi3SYuvN6hQ+EsepmhpkN9oulLz09uAs`), which is exactly what SEC-012's + ruling ("Dedicated MAAS->libvirt key" over reusing the dc0 service key) requires. +- dc0 having NO per-host block in the snap's `ssh config` (so it uses the default identity) + is the RULED design, not drift -- SEC-016 states it: "Host 172.31.0.6 -> that + IdentityFile, so 172.31.0.2/dc0 keeps SEC-012". + +So NO re-mint was performed and no live power path was touched. The private half was copied +from the voffice1 shadow store, sha256-verified byte-identical at both ends. The public half +had never existed anywhere and was DERIVED with `ssh-keygen -y`, its fingerprint verified +equal to the snap's installed key. + +Two things deliberately NOT done, both recorded rather than fixed: +- The key's INTERNAL comment still reads `maas-virsh-pod (D-123 amendment)` -- it was minted + for the qemu+ssh pod attempt that was refuted the same day, and the per-machine + `power_type=virsh` ruling correctly retained it. Rewriting a comment alters key material; + the file name is now authoritative and the manifest comment carries the explanation. +- Manifest `source` reads `local` to stay consistent with `--render` (which derives it and + would otherwise raise an S3 drift FAIL); the true provenance is in the manifest comment, + which S3 explicitly treats as out of scope. Writing `voffice1:` there would have + traded one finding for another, and writing a custody change would have created two NEW + S5 asymmetries, since S5 compares `custody`. + +Revert: `rm ~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519{,.pub}` and drop the two rows + +comment block from `creds-manifests/vr1-dc0.manifest`. The headend and snap copies were +never modified, so power control is unaffected either way. + +## 4. NetBox web-GUI admin password consolidated (SEC-025 OPENED) + +`~/vr1-office1-creds/netbox-admin-password` (0600), byte-identical to +`office1-netbox:/root/netbox-secrets/admin.pass` (sha256-verified both ends, 28 bytes, value +never printed or echoed). Register changes: the `netbox-admin` netbox row custody +`off-manifest-known` -> `source-of-record`; a NEW jumphost row with custody `verbatim-copy` +(so tier-3 V1 now asserts the two copies stay byte-identical); a manifest row; an extended +`n-netbox-vm-secrets` note. + +This was a HUMAN GUI login that had never left the VM it was minted on -- the third instance +of one miss class (SEC-009's `api.token`, SEC-020's `admin.pass`, now this). It produced no +finding before this session because `off-manifest-known` asserts nothing about the jumphost. + +SEC-025 is opened for the exposure the remediation CREATES (a plaintext human password at +rest on the jumphost), not for the gap it closed -- the same shape as SEC-007 and SEC-020, +with the same rotation obligation and the same both-copies-or-stale-trap hazard. + +Revert: `rm ~/vr1-office1-creds/netbox-admin-password`; revert the matrix row pair, the +manifest row, the note paragraph, and the SEC-025 ledger row. The VM copy is untouched. + +## 5. V2 learns the ruled-deferral state, and gains its first harness cases + +`scripts/creds-matrix.py`: `load_sec_states` now also reports whether a declared-dead SEC row +carries an operator-ruled deferral; `v2_declared_state` emits `V2 ACKNOWLEDGED DEFERRAL` +instead of a FAIL when it does. + +WHY: SEC-006 has read "**DEFERRED by operator ruling 2026-07-13: revoke at completion of this +deployment**" since long before this checker existed. V2 was reporting that standing operator +ruling as a defect on every run -- which is both wrong and corrosive, because a finding that +can never be actioned trains readers to skip findings. **Reissuing the token would have +contravened the ruling**, so the register was the thing that needed to change, not the +credential. + +The acknowledgement is NOT a pass. It names the id, the SEC row, the state word and the +ruling text, and states that the credential "is still live and still exposed until that +ruling is discharged". + +`DEFERRAL_RE` demands BOTH the emphasis-cased state word AND explicit attribution to an +operator ruling (the GA-R5 bar), and `[^|]` stops it crossing a ledger table cell. A looser +match would let prose that merely DISCUSSES deferring buy silence -- the false-green class +this register exists to prevent. T58 and T59 are the regression locks for exactly that. + +Also added `--ledger` (default unchanged). V2 read the SEC register via `--repo`, which the +harness must point at the REAL repo for S4 mint-ref resolution -- so there was no fixture path +to the ledger and **V2 had shipped with ZERO harness cases**. The override is what makes it +testable; it also de-couples the existing tier-3 cases from the live ledger. + +New cases T56-T59 (harness 56 -> 60): the founding invariant still fails; a ruled deferral is +acknowledged and still says EXPOSED; unruled "deferred" prose still fails; a claim spanning +two ledger cells does not count as ruled. + +NOTE: tier 3 is NOT part of preflight `P5` (which runs tier 1 + tier-2-local), so this does +not change the blocking deploy gate's behaviour. + +Revert: `git revert` this commit's changes to `scripts/creds-matrix.py` and +`tests/creds-matrix/run-tests.sh`; V2 returns to failing on SEC-006 every run. + +--- + +## Findings LOGGED, not actioned (hard rule 1) + +1. **dc0 edge API credential (`opnsense-api.txt`) is absent from every DECLARED location** -- + not the jumphost folder, not the headend shadow store. It exists only inside the dc0 + edge's own config. Recovery is a re-mint via `scripts/opnsense-bootstrap-apikey.sh` (the + dc1-proven path; dc0's edge SSH key IS present on vcloud), which is a live mutation on a + running edge and was explicitly OUT of this batch. This is the sole remaining SEC-021(a) + item and accounts for 2 of the 7 residual findings. +2. **No registered root/console credential at EITHER DC edge.** MEASURED: no matrix row, no + manifest entry, no SEC row. `office1-edge-root` is `cardinality=per-site` and office1-only. + NOT MEASURED, and deliberately not probed: what those root passwords actually are. The + phase-2 runbook says DC edges boot on FACTORY DEFAULTS and the D-112(c) chain installs + sshd + the service key without setting a root password -- but that is a repo reading, not + a measurement (hard rule 2). Root SSH is key-only (proven), so SSH is not the vector; the + web GUI is LAN-reachable at both edges and the serial console exists. Proposed as the + next-free SEC row once the operator gates a probe. +3. **Two structural blind spots that let (2) hide.** (a) `S5` compares only + `cardinality=per-DC` (`creds-matrix.py:380`); all SIX `per-site` rows are office1-only, so + a per-site credential missing at two of three sites is invisible to every check. This is + the "cardinality largely inert" item CURRENT-STATE already carries -- now with a concrete + consequence. (b) `creds-manifests/vm-secret-locations` declares only `jumphost`, + `headend`, `netbox`, while the checker also accepts `rack`, `edge`, `cloud`, `unit`, + `client` -- all with zero declared locations, therefore unaudited. SEC-015 was a + rack-resident edge-key copy, so the class is not hypothetical. +4. **The 3 residual S5 power-key asymmetries are RULED, not defects.** SEC-016 designs dc0 to + keep the snap's default identity while dc1 gets a named per-host key, so the filenames can + never match and `DC_TOKEN_RE` cannot normalise them. The register needs a way to record a + ruled exception; suppressing them silently would be wrong and renaming live files to + satisfy a checker would be worse. Needs an operator decision on mechanism. +5. **dc0/dc1 creds folders carry no README** while office1 does (`creds-folder-readme`, also + `per-site`). Cosmetic; noted for symmetry. +6. **The headend shadow stores (SEC-022) are now redundant for dc0's power key** -- vcloud + holds the copy SEC-021(b) asked for. Deleting the shadow copy is a plausible follow-up but + is destructive and out of scope. + +## Verification + +- `tests/creds-matrix/run-tests.sh` -- **60/60 PASS** (was 56). +- `scripts/creds-audit.sh` all three sites -- CLEAN (office1 14 entries, dc0 7, dc1 8). +- `creds-matrix.py --tier2 --tier3 --remote --privileged` -- 82 rows, findings **13 -> 7**. + Cleared: S2 x2, E1 x3 (2 power-key + 1 dc1 `.pub`), V2 x1 (now an acknowledged deferral). + Residual 7: dc0-edge-api (2, needs the excluded re-mint), S5 x3 (ruled, item 4 above), + S6 conflation x1 (the SEC-020 defect, separate), E4 uncheckable x2 (Stage-5/6 rows). + **The register is NOT green and is not expected to be.** +- `scripts/repo-lint.sh` + full gauntlet: see the commit message for the recorded results. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 9d39877..7de158d 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -43,6 +43,8 @@ | SEC-024 | 2026-07-26 | **`opentofu/terraform.tfstate.backup` carries the Office1 MAAS API key in PLAINTEXT at file mode 0664.** SEVERITY CORRECTED 2026-07-26 (measured, same day the row was opened): the original wording called it "group- and world-readable", which overstated it. The FILE BITS are 0664, but `opentofu/` is **0700** and `~` is 0750, so no other account can traverse to it and it is NOT effectively readable today. This is a DEFENCE-IN-DEPTH defect -- one `chmod` on the directory, a `cp -p` elsewhere, or an archive/extract would expose it -- not a live exposure. DOCFIX-175 records that `maas_api_key` reaches the OpenTofu state file in clear; the live `terraform.tfstate` is correctly 0600, but its automatically-written `.backup` sibling is 0664. MEASURED `stat -c '%n %a'` this session on the jumphost clone. FOUND BY the D-137 credential matrix on its first run after the committee fixes: the world-readable check had been gated to `custody=consolidated` rows (43 of 77 rows unchecked), and the tfstate siblings were not declared locations at all -- so neither the mode nor the file was visible to any control. Of the two `terraform.tfstate.pre-*` snapshots, `pre-G16-20260721` is also 0664 and `pre-G6-20260719` is 0600 -- an inconsistency consistent with an accidental umask at write time rather than a deliberate mode. Neither snapshot nor the backup is git-TRACKED (both match `.gitignore:72-73`), so SEC-004's public-repo exposure is NOT implicated. Other accounts exist on the host (`logxen` uid 1000, `libvirt-qemu`), which is why the directory mode is the only thing standing between them and the key. | measured 2026-07-26; `creds-matrix.py --tier2` E2 WORLD-READABLE; DOCFIX-175; capture `docs/audit/d137-tier2-postcommittee-20260726.txt` | operator | **OPEN -- logged, NOT actioned (hard rule 1).** Remediation is a `chmod` on the jumphost plus a decision about whether the historical snapshots should be retained at all; both are mutations and neither is in this step's scope. Rotation of the underlying key remains the SEC-018/-019 obligation. Custody off-repo per D-069. | +| SEC-025 | 2026-07-27 | **NetBox web-GUI `admin` password consolidated to the jumphost -- a HUMAN GUI login that had never left the VM it was minted on.** Found by the 2026-07-27 consolidation audit, which enumerated the accounts that ACTUALLY EXIST instead of the ones the register already knew about. The password was minted on `office1-netbox` at stage 2 (`operator-terminal`, no repo mint command -- research FINDING 1 class) and sat ONLY at `/root/netbox-secrets/admin.pass` (0600, root-owned, 28 bytes). This is the THIRD instance of one miss class: the founding SEC-009 defect (`api.token`), SEC-020 (`admin.pass` for MAAS), and now this -- a VM-minted human credential with no forcing function to reach the operator's folder, so the human who needs it cannot find it and no audit reports it missing. It was structurally invisible: the matrix carried the row as `off-manifest-known`, which asserts nothing about the jumphost. REMEDIATED THE SAME DAY (this row is opened for the exposure the remediation CREATES, not for the gap it closed): a byte-identical copy now sits at `~/vr1-office1-creds/netbox-admin-password` (0600 in a 0700 dir), sha256-verified at both ends, value never printed or echoed. The VM copy REMAINS source-of-record. | 2026-07-27 consolidation audit, capture `docs/audit/creds-consolidation-audit-20260727.txt`; matrix rows `netbox-admin` (netbox=source-of-record, jumphost=verbatim-copy); `creds-manifests/vr1-office1.manifest` | operator | **OPEN -- rotation obligation + a stale-copy hazard.** (1) ROTATE at v1 close, or immediately if the jumphost is rebuilt or shared -- the same obligation SEC-007 and SEC-020 carry for the other consolidated human passwords. (2) STALE-COPY HAZARD: two copies now exist and nothing keeps them in step, so any future rotation MUST update BOTH or the jumphost copy becomes a trap that reads healthy -- identical to the caveat SEC-020 records for `maas-admin-password`. D-137 tier 3 (V1) now asserts byte-identity between them, so a divergence is at least DETECTABLE, which it was not before. (3) Distinct identity from `netbox-secret-key` (a service secret) and from the two NetBox API tokens -- do not merge them; note SEC-006's upstream token is separately declared burned with revocation ruled-deferred. Custody off-repo per D-069. | + **CREDENTIAL-LIFECYCLE POLICY LIVES IN D-137 -- this register does not carry the rule.** DEMOTED 2026-07-26 per D-137 sub-ruling 4 ("D-137 is the authority"): the standing consolidation convention that used to be stated here has MOVED to diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 2bcea7a..15a9e65 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -39,11 +39,12 @@ (Roosevelt per-DC MAAS topology [ARCH], operator-pinned to the next deployment), D-136 (NetBox-coupled per-DC render pipeline [ARCH], non-gating). D-137 is ADOPTED and correctly drops off the scan. Status lines in `docs/design-decisions.md` are the only ruling authority. -- **OPEN security rows:** **20** per `bash scripts/ledger-scan.sh` (was 19). SEC-024 opened - 2026-07-26 -- `opentofu/terraform.tfstate.backup` at mode 0664 carrying the MAAS API key in - plaintext, found by the D-137 matrix on its first run after the committee fixes. The SEC - register of record is `docs/security-ledger.md`; row-level dispositions live THERE only - (GA-R4/F3) -- this block carries pointer + count, never rows. +- **OPEN security rows:** **21** per `bash scripts/ledger-scan.sh` (was 20). SEC-025 opened + 2026-07-27 -- the NetBox web-GUI `admin` password, a HUMAN login that had never left the VM + it was minted on, now consolidated to `~/vr1-office1-creds/`; the row covers the at-rest + exposure the copy CREATES, not the gap it closed. (SEC-024, 2026-07-26, was the previous + addition.) The SEC register of record is `docs/security-ledger.md`; row-level dispositions + live THERE only (GA-R4/F3) -- this block carries pointer + count, never rows. - **Next-free numbers:** D = **138**, DOCFIX = **204**, BUNDLEFIX = **053**. - **Standing numbering rule:** never write an identifier-shaped token (D-/DOCFIX-/BUNDLEFIX-NNN) ABOVE the real high-water mark anywhere in `docs/` or `runbooks/` prose -- historically a diff --git a/scripts/creds-matrix.py b/scripts/creds-matrix.py index 1b4afcd..dc95a92 100644 --- a/scripts/creds-matrix.py +++ b/scripts/creds-matrix.py @@ -834,9 +834,27 @@ # matched word so a human can see the basis rather than trusting the parse. REVOKED_WORDS = ("BURNED", "REVOKED", "COMPROMISED") +# A declared-dead credential whose REMEDIATION the operator has explicitly deferred is a +# different state from one nobody has ruled on, and V2 must not report them identically: +# SEC-006 has carried "**DEFERRED by operator ruling 2026-07-13: revoke at completion of +# this deployment.**" since before this checker existed, so failing it every run reports the +# operator's own standing decision as a defect and trains readers to skip the finding. +# +# THE MATCH IS DELIBERATELY NARROW, and the narrowness is the whole safety property. It +# demands BOTH the emphasis-cased state word AND an explicit attribution to an operator +# ruling -- the GA-R5 bar. A looser pattern (any "defer") would let a credential be silenced +# by a sentence merely *discussing* deferral, which converts this check into the false green +# the register exists to prevent. Ruled-deferred is still reported, as an ACKNOWLEDGED +# DEFERRAL naming the ruling; it is never dropped and never silent. +DEFERRAL_RE = re.compile(r"DEFERRED\b[^|]{0,120}?operator ruling") + def load_sec_states(path): - """{SEC-NNN: (state-word, snippet)} for rows the ledger declares dead.""" + """{SEC-NNN: (state-word, snippet, deferral)} for rows the ledger declares dead. + + `deferral` is the matched ruled-deferral phrase, or None when no operator ruling has + deferred the remediation. + """ out = {} if not os.path.exists(path): return out @@ -847,7 +865,9 @@ for w in REVOKED_WORDS: if w in line: i = line.index(w) - out[m.group(1)] = (w, line[max(0, i - 60):i + 80].strip()) + d = DEFERRAL_RE.search(line) + out[m.group(1)] = (w, line[max(0, i - 60):i + 80].strip(), + d.group(0).strip() if d else None) break return out @@ -861,16 +881,29 @@ of validity: a liveness probe is a snapshot that can be stale a minute later, but a declared revocation does not un-declare itself.""" bad = 0 + deferred = 0 for r in rows: if r.sec_ref == "-" or r.sec_ref not in sec_states: continue - word, snippet = sec_states[r.sec_ref] - if r.custody in IN_FOLDER or r.custody == "source-of-record": - bad += 1 - fails.append("V2 DECLARED-DEAD BUT REGISTERED HEALTHY: id '%s' (%s) is custody=%s, " - "but %s declares it %s -- a credential that must be revoked and " - "reissued cannot also be a good consolidated copy. Ledger text: ...%s..." - % (r.id, r.filename, r.custody, r.sec_ref, word, snippet[:90])) + word, snippet, deferral = sec_states[r.sec_ref] + if r.custody not in IN_FOLDER and r.custody != "source-of-record": + continue + if deferral: + # Reported, not dropped. The credential is still live and still exposed -- the + # ruling defers the REMEDIATION, it does not change the state -- so the notice + # says so rather than reading as an all-clear. + deferred += 1 + oks.append("V2 ACKNOWLEDGED DEFERRAL: id '%s' (%s) is custody=%s and %s declares " + "it %s, but remediation is DEFERRED BY OPERATOR RULING -- the credential " + "is still live and still exposed until that ruling is discharged. " + "Ruling text: '%s'" + % (r.id, r.filename, r.custody, r.sec_ref, word, deferral)) + continue + bad += 1 + fails.append("V2 DECLARED-DEAD BUT REGISTERED HEALTHY: id '%s' (%s) is custody=%s, " + "but %s declares it %s -- a credential that must be revoked and " + "reissued cannot also be a good consolidated copy. Ledger text: ...%s..." + % (r.id, r.filename, r.custody, r.sec_ref, word, snippet[:90])) if not bad: oks.append("V2 declared state: no credential is registered healthy while its SEC row " "declares it %s" % "/".join(w.lower() for w in REVOKED_WORDS)) @@ -971,6 +1004,13 @@ ap.add_argument("--all", action="store_true", help="run every implemented tier") ap.add_argument("--locations", default=os.path.join(repo, "creds-manifests", "vm-secret-locations")) + # V2 reads the SEC register. It was derived from --repo, which the harness must point at + # the REAL repo for S4 mint-ref resolution -- so V2 was the one check with no fixture + # path and therefore shipped with ZERO harness cases. This override is what makes it + # testable; the default is unchanged. + ap.add_argument("--ledger", + default=os.path.join(repo, "docs", "security-ledger.md"), + help="SEC register read by tier-3 V2 (default: /docs/security-ledger.md)") ap.add_argument("--tier2", action="store_true", help="run tier 2 (EXISTENCE) -- local locations only unless --remote") ap.add_argument("--remote", action="store_true", @@ -1042,8 +1082,7 @@ stages, oks, fails) if args.tier3 or args.all: print("=== creds-matrix: tier 3 (VALIDITY) ===") - v2_declared_state(rows, load_sec_states( - os.path.join(args.repo, "docs", "security-ledger.md")), oks, fails) + v2_declared_state(rows, load_sec_states(args.ledger), oks, fails) tier3_validity(rows, locations, args.remote, args.privileged, observed, oks, fails) else: diff --git a/tests/creds-matrix/run-tests.sh b/tests/creds-matrix/run-tests.sh index 6b69978..0d63218 100644 --- a/tests/creds-matrix/run-tests.sh +++ b/tests/creds-matrix/run-tests.sh @@ -47,9 +47,13 @@ # their own file before calling run(). [ -f "$d/stages" ] || awk '!/^#/ && NF==12 {print $9" reached"}' "$d/matrix.tsv" \ | sort -u > "$d/stages" + # V2's SEC register, fixture-scoped. Empty unless the case writes one -- WITHOUT this the + # tier-3 cases read the REAL docs/security-ledger.md (because --repo must point at the real + # repo for S4 mint-ref resolution), which is why V2 had no harness cases at all. + [ -f "$d/ledger.md" ] || : > "$d/ledger.md" python3 "$CHECK" --matrix "$d/matrix.tsv" --manifest-dir "$d/creds-manifests" \ --notes "$d/notes.md" --repo "$REPO" --stages "$d/stages" \ - --locations "$d/locations" "$@" > "$d/out" 2>&1 + --locations "$d/locations" --ledger "$d/ledger.md" "$@" > "$d/out" 2>&1 RC=$? } # A tier-2 fixture: a declared LOCAL location holding the baseline keypair. @@ -577,6 +581,54 @@ || { no "T55 stage reach must drive the verdict (pending=$PEND reached=$RC)"; sed 's/^/ /' "$D/out"; } +# ---- V2 (tier 3, declared state) ----------------------------------------------------- +# V2 shipped with ZERO cases: --repo had to point at the real repo, so there was no fixture +# path to the SEC register. T56-T59 exist because of that gap, not despite it. +secrow() { # secrow + printf '| SEC-900 | 2026-01-01 | fixture token is BURNED and must be revoked + reissued | fixture | operator | %s |\n' \ + "$2" > "$1/ledger.md" +} +# A consolidated row that cites the fixture SEC row. +v2env() { local d; d=$(mkenv); mkloc "$d" + row 'zeta-token singleton vr1-office1 jumphost zeta.txt api service consolidated stage2 operator-terminal SEC-900 n-alpha' "$d" + printf 'zeta.txt 600 local\n' >> "$d/creds-manifests/vr1-office1.manifest" + printf 'x\n' > "$d/store/zeta.txt"; chmod 600 "$d/store/zeta.txt" + echo "$d"; } + +# T56 -- the original invariant: dead in the ledger, healthy in the register -> FAIL. +D=$(v2env); secrow "$D" 'OPEN -- rotate at v1 close' +run "$D" --tier2 --tier3 +[ "$RC" = 1 ] && grep -q 'V2 DECLARED-DEAD BUT REGISTERED HEALTHY.*zeta-token' "$D/out" \ + && ok "T56 declared-dead + custody=consolidated -> V2 FAIL (the founding invariant)" \ + || { no "T56 a burned credential registered healthy must fail (rc=$RC)"; sed 's/^/ /' "$D/out"; } + +# T57 -- SEC-006's real shape: the operator RULED the revocation deferred. Reporting that as +# a defect every run reports the operator's own decision as a bug -- but it must NOT go silent. +D=$(v2env); secrow "$D" 'OPEN -- **DEFERRED by operator ruling 2026-01-02: revoke at completion of this deployment.** Until then the token is live and exposed' +run "$D" --tier2 --tier3 +[ "$RC" = 0 ] && grep -q 'V2 ACKNOWLEDGED DEFERRAL.*zeta-token' "$D/out" \ + && grep -q 'still live and still exposed' "$D/out" \ + && ok "T57 a RULED deferral -> acknowledged, exit 0, and the notice still says EXPOSED" \ + || { no "T57 a ruled deferral must be acknowledged, not failed and not silenced (rc=$RC)"; sed 's/^/ /' "$D/out"; } + +# T58 -- the evasion guard, and the reason the match is narrow: prose that merely DISCUSSES +# deferring, with no operator ruling, must not buy silence. A loose match here would turn V2 +# into the false green the register exists to prevent. +D=$(v2env); secrow "$D" 'OPEN -- deferred for now, we will revisit this later' +run "$D" --tier2 --tier3 +[ "$RC" = 1 ] && grep -q 'V2 DECLARED-DEAD BUT REGISTERED HEALTHY' "$D/out" \ + && ok "T58 'deferred' WITHOUT an operator ruling -> still FAIL (no silencing by prose)" \ + || { no "T58 unruled deferral prose must not silence V2 (rc=$RC)"; sed 's/^/ /' "$D/out"; } + +# T59 -- DEFERRED in one table cell and 'operator ruling' in another is NOT a ruled deferral; +# the cell boundary is load-bearing, so the pattern must not span `|`. +D=$(v2env) +printf '| SEC-900 | 2026-01-01 | fixture token is BURNED, DEFERRED | fixture | per an operator ruling | OPEN |\n' > "$D/ledger.md" +run "$D" --tier2 --tier3 +[ "$RC" = 1 ] && grep -q 'V2 DECLARED-DEAD BUT REGISTERED HEALTHY' "$D/out" \ + && ok "T59 a deferral claim spanning two ledger cells does NOT count as ruled" \ + || { no "T59 the deferral match must not cross a table cell (rc=$RC)"; sed 's/^/ /' "$D/out"; } + echo if [ "$F" = 0 ]; then echo "creds-matrix: $P/$P PASS"; exit 0; fi echo "creds-matrix: FAILURES: $F (passed $P)"; exit 1