Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260727-creds-consolidation.md
@JANeumatrix JANeumatrix 4 hours ago 28 KB Stage 4 cleanup: carve residue 108 fabrics -> 17; dc1 leftover nginx purged
Raw Blame History

changelog 2026-07-27 -- credential consolidation batch (SEC-021(b), SEC-025, V2 deferral)

Session changelog (ONE per session, GA-R2/D1). Status claims live ONLY in docs/CURRENT-STATE.md; this file records WHAT changed, WHY, and HOW TO REVERT.

Branch dc-dc-stage4-phase3-maas-deploy. Stage 4 stays OPEN; no gate closed.

Operator question that opened the work (2026-07-27): "Check and confirm that we have a consolidated set of login creds on vcloud for all accounts currently created. Are there any creds that should be in place that are missing, need to be created, or have not been moved into their vcloud folders?" Operator direction after the audit: "clear the whole consolidation batch first" -- i.e. the consolidation items, NOT the dc0 edge-API re-mint, which was presented as a separate live-mutation item and is deliberately NOT done here.

1. Audit capture (read-only)

docs/audit/creds-consolidation-audit-20260727.txt -- the full three-site sweep, metadata only, no credential value read anywhere. Sections: live MAAS account enumeration, creds-audit x3, both jumphost and headend folder listings, the full creds-matrix.py --tier2 --tier3 --remote --privileged run, the cardinality/per-site blind-spot evidence, the human-principal row set, and the host-role declaration gap.

The finding that drove everything else: creds-audit read CLEAN on all three sites while the matrix returned 13 findings, and the two most consequential gaps were in NEITHER surface -- they were found by enumerating the accounts that ACTUALLY EXIST. The register cannot report the absence of a credential it has no row for. creds-audit CLEAN remains not-evidence-of-completeness.

Revert: delete the capture file (it is a record, nothing depends on it).

2. dc1 svc public key backfilled to the headend store

voffice1:~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub placed (0644), sha256-verified identical to the vcloud copy. Clears one E1 EXPECTED-BUT-ABSENT (SEC-022). Public-key material only. dc0's shadow store already held both halves; dc1's held only the private -- the mirror image of the dc0 gap in item 3.

Revert: ssh voffice1 'rm ~/vr1-dc1-creds/vr1-dc1_svc_ed25519.pub'.

3. dc0 MAAS->libvirt power key consolidated to vcloud (SEC-021(b))

~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519 (0600) + .pub (0644), plus two rows in creds-manifests/vr1-dc0.manifest. This is SEC-021(b) executed as written: "dc0's key should gain a jumphost copy + manifest row". Clears S2 x2 and E1 x2.

MEASURED before acting, because the naming looked like a defect and was not:

  • The headend's maas-virsh_ed25519 and the region MAAS snap's id_ed25519 are the SAME key (fingerprint SHA256:Dt/YXTXSF4nXGj9cz8f0owL+qreVpMYBVm/igW10FXY).
  • It IS the dedicated SEC-012 key, not a reuse: the dc0 SERVICE key is a different key (SHA256:DBkeStCyf2qGi3SYuvN6hQ+EsepmhpkN9oulLz09uAs), which is exactly what SEC-012's ruling ("Dedicated MAAS->libvirt key" over reusing the dc0 service key) requires.
  • dc0 having NO per-host block in the snap's ssh config (so it uses the default identity) is the RULED design, not drift -- SEC-016 states it: "Host 172.31.0.6 -> that IdentityFile, so 172.31.0.2/dc0 keeps SEC-012".

So NO re-mint was performed and no live power path was touched. The private half was copied from the voffice1 shadow store, sha256-verified byte-identical at both ends. The public half had never existed anywhere and was DERIVED with ssh-keygen -y, its fingerprint verified equal to the snap's installed key.

Two things deliberately NOT done, both recorded rather than fixed:

  • The key's INTERNAL comment still reads maas-virsh-pod (D-123 amendment) -- it was minted for the qemu+ssh pod attempt that was refuted the same day, and the per-machine power_type=virsh ruling correctly retained it. Rewriting a comment alters key material; the file name is now authoritative and the manifest comment carries the explanation.
  • Manifest source reads local to stay consistent with --render (which derives it and would otherwise raise an S3 drift FAIL); the true provenance is in the manifest comment, which S3 explicitly treats as out of scope. Writing voffice1:<path> there would have traded one finding for another, and writing a custody change would have created two NEW S5 asymmetries, since S5 compares custody.

Revert: rm ~/vr1-dc0-creds/vr1-dc0-maas-power_ed25519{,.pub} and drop the two rows + comment block from creds-manifests/vr1-dc0.manifest. The headend and snap copies were never modified, so power control is unaffected either way.

4. NetBox web-GUI admin password consolidated (SEC-025 OPENED)

~/vr1-office1-creds/netbox-admin-password (0600), byte-identical to office1-netbox:/root/netbox-secrets/admin.pass (sha256-verified both ends, 28 bytes, value never printed or echoed). Register changes: the netbox-admin netbox row custody off-manifest-known -> source-of-record; a NEW jumphost row with custody verbatim-copy (so tier-3 V1 now asserts the two copies stay byte-identical); a manifest row; an extended n-netbox-vm-secrets note.

This was a HUMAN GUI login that had never left the VM it was minted on -- the third instance of one miss class (SEC-009's api.token, SEC-020's admin.pass, now this). It produced no finding before this session because off-manifest-known asserts nothing about the jumphost.

SEC-025 is opened for the exposure the remediation CREATES (a plaintext human password at rest on the jumphost), not for the gap it closed -- the same shape as SEC-007 and SEC-020, with the same rotation obligation and the same both-copies-or-stale-trap hazard.

Revert: rm ~/vr1-office1-creds/netbox-admin-password; revert the matrix row pair, the manifest row, the note paragraph, and the SEC-025 ledger row. The VM copy is untouched.

5. V2 learns the ruled-deferral state, and gains its first harness cases

scripts/creds-matrix.py: load_sec_states now also reports whether a declared-dead SEC row carries an operator-ruled deferral; v2_declared_state emits V2 ACKNOWLEDGED DEFERRAL instead of a FAIL when it does.

WHY: SEC-006 has read "DEFERRED by operator ruling 2026-07-13: revoke at completion of this deployment" since long before this checker existed. V2 was reporting that standing operator ruling as a defect on every run -- which is both wrong and corrosive, because a finding that can never be actioned trains readers to skip findings. Reissuing the token would have contravened the ruling, so the register was the thing that needed to change, not the credential.

The acknowledgement is NOT a pass. It names the id, the SEC row, the state word and the ruling text, and states that the credential "is still live and still exposed until that ruling is discharged".

DEFERRAL_RE demands BOTH the emphasis-cased state word AND explicit attribution to an operator ruling (the GA-R5 bar), and [^|] stops it crossing a ledger table cell. A looser match would let prose that merely DISCUSSES deferring buy silence -- the false-green class this register exists to prevent. T58 and T59 are the regression locks for exactly that.

Also added --ledger (default unchanged). V2 read the SEC register via --repo, which the harness must point at the REAL repo for S4 mint-ref resolution -- so there was no fixture path to the ledger and V2 had shipped with ZERO harness cases. The override is what makes it testable; it also de-couples the existing tier-3 cases from the live ledger.

New cases T56-T59 (harness 56 -> 60): the founding invariant still fails; a ruled deferral is acknowledged and still says EXPOSED; unruled "deferred" prose still fails; a claim spanning two ledger cells does not count as ruled.

NOTE: tier 3 is NOT part of preflight P5 (which runs tier 1 + tier-2-local), so this does not change the blocking deploy gate's behaviour.

Revert: git revert this commit's changes to scripts/creds-matrix.py and tests/creds-matrix/run-tests.sh; V2 returns to failing on SEC-006 every run.

6. Permission allowlist widened for read-only commands (.claude/settings.json)

Transcript scan of the 44 most-recent session JSONLs across both project dirs, counting Bash leading command+subcommand pairs (and splitting each candidate into LOCAL vs nested-inside-ssh, since a command that only ever runs inside ssh host '...' gains nothing from its own rule -- the outer ssh is what the permission layer matches).

ADDED to permissions.allow only -- 13 entries, 31 -> 44. ask, deny, env and hooks untouched; nothing removed or reordered; JSON re-validated and de-duplicated:

bash scripts/creds-audit.sh *          (33)   bash scripts/dc-dc-ceph-disk-budget.sh * (13)
bash scripts/site-baseleg.sh check*     (5)   tofu validate* + tofu -chdir=* validate*  (25)
tofu state list* + -chdir variant      (16)   tofu state show*                           (3)
tofu providers schema*                 (13)   tofu version*                             (11)
ip route*                              (22)   systemctl is-active *                      (6)

DELIBERATELY NOT ADDED, and this is the substantive half of the item:

  • ssh (693 calls), scp (47), virsh (234) -- the three biggest prompt sources, left alone. They are this cloud's MUTATION TRANSPORT. CLAUDE.md hard rule 3 makes the ask rules the enforcement mechanism for per-mutation approval and says not to work around them, and the Bash(ssh *) disposition is an OPEN operator item queued at the 2026-07-23 close. Allowlisting any ssh shape would bypass the gate AND pre-empt an unresolved operator decision. This item does NOT discharge that queued prune -- it is still open.
  • python3 scripts/creds-matrix.py (76 local runs -- the largest genuine miss) -- skipped because Bash(*creds-matrix.py *--privileged*) is an ask rule added 2026-07-26 precisely after the auto-mode classifier incident. Every prefix pattern wide enough to cover the observed ... 2>&1 | tail forms would ALSO match --privileged, and allow-vs-ask precedence was not verified this session. Adding a rule that might silently downgrade an operator-installed gate is exactly the class of change hard rule 3 forbids. Needs the precedence question answered first, then a narrow set of EXACT forms.
  • tofu plan (54) -- read-only in effect, but CURRENT-STATE section 10 says in terms "Do NOT re-run tofu plan casually against live state". The prompt is deliberate friction.
  • python3 -c/heredocs (324) and shells: arbitrary code execution. curl (114): a mutation vector via -X/-d. tofu fmt (26): rewrites files. tofu init (10): mutates .terraform/. nft list (10): normally needs sudo, which correctly stays ask.

TWO OBSERVATIONS, no action taken (flagged, not fixed): several existing allow entries are redundant with Claude Code's built-in auto-allow (Bash(git status*), Bash(git log*), Bash(grep *), Bash(ls *), Bash(jq *), Bash(cat scripts/*)) -- harmless, and removing entries was out of scope; and Bash(git pull*) sits in allow although it is a fetch-and-merge that mutates the working tree.

Revert: drop the 13 appended lines from permissions.allow in .claude/settings.json. No other field was touched, so the revert is a contiguous deletion.

7. Stage 4 close-out opened: dc-mirror.sh check was FALSE-GREENING the mirror gate

Operator direction: close Stage 4 before resuming the DC1 Stage-5 chain. The Stage 4 DoD (runbooks/dc-dc-phase3-maas-enlist-deploy.md:472-485) has six bullets; four are already met and captured. Verifying the fifth -- "per-DC mirror reachable" -- found that the check the gate would close on could not fail.

scripts/dc-mirror.sh:271-273 tested [ -f last-sync.status ] and printed the file's contents behind an unconditional OK. So the status word was displayed but never asserted on, and the check exited PASS regardless. MEASURED on both racks, 2026-07-27:

  • dc0 read FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0 -> printed OK, check PASS.
  • dc1 read RUNNING 2026-07-23T21:49:35Z -- a marker FOUR DAYS stale, left by the debmirror the D-135 amendment killed with a signal, so it never wrote a terminal state -> printed OK, check PASS.

FIXED: the status word is now case-analysed. OK* passes; FAIL* misses; an ABSENT file misses (it was a passing note, but nothing has attested the content); an unrecognised word REFUSES rather than defaulting to success. RUNNING* is deliberately neither -- in-flight and died-mid-run are indistinguishable from the file alone, so it reports an explicit UNKNOWN and cross-checks the unit: a RUNNING marker with no active unit is named as a corpse, not a success. Harness tests/dc-mirror/run-tests.sh 24/24 (was 19); T20-T24 lock each status word, T24 specifically locks out the return of the exact defective shape.

PROOF THE FIX WORKS -- re-run against both live racks, and the gate is now honestly NOT MET: dc-mirror check (dc0): FAIL on the failed sync, dc-mirror check (dc1): FAIL on the stale marker. Capture docs/audit/stage4-mirror-gate-20260727.txt.

SUBSTANCE BEHIND THE TWO FAILURES (measured, not inferred):

  • dc0's mirror CONTENT is complete and intact -- 949G ubuntu + 342M cloud-archive, all three dists (jammy, jammy-updates, jammy-security) plus pool and project. What failed is the overnight INCREMENTAL: Download of dists/jammy/Release failed: 500 read timeout -> "Failed to download some Release, Release.gpg or InRelease files!" -> debmirror aborted the ubuntu leg with 255. The UCA leg succeeded (uca=0). A transient upstream fetch failure, not mirror damage. The daily timer re-arms for 2026-07-28 00:14:48.
  • dc1's debmirror is dormant only ACCIDENTALLY. dc1-mirror-sync.timer is enabled with an EMPTY NextElapseUSecRealtime, because the service sits in failed/Result=signal from the 07-24 pause. The D-135 amendment RULED it paused and "kept dormant as the fallback", but nothing enforces that -- the dormancy is a side effect of a failed unit, not a declared state, so a systemctl reset-failed (or possibly a reboot) re-arms a 330G -> ~949G pull on a rack whose ruled artifact path is now the proxy. dc1's PROXY meanwhile checks PASS genuinely: apt-cacher-ng active on .4:3142, serving both archive and UCA Release with 200.

Revert: git revert this commit's scripts/dc-mirror.sh + tests/dc-mirror/run-tests.sh changes -- the check returns to passing unconditionally on last-sync. No live state was altered by this item; both racks were read only.

8. G17 opened by ruling; DoD bullets 5 and 6 repaired (DOCFIX-204)

RULING (GA-R5), 2026-07-27. Question as presented: "The node-side half of bullet 5. Nodes are powered off by the READY-handoff ruling, so no node-side probe can run as things stand. Either a gated rescue-boot check on one node per DC now (closes it inside Stage 4), or split it into its own gate row targeted at Stage 5 first boot (GA-R6 E3 explicitly permits this; a conditional close is not permitted)." Operator answer, exact utterance: "split it into its own gate row". Recorded in the G17 row of docs/CURRENT-STATE.md, which is the authority; committed and pushed before any dependent work (GA-R5).

G17 -- "Per-DC artifact source reachable FROM A NODE", [V], OPEN. Per DC: dc0 -> curl -sI http://10.12.8.4/ returns 200 from a booted node; dc1 -> apt fetches through the proxy at 10.12.68.4:3142, because dc1's ruled artifact path is the CACHING PROXY and it has no node-facing mirror (D-135 amendment) -- checking it as one fails by design. Natural trigger is Stage 5 first boot. The row states explicitly that G17 is NOT a Stage-5 precondition and must not be conflated with one: Stage 5's bootstrap needs OPEN edge egress for the juju agent stream and snaps (D-135 items 2-3 unbuilt), which is a different path from the apt artifact source.

What REMAINS in Stage 4 for bullet 5 is the rack-side half -- the source answers on its own address with an attested-current sync. dc1's proxy already PASSES; dc0 does not yet, pending the sync re-run (item 7).

DOCFIX-204 -- bullet 6 was unsatisfiable, and its stale text had spread to four surfaces. "NTP from the DC's own OPNsense edge working" is SUPERSEDED by D-129(iv) (RULED 2026-07-21, "Keep MAAS hierarchy (Recommended)", no NTP role on the edge). The DoD asked you to confirm the edge is the time source, which the ruling had already refused -- so the bullet could never be met by a correctly-built DC. Repaired in:

  • runbooks/dc-dc-phase3-maas-enlist-deploy.md DoD -- bullet 6 STRUCK with its supersession cited; bullet 5 rewritten per-DC with the split recorded.
  • the same runbook's Step 7, rewritten. It had THREE defects, each recorded rather than quietly patched because the old text is the kind that gets followed: it still branched on gap #5 ("if the mirror does not exist yet, STOP") which D-135 resolved; it applied one mirror check to both DCs, which fails on dc1 by design; and both its checks said "from a deployed node", which the READY-handoff ruling makes impossible in this stage. Now it invokes the two site checkers (with the hard-rule-2 note to take rack transit IPs from lib-hosts.sh, never memory) and carries the MAAS-hierarchy time check as a Stage-5/G17 observation.
  • docs/dc-dc-deployment-workflow.md:206 -- the Gate cell also still said "Nodes deployed", contradicting DOCFIX-200's READY handoff. Both fixed.
  • docs/dc-dc-buildout-design.md:120 -- corrected with the supersession noted inline.
  • runbooks/dc-dc-phase4-juju-bundle-per-dc.md:26 -- prerequisites said "nodes Deployed", which is precisely the expectation DOCFIX-200 removed to stop Stage 5 breaking; plus a new block making this stage the OWNER of G17's captures, with the warning that first boot is the observation window and missing it forces a deliberate rescue-boot.

Revert: git revert this commit's doc/runbook changes; the G17 row and the DOCFIX-204 edits are text-only, and no live state was touched.

9. dc0 sync re-run; dc1 full mirror REMOVED after moving the proxy's net layer (D-135 amendment)

dc0 (gated). systemctl start dc0-mirror-sync.service -> OK 2026-07-27T08:43:46Z ubuntu=0 uca=0 in 20 seconds, confirming the transient-500 diagnosis exactly. Fixed dc-mirror.sh check dc0 now reports PASS -- trustworthy precisely because the same check FAILED on the same rack twenty minutes earlier. Stage 4 DoD bullet 5 is now MET for both DCs.

dc1 -- the record was wrong about intent. CURRENT-STATE said the debmirror was "PAUSED ... kept dormant as the fallback". Operator, exact utterance: "DC0 is the test of a full mirror. DC1 is the test of the mirror proxy. DC1 needed the partially created full mirror and associated script/files/settings removed." Recorded as a D-135 AMENDMENT 2026-07-27: the per-DC split is the DELIBERATE EXPERIMENT, not an interim -- no fallback, no later convergence. That also supersedes the "DC1's proxy is REPLACED by a full mirror afterward" claim, which was in THREE places (D-135, CURRENT-STATE:109, dc-cache-proxy.sh:6); all three corrected.

The coupling trap, and why sequencing mattered. Asked whether to rip down and rebuild rather than pick and choose, I found the rebuild path was BLOCKED, not merely slower: dc-cache-proxy.sh did not own its utility net layer -- its header said it REUSES dc-mirror.sh's <site>-mirror-net.service and "fails loud if absent". So on dc1 three files named dc1-mirror-* (net unit, apply helper, resolved DNS drop-in) were load-bearing FOR THE PROXY. Therefore (a) the 07-23 changelog's recorded removal ("stop/disable ... -net.service, rm the generated files") would have silently killed the proxy, the Stage-4 bullet-5 PASS, and the Stage-5 dc1 apt path; and (b) the proxy could not be REBUILT afterward, since nothing but dc-mirror.sh install created the net layer -- rebuilding a PROXY meant reinstalling the whole MIRROR apparatus, enabled daily timer included. Operator direction: "Do the net layer first, then rip it all down."

Step 1 -- net layer moved (script change). dc-cache-proxy.sh now owns <site>-cache-proxy-net.service + <site>-cache-proxy-net-apply + its own resolved.conf.d/<site>-cache-proxy.conf, reusing dc-mirror's proven mechanism (bridge resolved from the libvirt net NAME at runtime, hard rule 3). Neutrally named so a proxy-only DC carries NO mirror-named dependency. Idempotent ip addr replace / ip route replace on identical MEASURED values, so the two net units coexist safely where a site runs both -- dc0 is unaffected. check asserts the ARTIFACTS, not just the live address/route, because asserting only the live effect would PASS while the layer was really the mirror's: it would look independent and break on teardown, the exact failure being removed.

Harness 20/20 (was 15). T12 was REPLACED, not deleted -- it asserted the superseded design ("net layer reused ... no ip addr/route writes; points at dc-mirror.sh"), so it failed by design once the ruling landed. It is now INVERTED (T12: owns the layer) plus T12b-T12f (no operative dc-mirror referral; proxy-named artifacts; enabled-not-just-started; check asserts artifacts; idempotent verbs only). T12b is deliberately scoped to NON-COMMENT lines: the header's account of WHY the coupling was removed cites dc-mirror.sh install, and deleting that history to satisfy a grep would lose the reason the trap existed.

Step 2 -- independence PROVEN before deleting anything. A oneshot with RemainAfterExit=yes does not undo its work on stop, so merely stopping dc1-mirror-net.service would have proven nothing. The live state was removed outright -- ip addr del 10.12.68.4/22 (count -> 0) and ip route del default via 10.12.64.1 (count -> 0) -- then the proxy-owned unit applied ALONE restored both, and check dc1 returned PASS with the mirror net unit disabled.

Step 3 -- wholesale removal, no picking and choosing. All mirror units, the timer, the sync helper, the nginx vhost, the resolved drop-in, the debmirror package, and the 330G of partial mirror (33,037 files; rm -rf 47.6s; rack disk 339G -> 9.0G). Zero mirror-named residue; every proxy artifact present; proxy PASS throughout and never down. Capture docs/audit/dc1-mirror-teardown-20260727.txt.

nginx left installed, deliberately: it was a mirror prereq, but with the vhost gone it is a generic server on its stock default vhost. Removing the metapackage while nginx-core keeps running is a half-measure worse than either extreme, and a full purge is scope creep on a rack host. If dc1 is ever rebuilt as a mirror, dc-mirror.sh install fail-louds with the exact apt line.

Revert: reinstall via dc-mirror.sh install dc1 (recreates units/vhost/timer; needs the debmirror package reinstalled first) then sync dc1 to re-pull ~950G. The 330G partial is NOT recoverable -- deliberately, per the amendment. The proxy net layer reverts by git revert of scripts/dc-cache-proxy.sh + its harness, then re-running dc-mirror.sh install dc1 to restore the mirror-owned layer.

10. Carve residue cleaned (108 fabrics -> 17); dc1's leftover nginx removed

Capture docs/audit/stage4-carve-residue-cleanup-20260727.txt. Both items were queued to the stage close at the 2026-07-23 carve.

nginx on dc1. Purged nginx + nginx-common (dry-run first: nothing else depended on them, and no nginx-core was installed), plus autoremove of liblockfile-simple-perl and liblog-agent-perl -- debmirror's orphaned perl deps. /etc/nginx gone, nothing listening on :80, proxy still PASS on :3142. dc0 keeps its nginx and was re-verified unaffected (it serves the mirror).

Carve residue: 108 fabrics -> 17. Deleted subnet id=8 192.168.1.0/24 (the superseded OPNsense FACTORY LAN -- both edges were re-addressed to 10.12.4.1 / 10.12.64.1), then fabric-5, then the 90 auto-created empties (ids 6..95).

The audit caught an ordering dependency the original flag did not state: the 192.168.1.0/24 subnet was the ONLY occupant of fabric-5, so deleting the fabric first would have cascaded the subnet away rather than removing it cleanly. Subnet first, then the fabric.

METHOD, chosen because of this repo's own history: a MAAS fabric delete cascades to its VLANs, and the 2026-07-21 pod-delete incident cost 9 machine records precisely because the association check ran AFTER the delete. So emptiness was PROVEN per fabric -- zero subnets, zero ipranges, zero node interfaces across all of its VLANs -- against a fresh occupancy snapshot (fabrics + subnets + ipranges + every machine's interface_set) taken immediately before the batch, and the cascade check was re-run afterwards as well.

KEPT, all verified occupied and enumerated in the capture: office1 base + GUA (fabric-0), office1 compose with its 2 LXD guests (fabric-1), both transits (fabric-2 / fabric-141), both metal-admin/boot fabrics with 9 nodes each (fabric-4 / fabric-142), libvirt default (fabric-3), and the 10 named plane fabrics (vr1-dc0-* / vr1-dc1-*, 9-18 interfaces each).

POST-STATE: 18 Ready + 2 Deployed office1 guests, all 18 still power_type=virsh, 7 interfaces each (6 flat planes per D-133 + br-ex), ZERO orphaned interfaces, placement tags 9+9 intact, and both DCs' artifact paths re-verified PASS after everything.

Revert: fabrics and subnets are MAAS records, not config -- there is no undo. Recreating them would mean maas admin fabrics create + subnets create, but nothing needs them: the 90 were auto-created empties, and 192.168.1.0/24 was a superseded factory address. The nginx purge reverts with apt-get install nginx (dc1 has no use for it while it is the proxy DC).

Findings LOGGED, not actioned (hard rule 1)

  1. dc0 edge API credential (opnsense-api.txt) is absent from every DECLARED location -- not the jumphost folder, not the headend shadow store. It exists only inside the dc0 edge's own config. Recovery is a re-mint via scripts/opnsense-bootstrap-apikey.sh (the dc1-proven path; dc0's edge SSH key IS present on vcloud), which is a live mutation on a running edge and was explicitly OUT of this batch. This is the sole remaining SEC-021(a) item and accounts for 2 of the 7 residual findings.
  2. No registered root/console credential at EITHER DC edge. MEASURED: no matrix row, no manifest entry, no SEC row. office1-edge-root is cardinality=per-site and office1-only. NOT MEASURED, and deliberately not probed: what those root passwords actually are. The phase-2 runbook says DC edges boot on FACTORY DEFAULTS and the D-112(c) chain installs sshd + the service key without setting a root password -- but that is a repo reading, not a measurement (hard rule 2). Root SSH is key-only (proven), so SSH is not the vector; the web GUI is LAN-reachable at both edges and the serial console exists. Proposed as the next-free SEC row once the operator gates a probe.
  3. Two structural blind spots that let (2) hide. (a) S5 compares only cardinality=per-DC (creds-matrix.py:380); all SIX per-site rows are office1-only, so a per-site credential missing at two of three sites is invisible to every check. This is the "cardinality largely inert" item CURRENT-STATE already carries -- now with a concrete consequence. (b) creds-manifests/vm-secret-locations declares only jumphost, headend, netbox, while the checker also accepts rack, edge, cloud, unit, client -- all with zero declared locations, therefore unaudited. SEC-015 was a rack-resident edge-key copy, so the class is not hypothetical.
  4. The 3 residual S5 power-key asymmetries are RULED, not defects. SEC-016 designs dc0 to keep the snap's default identity while dc1 gets a named per-host key, so the filenames can never match and DC_TOKEN_RE cannot normalise them. The register needs a way to record a ruled exception; suppressing them silently would be wrong and renaming live files to satisfy a checker would be worse. Needs an operator decision on mechanism.
  5. dc0/dc1 creds folders carry no README while office1 does (creds-folder-readme, also per-site). Cosmetic; noted for symmetry.
  6. The headend shadow stores (SEC-022) are now redundant for dc0's power key -- vcloud holds the copy SEC-021(b) asked for. Deleting the shadow copy is a plausible follow-up but is destructive and out of scope.

Verification

  • tests/creds-matrix/run-tests.sh -- 60/60 PASS (was 56).
  • scripts/creds-audit.sh all three sites -- CLEAN (office1 14 entries, dc0 7, dc1 8).
  • creds-matrix.py --tier2 --tier3 --remote --privileged -- 82 rows, findings 13 -> 7. Cleared: S2 x2, E1 x3 (2 power-key + 1 dc1 .pub), V2 x1 (now an acknowledged deferral). Residual 7: dc0-edge-api (2, needs the excluded re-mint), S5 x3 (ruled, item 4 above), S6 conflation x1 (the SEC-020 defect, separate), E4 uncheckable x2 (Stage-5/6 rows). The register is NOT green and is not expected to be.
  • scripts/repo-lint.sh + full gauntlet: see the commit message for the recorded results.
  • .claude/settings.json (item 6): json.load parses, 44 allow entries, zero duplicates, ask 52 / deny 15 / env / hooks byte-unchanged.