============================================================================== SAVEGAME SWEEP -- 2026-08-04 session (ovn-central cert reeval -> root cause -> fix PROVEN live -> wired for redeploy; plus the 08-03 provider-bundle-check reconcile that opened this working session). Method (ruled 2026-07-31): read the whole session, enumerate every finding / decision / measurement / mistake, grep each against repo surfaces. Hit = ALREADY ON SURFACE. No hit = FIRST SURFACE, would be lost on /clear. Status claims live in docs/CURRENT-STATE.md ONLY. ============================================================================== -------------------------------------------------------------------------------------- FIRST SURFACE -- existed ONLY in the transcript. Listed first, by consequence. -------------------------------------------------------------------------------------- F1. >>> SEC-033 was NOT filed on the security-ledger register. <<< The relation-databag exposure (juju show-unit on ANY tls-certificates consumer prints vault's GLOBAL-CLIENT private key in cleartext to any model reader) was written into CURRENT-STATE + the reeval doc PROSE, but the register of record is docs/security-ledger.md and it has no SEC-033 row (ledger-scan: SEC still 28, highest SEC-032). This savegame ADDS the SEC-033 row (graduating the observation to its surface -- GA-R7). grep 'SEC-033' docs/security-ledger.md before this session: 0 hits -> FIRST SURFACE on the register. F2. >>> LIVE dc0 MODEL DRIFTED FROM CURRENT-STATE via this session's reversible tests. <<< CURRENT-STATE describes the before-state (certificates:142, all 3 ovn-central units broken). MEASURED now: the certificates relation is certificates:145 (bounced 3x during the fix proof), and ovn-central/0 carries an ISSUED vault server cert + /etc/ovn/{cert_host,key_host, ovn-central.crt} (from the single-unit proof; it persists) while /1,/2 are broken. Bindings were reverted to all-metal-internal (= before-state). This is teardown-bound (the redeploy wipes it) but recorded so the next session does not read "/0 has a cert, relation is :145" as a new fault. Measurement wins (GA-R1 C2); no doc correction owed since it is a transient test residue on a teardown-bound model, not a config-of-record claim. F3. >>> REDEPLOY PREREQ: the dc0 rack ~/repo-stage does NOT carry the new scripts/dc-node-etchosts.sh. <<< Step 1.2b runs it FROM the rack (D-138), so before the redeploy the rack stage must be updated (scp + sha256-verify, per the D-138 staged-copy rule) OR the step run from wherever the juju client + repo are co-located. Not staged as of close. F4. Developer-doc research finding worth keeping (partially on surface): juju bindings are explicitly NOT an L3 routing mechanism (Juju charm-guide) and default-route selection is non-deterministic ("first space", LP#1781856) -- this is WHY moving ovn-central's default binding did not restore a route and the CN stayed empty. Captured in the reeval doc's CORRECTION/RESOLUTION and CURRENT-STATE; recorded here so the reasoning is not lost if those are later condensed. -------------------------------------------------------------------------------------- ALREADY ON SURFACE (grep-confirmed) -- recorded for completeness. -------------------------------------------------------------------------------------- - provider-bundle-check reconciled to D-141 (v4-only) + FN1/FN2/FN3: commit 5c9da99; docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt (AMENDED block); changelog-20260803. - ovn-central root cause (isolated-plane metal-internal, CN=get_hostname(metal-internal addr)), LP #2044324 NO MATCH, vault signs any non-empty CN, OVN no CN-content rule: reeval doc + CURRENT-STATE + changelog-20260804. - rdns_mode fix VOID / dns_servers fix VOID / binding approach refuted (3 configs): reeval doc CORRECTION+RESOLUTION, remediation-plan VOID banner, sweep-misses F1 correction, CURRENT-STATE. - Fix PROVEN (controlled single-unit test: cert issues only with the /etc/hosts entry): reeval RESOLUTION + changelog-20260804 Item 2. - NEW mechanism scripts/dc-node-etchosts.sh + tests/dc-node-etchosts (harness 9/9) + runbook Step 1.2b; a NEW mechanism borrowing D-008's shape, not D-008: committed 9b49b84. - F2 (hacluster stale IPv6 VIP CIB, 4 units; designate live vip still v6; v6 naming-collision defect) + F3 (octavia downstream of ovn-central; neutron-api false-green): stage5-sweep-misses (committed); both resolve via the clean redeploy. - app STAYS on metal-internal (D-052 data-type placement): reeval RESOLUTION + CURRENT-STATE. -------------------------------------------------------------------------------------- MANDATORY FIVE (structurally invisible) -------------------------------------------------------------------------------------- 1. Gitignored settings.local.json: allow 313 / ask 11 / deny 0 -- UNCHANGED this session (was 313 at 08-03 close). Broad ssh allows present as before (Bash(ssh voffice1 *), the -J voffice1 172.31.0.2/.6 dc-rack forms). No new allow rule added this session. Same broad-ssh posture the 08-03 sweep flagged; still gitignored (only recovery copy). 2. Dangling refs: new commits cite scripts/dc-node-etchosts.sh, tests/dc-node-etchosts, runbooks/phase-01-bundle-deploy.md Step 1.2b, the audit docs -- all resolve (repo-lint 0 fail). 3. Ruling fidelity: NO new GA-R5 rulings this session (all OPS: scripts/runbook/audit). No new D-/DOCFIX-/BUNDLEFIX numbers assigned (next-free unchanged D-142/DOCFIX-209/BUNDLEFIX-053). 4. As-executed log: not opened (run-logged.sh interactive-only). This session's mutations were reversible live tests, all reverted; the durable deliverables are committed. Gap declared here. 5. Contradiction detector: F2 above (live model vs CURRENT-STATE before-state) is the residue; no config-of-record contradiction (deploy input verified v4-only clean; the rdns_mode/dns_servers corrections already landed in the docs this session). -------------------------------------------------------------------------------------- OWNED (mistakes this session) -------------------------------------------------------------------------------------- - Root cause called WRONG THREE TIMES before it was right: (i) "metal-internal rdns_mode disabled" -- refuted, rdns_mode=2 on all planes; (ii) "dns_servers advertises unroutable 10.12.4.6" -- refuted, both links set to the reachable resolver still failed; (iii) landed on the real cause (isolated plane, CN from metal-internal addr). Each corrected by MEASURING before committing further -- the discipline worked, but the first two shipped into committed records and needed VOID/CORRECTION passes. - Explained away a null resolvectl result ("resolved binds to eth1") as a rationalization instead of measuring; advisor caught it; the discriminating test then established the real constraint. - TWO false-negative binding tests: concluded the binding fix failed without bouncing the relation to re-fire the request (Test 1), then mis-attributed which address the CN uses; only the direct get_hostname measurement + a fresh bounce settled it. Same false-negative shape twice. - Over-claimed "metal-admin-only DNS" (metal-internal is the outlier that matters, not a posture) and "the default binding is not the lever" (it is ON the causal path) -- both corrected in-record. - Guessed dc1's metal-internal CIDR (10.12.68) writing the harness; the harness caught it (real value 10.12.72.0/22, from lib-net). Fixed; a case that could not have passed on the guess. - Nearly built the cloudinit wiring on the get_hostname-layer proof WITHOUT confirming a server cert actually issues (the exact false-negative shape); advisor made the cert-issuance test blocking; it passed and closed the chain.