Newer
Older
openstack-caracal-dc-dc / docs / audit / queued-findings-20260804-ovn-cert-fix.txt
==============================================================================
SAVEGAME SWEEP -- 2026-08-04 session (ovn-central cert reeval -> root cause ->
fix PROVEN live -> wired for redeploy; plus the 08-03 provider-bundle-check
reconcile that opened this working session).
Method (ruled 2026-07-31): read the whole session, enumerate every finding /
decision / measurement / mistake, grep each against repo surfaces.
Hit = ALREADY ON SURFACE. No hit = FIRST SURFACE, would be lost on /clear.
Status claims live in docs/CURRENT-STATE.md ONLY.
==============================================================================

--------------------------------------------------------------------------------------
FIRST SURFACE -- existed ONLY in the transcript. Listed first, by consequence.
--------------------------------------------------------------------------------------

F1. >>> SEC-033 was NOT filed on the security-ledger register. <<< The relation-databag
    exposure (juju show-unit on ANY tls-certificates consumer prints vault's GLOBAL-CLIENT
    private key in cleartext to any model reader) was written into CURRENT-STATE + the reeval
    doc PROSE, but the register of record is docs/security-ledger.md and it has no SEC-033 row
    (ledger-scan: SEC still 28, highest SEC-032). This savegame ADDS the SEC-033 row (graduating
    the observation to its surface -- GA-R7). grep 'SEC-033' docs/security-ledger.md before this
    session: 0 hits -> FIRST SURFACE on the register.

F2. >>> LIVE dc0 MODEL DRIFTED FROM CURRENT-STATE via this session's reversible tests. <<<
    CURRENT-STATE describes the before-state (certificates:142, all 3 ovn-central units broken).
    MEASURED now: the certificates relation is certificates:145 (bounced 3x during the fix proof),
    and ovn-central/0 carries an ISSUED vault server cert + /etc/ovn/{cert_host,key_host,
    ovn-central.crt} (from the single-unit proof; it persists) while /1,/2 are broken. Bindings
    were reverted to all-metal-internal (= before-state). This is teardown-bound (the redeploy
    wipes it) but recorded so the next session does not read "/0 has a cert, relation is :145" as
    a new fault. Measurement wins (GA-R1 C2); no doc correction owed since it is a transient test
    residue on a teardown-bound model, not a config-of-record claim.

F3. >>> REDEPLOY PREREQ: the dc0 rack ~/repo-stage does NOT carry the new
    scripts/dc-node-etchosts.sh. <<< Step 1.2b runs it FROM the rack (D-138), so before the
    redeploy the rack stage must be updated (scp + sha256-verify, per the D-138 staged-copy rule)
    OR the step run from wherever the juju client + repo are co-located. Not staged as of close.

F4. Developer-doc research finding worth keeping (partially on surface): juju bindings are
    explicitly NOT an L3 routing mechanism (Juju charm-guide) and default-route selection is
    non-deterministic ("first space", LP#1781856) -- this is WHY moving ovn-central's default
    binding did not restore a route and the CN stayed empty. Captured in the reeval doc's
    CORRECTION/RESOLUTION and CURRENT-STATE; recorded here so the reasoning is not lost if those
    are later condensed.

--------------------------------------------------------------------------------------
ALREADY ON SURFACE (grep-confirmed) -- recorded for completeness.
--------------------------------------------------------------------------------------
- provider-bundle-check reconciled to D-141 (v4-only) + FN1/FN2/FN3: commit 5c9da99;
  docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt (AMENDED block); changelog-20260803.
- ovn-central root cause (isolated-plane metal-internal, CN=get_hostname(metal-internal addr)),
  LP #2044324 NO MATCH, vault signs any non-empty CN, OVN no CN-content rule: reeval doc +
  CURRENT-STATE + changelog-20260804.
- rdns_mode fix VOID / dns_servers fix VOID / binding approach refuted (3 configs): reeval doc
  CORRECTION+RESOLUTION, remediation-plan VOID banner, sweep-misses F1 correction, CURRENT-STATE.
- Fix PROVEN (controlled single-unit test: cert issues only with the /etc/hosts entry): reeval
  RESOLUTION + changelog-20260804 Item 2.
- NEW mechanism scripts/dc-node-etchosts.sh + tests/dc-node-etchosts (harness 9/9) + runbook
  Step 1.2b; a NEW mechanism borrowing D-008's shape, not D-008: committed 9b49b84.
- F2 (hacluster stale IPv6 VIP CIB, 4 units; designate live vip still v6; v6 naming-collision
  defect) + F3 (octavia downstream of ovn-central; neutron-api false-green): stage5-sweep-misses
  (committed); both resolve via the clean redeploy.
- app STAYS on metal-internal (D-052 data-type placement): reeval RESOLUTION + CURRENT-STATE.

--------------------------------------------------------------------------------------
MANDATORY FIVE (structurally invisible)
--------------------------------------------------------------------------------------
1. Gitignored settings.local.json: allow 313 / ask 11 / deny 0 -- UNCHANGED this session (was
   313 at 08-03 close). Broad ssh allows present as before (Bash(ssh voffice1 *), the -J voffice1
   172.31.0.2/.6 dc-rack forms). No new allow rule added this session. Same broad-ssh posture the
   08-03 sweep flagged; still gitignored (only recovery copy).
2. Dangling refs: new commits cite scripts/dc-node-etchosts.sh, tests/dc-node-etchosts,
   runbooks/phase-01-bundle-deploy.md Step 1.2b, the audit docs -- all resolve (repo-lint 0 fail).
3. Ruling fidelity: NO new GA-R5 rulings this session (all OPS: scripts/runbook/audit). No new
   D-/DOCFIX-/BUNDLEFIX numbers assigned (next-free unchanged D-142/DOCFIX-209/BUNDLEFIX-053).
4. As-executed log: not opened (run-logged.sh interactive-only). This session's mutations were
   reversible live tests, all reverted; the durable deliverables are committed. Gap declared here.
5. Contradiction detector: F2 above (live model vs CURRENT-STATE before-state) is the residue;
   no config-of-record contradiction (deploy input verified v4-only clean; the rdns_mode/dns_servers
   corrections already landed in the docs this session).

--------------------------------------------------------------------------------------
OWNED (mistakes this session)
--------------------------------------------------------------------------------------
- Root cause called WRONG THREE TIMES before it was right: (i) "metal-internal rdns_mode
  disabled" -- refuted, rdns_mode=2 on all planes; (ii) "dns_servers advertises unroutable
  10.12.4.6" -- refuted, both links set to the reachable resolver still failed; (iii) landed on
  the real cause (isolated plane, CN from metal-internal addr). Each corrected by MEASURING before
  committing further -- the discipline worked, but the first two shipped into committed records and
  needed VOID/CORRECTION passes.
- Explained away a null resolvectl result ("resolved binds to eth1") as a rationalization instead
  of measuring; advisor caught it; the discriminating test then established the real constraint.
- TWO false-negative binding tests: concluded the binding fix failed without bouncing the relation
  to re-fire the request (Test 1), then mis-attributed which address the CN uses; only the direct
  get_hostname measurement + a fresh bounce settled it. Same false-negative shape twice.
- Over-claimed "metal-admin-only DNS" (metal-internal is the outlier that matters, not a posture)
  and "the default binding is not the lever" (it is ON the causal path) -- both corrected in-record.
- Guessed dc1's metal-internal CIDR (10.12.68) writing the harness; the harness caught it
  (real value 10.12.72.0/22, from lib-net). Fixed; a case that could not have passed on the guess.
- Nearly built the cloudinit wiring on the get_hostname-layer proof WITHOUT confirming a server
  cert actually issues (the exact false-negative shape); advisor made the cert-issuance test
  blocking; it passed and closed the chain.