Newer
Older
openstack-caracal-dc-dc / docs / audit / queued-findings-20260730.txt
QUEUED FINDINGS / TRANSCRIPT-ONLY MATERIAL -- 2026-07-30 successor session
==========================================================================
Written BEFORE the GA-R4 close bookend, on operator instruction: "I'm worried that the fixes
we made to the defective commands will be lost when we close this session ... review and
confirm that there is nothing that should be saved will be lost during the bookend."

Precedent: docs/audit/queued-findings-20260726.txt / -20260727.txt / -20260729.txt.
A bookend is a BOUNDED summary (GA-R4 rule 1, 15-line cap). Anything that must survive has to
be on a durable surface BEFORE it, not inside it.

THE CONCERN WAS WELL FOUNDED. The operator's corrected commands -- the ones that ACTUALLY
produced both DCs' live PKI -- were NOT in the repo when the question was asked. They had been
reviewed in conversation, judged equivalent, and never folded in. Four items, now landed:

  (1) PORTABLE base64. GEN.d shipped `base64 -w0 <file>`, a GNU-only flag. What actually ran
      was `base64 < file | tr -d '\r\n'` -- identical single-line output, portable. The runbook
      described a command NOBODY EXECUTED. Adopted verbatim, with the reason recorded inline.
  (2) VIP_OVERLAY guard. The operator added `VIP_OVERLAY="${VIP_OVERLAY:?...}"` to GEN.c so it
      fails fast instead of reaching the readability test with an empty path. Adopted.
  (3) THE HEREDOC COLUMN-1 REQUIREMENT (F8). This is the one that cost real time live. It is now
      a CAUTION HEADING above GEN.c, not a footnote, because the failure is silent: an indented
      terminator yields a config with no [alt_names] and therefore a controller certificate with
      NO SANs, while every openssl command still prints OK.
  (4) ${DC_LABEL} guard -- F8's sibling. GEN.a/GEN.b guarded ${DC:?} but used ${DC_LABEL}
      unguarded, so an unset label bakes a DC-less CA subject into a 10-YEAR CA with no error
      anywhere. Both call sites now `${DC_LABEL:?}`.

DELIBERATELY NOT DONE, with the reason: GEN.c's heredoc was NOT rewritten as printf lines. That
would be more paste-proof, but it is an untested rewrite of a step that mints 10-year CA
material and cannot be exercised end-to-end from an agent session (the guard blocks the mint by
design). The residual risk is bounded instead -- octavia-pki.sh A9 asserts the SAN set and
harness T2 pins it, so a SAN-less cert now FAILS a gate rather than shipping. Converting to
printf is a reasonable follow-up for whoever can run a real generation.

--------------------------------------------------------------------------
RULING-SHAPED, NOT YET PUT TO THE OPERATOR
--------------------------------------------------------------------------
Q1. THE PER-DC DNS LABEL IS AMBIGUOUS INSIDE A RULED DECISION, and it now blocks any FQDN cert.
    D-008 fixes `<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local`. D-106:2563 instantiates
    VR1 as `...omega.dc1.vr1...` and `...dc2.vr1...`. The substrate's DCs are `vr1-dc0` and
    `vr1-dc1`. So substrate `vr1-dc1` is `dc1` by TOKEN or `dc2` by POSITION -- the DC1/DC2
    ambiguity item 3.1 retired elsewhere, surviving inside a ruling, where it decides
    CERTIFICATE IDENTITY. Both live certs currently carry `dc0.vr0`, the VR0 region, which is
    wrong for a VR1 DC under either reading.
    ENFORCED, NOT FILED: octavia-pki.sh A12 REFUSES (exit 3) on the label once
    `os-public-hostname` is set, rather than blessing a name it cannot validate. So the ruling
    is now blocking rather than forgettable.

Q2. D-137 OPEN FORK 1 (enforcement strength) -- still unruled, and this session produced
    unusually good evidence for it. The guard blocked CORRECTLY once (the PKI mint) and MISFIRED
    five times: twice on prose (a filename appearing in a register row; a secret directory path
    in a commit message), once on a `chmod` that strictly IMPROVED posture, and twice on `awk`
    reading a runbook. Option (c) -- guard refuses secret-writing shapes OUTSIDE a sanctioned
    minter -- is the option that makes the carve-out deliberate and narrow instead of incidental.
    Relevant: `scripts/octavia-pki.sh generate` is deliberately unimplemented pending this.

--------------------------------------------------------------------------
FINDINGS WITH NO OTHER HOME
--------------------------------------------------------------------------
F10. MINT-REF LINE NUMBERS DRIFT SILENTLY, AND S4 CANNOT SEE IT. `creds-matrix.py`'s S4 asserts
     only that a `runbook:<path>:<line>` ref is WITHIN EOF. Any edit that adds lines ABOVE an
     anchor leaves the ref pointing at unrelated text and S4 still reports clean. This bit
     THREE times in two sessions (the F1 rename, the backup step, today's F8 fixes); each time
     it was caught by hand, never by a gate. All 24 octavia refs were re-anchored today.
     Worth noting the correct re-anchor is a SINGLE-PASS mapping keyed by row id, not sequential
     seds: 391 was simultaneously an OLD value (issuing cert) and a NEW value (issuing secret),
     so replace-one-then-the-other corrupts the first.
     FIX SHAPE (not built): have S4 additionally assert the referenced line still MATCHES a
     stored fingerprint, or move mint-refs to a stable anchor (a named step id) rather than a
     line number.

F11. `~/.ssh/config.d/vr1-sites` LABELS `office1-tailscale` "Office1 subnet router (D-107)" AND
     IT ADVERTISES NO ROUTES. Measured `AdvertisedRoutes: <none>`. Also: the operating skill's
     routing table claims the workstation reaches voffice1 over the tailnet citing D-107, which
     is titled "Airgap posture, per-DC artifact mirror, and NTP" and rules nothing of the sort.
     Both queued to the deferred skill sweep. The ssh-config label is OUTSIDE the repo, so it
     will not be caught by any gate -- recorded here so it is not lost.

F12. COMMANDS IN THIS REPO ARE CWD-SENSITIVE IN A WAY THAT BITES REPEATEDLY. The operator's
     `verify` failed twice with "No such file or directory" because the shell was in
     `~/octavia-pki/vr1-dc0` rather than the repo; the session then made the SAME class of error
     four consecutive times, describing a `cd` it was not actually sending. Worth a runbook
     convention (absolute `$REPO/scripts/...` in pasteable blocks) rather than trusting cwd.

F13. THE TWO LIVE INNER tfstates ARE MODE 664 -- group-writable, and they are the authority
     `tofu` trusts, so a rewrite can make it destroy or orphan real resources. Not fixed:
     tightening needs its own gated change PLUS proof the libvirt provider preserves the mode
     across the rewrite it performs on every apply. Recorded in n-tfstate-backup too.

--------------------------------------------------------------------------
MEASURED VALUES WORTH KEEPING (all also in CURRENT-STATE)
--------------------------------------------------------------------------
- Octavia PKI, both DCs: `octavia-pki.sh verify` PASS 29/0 (26 before A12 was added).
- Inner tfstate serials at backup time: vr1-dc0 = 9, vr1-dc1 = 4; 29 resources each.
  sha256 head: dc0 c16995a1a9a82219, dc1 fbd506360ba4720f -- unchanged since session open,
  so the backups match what tofu actually uses.
- P5: 7 findings on the jumphost, 6 on the headend. ZERO E2 and ZERO E3 on any octavia or
  tfstate artifact. The residual set is entirely pre-existing (dc0-edge-api, three power-key
  asymmetries, the maas-region-admin principal conflation, two uncheckable rows).
- Gauntlet ALL GREEN (89) in the default locale AND under LC_ALL=C, identical.
- Harnesses added/extended today: octavia-pki 21/21 (new), creds-matrix 65/65,
  pre-flight-checks 31/31 (new), preflight 26/26, dc-selector 69.

--------------------------------------------------------------------------
CONFIRMED NOT AT RISK
--------------------------------------------------------------------------
- Every command that generated or backed up the PKI is in `phase-01-bundle-deploy.md`
  (1.0-GEN.0/.a/.b/.c/.d/.e) in the form that ACTUALLY RAN.
- The tfstate backup procedure is `dc-dc-phase2-tofu-dc-substrate.md` step 13.
- The backup artifacts themselves are on disk, declared in the register, and their restores
  were PROVEN (PKI 12/12 sha256 vs live; tfstate byte-identical, parses, correct serial).
- All 31 commits are pushed to origin. Nothing in the repo lives only on this jumphost.
- The workstation ProxyJump config is operator-side and intentionally NOT in the repo; F11
  records the reasoning it depends on.

--------------------------------------------------------------------------
APPENDED CORRECTION 2026-07-30 (DOCFIX-205) -- Q1 ABOVE IS WITHDRAWN
--------------------------------------------------------------------------
Appended, not rewritten, per the precedent set by stage4-mirror-gate-20260727.txt (a wrong
causal claim superseded by an appended correction). The Q1 entry above stands as the record of
what was believed; this block records what was MEASURED when it was put to the operator.

Q1 IS NOT A RULING QUESTION. It was ruled on 2026-07-13 by D-117, whose Status line names the
supersession directly -- "Supersedes ... and the D-106 `dc1`/`dc2` zone labels" -- and whose
amendment rules the replacement: "The repo's `dc1`/`dc2` labels are retired in favour of
`dc0`/`dc1`." So substrate `vr1-dc0` -> `dc0`, `vr1-dc1` -> `dc1`, and the VR1 zones are
`omega.dc0.vr1.cloud.neumatrix.local` / `omega.dc1.vr1.cloud.neumatrix.local`. D-008's
four-label shape is unchanged; D-119's single `vr1-dc0` token is NOT adopted for DNS.

TWO CLAIMS IN THE Q1 ENTRY ABOVE ARE WRONG AND WOULD MISLEAD A READER:
  - "it now blocks any FQDN cert" -- it did not. Measured live on the headend, A12 was in its
    INERT branch and PASSED on BOTH DCs, because `os-public-hostname` is set in no deploy
    artifact (B5 IP-only). It would have armed only at D-106's Stage-7 work.
  - "ENFORCED, NOT FILED: A12 REFUSES (exit 3)" -- that refusal NO LONGER EXISTS. A12 now
    ASSERTS the full expected zone, derived from the site token (`${SITE%%-*}` region,
    `${SITE#*-}` DC label). Harness 21/21 -> 23/23.

WHY IT RESURFACED (the reusable half): D-117 ruled that the ADOPTED decision texts
D-101/D-106/D-111/D-115 be ANNOTATED in place. ZERO of the four carried the annotation, so
D-106 read as though its retired labels were live. It stayed invisible because D-117's OWN
Status line claimed "FULLY EXECUTED BY D-119", while D-119 scopes its discharge to the
SELECTOR half only. All four are annotated and D-117's Status is corrected.

Q2 (D-137 open fork 1) IS UNAFFECTED and still stands as written.

Evidence: docs/audit/docfix205-d117-annotation-20260730.txt; changelog
docs/changelog-20260730-docfix205-d117-annotation.md. Status ONLY in CURRENT-STATE.md.