Newer
Older
openstack-caracal-dc-dc / docs / audit / stage5-findings-20260729-successor.md

Stage-5 precondition findings -- successor session 2026-07-29

Findings raised by the successor session that opened after the 2026-07-29 close bookend (1cd45cd). Each is an OBSERVATION MEASURED BEFORE BEING PUT to the operator, per the skill's standing rule. Nothing here is adopted; nothing here is executed. Status authority remains docs/CURRENT-STATE.md.


F1 -- R7's per-DC Octavia PKI is HALF BUILT: the artifact PATHS are still single and unscoped

Class: RULED-IS-NOT-BUILT. Severity: HIGH -- silent destruction of a live DC's CA material, with no gate anywhere that can detect it.

The ruling

D-109 AMENDMENT (2026-07-27, R7), operator utterance "Per-DC independent Octavia PKI; fix the generator first (Recommended)". Decision text, verbatim: "each DC gets its own Octavia CA; no cross-DC amphora root-of-trust." Reuse was refused explicitly on posture -- one amphora CA private key across two clouds D-100 defines as independent would widen the SEC-004 exposure rather than contain it.

What was built (verified present)

  • The CA SUBJECT is per-DC: runbooks/phase-01-bundle-deploy.md:313-316 exports DC / DC_LABEL / VIP_OVERLAY; :328 uses /CN=${DC_LABEL} ....
  • The VIP gate reads the per-DC merged overlay rather than bundle.yaml's dc0 band (R7's explicit caveat), and the D-109 ruling note of 2026-07-29 added the IPv6 IP SAN (IP.1 v4 + IP.2 v6, both derived per-DC).

What was NOT built (the defect)

The artifact LOCATIONS are not parameterised by $DC at all:

Location path:line Value
Workspace root phase-01-bundle-deploy.md:293 WORKDIR="$HOME/octavia-pki"
Issuing CA dir :320 "$WORKDIR/issuing-ca"
Controller CA dir :342 "$WORKDIR/controller-ca"
Controller dir :364 "$WORKDIR/controller"
Overlay write :465,473,483 "$REPO/overlays/octavia-pki.yaml"

$DC appears in the subject and the VIP gate and NOWHERE in a path.

Failure mode, stated concretely

Generate dc0's PKI, deploy dc0. Later generate dc1's PKI per the same ruled step. 1.0-GEN.0 runs mkdir -p over the SAME three directories and every subsequent openssl write lands on dc0's files: passphrase.txt, issuing-ca.key.enc, issuing-ca.cert.pem, controller-ca.*, controller.key, controller.bundle.pem. 1.0-GEN.d then overwrites overlays/octavia-pki.yaml. dc0's issuing CA private key, its controller CA private key and both passphrases are destroyed on disk, and the single fixed-name overlay now carries dc1's CA.

Consequences beyond the loss itself:

  1. Any dc0 redeploy or juju config refresh reads the fixed overlay name at phase-01:34,86,148-153,180,241 and silently applies dc1's CA to dc0's Octavia -- a cross-DC shared amphora root-of-trust, precisely the outcome R7 refused.
  2. Controller-cert rotation for dc0 becomes impossible: the controller CA key it must be signed by is gone.
  3. Nothing detects it. The existence checks are DC-blind -- phase-01:86 is [ -f overlays/octavia-pki.yaml ] and scripts/pre-flight-checks.sh:65 sets OVL="overlays/octavia-pki.yaml". Both ask does a file exist, never whose CA is it. This is the assert-on-existence-not-content class the repo fixed in dc-mirror.sh check on 2026-07-27.

Why the ruling did not catch it

R7's own "Work implied" list enumerates: parameterise the CA subject per DC; DC-parameterise the VIP gate; then generate dc1's own PKI. It does not name the paths. The 2026-07-29 execution followed that list faithfully and completely -- the gap is in the list, not in the execution.

Corroborating contradiction already on a status surface: docs/CURRENT-STATE.md:990 describes overlays/octavia-pki.yaml as "per-DC generated CA material under the R7 amendment" -- a single unscoped path described as holding per-DC material.

Blast radius of the fix (measured, grep -rn)

runbooks/phase-01-bundle-deploy.md (5 path blocks + 6 gate/deploy references), scripts/pre-flight-checks.sh:65, .gitignore:40-41, creds-matrix.tsv:103-111 (see F2), bundle.yaml:40,649,654 (comments), README.md:35 (already stale -- says overlays/ holds one overlay; there are four tracked), and runbooks/dc-dc-phase4-juju-bundle-per-dc.md's deploy command.

Disposition

LOGGED, NOT EXECUTED (hard rule 1). The PRINCIPLE is already ruled by the D-109 amendment, so conformance is OPS under GA-R3 and needs no new D-number. The PATH SCHEME is a choice on a secret-handling surface and is put to the operator.


F2 -- the credential register cannot see a missing second Octavia PKI set

Class: a register cannot report the absence of something it has no ROW for. Severity: MEDIUM. Direct consequence of F1 and fixed with it.

Measured: creds-matrix.tsv:103-111 declares all nine Octavia PKI credentials with cardinality singleton, site-key -, host-role jumphost, mint-stage vr0-phase01:

octavia-issuing-ca-passphrase / -key / -cert
octavia-controller-ca-pass / -key / -cert
octavia-controller-key / -bundle
octavia-pki-overlay

scripts/creds-matrix.py:44 defines the vocabulary {singleton, per-site, per-DC, per-node, per-tenant}, and s5_per_dc_symmetry (:368-398) enforces both-DC existence only for rows whose cardinality is per-DC -- it selects on r.cardinality != "per-DC" and skips. So under R7 the register would match one PKI set and be structurally blind to whether the second exists. That is the exact class the D-137 register was built to eliminate, and it sits behind the BLOCKING preflight gate P5 (scripts/preflight.sh:144).

Baseline measured this session, so the delta is attributable: python3 scripts/creds-matrix.py --tier2 -> 82 rows, 15 check groups clean, 7 findings, exit 1. The nine Octavia rows are currently E0-SKIPPED because ~/octavia-pki/* does not exist ("a mint stage this deployment has not reached"), which is correct today and is exactly why the blindness has not yet bitten.

Correct fix is derived from the generator, not invented: the rows become per-DC with region-qualified site-keys and whatever per-DC paths F1 settles on. F2 must land BEFORE the artifacts are generated -- generating dc0's set against a singleton register produces a register that looks satisfied while dc1's set is absent.


F3 -- ~/octavia-pki/ and overlays/octavia-pki.yaml are ABSENT on this jumphost

Measured, existence only, no contents read: test -d ~/octavia-pki -> ABSENT; ls overlays/ -> dc-dc-ipv6-family-matrix.yaml dc-ha-scaleup.yaml vr1-dc0-vips.yaml vr1-dc1-machines.yaml vr1-dc1-vips.yaml (five tracked overlays, no octavia-pki). So this is generation FROM SCRATCH for both DCs, not a regeneration and not a reuse decision -- which removes the reuse-vs-regenerate choice dc-dc-phase4-juju-bundle-per-dc.md:204-209 frames as an operator call. Nothing exists to reuse.

Consequence for sequencing: F1 and F2 can both be fixed with ZERO risk to existing material, because there is no existing material. The window in which this fix is free closes the moment the first DC's PKI is generated.


F4 -- renaming the overlay would SILENTLY UN-IGNORE a CA private key

Class: coupling trap. Severity: HIGH, but only if F1 is fixed carelessly. It is recorded here so the F1 fix cannot be taken without it.

.gitignore:40 is an EXACT path, not a glob:

overlays/octavia-pki.yaml
octavia-pki/
passphrase.txt

F1's fix renames the overlay per DC (e.g. overlays/vr1-dc0-octavia-pki.yaml). That new name does not match .gitignore:40, so the moment the generator writes it, a file containing base64 CA private-key blobs plus the issuing-CA passphrase IN PLAINTEXT becomes an untracked-but-committable file -- in a repo SEC-004 records as currently PUBLIC. docs/design-decisions.md (R7 posture paragraph) states that this gitignore is "the only thing standing between a CA private key and publication"; creds-matrix-notes.md:182-185 (n-overlay-in-clone) says the same.

The F1 edit MUST therefore widen line 40 to a glob covering every per-DC name (overlays/*octavia-pki.yaml) in the same commit that renames anything, and the generator should verify its own output is ignored (git check-ignore -q "$OUT") before writing key material into it -- an assert-on-content gate rather than a convention anybody must remember. octavia-pki/ at line 41 is already a directory glob and covers ~/octavia-pki/<dc>/ without change.


F5 -- preflight's DC selector does not reach P4, and P4 is dc0-frozen

Class: gate does not measure what the operator asked it to measure. Severity: MEDIUM. Adjacent to, but not covered by, the 2026-07-29 P2 DC-awareness fix.

Measured:

  • scripts/preflight.sh:99 sets DC="${DC:-vr1-dc0}" and never exports it. Invoked as DC=vr1-dc1 bash scripts/preflight.sh the variable arrives in the environment and survives into children; invoked BARE it is a plain shell variable and children see nothing. So propagation depends on the caller's invocation form -- two call shapes, two different behaviours, neither documented.
  • It does not matter yet, because scripts/pre-flight-checks.sh (P4) never reads DC at all: grep -n 'DC' returns exactly one hit, inside a comment at :80. Its own CHECK 1 comment admits the state -- "this check is still dc0-only -- it never calls lib_net_select_dc" -- but that admission is scoped to CHECK 1, while CHECK 0's OVL="overlays/octavia-pki.yaml" (:65) is equally unscoped.

Net effect: DC=vr1-dc1 bash scripts/preflight.sh runs a DC-aware P2 and a dc0-frozen P4 in the same invocation and prints one combined verdict. A dc1 operator reading PREFLIGHT output cannot tell which halves measured dc1.

Also note preflight.sh:102 hardcodes overlays/octavia-pki.yaml in P2's overlay assembly loop, so it inherits F1's rename and must change with it.


F7 -- office1-tailscale is labelled a SUBNET ROUTER and advertises NO ROUTES

Class: RULED-IS-NOT-BUILT / wrong record. Severity: MEDIUM operationally, but it is the THIRD instance in one session of a record that points somewhere real and says something false.

Raised because the operator reported Permission denied (publickey) when trying to reach voffice1 "directly from my workstation via tailscale". Measured rather than assumed:

  • tailscale is absent from voffice1 AND from vcloud -- no binary in any standard location, no /var/lib/tailscale, no interface, service not running. Checked with explicit path probes after an initial command -v, because "tool absent from PATH" reported as "feature absent" is a trap this repo has hit repeatedly.
  • The tailnet node is a SEPARATE LXD guest INSIDE voffice1: office1-tailscale, RUNNING, 100.64.0.53 / fd7a:115c:a1e0::35, site leg 10.10.1.11.
  • tailscale status --json -> AdvertisedRoutes: <none>. So nothing on the tailnet can reach 10.10.0.20 at all. There is no route, and there never was one.

The two records that made the belief look supported:

  1. ~/.ssh/config.d/vr1-sites labels the stanza Host office1-tailscale # Office1 subnet router (D-107). It is not routing anything.
  2. The operating skill's routing table states "This is the vcloud path; the workstation reaches the same over the tailnet (D-107)". D-107 is titled "Airgap posture, per-DC artifact mirror, and NTP (VR1)" and rules nothing about a workstation tailnet path -- the citation is real but does not support the claim. Exactly the "a citation is an existence claim; only its CONTENT is evidence" rule.

FIXED (operational, outside the repo): ~/.ssh/config.d/vr1-sites on vcloud now matches Host voffice1 10.10.0.20, so the raw IP resolves to the office1 service key with IdentitiesOnly yes instead of falling through to the ~/.ssh/id_* defaults -- which was the whole cause of the original denial. Verified both forms connect, with a NEGATIVE CONTROL (ssh -G 10.10.0.99 still reads identitiesonly no) proving the key is not offered to unrelated hosts.

RULED (operator, 2026-07-29): option (a), ProxyJump. The workstation reaches voffice1 through vcloud rather than over the tailnet. Making office1-tailscale advertise 10.10.0.0/24 was presented as option (b) and NOT taken -- it exposes the isolated site net onto the tailnet, which is a posture change against the airgap subject D-107 actually governs, and would need its own ruling.

QUEUED, NOT EXECUTED: correct the skill's routing-table citation, and correct or remove the "subnet router" label in the ssh stanza. The skill sweep is deferred to the Stage-5 close by standing operator direction, so this is input to it.

WORTH A TARGETED SWEEP: three wrong records in one session (D-124's MAC sourcing instruction, the skill's tailnet citation, this label), and two of the three were found only because something adjacent was being fixed. Nothing in this repo gates prose, so none of them could have been caught by a check.