# Stage-5 precondition findings -- successor session 2026-07-29

Findings raised by the successor session that opened after the 2026-07-29 close
bookend (`1cd45cd`). Each is an OBSERVATION MEASURED BEFORE BEING PUT to the
operator, per the skill's standing rule. Nothing here is adopted; nothing here is
executed. Status authority remains `docs/CURRENT-STATE.md`.

---

## F1 -- R7's per-DC Octavia PKI is HALF BUILT: the artifact PATHS are still single and unscoped

**Class: RULED-IS-NOT-BUILT.** Severity: HIGH -- silent destruction of a live DC's
CA material, with no gate anywhere that can detect it.

### The ruling

D-109 AMENDMENT (2026-07-27, R7), operator utterance *"Per-DC independent Octavia
PKI; fix the generator first (Recommended)"*. Decision text, verbatim: **"each DC
gets its own Octavia CA; no cross-DC amphora root-of-trust."** Reuse was refused
explicitly on posture -- one amphora CA private key across two clouds D-100
defines as independent would widen the SEC-004 exposure rather than contain it.

### What was built (verified present)

- The CA SUBJECT is per-DC: `runbooks/phase-01-bundle-deploy.md:313-316` exports
  `DC` / `DC_LABEL` / `VIP_OVERLAY`; `:328` uses `/CN=${DC_LABEL} ...`.
- The VIP gate reads the per-DC merged overlay rather than `bundle.yaml`'s dc0
  band (R7's explicit caveat), and the D-109 ruling note of 2026-07-29 added the
  IPv6 IP SAN (`IP.1` v4 + `IP.2` v6, both derived per-DC).

### What was NOT built (the defect)

The artifact LOCATIONS are not parameterised by `$DC` at all:

| Location | path:line | Value |
|---|---|---|
| Workspace root | `phase-01-bundle-deploy.md:293` | `WORKDIR="$HOME/octavia-pki"` |
| Issuing CA dir | `:320` | `"$WORKDIR/issuing-ca"` |
| Controller CA dir | `:342` | `"$WORKDIR/controller-ca"` |
| Controller dir | `:364` | `"$WORKDIR/controller"` |
| Overlay write | `:465,473,483` | `"$REPO/overlays/octavia-pki.yaml"` |

`$DC` appears in the subject and the VIP gate and NOWHERE in a path.

### Failure mode, stated concretely

Generate dc0's PKI, deploy dc0. Later generate dc1's PKI per the same ruled step.
`1.0-GEN.0` runs `mkdir -p` over the SAME three directories and every subsequent
`openssl` write lands on dc0's files: `passphrase.txt`, `issuing-ca.key.enc`,
`issuing-ca.cert.pem`, `controller-ca.*`, `controller.key`, `controller.bundle.pem`.
`1.0-GEN.d` then overwrites `overlays/octavia-pki.yaml`. **dc0's issuing CA private
key, its controller CA private key and both passphrases are destroyed on disk**, and
the single fixed-name overlay now carries dc1's CA.

Consequences beyond the loss itself:

1. Any dc0 redeploy or `juju config` refresh reads the fixed overlay name at
   `phase-01:34,86,148-153,180,241` and silently applies **dc1's CA** to dc0's
   Octavia -- a cross-DC shared amphora root-of-trust, precisely the outcome R7
   refused.
2. Controller-cert rotation for dc0 becomes impossible: the controller CA key it
   must be signed by is gone.
3. Nothing detects it. The existence checks are DC-blind -- `phase-01:86` is
   `[ -f overlays/octavia-pki.yaml ]` and `scripts/pre-flight-checks.sh:65` sets
   `OVL="overlays/octavia-pki.yaml"`. Both ask *does a file exist*, never *whose
   CA is it*. This is the assert-on-existence-not-content class the repo fixed in
   `dc-mirror.sh check` on 2026-07-27.

### Why the ruling did not catch it

R7's own "Work implied" list enumerates: parameterise the CA subject per DC;
DC-parameterise the VIP gate; then generate dc1's own PKI. It does not name the
paths. The 2026-07-29 execution followed that list faithfully and completely --
the gap is in the list, not in the execution.

Corroborating contradiction already on a status surface: `docs/CURRENT-STATE.md:990`
describes `overlays/octavia-pki.yaml` as "per-DC generated CA material under the R7
amendment" -- a single unscoped path described as holding per-DC material.

### Blast radius of the fix (measured, `grep -rn`)

`runbooks/phase-01-bundle-deploy.md` (5 path blocks + 6 gate/deploy references),
`scripts/pre-flight-checks.sh:65`, `.gitignore:40-41`, `creds-matrix.tsv:103-111`
(see F2), `bundle.yaml:40,649,654` (comments), `README.md:35` (already stale --
says `overlays/` holds one overlay; there are four tracked), and
`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`'s deploy command.

### Disposition

LOGGED, NOT EXECUTED (hard rule 1). The PRINCIPLE is already ruled by the D-109
amendment, so conformance is OPS under GA-R3 and needs no new D-number. The PATH
SCHEME is a choice on a secret-handling surface and is put to the operator.

---

## F2 -- the credential register cannot see a missing second Octavia PKI set

**Class: a register cannot report the absence of something it has no ROW for.**
Severity: MEDIUM. Direct consequence of F1 and fixed with it.

Measured: `creds-matrix.tsv:103-111` declares all nine Octavia PKI credentials with
cardinality **`singleton`**, site-key `-`, host-role `jumphost`, mint-stage
`vr0-phase01`:

    octavia-issuing-ca-passphrase / -key / -cert
    octavia-controller-ca-pass / -key / -cert
    octavia-controller-key / -bundle
    octavia-pki-overlay

`scripts/creds-matrix.py:44` defines the vocabulary
`{singleton, per-site, per-DC, per-node, per-tenant}`, and `s5_per_dc_symmetry`
(`:368-398`) enforces both-DC existence **only for rows whose cardinality is
`per-DC`** -- it selects on `r.cardinality != "per-DC"` and skips. So under R7 the
register would match one PKI set and be structurally blind to whether the second
exists. That is the exact class the D-137 register was built to eliminate, and it
sits behind the BLOCKING preflight gate P5 (`scripts/preflight.sh:144`).

Baseline measured this session, so the delta is attributable:
`python3 scripts/creds-matrix.py --tier2` -> **82 rows, 15 check groups clean,
7 findings, exit 1**. The nine Octavia rows are currently E0-SKIPPED because
`~/octavia-pki/*` does not exist ("a mint stage this deployment has not reached"),
which is correct today and is exactly why the blindness has not yet bitten.

Correct fix is derived from the generator, not invented: the rows become `per-DC`
with region-qualified site-keys and whatever per-DC paths F1 settles on. **F2 must
land BEFORE the artifacts are generated** -- generating dc0's set against a
`singleton` register produces a register that looks satisfied while dc1's set is
absent.

---

## F3 -- `~/octavia-pki/` and `overlays/octavia-pki.yaml` are ABSENT on this jumphost

Measured, existence only, no contents read: `test -d ~/octavia-pki` -> ABSENT;
`ls overlays/` -> `dc-dc-ipv6-family-matrix.yaml dc-ha-scaleup.yaml
vr1-dc0-vips.yaml vr1-dc1-machines.yaml vr1-dc1-vips.yaml` (five tracked overlays,
no octavia-pki). So this is generation FROM SCRATCH for both DCs, not a
regeneration and not a reuse decision -- which removes the reuse-vs-regenerate
choice `dc-dc-phase4-juju-bundle-per-dc.md:204-209` frames as an operator call.
Nothing exists to reuse.

Consequence for sequencing: F1 and F2 can both be fixed with ZERO risk to existing
material, because there is no existing material. The window in which this fix is
free closes the moment the first DC's PKI is generated.

---

## F4 -- renaming the overlay would SILENTLY UN-IGNORE a CA private key

**Class: coupling trap.** Severity: HIGH, but only if F1 is fixed carelessly. It is
recorded here so the F1 fix cannot be taken without it.

`.gitignore:40` is an EXACT path, not a glob:

    overlays/octavia-pki.yaml
    octavia-pki/
    passphrase.txt

F1's fix renames the overlay per DC (e.g. `overlays/vr1-dc0-octavia-pki.yaml`). That
new name **does not match `.gitignore:40`**, so the moment the generator writes it,
a file containing base64 CA private-key blobs plus the issuing-CA passphrase IN
PLAINTEXT becomes an untracked-but-committable file -- in a repo `SEC-004` records
as currently **PUBLIC**. `docs/design-decisions.md` (R7 posture paragraph) states
that this gitignore is "the only thing standing between a CA private key and
publication"; `creds-matrix-notes.md:182-185` (`n-overlay-in-clone`) says the same.

The F1 edit MUST therefore widen line 40 to a glob covering every per-DC name
(`overlays/*octavia-pki.yaml`) **in the same commit that renames anything**, and the
generator should verify its own output is ignored (`git check-ignore -q "$OUT"`)
before writing key material into it -- an assert-on-content gate rather than a
convention anybody must remember. `octavia-pki/` at line 41 is already a directory
glob and covers `~/octavia-pki/<dc>/` without change.

---

## F5 -- preflight's `DC` selector does not reach P4, and P4 is dc0-frozen

**Class: gate does not measure what the operator asked it to measure.**
Severity: MEDIUM. Adjacent to, but not covered by, the 2026-07-29 P2 DC-awareness fix.

Measured:

- `scripts/preflight.sh:99` sets `DC="${DC:-vr1-dc0}"` and **never exports it**.
  Invoked as `DC=vr1-dc1 bash scripts/preflight.sh` the variable arrives in the
  environment and survives into children; invoked BARE it is a plain shell variable
  and children see nothing. So propagation depends on the caller's invocation form
  -- two call shapes, two different behaviours, neither documented.
- It does not matter yet, because `scripts/pre-flight-checks.sh` (P4) never reads
  `DC` at all: `grep -n 'DC'` returns exactly one hit, inside a comment at `:80`.
  Its own CHECK 1 comment admits the state -- "this check is still dc0-only -- it
  never calls `lib_net_select_dc`" -- but that admission is scoped to CHECK 1, while
  CHECK 0's `OVL="overlays/octavia-pki.yaml"` (`:65`) is equally unscoped.

Net effect: `DC=vr1-dc1 bash scripts/preflight.sh` runs a DC-aware P2 and a
dc0-frozen P4 in the same invocation and prints one combined verdict. A dc1 operator
reading `PREFLIGHT` output cannot tell which halves measured dc1.

Also note `preflight.sh:102` hardcodes `overlays/octavia-pki.yaml` in P2's overlay
assembly loop, so it inherits F1's rename and must change with it.

---

## F7 -- `office1-tailscale` is labelled a SUBNET ROUTER and advertises NO ROUTES

**Class: RULED-IS-NOT-BUILT / wrong record.** Severity: MEDIUM operationally, but it is the
THIRD instance in one session of a record that points somewhere real and says something false.

Raised because the operator reported `Permission denied (publickey)` when trying to reach
`voffice1` "directly from my workstation via tailscale". Measured rather than assumed:

- **`tailscale` is absent from `voffice1` AND from `vcloud`** -- no binary in any standard
  location, no `/var/lib/tailscale`, no interface, service not running. Checked with explicit
  path probes after an initial `command -v`, because "tool absent from PATH" reported as
  "feature absent" is a trap this repo has hit repeatedly.
- The tailnet node is a SEPARATE LXD guest INSIDE voffice1: `office1-tailscale`, RUNNING,
  `100.64.0.53` / `fd7a:115c:a1e0::35`, site leg `10.10.1.11`.
- **`tailscale status --json` -> `AdvertisedRoutes: <none>`.** So nothing on the tailnet can
  reach `10.10.0.20` at all. There is no route, and there never was one.

**The two records that made the belief look supported:**

1. `~/.ssh/config.d/vr1-sites` labels the stanza `Host office1-tailscale  # Office1 subnet
   router (D-107)`. It is not routing anything.
2. The operating skill's routing table states "This is the vcloud path; the workstation reaches
   the same over the tailnet (D-107)". **D-107 is titled "Airgap posture, per-DC artifact
   mirror, and NTP (VR1)"** and rules nothing about a workstation tailnet path -- the citation
   is real but does not support the claim. Exactly the "a citation is an existence claim; only
   its CONTENT is evidence" rule.

**FIXED (operational, outside the repo):** `~/.ssh/config.d/vr1-sites` on vcloud now matches
`Host voffice1 10.10.0.20`, so the raw IP resolves to the office1 service key with
`IdentitiesOnly yes` instead of falling through to the `~/.ssh/id_*` defaults -- which was the
whole cause of the original denial. Verified both forms connect, with a NEGATIVE CONTROL
(`ssh -G 10.10.0.99` still reads `identitiesonly no`) proving the key is not offered to
unrelated hosts.

**RULED (operator, 2026-07-29): option (a), ProxyJump.** The workstation reaches `voffice1`
through `vcloud` rather than over the tailnet. Making `office1-tailscale` advertise
`10.10.0.0/24` was presented as option (b) and NOT taken -- it exposes the isolated site net
onto the tailnet, which is a posture change against the airgap subject D-107 actually governs,
and would need its own ruling.

**QUEUED, NOT EXECUTED:** correct the skill's routing-table citation, and correct or remove the
"subnet router" label in the ssh stanza. The skill sweep is deferred to the Stage-5 close by
standing operator direction, so this is input to it.

**WORTH A TARGETED SWEEP:** three wrong records in one session (D-124's MAC sourcing
instruction, the skill's tailnet citation, this label), and two of the three were found only
because something adjacent was being fixed. Nothing in this repo gates prose, so none of them
could have been caught by a check.

---

## F8 -- Step 1.0-GEN.c's heredoc is a PASTE HAZARD, hit live during execution

**Class: authoring defect in an execution surface.** Found by the operator while generating,
2026-07-29 -- they had to correct the block before it would run, and recorded the reason as
"CNF heredoc must start at column 1".

`1.0-GEN.c` opens `cat > controller.cnf <<'CNF'` from inside an INDENTED `( { ... } )`
subshell. The heredoc body and its `CNF` terminator sit at column 1 in the file, which is
correct -- but the surrounding block is indented, so anyone who copies the step and pastes it
with the indentation preserved gets a terminator that no longer matches. The heredoc then
never closes and the rest of the block is swallowed as config text.

**Why it matters more than a formatting nit:** the failure is not clean. Depending on where the
paste breaks, `controller.cnf` can end up holding shell code, or the block can run with a cnf
that lacks `[alt_names]` -- which would produce a controller cert with NO SANs at all while
every other step still reports OK. Nothing downstream asserts the SAN set.

FIX OPTIONS (not executed -- the operator was mid-execution and hard rule 1 forbids editing the
step under them): use `<<-'CNF'` with tab-indented body, or de-indent the heredoc explicitly
with a comment saying why, or generate the cnf with `printf` lines instead of a heredoc. The
third is the most paste-proof.

QUEUED ALSO: a post-generation assertion that the controller cert actually carries the expected
SAN set, since an empty `[alt_names]` currently passes every existing check.

---

## F9 -- the Octavia controller cert's DNS SANs are COUPLED to D-106, and nothing records it

**Class: unrecorded cross-stage coupling.** Raised by an operator question during generation:
"Why isn't `os-public-hostname` set before this process?"

The answer is that it is deliberately set NOWHERE (measured: no option key in `bundle.yaml` or
any overlay -- every hit is prose). `bundle.yaml:11` posture B5 is IP-ONLY, "the dual VIPs ARE
the catalog endpoints ... No control-plane DNS dependency", and R5 explicitly REFUSED option
(c), setting it at Stage 5, because that recreates the D-019 root cause -- metal-only charms
pulling a public FQDN endpoint they cannot resolve before FQDN-SAN certs exist. D-106's order
also puts it structurally later: designate DEPLOYS at Stage 5 but is CONFIGURED at Stage 7, so
at PKI-generation time no zone exists and no FQDN resolves.

**So today the controller cert's two DNS SANs are inert strings** -- amphorae reach the
controller on `o-hm0`'s charm-generated ULA by ADDRESS, and the controller verifies amphorae by
UUID. That is why the known `dc0.vr0` residue is harmless right now.

**THE COUPLING NOBODY HAS WRITTEN DOWN:** when D-106 lands at Stage 7 and `os-public-hostname`
is set, those DNS SANs STOP being cosmetic -- and **both DCs' controller certs carry
`octavia-controller.omega.dc0.vr0...`**, so dc1's would be wrong at exactly the moment it starts
mattering. D-106's own sequence includes "Vault issues FQDN-SAN certs", so the likely correct
disposition is that the amphora controller cert is REISSUED as part of that step rather than
patched now (it is a 2-year cert, and the amphora PKI is a separate trust domain from the
Vault-issued API TLS).

Recorded because otherwise Stage 7 arrives and nothing connects "issue FQDN-SAN certs" to "the
Octavia amphora control-plane cert also needs redoing, per DC". NOT a Stage-5 blocker.

