# Stage-5 sweep for "other misses like the ovn-central one" -- 2026-08-03

Triggered after the ovn-central cert re-eval. Class hunted: silent failures / latent
misconfigurations, especially per-plane facility gaps (like metal-internal's missing reverse DNS)
and incomplete-transition fallout, that were not surfaced in CURRENT-STATE. Method: live census
from the dc0 rack (read-only) + per-plane DNS matrix + targeted probes. All MEASURED this session.

## Census (juju status, model vr1-dc0): 55 active, 6 blocked, 4 waiting, 1 unknown, 1 ERROR.

## FINDING 1 -- metal-internal is the ONLY plane without reverse DNS; standup tooling ignores rdns_mode
> **CORRECTED 2026-08-04 (measured):** F1's framing is WRONG on two counts. (i) `rdns_mode=2` on ALL
> planes incl. metal-internal (not an rdns outlier); the PTR exists at the region BIND. (ii) The
> `phase-00-maas-standup.sh` "never sets rdns_mode" claim is doubly void -- rdns_mode is not the
> cause, and that script is VR0-only, not the VR1 path (`dc-region-topology.sh` is). The REAL cause:
> metal-internal is an isolated plane with no reachable resolver, so a metal-internal-bound container
> cannot reverse-resolve its own address -> empty cert CN. Full corrected mechanism +
> discriminating test: `ovn-central-cert-reeval-20260803.md` CORRECTION 2026-08-04. The text below is
> retained as the refuted reasoning only.
Per-plane reverse-DNS (MEASURED via a unit's resolver, one live addr each):
  provider-public 10.12.4.x  -> resolves (br-ex.<node>.maas)
  metal-admin     10.12.8.x  -> resolves (juju-<node>.maas)
  metal-internal  10.12.12.x -> **NONE**  <-- the outlier
  data-tenant     10.12.16.x -> resolves (eth2.<node>.maas)
  storage         10.12.32.x -> resolves (eth1.<node>.maas)
  10.12.36.x                 -> resolves (eth2.<node>.maas)
So this is NOT a "metal-admin-only DNS" posture (my earlier framing, corrected): FIVE of six planes
have reverse DNS and metal-internal alone lacks it. Origin (MEASURED): `phase-00-maas-standup.sh`
sets per-subnet `gateway_ip`, `managed=true`, and `dns_servers`, but **never `rdns_mode`** -- so
reverse-DNS enablement is unmanaged and inconsistent across planes, and metal-internal ended up
without it. This is the ROOT of the ovn-central cert failure and a TOOLING GAP.
- Remediation for the live symptom: `docs/audit/ovn-central-cert-remediation-plan-20260803.md`.
- Systemic fix (Roosevelt-delta): have the standup tooling MANAGE `rdns_mode` per subnet
  (assert/set it, with a harness), so every DC's planes are reverse-DNS-consistent by construction.
  Every future DC would otherwise inherit the same inconsistency. Classify OPS-vs-D-NNN with the
  operator (leans toward a standup-DoD addition).
- Exact per-subnet `rdns_mode` values need an operator MAAS read (`maas vr1-dc0-region subnets read`;
  key behind the secrets guard). Behaviorally, metal-internal reads as rdns-off, the rest rds-on.

## FINDING 2 -- 4 hacluster subordinates BLOCKED on stale/malformed IPv6 VIP resources (D-141 fallout)
barbican-hacluster, designate-hacluster, magnum-hacluster, placement-hacluster all blocked:
"Resource: res_<svc>_None_ipv6addr_vip not running". MEASURED on barbican-hacluster/0:
- `grp_barbican_vips` contains the three v4 IPaddr2 VIPs (Started) PLUS **TWO** IPv6addr resources
  for the SAME v6 address `2602:f3e2:f02:21::51`:
    res_barbican_eth0_ipv6addr_vip  (IPv6addr, nic=eth0)  -> Started
    res_barbican_None_ipv6addr_vip  (IPv6addr, nic=None)  -> NOT running  <-- the block
  An IPv6addr resource with `nic=None` cannot start; hence blocked.
- The `ha` relation data (json_groups / json_resource_params) still carries these v6 VIP resources,
  even though the PRINCIPAL's `juju config barbican vip` is now v4-only ("10.12.4.51 10.12.8.51
  10.12.12.51"). So the LIVE hacluster retains IPv6 VIP resources from the PRE-D-141 dual-stack
  deploy that were never cleaned up when the model was reverted to v4-only.
- CLASS: incomplete D-141 v4-only transition on the LIVE model -- the SAME family as the
  provider-bundle-check harness reconcile (a built-surface reverted without its companion change).
  Not in CURRENT-STATE. Silently blocking 4 units.
- Follow-up (not done): determine why the v6 resources persist (stale pacemaker CIB not reconciled
  after the vip config change, vs the charm still computing a v6 VIP from somewhere) and clean them
  (likely `juju run <principal>/leader update-status` won't clear a CIB resource; may need the
  hacluster charm to re-render, or a manual `crm configure delete res_*_None_ipv6addr_vip` GATED).
  Also confirm whether the prefer-ipv6 charms' haclusters are clean (census: they were active).

## FINDING 3 -- octavia/0 in ERROR (not a cert bug; downstream TLS to neutron)
octavia/0: hook failed "certificates-relation-changed", looping every ~5 min. Root (MEASURED
traceback): the hook calls neutron at `https://10.12.4.55:9696/v2.0/networks` and gets
`SSLError(SSLEOFError ... UNEXPECTED_EOF_WHILE_READING)`. Probing neutron-api's VIP directly:
`openssl s_client -connect 10.12.4.55:9696` -> CONNECTED but **"no peer certificate available"** --
neutron-api's VIP is not presenting a server cert / not serving TLS cleanly. So octavia's error is
DOWNSTREAM of neutron-api's VIP TLS being unhealthy, itself plausibly tied to the OVN NB/SB outage
(neutron/OVN ML2 unhealthy) OR a separate neutron HAProxy/cert issue. Not in CURRENT-STATE.
- Follow-up (not done): determine why neutron's VIP presents no cert -- is HAProxy up with the vault
  cert? is the neutron backend down because OVN is down? Fixing ovn-central may clear this; verify
  after. Distinct from the ovn-central cert root cause.

## KNOWN / EXPECTED TAIL (for completeness -- NOT new misses)
ceph-rbd-mirror/0 (cross-DC, 'ceph-remote' missing), barbican-vault/0 ('secrets-storage'
incomplete), designate/0 ('nameservers must be set' -- Stage 7), glance-simplestreams-sync/0
(unknown) -- all previously known/expected.

## META-OBSERVATION
Findings 1 and 2, plus the already-fixed provider-bundle-check harness, are all tails of INCOMPLETE
TRANSITIONS: the D-141 v4-only revert and the plane build-out each left companion changes undone
(hacluster CIB, harness) or a facility inconsistent (rdns per plane). Recommend a deliberate
"D-141 transition completeness" and "per-plane facility parity" pass rather than fixing each tail
as it surfaces. Severity order for action: F1 (blocks ovn-central + Roosevelt-delta) > F2 (blocks 4
units) > F3 (1 unit, likely downstream).