Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / stage5-sweep-misses-20260803.md
@JANeumatrix JANeumatrix 22 hours ago 5 KB audit: ovn-central cert reeval + remediation plan + Stage-5 misses sweep
Raw Blame History

Stage-5 sweep for "other misses like the ovn-central one" -- 2026-08-03

Triggered after the ovn-central cert re-eval. Class hunted: silent failures / latent misconfigurations, especially per-plane facility gaps (like metal-internal's missing reverse DNS) and incomplete-transition fallout, that were not surfaced in CURRENT-STATE. Method: live census from the dc0 rack (read-only) + per-plane DNS matrix + targeted probes. All MEASURED this session.

Census (juju status, model vr1-dc0): 55 active, 6 blocked, 4 waiting, 1 unknown, 1 ERROR.

FINDING 1 -- metal-internal is the ONLY plane without reverse DNS; standup tooling ignores rdns_mode

Per-plane reverse-DNS (MEASURED via a unit's resolver, one live addr each): provider-public 10.12.4.x -> resolves (br-ex..maas) metal-admin 10.12.8.x -> resolves (juju-.maas) metal-internal 10.12.12.x -> NONE <-- the outlier data-tenant 10.12.16.x -> resolves (eth2..maas) storage 10.12.32.x -> resolves (eth1..maas) 10.12.36.x -> resolves (eth2..maas) So this is NOT a "metal-admin-only DNS" posture (my earlier framing, corrected): FIVE of six planes have reverse DNS and metal-internal alone lacks it. Origin (MEASURED): phase-00-maas-standup.sh sets per-subnet gateway_ip, managed=true, and dns_servers, but never rdns_mode -- so reverse-DNS enablement is unmanaged and inconsistent across planes, and metal-internal ended up without it. This is the ROOT of the ovn-central cert failure and a TOOLING GAP.

  • Remediation for the live symptom: docs/audit/ovn-central-cert-remediation-plan-20260803.md.
  • Systemic fix (Roosevelt-delta): have the standup tooling MANAGE rdns_mode per subnet (assert/set it, with a harness), so every DC's planes are reverse-DNS-consistent by construction. Every future DC would otherwise inherit the same inconsistency. Classify OPS-vs-D-NNN with the operator (leans toward a standup-DoD addition).
  • Exact per-subnet rdns_mode values need an operator MAAS read (maas vr1-dc0-region subnets read; key behind the secrets guard). Behaviorally, metal-internal reads as rdns-off, the rest rds-on.

FINDING 2 -- 4 hacluster subordinates BLOCKED on stale/malformed IPv6 VIP resources (D-141 fallout)

barbican-hacluster, designate-hacluster, magnum-hacluster, placement-hacluster all blocked: "Resource: res__None_ipv6addr_vip not running". MEASURED on barbican-hacluster/0:

  • grp_barbican_vips contains the three v4 IPaddr2 VIPs (Started) PLUS TWO IPv6addr resources for the SAME v6 address 2602:f3e2:f02:21::51: res_barbican_eth0_ipv6addr_vip (IPv6addr, nic=eth0) -> Started res_barbican_None_ipv6addr_vip (IPv6addr, nic=None) -> NOT running <-- the block An IPv6addr resource with nic=None cannot start; hence blocked.
  • The ha relation data (json_groups / json_resource_params) still carries these v6 VIP resources, even though the PRINCIPAL's juju config barbican vip is now v4-only ("10.12.4.51 10.12.8.51 10.12.12.51"). So the LIVE hacluster retains IPv6 VIP resources from the PRE-D-141 dual-stack deploy that were never cleaned up when the model was reverted to v4-only.
  • CLASS: incomplete D-141 v4-only transition on the LIVE model -- the SAME family as the provider-bundle-check harness reconcile (a built-surface reverted without its companion change). Not in CURRENT-STATE. Silently blocking 4 units.
  • Follow-up (not done): determine why the v6 resources persist (stale pacemaker CIB not reconciled after the vip config change, vs the charm still computing a v6 VIP from somewhere) and clean them (likely juju run <principal>/leader update-status won't clear a CIB resource; may need the hacluster charm to re-render, or a manual crm configure delete res_*_None_ipv6addr_vip GATED). Also confirm whether the prefer-ipv6 charms' haclusters are clean (census: they were active).

FINDING 3 -- octavia/0 in ERROR (not a cert bug; downstream TLS to neutron)

octavia/0: hook failed "certificates-relation-changed", looping every ~5 min. Root (MEASURED traceback): the hook calls neutron at https://10.12.4.55:9696/v2.0/networks and gets SSLError(SSLEOFError ... UNEXPECTED_EOF_WHILE_READING). Probing neutron-api's VIP directly: openssl s_client -connect 10.12.4.55:9696 -> CONNECTED but "no peer certificate available" -- neutron-api's VIP is not presenting a server cert / not serving TLS cleanly. So octavia's error is DOWNSTREAM of neutron-api's VIP TLS being unhealthy, itself plausibly tied to the OVN NB/SB outage (neutron/OVN ML2 unhealthy) OR a separate neutron HAProxy/cert issue. Not in CURRENT-STATE.

  • Follow-up (not done): determine why neutron's VIP presents no cert -- is HAProxy up with the vault cert? is the neutron backend down because OVN is down? Fixing ovn-central may clear this; verify after. Distinct from the ovn-central cert root cause.

KNOWN / EXPECTED TAIL (for completeness -- NOT new misses)

ceph-rbd-mirror/0 (cross-DC, 'ceph-remote' missing), barbican-vault/0 ('secrets-storage' incomplete), designate/0 ('nameservers must be set' -- Stage 7), glance-simplestreams-sync/0 (unknown) -- all previously known/expected.

META-OBSERVATION

Findings 1 and 2, plus the already-fixed provider-bundle-check harness, are all tails of INCOMPLETE TRANSITIONS: the D-141 v4-only revert and the plane build-out each left companion changes undone (hacluster CIB, harness) or a facility inconsistent (rdns per plane). Recommend a deliberate "D-141 transition completeness" and "per-plane facility parity" pass rather than fixing each tail as it surfaces. Severity order for action: F1 (blocks ovn-central + Roosevelt-delta) > F2 (blocks 4 units) > F3 (1 unit, likely downstream).