Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / ovn-central-cert-reeval-20260803.md
@JANeumatrix JANeumatrix 6 hours ago 15 KB ovn-central cert: root-caused, fix PROVEN live, and wired for redeploy
Raw Blame History

ovn-central "awaiting server certificate data" -- ROOT-CAUSE RE-EVALUATION, 2026-08-03

Re-evaluation of the prior 2026-08-03 diagnosis (the one the operator flagged as unreliable, made under context-length pressure). Method: fresh live measurement (read-only, from the dc0 rack per D-138) + three parallel deployed-source agents + one adversarial-review agent. Every claim below is tagged MEASURED (read live/source this session) or INFERRED/UNKNOWN. Model vr1-dc0; ovn-central rev 311 (24.03/stable); vault rev 724 (1.8/stable).

VERDICT: the prior root cause was WRONG. Corrected root cause below, confirmed end-to-end.

Corrected root cause (MEASURED at every hop)

ovn-central publishes a certificate request with an EMPTY common_name, so vault never creates a server-cert request for it, so ovn-central blocks -- and the empty CN is because ovn-central's address has no reverse-DNS PTR:

  1. vault issues a per-unit SERVER cert ONLY when the requester databag carries a truthy top-level common_name OR a cert_requests map (vault charm hooks/relations/tls-certificates/ provides.py:179,191). Otherwise it publishes ONLY the global client cert (ca+client.cert+ client.key), unconditionally (reactive/vault_handlers.py:1094-1121). BY DESIGN.
  2. ovn-central uses the default charms.openstack cert path (it does NOT override get_certificate_requests; reactive/layer_openstack.py:127-137 -> charms_openstack/charm/ classes.py:284-291 -> charmhelpers/contrib/openstack/cert_utils.py). That path sets cn = get_hostname(local_address(unit_get_fallback='private-address')) (cert_utils.py add_hostname_cn).
  3. private-address = eth1 10.12.12.122 (metal-internal). MEASURED unit-get private-address. get_hostname("10.12.12.122") returns None -- gethostbyaddr raises herror; the metal-internal plane 10.12.12.0/22 has NO PTR records. Contrast MEASURED: eth0 10.12.8.185 (metal-admin) reverse-resolves to juju-237a74-0-lxd-15.maas.
  4. => published databag = certificate_name+sans:["10.12.12.122"], no common_name => vault's all_requests skips it => client-cert-only => certificates.server.certs.available never sets => /etc/ovn/{cert_host,key_host,ovn-central.crt} never written (MEASURED absent) => raft NB/SB cluster can't SSL-handshake on 6643/6644 (cluster/status = "joining cluster, Cluster ID: not yet known") => the pssl: client listeners 6641/6642 open only on a cluster leader (configure_ovn_listener, if status.is_cluster_leader) => nothing listens on 6641/6642.
  5. Sibling comparison that proves the mechanism generic (MEASURED): mysql-innodb-cluster publishes the IDENTICAL CN-less shape, gets the IDENTICAL client-cert-only response, and is healthy -- it falls back to MySQL's own auto-generated self-signed cert (issuer CN=MySQL_Server_8.0.46_Auto_Generated_CA_Certificate). ovn-chassis WORKS because it OVERRIDES the CN with the OVS-DB hostname (vr1-dc0-compute-01.maas), never DNS. ovn-central has neither a fallback nor an override.

What the prior diagnosis got wrong

  • "LP #2044324 CONFIRMED root cause" -> NO MATCH (MEASURED against the bug's content). The bug blocks on "Ports which should be open, but are not: 6641, 6642" (a post-cert port check) with NO binding at all and "inconsistently"; ours is "awaiting server certificate data", binding PRESENT, request PUBLISHED, DETERMINISTIC. The bug is untriaged 2.5y, zero comments, lxd-cloud. No upstream fix exists (github.com/canonical/charm-ovn-central; opendev mirror is archived), and no channel/revision changes cert-request construction (24.03/stable moved 311->332, nothing relevant; no 24.09 track). A channel bump will NOT fix this.
  • "vault cannot sign because common_name is missing." -> vault is never ASKED to sign; no server request object is created for a CN-less databag. mysql demonstrates identical treatment + health.
  • The "Skipping request for certificate for ip in , no local address found" log line the prior session leaned on does NOT causally drive this, and per the reviewer's grep does not even appear as that exact string in rev 311 -- the prior session likely quoted a different charm version's log.

CORRECTION to an over-claim MADE DURING this re-eval (owned)

An interim framing here said "the default binding is NOT the lever, PROVEN by: broken at deploy under metal-admin default + the rebind changed nothing." That "proven" is WITHDRAWN. Source shows the CN derives from private-address, which FOLLOWS the default binding. So the D-052 amendment (default '' metal-admin -> metal-internal, commit 53e046e) is ON the causal path: it moves private-address from PTR-bearing eth0 to PTR-less eth1. Whether ovn-central was already CN-broken at deploy (default still metal-admin) is UNKNOWN and not reconstructable (debug-log rotated; juju keeps no historical relation data). The corrected root cause rests on the CURRENT measured state, which is solid; the deploy-time story is not needed for it.

Remedies (ranked; all are MUTATIONS -> operator-gated, NOT executed)

PRIMARY -- add reverse-DNS for the metal-internal ovn-central addresses, then re-fire.

  • vault's PKI role signs with allow_any_name=True/enforce_hostnames=False (MEASURED by the vault-source agent, vault_pki.py), and OVN peers connect by IP (already in the SAN), so the CN CONTENT is near-irrelevant -- it only needs to be NON-EMPTY.
  • Add PTRs in MAAS for 10.12.12.122/.108/.104 (the three ovn-central units). FEASIBILITY (MEASURED): the reverse zone 12.12.10.in-addr.arpa IS served by MAAS (has SOA), but has NO PTRs for these juju-assigned addresses (MAAS did not allocate them), so this is "register records", not "already managed". CONFIRM FIRST (operator, read-only): maas <profile> subnets read -> rdns_mode/ managed for the 10.12.12.0/22 row.
  • Re-fire: the request handler is gated when_not is-update-status-hook, so DNS change alone won't re-run it -- a no-op juju config touch or a certificates relation cycle re-publishes; request_server_cert overwrites the empty CN.
  • APPLY TO ALL THREE UNITS. The raft cluster + 6641/6642 need all three to hold certs at once.
  • ACCEPTANCE CRITERIA (do NOT use "unit goes active" -- it is a FALSE GREEN; workload status gates only on the cert flag, not cluster health): (1) ovn-central_N.server.cert appears in the databag; (2) /etc/ovn/{cert_host,key_host,ovn-central.crt} written on each unit; (3) after all three, ovs-appctl cluster/status OVN_Northbound shows a real Cluster ID + a leader, and 6641/6642 actually listen.

PROOF-FIRST DIAGNOSTIC (recommended before the DNS change; low-risk, one unit). Add an /etc/hosts line on ovn-central/0 mapping 10.12.12.122 -> a .maas name (nsswitch = files dns), poke the certificates relation, and confirm criteria (1)+(2) appear. This proves the CN is the only issuance blocker WITHOUT touching MAAS DNS. Caveats (MEASURED/reviewer): juju's machine agent rewrites /etc/hosts, so it is EPHEMERAL -- read the result immediately after the poke; and it can only establish (1)+(2), never cluster health.

ALTERNATIVE, D-NNN-shaped (NOT recommended without a ruling) -- revert the default binding to metal-admin. Since eth0/metal-admin already has a working PTR, moving private-address back there would yield a resolvable CN with NO DNS change. BUT it reverses the standing metal-internal-binding decision (commits 580fc06/53e046e) and has a LARGE blast radius: it moves ovsdb-peer bound-address, the SAN set, the firewall cluster_remote_addrs, and the raft addresses already baked into the joining DBs (Address: ssl:10.12.12.122:6643). Flag for the operator as a decision, not a fix.

RULED OUT (MEASURED): charm-config CN lever -- rev 311 config.yaml has no os-*-network/ os-*-hostname/ssl_* option, metadata.yaml declares no internal/admin/public extra-bindings. Accept-degraded -- total OVN control-plane outage; no non-TLS or manual-cert escape in rev 311 (actions.yaml has no cert action).

WHY NO PTR FOR metal-internal (operator asked; MEASURED from the unit resolver)

MAAS DID register the metal-internal addresses in FORWARD DNS but never generated their REVERSE (PTR) records. Measured getent hosts on ovn-central/0 (resolver 127.0.0.53 -> maas):

  • metal-admin 10.12.8.185 : forward juju-237a74-0-lxd-15.maas AND reverse PTR -> BOTH present.
  • metal-internal 10.12.12.122: forward eth1.juju-237a74-0-lxd-15.maas present; reverse PTR ABSENT.
  • metal-internal 10.12.12.114 (keystone): forward eth1.juju-237a74-0-lxd-7.maas; reverse ABSENT. So MAAS knows these addresses (forward A records exist under eth1.<node>.maas); it simply does not publish PTRs for the metal-internal plane. Reverse-DNS GENERATION (MAAS subnet rdns_mode) is effectively enabled for the metal-admin subnet (10.12.8.0/22, the PXE/provisioning plane MAAS fully manages) and NOT for metal-internal (10.12.12.0/22, a juju-consumed L3 segment). This is consistent with the deployment's deliberate metal-admin-centric DNS posture (D-131 scopes the node-DNS forwarder to metal-admin only). It is default/plane-scoping behavior, not a defect -- it only became load-bearing because ovn-central's default cert path derives its CN via a REVERSE lookup of its metal-internal address, which the API charms' cert path does not do the same way. CONSEQUENCE FOR THE REMEDY (refines Remedy 1): because the FORWARD records already exist, the cleanest fix is to ENABLE reverse-DNS on the metal-internal subnet in MAAS (rdns_mode), which makes MAAS auto-generate PTRs for the addresses it already knows -- one subnet-config toggle + DNS reload, rather than hand-creating per-address PTRs. Confirm the current rdns_mode/managed for 10.12.12.0/22 via maas <profile> subnets read first (operator; the MAAS API key is behind the secrets guard so this session did not read it). NOTE the reverse zone itself is already MAAS-served (SOA present for 12.12.10.in-addr.arpa) but unpopulated -- consistent with rdns generation being off for that subnet.

>>> CORRECTION 2026-08-04 (MEASURED): the "no PTR / enable rdns_mode" diagnosis above is WRONG. <<<

Measured from the dc0 region (profile vr1-dc0-region, guarded) and the ovn-central/0 unit:

  • rdns_mode=2, managed=True on ALL SIX planes, metal-internal included. rdns is NOT the cause; the "enable rdns_mode" remediation is VOID (a no-op).
  • The PTR EXISTS. The region BIND at 10.12.8.6 answers dig -x 10.12.12.122 -> eth1.juju-...maas correctly. The failure is the RESOLUTION PATH from the unit, not PTR generation.
  • The metal-internal plane is ISOLATED and has NO reachable resolver. The region controller hot-kid has interfaces ONLY on metal-admin (10.12.8.6) and provider-public (10.12.4.6) -- none on metal-internal. metal-internal subnet dns_servers=[], so MAAS advertises the provider-public 10.12.4.6, which is unroutable from metal-internal. On the unit, routes are LINK-SCOPED only (10.12.8.0/22 dev eth0, 10.12.12.0/22 dev eth1); ping -I eth1 10.12.8.6 -> NO ROUTE.
  • systemd-resolved scopes the reverse query of a container's OWN metal-internal address to eth1, which cannot route to any resolver -> "No route to host" -> get_hostname()=None -> empty CN. DISCRIMINATING TEST (advisor-directed): setting BOTH links to only 10.12.8.6 (confirmed applied) + flush STILL failed "No route to host" for 10.12.12.122, while 10.12.16.1 resolved via eth0. So it is NOT a dns_servers/allow_dns change either. Non-local addresses resolve via eth0; a container's OWN metal-internal address cannot, because metal-internal is isolated by design.
  • CORRECTED root cause: ovn-central (rev 311) derives its cert CN from a REVERSE lookup of its default-binding (private) address, which is on the DELIBERATELY-ISOLATED metal-internal plane that has no reachable resolver -- so the CN is empty and no server cert is issued. This is a structural incompatibility between charm-ovn-central's CN-from-reverse-DNS behaviour and binding the OVN DB plane to an isolated network. The refutation of LP #2044324 and the vault-issuance rule above STAND.
  • CORRECTED fix framing (a real decision -- see the remediation plan's CORRECTION): the fix must DECOUPLE ovn-central's cert CN from metal-internal reverse-DNS. It is a PRECONDITION for any redeploy (step 3 redeploys onto the SAME MAAS; fresh containers fail identically otherwise), and it must be verified at PROVISION time (what MAAS hands a NEW container), not just runtime-poked.

>>> RESOLUTION 2026-08-04 (PROVEN live, then wired for redeploy). <<<

Root cause (final, measured): charm-ovn-central (rev 311) computes its TLS common_name as get_hostname(local_address()), and local_address() is the unit's metal-internal address (10.12.12.x) -- the DELIBERATELY-ISOLATED D-052 plane with no gateway and no reachable DNS resolver. get_hostname() -> None -> empty CN -> vault issues no server cert. rdns_mode=2 and the MAAS forward+reverse records exist at the region BIND; the reverse is simply UNREACHABLE from the isolated plane. The charm has NO config override for the CN (measured: config.yaml).

Binding approach REFUTED, live (three configs): ''->metal-admin, certificates->metal-admin, and ''->metal-admin+bounce all left the CN empty -- the charm derives the CN from the metal-internal address regardless of which endpoint is rebound. Also juju/MAAS default-route selection is non-deterministic and not tied to the default binding (Juju charm-guide: a binding "is not a mechanism that alters or verifies routing"; LP#1781856). So the app STAYS on metal-internal (D-052-correct for its OVSDB/certificates data type); the fix does NOT move it.

Fix PROVEN end-to-end, live 2026-08-04 (controlled single-unit test): an /etc/hosts entry 10.12.12.122 <name> on ovn-central/0 made the charm's own get_hostname("10.12.12.122") resolve, the charm published common_name, vault issued ovn-central_0.server.cert, and the charm wrote /etc/ovn/{cert_host,key_host,ovn-central.crt} -- while control units /1 and /2 (no entry) stayed broken. The entry is the sole cause of the fix. OVN imposes NO CN-content rule on the server/DB cert and vault signs any non-empty CN, so the name only needs to be non-empty and stable.

Wired for the redeploy (verify-at-provision): scripts/dc-node-etchosts.sh renders a model-wide cloudinit-userdata (harness tests/dc-node-etchosts 9/9) that, at machine provision, adds each node's metal-internal address -> hostname to /etc/hosts, scoped to THIS DC's metal-internal CIDR pinned from lib-net.sh (dc0 10.12.12.0/22, dc1 10.12.72.0/22). The rendered runcmd was executed live and correctly added the metal-internal entry, skipped the metal-admin address, and made get_hostname resolve. Applied as gated Step 1.2b in runbooks/phase-01-bundle-deploy.md (after add-model, before deploy). This is a NEW mechanism that BORROWS D-008's static-/etc/hosts SHAPE but is NOT D-008 (D-008's /etc/hosts is FQDN->VIP for os-public-hostname, unset in VR1; no /etc/hosts script existed before this). The DELIVERY (does juju apply it to fresh containers) is only verifiable at the redeploy; the rendered content and the cert-issuance chain are proven.

Residual UNKNOWNs (before executing)

  • rdns_mode/managed for 10.12.12.0/22 in MAAS (needs the MAAS API; not read this session -- secrets rule kept the reviewer off the API key). Decides whether PTRs can simply be added.
  • End-to-end confirmation that a non-empty CN yields a usable server cert AND a formed cluster -- the proof-first diagnostic + all-three rollout establishes this.

SEC observation (file as SEC-033)

juju show-unit <any tls-certificates consumer> exposes vault's GLOBAL-CLIENT private key (client.key PEM) in the relation databag in cleartext to any juju model reader. Inherent to the tls-certificates global-client model, not newly introduced; mitigate via juju model RBAC.