Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / ovn-central-cert-remediation-plan-20260803.md
@JANeumatrix JANeumatrix 19 hours ago 8 KB CORRECT ovn-central root cause: NOT rdns_mode -- metal-internal is isolated (no reachable resolver)
Raw Blame History

ovn-central server-cert remediation -- rdns_mode + re-fire (DRAFT, operator-gated)

>>> VOID 2026-08-04 -- DO NOT EXECUTE THE rdns_mode STEPS BELOW. <<< Measurement refuted the premise: rdns_mode=2 on all planes (already enabled), the PTR exists at the region BIND, and the real cause is that the metal-internal plane is ISOLATED with no reachable resolver, so a metal-internal-bound container cannot reverse-resolve its own address (confirmed: both links set to the reachable resolver STILL "No route to host"). See the CORRECTION in ovn-central-cert-reeval-20260803.md. The fix is NOT an rdns_mode/dns_servers change; it is a DECISION to decouple ovn-central's cert CN from metal-internal reverse-DNS (candidates: land the cert/CN address on the DNS-working metal-admin plane via the default binding -- needs verifying private-address follows the binding, and note ovn-central was already broken at deploy when the default WAS metal-admin; or a static-hostname//etc/hosts bootstrap per D-008; or a charm CN source). Operator decision pending; the steps below are retained only as the record of the refuted approach. The corrected acceptance criteria (server cert on all 3 -> /etc/ovn/* -> cluster formed -> 6641/6642 -> neutron+octavia cascade heal) and the "verify at provision time, not runtime" rule still apply to whatever fix is chosen.

Companion to docs/audit/ovn-central-cert-reeval-20260803.md. Every mutating step is GATED (present -> justify -> individual human approval). Nothing here is executed by the drafting session. Root cause (measured): ovn-central derives its cert CN from a REVERSE lookup of its metal-internal address (10.12.12.x); the metal-internal plane has forward A records but NO reverse PTRs, so get_hostname() -> None -> empty common_name -> vault issues no server cert -> ovn-central blocks. Fix: enable reverse-DNS generation on the metal-internal subnet so PTRs auto-populate for the addresses MAAS already knows, then re-fire the cert request.

Access + guards (all MAAS ops)

  • MAAS admin for dc0 runs FROM voffice1 via the SSH tunnel to the dc0 region API (10.12.8.6:5240), profile vr1-dc0-region (CURRENT-STATE:447-456). voffice1 cannot reach 10.12.8.6:5240 directly -- bring up the rack-originated tunnel -L 127.0.0.1:5241:10.12.8.6:5240 first (the profile points at the tunnel endpoint; if the tunnel is down the profile REFUSES, exit 2 -- it cannot silently fall back to Office1).
  • BEFORE ANY dc0 MAAS command, assert the profile resolves to the dc0 rack, NOT Office1 (a wrong-region mutation is an idempotent no-op that prints success while destroying nothing -- or worse; this is the repo's most-warned MAAS hazard): bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid # expect exit 0
  • juju ops for ovn-central run FROM the dc0 rack (ssh vr1-dc0-rack 'juju ...', D-138).
  • The MAAS API key is behind the PreToolUse secrets guard; the operator runs the MAAS commands.

STEP 0 -- MEASURE FIRST (read-only; operator runs on voffice1 after the tunnel + assert)

Capture the metal-internal subnet id and its current DNS config, and the metal-admin values to mirror. Do NOT proceed if managed is false on metal-internal (rdns needs a managed subnet -- that is a bigger change; stop and re-scope). maas vr1-dc0-region subnets read |
jq -r '.[] | select(.cidr|startswith("10.12.8.") or startswith("10.12.12.")) | "(.id)\t(.cidr)\t(.name)\tmanaged=(.managed)\trdns_mode=(.rdns_mode)\tallow_dns=(.allow_dns)"' Record: METAL_INTERNAL_ID, its current rdns_mode (expected 0/Disabled), and METAL_ADMIN rdns_mode (the value to mirror -- expected 2). ACCEPTANCE for this step: metal-internal rdns_mode is currently 0 (or !=metal-admin) AND managed=true. If managed=false -> STOP, re-scope.

STEP 1 -- (OPTIONAL, RECOMMENDED) proof-first: confirm CN is the ONLY issuance blocker

Low-risk, one unit, no MAAS change. Proves a non-empty CN yields a server cert before touching DNS. GATED. On ovn-central/0 (nsswitch = files dns, MEASURED): ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- bash -c ' echo \"10.12.12.122 ovn-central-0.proof.local\" | sudo tee -a /etc/hosts '" Then re-fire ONLY this unit's request (see STEP 3's per-unit form) and read the result IMMEDIATELY (juju's machine agent rewrites /etc/hosts, so the entry is EPHEMERAL):

  • PASS iff ovn-central_0.server.cert appears in relation-get -r certificates:142 - vault/0's view (or juju show-unit ovn-central/0 shows it) AND /etc/ovn/{cert_host,key_host,ovn-central.crt} get written. This proves ONLY cert issuance (criteria 1-2), never cluster health. Revert: remove the /etc/hosts line (or let the next hook rewrite it). This is a diagnostic, NOT the fix.

STEP 2 -- ENABLE reverse-DNS on the metal-internal subnet (MUTATION, GATED)

Set metal-internal's rdns_mode to match metal-admin's captured value (expected 2 = Enabled + RFC2317).

# re-assert profile first
bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid
maas vr1-dc0-region subnet update <METAL_INTERNAL_ID> rdns_mode=<METAL_ADMIN_RDNS_MODE>

MAAS regenerates DNS on the change. Then VERIFY PTRs populated (read-only, from a unit's resolver): ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- bash -c ' for ip in 10.12.12.122 10.12.12.108 10.12.12.104; do printf \"%s -> \" \$ip; getent hosts \$ip || echo NONE; done'" ACCEPTANCE: all three now resolve to eth1.<node>.maas (was NONE). If still NONE after a minute, the region BIND may need a reload -- capture, do NOT loop blindly; investigate before re-running. Revert: maas vr1-dc0-region subnet update <METAL_INTERNAL_ID> rdns_mode=<PRIOR_VALUE> (from STEP 0).

STEP 3 -- RE-FIRE the cert request on ALL THREE units (MUTATION, GATED)

The request handler is gated when_not is-update-status-hook, so DNS alone will not re-run it. Cycle the app-level certificates relation once -- this re-runs the joined/changed handlers on all three units together, and request_server_cert overwrites the empty CN with the now-resolvable name: ssh vr1-dc0-rack "juju remove-relation ovn-central:certificates vault:certificates"

# wait for the relation to fully depart, then:
ssh vr1-dc0-rack "juju integrate ovn-central:certificates vault:certificates"

(Least-disruptive alternative if a full bounce is undesirable: juju config ovn-central ovsdb-server-election-timer=5 then back to =4 to drive config-changed -- benign, the cluster is already dead. The relation bounce is the deterministic, prev-session-proven trigger and is preferred because it re-fires all three units at once.) NOTE the relation currently in use is certificates:142; a bounce mints a new relation id -- expected. Revert: re-integrate (the relation is self-healing); no data loss (ovn-central has no working server cert to lose).

STEP 4 -- ACCEPTANCE (read-only). DO NOT use "unit active" as the gate -- it is a FALSE GREEN.

Workload status gates only on the cert flag, not cluster health; a single unit can read active while the OVN cluster stays dead. Require, IN ORDER:

  1. Each unit publishes a server cert: juju show-unit ovn-central/N -> vault's view carries ovn-central_N.server.cert (not just ca/client.cert). All three.
  2. /etc/ovn/{cert_host,key_host,ovn-central.crt} written on all three units.
  3. Cluster formed (only after ALL THREE hold certs): ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound" -> a real Cluster ID and a Leader (not "joining cluster / not yet known"); same for OVN_Southbound.
  4. Listeners up: ss -tlnp | grep -E '6641|6642' present on the cluster leader.
  5. THEN juju status ovn-central -> active (consequence, not criterion), and the OVN consumers (neutron-api-plugin-ovn, ovn-chassis) settle. Re-run bash scripts/cloud-assert.sh for the behavioral verdict.

Classification + follow-ups (operator)

  • Is enabling rdns on metal-internal an OPS change or a D-NNN? It unblocks a charm (OPS-leaning) but mildly adjusts the metal-admin-only DNS posture (D-131). Roosevelt-delta: EVERY DC where ovn-central binds metal-internal needs this same rdns enablement, OR the CN derivation should not depend on a reverse lookup. Recommend recording the ruling and, if adopted, folding rdns-on-east-west-planes into the DC-standup definition-of-done (dc-plane-ipam.sh / the region MAAS config step).
  • No tested artifact exists for "enable rdns + re-fire ovn-central cert" (raw maas/juju here, gated). If this recurs per-DC, build one with a harness (repo norm). Flagged, not a licence to improvise beyond this one-off.
  • SEC-033 (relation-databag exposes vault's global-client private key) is independent -- file separately.