# ovn-central server-cert remediation -- rdns_mode + re-fire (DRAFT, operator-gated)

Companion to `docs/audit/ovn-central-cert-reeval-20260803.md`. Every mutating step is GATED
(present -> justify -> individual human approval). Nothing here is executed by the drafting session.
Root cause (measured): ovn-central derives its cert CN from a REVERSE lookup of its metal-internal
address (10.12.12.x); the metal-internal plane has forward A records but NO reverse PTRs, so
get_hostname() -> None -> empty common_name -> vault issues no server cert -> ovn-central blocks.
Fix: enable reverse-DNS generation on the metal-internal subnet so PTRs auto-populate for the
addresses MAAS already knows, then re-fire the cert request.

## Access + guards (all MAAS ops)
- MAAS admin for dc0 runs FROM voffice1 via the SSH tunnel to the dc0 region API
  (`10.12.8.6:5240`), profile **`vr1-dc0-region`** (CURRENT-STATE:447-456). voffice1 cannot reach
  `10.12.8.6:5240` directly -- bring up the rack-originated tunnel `-L 127.0.0.1:5241:10.12.8.6:5240`
  first (the profile points at the tunnel endpoint; if the tunnel is down the profile REFUSES,
  exit 2 -- it cannot silently fall back to Office1).
- BEFORE ANY dc0 MAAS command, assert the profile resolves to the dc0 rack, NOT Office1
  (a wrong-region mutation is an idempotent no-op that prints success while destroying nothing --
  or worse; this is the repo's most-warned MAAS hazard):
    bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid    # expect exit 0
- juju ops for ovn-central run FROM the dc0 rack (`ssh vr1-dc0-rack 'juju ...'`, D-138).
- The MAAS API key is behind the PreToolUse secrets guard; the operator runs the MAAS commands.

## STEP 0 -- MEASURE FIRST (read-only; operator runs on voffice1 after the tunnel + assert)
Capture the metal-internal subnet id and its current DNS config, and the metal-admin values to
mirror. Do NOT proceed if `managed` is false on metal-internal (rdns needs a managed subnet -- that
is a bigger change; stop and re-scope).
    maas vr1-dc0-region subnets read | \
      jq -r '.[] | select(.cidr|startswith("10.12.8.") or startswith("10.12.12.")) |
             "\(.id)\t\(.cidr)\t\(.name)\tmanaged=\(.managed)\trdns_mode=\(.rdns_mode)\tallow_dns=\(.allow_dns)"'
Record: METAL_INTERNAL_ID, its current rdns_mode (expected 0/Disabled), and METAL_ADMIN rdns_mode
(the value to mirror -- expected 2). ACCEPTANCE for this step: metal-internal rdns_mode is currently
0 (or !=metal-admin) AND managed=true. If managed=false -> STOP, re-scope.

## STEP 1 -- (OPTIONAL, RECOMMENDED) proof-first: confirm CN is the ONLY issuance blocker
Low-risk, one unit, no MAAS change. Proves a non-empty CN yields a server cert before touching DNS.
GATED. On ovn-central/0 (nsswitch = files dns, MEASURED):
    ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- bash -c '
      echo \"10.12.12.122 ovn-central-0.proof.local\" | sudo tee -a /etc/hosts
    '"
Then re-fire ONLY this unit's request (see STEP 3's per-unit form) and read the result IMMEDIATELY
(juju's machine agent rewrites /etc/hosts, so the entry is EPHEMERAL):
  - PASS iff `ovn-central_0.server.cert` appears in `relation-get -r certificates:142 - vault/0`'s
    view (or `juju show-unit ovn-central/0` shows it) AND `/etc/ovn/{cert_host,key_host,ovn-central.crt}`
    get written. This proves ONLY cert issuance (criteria 1-2), never cluster health.
Revert: remove the /etc/hosts line (or let the next hook rewrite it). This is a diagnostic, NOT the fix.

## STEP 2 -- ENABLE reverse-DNS on the metal-internal subnet (MUTATION, GATED)
Set metal-internal's rdns_mode to match metal-admin's captured value (expected 2 = Enabled + RFC2317).
    # re-assert profile first
    bash scripts/maas-profile-assert.sh vr1-dc0-region hot-kid
    maas vr1-dc0-region subnet update <METAL_INTERNAL_ID> rdns_mode=<METAL_ADMIN_RDNS_MODE>
MAAS regenerates DNS on the change. Then VERIFY PTRs populated (read-only, from a unit's resolver):
    ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- bash -c '
      for ip in 10.12.12.122 10.12.12.108 10.12.12.104; do printf \"%s -> \" \$ip; getent hosts \$ip || echo NONE; done'"
  ACCEPTANCE: all three now resolve to `eth1.<node>.maas` (was NONE). If still NONE after a minute,
  the region BIND may need a reload -- capture, do NOT loop blindly; investigate before re-running.
Revert: `maas vr1-dc0-region subnet update <METAL_INTERNAL_ID> rdns_mode=<PRIOR_VALUE>` (from STEP 0).

## STEP 3 -- RE-FIRE the cert request on ALL THREE units (MUTATION, GATED)
The request handler is gated `when_not is-update-status-hook`, so DNS alone will not re-run it. Cycle
the app-level certificates relation once -- this re-runs the joined/changed handlers on all three
units together, and `request_server_cert` overwrites the empty CN with the now-resolvable name:
    ssh vr1-dc0-rack "juju remove-relation ovn-central:certificates vault:certificates"
    # wait for the relation to fully depart, then:
    ssh vr1-dc0-rack "juju integrate ovn-central:certificates vault:certificates"
(Least-disruptive alternative if a full bounce is undesirable: `juju config ovn-central
ovsdb-server-election-timer=5` then back to `=4` to drive config-changed -- benign, the cluster is
already dead. The relation bounce is the deterministic, prev-session-proven trigger and is preferred
because it re-fires all three units at once.) NOTE the relation currently in use is certificates:142;
a bounce mints a new relation id -- expected.
Revert: re-integrate (the relation is self-healing); no data loss (ovn-central has no working server
cert to lose).

## STEP 4 -- ACCEPTANCE (read-only). DO NOT use "unit active" as the gate -- it is a FALSE GREEN.
Workload status gates only on the cert flag, not cluster health; a single unit can read `active` while
the OVN cluster stays dead. Require, IN ORDER:
  1. Each unit publishes a server cert: `juju show-unit ovn-central/N` -> vault's view carries
     `ovn-central_N.server.cert` (not just ca/client.cert). All three.
  2. `/etc/ovn/{cert_host,key_host,ovn-central.crt}` written on all three units.
  3. Cluster formed (only after ALL THREE hold certs):
     `ssh vr1-dc0-rack "juju exec --unit ovn-central/0 -- ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound"`
     -> a real `Cluster ID` and a `Leader` (not "joining cluster / not yet known"); same for OVN_Southbound.
  4. Listeners up: `ss -tlnp | grep -E '6641|6642'` present on the cluster leader.
  5. THEN `juju status ovn-central` -> active (consequence, not criterion), and the OVN consumers
     (neutron-api-plugin-ovn, ovn-chassis) settle. Re-run `bash scripts/cloud-assert.sh` for the
     behavioral verdict.

## Classification + follow-ups (operator)
- Is enabling rdns on metal-internal an OPS change or a D-NNN? It unblocks a charm (OPS-leaning) but
  mildly adjusts the metal-admin-only DNS posture (D-131). Roosevelt-delta: EVERY DC where ovn-central
  binds metal-internal needs this same rdns enablement, OR the CN derivation should not depend on a
  reverse lookup. Recommend recording the ruling and, if adopted, folding rdns-on-east-west-planes into
  the DC-standup definition-of-done (dc-plane-ipam.sh / the region MAAS config step).
- No tested artifact exists for "enable rdns + re-fire ovn-central cert" (raw maas/juju here, gated). If
  this recurs per-DC, build one with a harness (repo norm). Flagged, not a licence to improvise beyond
  this one-off.
- SEC-033 (relation-databag exposes vault's global-client private key) is independent -- file separately.