diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index fc59bce..7037da8 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2158,6 +2158,20 @@ server cert (neutron VIP TLS unhealthy, plausibly tied to the OVN outage). F2/F3 being dug into; in-place-vs-redeploy strategy under discussion. SEC-033 noted (relation databag exposes vault's global-client private key to any juju model reader). + **>>> CORRECTION 2026-08-04 (MEASURED, supersedes the rdns_mode framing just above). <<<** Read the + live dc0 region: **`rdns_mode=2` on ALL planes** (metal-internal included) and the PTR EXISTS at the + region BIND (10.12.8.6 answers `dig -x 10.12.12.122`). rdns_mode is NOT the cause and "enable + rdns_mode" is a VOID no-op; sweep F1's "standup never sets rdns_mode" is also void (that script is + VR0-only, not the VR1 path). CORRECTED cause: the metal-internal plane is DELIBERATELY ISOLATED + (link-scoped routes only; `ping -I eth1 10.12.8.6` -> NO ROUTE) with NO resolver reachable on it -- + the region controller has no metal-internal interface. systemd-resolved scopes the reverse query of + a container's OWN metal-internal address to eth1, which cannot reach any resolver -> empty cert CN. + Discriminating test: both links set to the reachable 10.12.8.6 STILL failed "No route to host", so + it is NOT a dns_servers change either. The FIX is now a DECISION (decouple ovn-central's cert CN + from metal-internal reverse-DNS) and a PRECONDITION for any redeploy (same MAAS; must be verified at + provision time, not runtime-poked). Full record + fix candidates: + `docs/audit/ovn-central-cert-reeval-20260803.md` CORRECTION 2026-08-04 and the VOID banner in + `ovn-central-cert-remediation-plan-20260803.md`. **>>> SESSION CLOSE 2026-08-03 (GA-R4 bookend). Stage 5 remains OPEN; this is a session bookend, NOT a stage close. <<<** DEPLOY IS MID-STAGE-5: bundle deployed + mostly converged (9 machines started, mysql ONLINE, vault init+unseal+root-CA, ~25 units active, 0 error), diff --git a/docs/audit/ovn-central-cert-reeval-20260803.md b/docs/audit/ovn-central-cert-reeval-20260803.md index 9d944fa..aeedc6f 100644 --- a/docs/audit/ovn-central-cert-reeval-20260803.md +++ b/docs/audit/ovn-central-cert-reeval-20260803.md @@ -126,6 +126,33 @@ this session did not read it). NOTE the reverse zone itself is already MAAS-served (SOA present for 12.12.10.in-addr.arpa) but unpopulated -- consistent with rdns generation being off for that subnet. +## >>> CORRECTION 2026-08-04 (MEASURED): the "no PTR / enable rdns_mode" diagnosis above is WRONG. <<< +Measured from the dc0 region (profile vr1-dc0-region, guarded) and the ovn-central/0 unit: +- **`rdns_mode=2, managed=True` on ALL SIX planes**, metal-internal included. rdns is NOT the cause; + the "enable rdns_mode" remediation is VOID (a no-op). +- **The PTR EXISTS.** The region BIND at 10.12.8.6 answers `dig -x 10.12.12.122` -> `eth1.juju-...maas` + correctly. The failure is the RESOLUTION PATH from the unit, not PTR generation. +- **The metal-internal plane is ISOLATED and has NO reachable resolver.** The region controller + `hot-kid` has interfaces ONLY on metal-admin (10.12.8.6) and provider-public (10.12.4.6) -- none on + metal-internal. metal-internal subnet `dns_servers=[]`, so MAAS advertises the provider-public + `10.12.4.6`, which is unroutable from metal-internal. On the unit, routes are LINK-SCOPED only + (`10.12.8.0/22 dev eth0`, `10.12.12.0/22 dev eth1`); `ping -I eth1 10.12.8.6` -> **NO ROUTE**. +- **systemd-resolved scopes the reverse query of a container's OWN metal-internal address to eth1**, + which cannot route to any resolver -> "No route to host" -> `get_hostname()`=None -> empty CN. + DISCRIMINATING TEST (advisor-directed): setting BOTH links to only 10.12.8.6 (confirmed applied) + + flush STILL failed "No route to host" for 10.12.12.122, while 10.12.16.1 resolved via eth0. So it + is NOT a `dns_servers`/`allow_dns` change either. Non-local addresses resolve via eth0; a + container's OWN metal-internal address cannot, because metal-internal is isolated by design. +- **CORRECTED root cause:** ovn-central (rev 311) derives its cert CN from a REVERSE lookup of its + default-binding (private) address, which is on the DELIBERATELY-ISOLATED metal-internal plane that + has no reachable resolver -- so the CN is empty and no server cert is issued. This is a structural + incompatibility between charm-ovn-central's CN-from-reverse-DNS behaviour and binding the OVN DB + plane to an isolated network. The refutation of LP #2044324 and the vault-issuance rule above STAND. +- **CORRECTED fix framing (a real decision -- see the remediation plan's CORRECTION):** the fix must + DECOUPLE ovn-central's cert CN from metal-internal reverse-DNS. It is a PRECONDITION for any + redeploy (step 3 redeploys onto the SAME MAAS; fresh containers fail identically otherwise), and it + must be verified at PROVISION time (what MAAS hands a NEW container), not just runtime-poked. + ## Residual UNKNOWNs (before executing) - `rdns_mode`/`managed` for 10.12.12.0/22 in MAAS (needs the MAAS API; not read this session -- secrets rule kept the reviewer off the API key). Decides whether PTRs can simply be added. diff --git a/docs/audit/ovn-central-cert-remediation-plan-20260803.md b/docs/audit/ovn-central-cert-remediation-plan-20260803.md index 4f3c804..29b3549 100644 --- a/docs/audit/ovn-central-cert-remediation-plan-20260803.md +++ b/docs/audit/ovn-central-cert-remediation-plan-20260803.md @@ -1,5 +1,20 @@ # ovn-central server-cert remediation -- rdns_mode + re-fire (DRAFT, operator-gated) +> **>>> VOID 2026-08-04 -- DO NOT EXECUTE THE rdns_mode STEPS BELOW. <<<** Measurement refuted the +> premise: `rdns_mode=2` on all planes (already enabled), the PTR exists at the region BIND, and the +> real cause is that the metal-internal plane is ISOLATED with no reachable resolver, so a +> metal-internal-bound container cannot reverse-resolve its own address (confirmed: both links set to +> the reachable resolver STILL "No route to host"). See the CORRECTION in +> `ovn-central-cert-reeval-20260803.md`. The fix is NOT an rdns_mode/dns_servers change; it is a +> DECISION to decouple ovn-central's cert CN from metal-internal reverse-DNS (candidates: land the +> cert/CN address on the DNS-working metal-admin plane via the default binding -- needs verifying +> private-address follows the binding, and note ovn-central was already broken at deploy when the +> default WAS metal-admin; or a static-hostname/`/etc/hosts` bootstrap per D-008; or a charm CN +> source). Operator decision pending; the steps below are retained only as the record of the refuted +> approach. The corrected acceptance criteria (server cert on all 3 -> /etc/ovn/* -> cluster formed -> +> 6641/6642 -> neutron+octavia cascade heal) and the "verify at provision time, not runtime" rule +> still apply to whatever fix is chosen. + Companion to `docs/audit/ovn-central-cert-reeval-20260803.md`. Every mutating step is GATED (present -> justify -> individual human approval). Nothing here is executed by the drafting session. Root cause (measured): ovn-central derives its cert CN from a REVERSE lookup of its metal-internal diff --git a/docs/audit/stage5-sweep-misses-20260803.md b/docs/audit/stage5-sweep-misses-20260803.md index 645e7ab..a9111e6 100644 --- a/docs/audit/stage5-sweep-misses-20260803.md +++ b/docs/audit/stage5-sweep-misses-20260803.md @@ -8,6 +8,14 @@ ## Census (juju status, model vr1-dc0): 55 active, 6 blocked, 4 waiting, 1 unknown, 1 ERROR. ## FINDING 1 -- metal-internal is the ONLY plane without reverse DNS; standup tooling ignores rdns_mode +> **CORRECTED 2026-08-04 (measured):** F1's framing is WRONG on two counts. (i) `rdns_mode=2` on ALL +> planes incl. metal-internal (not an rdns outlier); the PTR exists at the region BIND. (ii) The +> `phase-00-maas-standup.sh` "never sets rdns_mode" claim is doubly void -- rdns_mode is not the +> cause, and that script is VR0-only, not the VR1 path (`dc-region-topology.sh` is). The REAL cause: +> metal-internal is an isolated plane with no reachable resolver, so a metal-internal-bound container +> cannot reverse-resolve its own address -> empty cert CN. Full corrected mechanism + +> discriminating test: `ovn-central-cert-reeval-20260803.md` CORRECTION 2026-08-04. The text below is +> retained as the refuted reasoning only. Per-plane reverse-DNS (MEASURED via a unit's resolver, one live addr each): provider-public 10.12.4.x -> resolves (br-ex..maas) metal-admin 10.12.8.x -> resolves (juju-.maas)