diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index a6d32de..d8a1a28 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -572,7 +572,38 @@ rewriting regardless. Option (c), setting `os-public-hostname` at Stage 5, was refused as the one branch that genuinely collides with D-106 -- it recreates the D-019 root cause (metal-only charms pulling a public FQDN endpoint they cannot resolve) before the FQDN-SAN - certs exist. Remaining: R6-R11 blocking, R12-R15 standing. + certs exist. + **PRE-RULING MEASUREMENTS TAKEN FOR R6-R15, 2026-07-27** (operator: "Measure all the rest + then we can work through them with relevant data at hand"). Capture: + `docs/audit/r6-r15-measurements-20260727.txt`. Consequential results: + **(R6/R11) vault's ENTIRE HA apparatus exists only in `dc-ha-scaleup.yaml`** -- base vault + is `num_units: 1` with an EMPTY options block, no hacluster, no relation; the overlay adds + the `vault-hacluster` application, `cluster_count: 3` and the `vault:ha` relation, and + still NO vip. So applying the overlay creates a 3-node pacemaker cluster with nothing to + manage. **23 relations consume `vault:certificates`** plus barbican's secrets backend, so + every consumer binds a UNIT address with nothing to fail over to. Of the 12 base hacluster + subordinates, exactly ONE principal lacks a vip (designate); octavia by contrast carries a + proper triple, so this is a 2-app gap, not a pattern. + **(R10) THE QUESTION LARGELY DISSOLVES: preflight has been run on the WRONG HOST.** + Measured client availability -- vcloud: `maas` ABSENT, `juju` ABSENT, `openstack` ABSENT; + voffice1: `maas` PRESENT, `juju` PRESENT, `openstack` ABSENT. So P3's 33 warns and P4's + "MAAS unreachable" BOTH clear by running preflight on the D-128 Plane-2 host where it + belongs. Only the octavia-pki absence and the 7 credential findings are host-independent. + Same "tool absence reported as something else" class as U6. + **(R9) 8 of 28 `lib-net.sh` consumers call the DC selector**; the 20 that do not include + the whole `phase-02`..`phase-06` family Stage 5+ runs. + **(R8) the v4-only lb-mgmt shape is already pre-analysed in-repo** -- + `overlays/dc-dc-ipv6-family-matrix.yaml` cites LP #1911788 and #1913409 and carries a + drafted block for it (NOT independently verified upstream this session). + **(R12) `chronyc` appears ZERO times in this document** -- L1-8 confirmed, G17's row does + not carry the time check. **(R14)** the matrix has NO exception field, so SEC-016's ruled + power-key asymmetries report as findings forever. **(R15)** 81 harnesses on disk and 81 + reported, but `81` is pinned nowhere executable. + **METHOD NOTE:** an early `awk -F'\t'` parse of `creds-matrix.tsv` returned ZERO + operator-terminal rows, contradicting lens 7's 30. The file is SPACE-ALIGNED, not + tab-separated; re-measured correctly it is 30 rows / 16 ids and lens 7 was right. A + disagreement with a prior finding was treated as a reason to re-check the instrument. + Remaining: R6-R11 blocking, R12-R15 standing -- all now measured. - Position inside Stage 3: deploy step A EXECUTED 2026-07-19 (6/0/6 exact; convergence zero -- `docs/audit/outer-plan-20260719-postA-converged.txt`). **Deploy step B diff --git a/docs/audit/r6-r15-measurements-20260727.txt b/docs/audit/r6-r15-measurements-20260727.txt new file mode 100644 index 0000000..251b14b --- /dev/null +++ b/docs/audit/r6-r15-measurements-20260727.txt @@ -0,0 +1,206 @@ +===================================================================== +PRE-RULING MEASUREMENTS for R6-R15 +Date: 2026-07-27 Branch: dc-dc-stage5-grounding-audit +Taken by the session directly (not delegated), after the UNMEASURED-gap +sweep established that lens findings needed measurement before being +put to the operator as questions. +===================================================================== + +METHOD NOTE, recorded because it nearly produced a wrong figure: an early +attempt to parse `creds-matrix.tsv` with `awk -F'\t'` returned ZERO +operator-terminal rows, contradicting lens 7's figure of 30. The file is +SPACE-ALIGNED, not tab-separated (measured: zero TABs in the first 20 lines), +which is why the repo's own tooling splits on default whitespace. Re-measured +correctly, the count is 30 rows / 16 ids -- lens 7 was right. A disagreement +with a prior finding was treated as a reason to re-check the instrument, not +to publish the new number. + +--------------------------------------------------------------------- +R6 -- HA SCALE-UP OVERLAY +--------------------------------------------------------------------- +`overlays/dc-ha-scaleup.yaml` scales FOURTEEN applications 1 -> 3: + barbican, ceph-radosgw, cinder, designate, glance, keystone, magnum, + neutron-api, nova-cloud-controller, octavia, openstack-dashboard, + placement, rabbitmq-server, vault + +TWELVE hacluster subordinates exist in the base bundle, every one moving +cluster_count 1 -> 3 under the overlay, each principal reaching 3 units. +Of those twelve principals, exactly ONE has no VIP: **designate**. + +THE OVERLAY ALSO INTRODUCES AN APPLICATION THE BASE DOES NOT HAVE: + `vault-hacluster` (charm hacluster, 2.4/stable, cluster_count: 3) +plus the relation `vault:ha <-> vault-hacluster:ha`. +So vault's HA exists ONLY in the overlay. Base vault is `num_units: 1` +with an EMPTY options block -- no vip, no hacluster, no relation. + +CONSEQUENCE, stated precisely: applying the overlay creates a 3-unit vault +with a 3-node pacemaker cluster and NO VIP for it to manage. That is the +decorative-HA defect in its purest form -- hacluster with nothing to do. + +WHY VAULT IS THE SHARP CASE: **23 relations consume `vault:certificates`** -- +mysql-innodb-cluster, keystone, glance, glance-simplestreams-sync, +nova-cloud-controller, placement, neutron-api, neutron-api-plugin-ovn, +ovn-central, ovn-chassis, cinder, ceph-radosgw, openstack-dashboard, octavia, +ovn-chassis-octavia, barbican, barbican-vault, magnum, designate -- plus +`barbican-vault:secrets-storage <-> vault:secrets`. Vault is the certificate +authority for the whole cloud, and with no VIP every consumer binds a UNIT +address. There is nothing for HA to fail over TO. + +Stage-6 coupling (unchanged from L6-4, restated with the scale figures): the +overlay takes ceph-radosgw and designate to 3, while the Stage-6 radosgw +multisite path is single-unit-shaped (`--unit U`, one restart). + +--------------------------------------------------------------------- +R7 -- OCTAVIA CA SCOPE +--------------------------------------------------------------------- +The generator (`runbooks/phase-01-bundle-deploy.md` step 1.0-GEN) is +dc0-frozen in TWO distinct ways, and they differ in severity: + - CA SUBJECT is a BAKED LITERAL: `-subj "/CN=VR0 DC0 Omega Cloud Octavia + Controller CA/O=Neumatrix"`. Reusing it puts a VR0-DC0-named CA at the + root of dc1's amphora trust chain. + - The controller cert's SAN is DERIVED, not baked -- the step states the SAN + carries the controller FQDN, the octavia API FQDN and the Octavia API VIP + "derived from the bundle at generation time -- DOCFIX-067; never a baked + literal". So the SAN half is already per-DC-correct BY DESIGN. + - But the VIP it derives from is gated `grep -qE '^10\.12\.4\.'` (L5-3), + which is dc0's band, and dc1's octavia VIP lives in an overlay. + +D-109 is the governing precedent and it is unambiguous: "Per-DC INDEPENDENT +Vault roots (each DC's Vault is its own root CA; no regional root-of-trust +this phase)", rationale "independent roots keep the two clouds +cryptographically independent (DR-honest)". D-109 does not mention Octavia PKI, +so it is precedent, not a ruling on this question. + +--------------------------------------------------------------------- +R8 -- OCTAVIA lb-mgmt-net ADDRESS FAMILY +--------------------------------------------------------------------- +MEASURED: octavia in the base bundle HAS a VIP triple -- +`10.12.4.57 10.12.8.57 10.12.12.57` (dc0 band) -- unlike vault and designate. +num_units 1, `to: [lxd:0]`, `public: provider-public`, `ovsdb-cms: data-tenant`. + +`overlays/dc-dc-ipv6-family-matrix.yaml` has ALREADY DONE THIS ANALYSIS and +recommends an answer. Its own text: + - lb-mgmt-net "is a NEUTRON tenant network the octavia charm itself creates" + - two upstream reports cited: **Launchpad #1911788** ("IPv6 mgmt network not + working, octavia can't talk to amphora") and **#1913409** + ("octavia_amp_network does not support IPv6", kolla-ansible, same + underlying Octavia limitation) + - `:78` the overlay "deliberately does NOT attempt an Octavia lb-mgmt-net + IPv6 config change" + - `:140` "octavia: deliberately NOT included here" + - `:147` carries a prepared block for "leaving lb-mgmt-net v4-only as a + deliberate exception" +So the v4-only-lb-mgmt shape is pre-analysed, evidence-backed and already +drafted in-repo. NOT independently verified this session: the two LP bug +numbers were read from the overlay, NOT fetched upstream. + +--------------------------------------------------------------------- +R9 -- WHERE dc1's OPENSTACK-LAYER LITERALS LIVE +--------------------------------------------------------------------- +MEASURED: **8 of 28** `lib-net.sh` consumers call `lib_net_select_dc`. +The 20 that do NOT include the entire Stage-5+ phase family: + phase-02-vault-preflight.sh, phase-03-admin-openrc.sh, phase-03-core-verify.sh, + phase-04-internal-cert-san-verify.sh, phase-04-network-create.sh, + phase-04-network-verify.sh, phase-05-octavia-verify.sh, + phase-06-k8s-bootstrap.sh, phase-06-kubeconfig-gate.sh, + pre-flight-checks.sh, juju-spaces-check.sh, deploy-watch.sh, + tenant-onboard.sh, tenant-acceptance.sh, tenant-offboard.sh, + vault-kv-health.sh, osd-blank-check.sh, dc-rack-net.sh, site-baseleg.sh, + creds-matrix.py +Sourcing `lib-net.sh` without the selector yields VR0/dc0 literals by design +(`lib-net.sh:76-78` says so explicitly). So on a dc1 run these silently use +dc0's 10.12.4/8/12 values -- EXCEPT where the dc1 arm `unset`s them, where +they instead die under `set -u`. Both failure modes are live. + +--------------------------------------------------------------------- +R10 -- STAGE-5 ENTRY BASIS AGAINST A RED PREFLIGHT +--------------------------------------------------------------------- +THIS QUESTION LARGELY DISSOLVES ON MEASUREMENT. Client availability: + + vcloud (jumphost) voffice1 (D-128 Plane-2 host) + maas ABSENT PRESENT + juju ABSENT PRESENT + openstack ABSENT ABSENT + +So of the current red set: + P1 lint -- 0 fail today; the earlier FAIL was this session's own + uncommitted captures. ALREADY CLEAR. + P3 33 warns -- caused by `juju` missing. CLEARS when run on voffice1. + P4 "MAAS unreachable" -- caused by the `maas` binary missing, NOT by + unreachability. CLEARS when run on voffice1. + P4 octavia-pki -- genuinely absent (gitignored secret), and the generator + is dc0-frozen (R7). Does NOT clear by changing host. + P5 creds 7 -- 3x S5 RULED CORRECT by SEC-016, 1x S6 (the SEC-020 defect), + 2x E4 uncheckable Stage-5/6 rows, 1x E1+S2 dc0-edge-api + (SEC-021). Does NOT clear by changing host. + +Preflight has been run on the wrong host. That is not an exception to record, +it is a defect to fix -- and it is the same "tool absence reported as +something else" class as U6. + +--------------------------------------------------------------------- +R11 -- VAULT AND DESIGNATE VIPs +--------------------------------------------------------------------- +Superseded framing: the audit said "vault and designate both have HA intent +and no VIP", which is true but understates the difference between them. +MEASURED: + - designate: IS in the base bundle with an hacluster subordinate + (`cluster_count: 1`) and an `ha: metal-internal` binding, and NO vip. + Consistent at base scale (1 unit / cc 1); becomes decorative at 3. + - vault: has NO hacluster in the base bundle AT ALL and an EMPTY options + block. Its entire HA apparatus -- the `vault-hacluster` application, its + `cluster_count: 3`, and the `vault:ha` relation -- exists ONLY in + `dc-ha-scaleup.yaml`. And still no vip. + - octavia, by contrast, DOES carry a proper VIP triple, so the pattern is + not uniform and "add VIPs to the apps missing them" is a 2-app change. +The blast radius is asymmetric: designate serves DNS; vault is the CA for 23 +relations plus barbican's secrets backend. + +--------------------------------------------------------------------- +R12 -- G17 SCOPE (does it carry the node time check?) +--------------------------------------------------------------------- +MEASURED: `chronyc` appears exactly ONCE in the repo's Stage-5-relevant +surfaces -- `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:46` ("both: `chronyc +sources` shows the MAAS-served time source, NOT the DC edge"). It appears +**ZERO times in `docs/CURRENT-STATE.md`**, i.e. the G17 gate row does not +carry it. L1-8 CONFIRMED: the runbook and the workflow doc both assign a node +time check to G17; the gate row's own [V] column names only the two artifact +fetches. + +--------------------------------------------------------------------- +R13 -- CREDENTIAL REPRODUCIBILITY +--------------------------------------------------------------------- +MEASURED (correctly, see the method note): **30 rows across 16 distinct ids** +carry `mint-ref=operator-terminal` -- exactly lens 7's figure. +The UNRECOVERABLE-IN-PLACE subset, i.e. SSH keypairs whose loss means losing +the only access path, is SEVEN ids: + dc0-edge-key, dc1-edge-key (edge SSH -- SEC-007/-015 make this the + ONLY management path to the DC edges) + dc0-svc-key, dc1-svc-key (rack service access) + dc0-maas-power-key, dc1-maas-power-key (MAAS -> libvirt power control) + office1-svc-key (headend access) +Plus `office1-edge-root` (password + hash, not a keypair) and the remaining +office1/netbox/vault/rgw rows. + +--------------------------------------------------------------------- +R14 -- CAN THE REGISTER EXPRESS A RULED EXCEPTION? +--------------------------------------------------------------------- +MEASURED: the matrix schema has NO exception/waiver/ruled field. The four +grep hits for those words are prose in the header comment block, not columns. +So the three S5 power-key asymmetries that SEC-016 RULED to be correct by +design have no representable disposition, and P5 reports them as findings on +every run -- a permanent red that a reader learns to scroll past. + +--------------------------------------------------------------------- +R15 -- GAUNTLET / REPO-LINT FLOOR +--------------------------------------------------------------------- +MEASURED: 81 harnesses on disk, 81 reported by the gauntlet -- consistent +today. `81` is pinned NOWHERE executable (grep of `run-tests-all.sh`: no +match); it exists only as prose in CURRENT-STATE. So a renamed or deleted +harness silently lowers the count and still prints ALL GREEN (L4-8, verified +in a fixture). +`repo_lint.py` DOES contain 8 matches for `is_dir|floor|len(files)`, so some +floor logic exists in the file -- but NOT on the resolved ROOT, which is the +L4-1 defect: a mistyped flag or path resolves to a nonexistent directory, +rglob yields nothing, and it reports `PASS (0 fail, 0 warn)` over zero files. +Both verified independently by this session earlier.