Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / audit / stage5-expansion-review-20260724.md
@JANeumatrix JANeumatrix 22 hours ago 7 KB vr1-dc1 bundle render (9-node role-separated) + expansion review record
Raw Blame History

Deployment-expansion review -- vr1-dc1 (4-node hyperconverged -> 9-node role-separated)

Date: 2026-07-24. Branch: dc-dc-stage4-phase3-maas-deploy. Convened by: operator ("Have subagents deploy a deployment script review for additional cross connections, updates, associations, and pinning updates that are also needed based on the stack expansion").

Status: ADVISORY / audit record. Findings dispositioned below (handled in the render / follow-up / VERIFY-LIVE / no-change). Adopts nothing. docs/CURRENT-STATE.md remains the status authority (GA-R1). Companion to docs/audit/committee-20260724-track2-bundle-render.md.

Three independent read-only reviewers (relations/bindings/cross-DC; config/pins/channels; deployment-scripts). Every load-bearing claim was re-checked against the repo before recording.

A. Handled IN the render (this session; validated)

  • ovn-chassis provider MACs -- bundle.yaml carried VR0's 4 openstack0-3 provider MACs (52:54:00:*); re-homed to the 2 dc1 COMPUTE nodes' provider-public (enp2s0) MACs 52:54:01:d1:04:02 / 52:54:01:d1:05:02 (ovn-chassis is subordinate to the 2-unit nova-compute). NOTE: these are the substrate-PINNED values (scheme 52:54:01:d1:NN:PP, provider = NIC ordinal :02, opentofu/vr1-dc1-substrate/main.tf) -- config-carried as-built, not inferred; still confirm at juju deploy --dry-run.
  • nova-compute num_units 3 -> 2 (2 compute nodes, D-121 Option C).
  • ceph-osd = 4 was already correct (matches R-3's 4 storage nodes); the stale "ceph-osd 3" comment in overlays/dc-ha-scaleup.yaml was corrected to 4.
  • provider-bundle-check.py ovn-chassis check: skip the VR0 Pattern-A MAC set on a role-separated bundle (verify present + well-formed instead).

B. Follow-up (deploy scripts / gates -- NOT render-blocking, needed before their gate)

  • preflight.sh gates the wrong artifact -- runs provider-bundle-check.py bundle.yaml BARE (base bundle, default dc0 bands, role tags absent so placement self-skips). Must invoke the real dc1 deploy input: --overlay overlays/vr1-dc1-machines.yaml --overlay overlays/vr1-dc1-vips.yaml --overlay overlays/dc-ha-scaleup.yaml --dc vr1-dc1 (the checker already supports this; only the call site is VR0-frozen). RENDER-ADJACENT.
  • pre-flight-checks.sh is VR0-frozen -- never calls the $DC selectors; VIP band vars are dc0 (and unset under the vr1-dc1 lib-net arm -> trips set -u); host loops resolve by hostname (host_sysid) but VR1 MAAS hostnames are random -- must use host_sysid_by_bootmac "${HOST_BOOT_MAC[$h]}". Fails every dc1 node as written.
  • cloud-assert.sh false-passes decorative HA -- only quorum-checks mysql (A2) + ovn-central (A3); the 12 scaled hacluster-fronted API services get no pacemaker-quorum / VIP-ownership assertion, so a VIP parked on one node (the exact cluster_count-still-1 failure D-121 targets) passes clean. Needs a per-hacluster "3 online / VIP owned" check before the post-deploy acceptance gate.
  • juju bootstrap unconstrained (phase-4 Step 2) -- with 9 Ready nodes tagged openstack-vr1-dc1 in a region that sees both DCs, an unconstrained bootstrap can grab a bundle node. Needs a distinct controller tag + --constraints tags=<controller-tag> (pairs with the D-104-amendment 10th VM).
  • phase-4 runbook drift -- claims lib_hosts_select_dc vr1-dc1 "fails loud until populated"; it was populated 2026-07-23. Fold into the Step-4 de-stale.
  • Cross-DC (D-108) only the LOCAL leg wired -- rbd-mirror ceph-remote (peer ceph-mon) and radosgw multisite (realm/zonegroup/zone + sync) are absent; both are cross-model / cross-controller (D-104) and staged as DR follow-ups per D-108's "prove one-way first".

C. VERIFY-LIVE gates (into the deploy runbook; not render changes)

  • vault 1.8/stable real HA-on-MySQL at 3 units (D-121's own GATE) -- confirm leader election on the MySQL backend before commit; if it fails, evaluate Raft -> D-068, NEVER fall back to etcd.
  • hacluster 2.4/stable cluster_count -- confirm it means "peers required before VIP binds"; else the VIP can bootstrap off one node (decorative HA).
  • Ceph pools land at defaults size=3 / min_size=2 / failure-domain=host -- not set anywhere; the D-121 R-3 rebuild-headroom argument silently depends on these holding on 4 storage nodes. Post-deploy assertion.
  • ovn-chassis dc1 compute provider MACs present locally (section A).

D. RESOLVE-BEFORE-DEPLOY (operator decisions; VIP-overlay layer, separate from the render)

  • Vault VIP (decorative-HA regression). dc-ha-scaleup.yaml scales vault -> 3 + vault-hacluster cluster_count: 3 + re-adds [vault:ha, vault-hacluster:ha], but NO vip: is set on vault anywhere (base or overlays) -- it was dropped with BUNDLEFIX-002 when vault was de-HA'd. D-036 lists vault among the apps carrying provider+metal VIPs. Without a VIP, kv consumers (barbican-vault -> vault:secrets) get one unit's vault_url, not a stable HA endpoint = decorative HA. Fix = add a dc1-band vault VIP in an overlay (needs a band octet) + the vault-HA VERIFY-LIVE (section C). designate is VIP-less BY DESIGN (dnsaas), not a regression -- the contrast confirms the class.
  • v4-only vs dual-stack phasing. overlays/vr1-dc1-vips.yaml (v4-only, concrete dc1 bands) and overlays/dc-dc-ipv6-family-matrix.yaml (v4+v6, unrendered {{TOKEN}}s) BOTH own vip: for the same 10 apps; overlay merge is per-key REPLACE, so applying both = last-wins (dual-stack lost, or literal tokens deployed). Only bites if both are applied together: a v4-first phase-4 deploy uses only vr1-dc1-vips (no collision). Operator decides whether dc1 phase-4 is v4-only (defer IPv6) or dual-stack (forces reconciling the two overlays into ONE vip string per app now).
  • machines-block overlay merge (vr1-dc1-machines.yaml) -- documented juju behavior, but no offline merger exists to prove it here; confirm at juju deploy --dry-run. Fallback: inline the dc1 tags into a rendered dc1 bundle.

E. Confirmed CORRECT under the expansion (no change)

  • All app relations / space bindings / subordinate associations HOLD under role separation -- because all 9 role nodes get the uniform 6-NIC set (SEC-011, uniform-NIC ruling); ceph clients on control LXD, octavia:ovsdb-cms -> data-tenant + ovn-chassis-octavia on control, ovn-chassis on the 2 compute nodes -- all satisfied. (Caveat: holds ONLY while the uniform-6-NIC decision stands.)
  • No channel/pin change required by 3-unit HA or role separation (D-002 pins hold).
  • lib-hosts.sh vr1-dc1 arm is complete + measured (9 role nodes, octet bands, boot MACs, per-DC tag) -- not a placeholder.
  • provider-bundle-check.py placement section is role-aware + harnessed (this session).
  • cloud-assert.sh A2/A3 already expect 3 units (mysql/ovn). ceph-osd=4, expected-osd-count: 4, monitor-count: 3, reserved-host-memory: 8192 (D-040, ruled retained) all correct.
  • dc-dc phase-3/4 runbooks are discover-live (do not hardcode a node count); the openstack0-3 loops live only in the VR0 teardown/reenroll path, not the dc1 deploy path.

F. Design signal

These findings motivate a NetBox->deploy coupling (a per-DC renderer/pipeline that GENERATES the overlays + tfvars from NetBox IPAM, making the deploy gates DC-aware and eliminating the hand-render fragility + the VIP-overlay collision by construction). Being scoped as a new PROPOSED D-number (NetBox-deploy coupling; next-free per ledger-scan) in the design-discussion seat (Chat). This is the escalation of the committee's Fork-2 (deferred renderer tool).