Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / runbooks / dc-dc-phase6-designate-cos-magnum.md
@JANeumatrix JANeumatrix 9 hours ago 34 KB DOCFIX-157/158/159: Stage 4/6/7 runbooks -- MAAS deploy, DR drill, Designate/COS/Magnum
Raw Blame History

DC-DC Phase 6 -- Designate, COS, Magnum/CAPI (per DC) (Stage 7)

Complete the per-DC service set: reactivate Designate (cloud-internal DNS), stand up COS observability, and bring up the Magnum/CAPI management + workload cluster path -- once per DC, against that DC's OWN Juju model and Keystone. This is the LAST per-DC build stage before the buildout is functionally complete in both DCs; DR wiring (Stage 6 / Phase 5) is a peer stage this runbook does not re-verify beyond confirming its gate passed.

This runbook is written PREP-ONLY: no DC1/DC2 cloud exists or is reachable this session. Every command below is the procedure to run LATER, once each DC has a live Charmed OpenStack deployment and its own Juju controller (Stage 4 / Phase 4 in docs/dc-dc-buildout-design.md). Every literal that depends on that future live state is marked MEASURE and must be resolved from real output at execution time -- nothing here is carried over from the VR0 DC0 as-built values quoted for reference.

Entry condition: Stage 6's gate (DR wiring / failover drill) has passed for the DC you are about to work on. This runbook does not re-verify Stage 6; it only requires that gate closed before starting.

Decisions: D-106 (Designate reactivation -- supersedes D-019, reactivates D-008), D-008 (original DNS architecture / bootstrap order, now reactivated verbatim), D-019 (the v1 Designate deferral D-106 supersedes; its root cause is what this stage closes), D-021 (the amphora-pipeline constraint whose root cause is the D-019 no-DNS posture -- resolved here), D-105 (CAPI mgmt-cluster placement + COS/Magnum scope, per DC, carries D-035 and D-046), D-046 (Magnum trustee domain-setup re-run trap), D-101 (IPv6 family matrix -- gates whether AAAA records are populated), D-110 (numbering; no new D-NNN is assigned by this runbook).

Troubleshooting: runbooks/appendix-A-troubleshooting.md for anything charm/ driver-specific inherited from phase-06/07/08 below; this runbook does not duplicate that index.

!!! D-046 TRAP -- READ BEFORE TOUCHING MAGNUM, EVERY TIME, PER DC: "the magnum charm action domain-setup is MANUAL, not automatic; magnum reports active / 'Unit is ready' regardless of whether it has run" (D-046). Per D-105: "The Magnum domain-setup action is re-run after every teardown / redeploy per DC (D-046 trap: magnum reports 'Unit is ready' regardless)." Magnum's unit status is NOT evidence the trustee domain exists. Re-run and re-assert Step 7.0 of phase-07-conductor-graft.md (reproduced as Step 6.3 below) after ANY teardown/redeploy of either DC's magnum unit -- do not skip it because the unit looks healthy.

!!! BUNDLE.YAML IS A BUILT SURFACE -- this runbook's Step 1 changes it. Per this repo's standing discipline ("grep design-decisions for the governing D-NNN before ANY change to a built surface"): D-106 is the governing decision and its Status line reads ADOPTED (2026-07-09 operator ruling), so the edit below IS authorized in principle. That is NOT a license to wave the edit through unreviewed -- it is still a verify-before-mutate, individually-gated repo change like any other built-surface edit (repo-lint clean, preflight.sh PASS before any live apply, changelog entry with a revert). Do not fold it into a "just add designate" one-liner; read Step 1 in full before editing.

!!! COS MECHANISM IS UNDETERMINED -- do not invent it. D-105 rules the SCOPE (COS in-scope, per-DC, no Office1 roll-up) but no D-NNN or design doc in this repo specifies the COS deployment MECHANISM (bundle vs. separate Juju model, exact channel, cross-model-relation topology). Step 5 below presents the standard options for the operator to pick from at execution time -- it does not pre-select one.

Sequence (run ONCE PER DC -- DC1 first, then DC2, independently; do not

assume DC2 repeats DC1's literals -- each DC has its own Keystone)

0.  Entry gate + DC context            (read-only: confirm Stage 6 gate + which
                                         DC's controller/model you are targeting)
1.  bundle.yaml edit (Designate)       [REPO CHANGE: reviewed once, applies to
                                         both DCs -- gated behind D-106]
2.  Static /etc/hosts bootstrap        (D-008 step 1, per DC)
3.  os-public-hostname per API charm   [MUTATION: juju config, per DC]
4.  Vault issues FQDN-SAN certs        [MUTATION: cert cascade, per DC]
5.  Deploy designate + designate-bind  [MUTATION: bundle-diff apply into the
    into the LIVE per-DC model          live model, per DC]
6.  Designate zone + A/AAAA records    [MUTATION: per DC, from measured values]
7.  Neutron default_dns_domain/        [MUTATION: per DC]
    dns_servers -> Designate
8.  Tenant subnets --dns-nameserver    [MUTATION: per DC, incl. capi-mgmt-subnet]
9.  Verify D-008/D-106 bootstrap       (read-only: resolution + cert validity)
10. COS observability                  [DECISION POINT + MUTATION, per DC]
11. CAPI mgmt cluster                  (reuse phase-06-incloud-mgmt-cluster.md,
                                         per DC, DC-parameterized)
12. Magnum conductor graft             (reuse phase-07-conductor-graft.md, per
                                         DC -- D-046 trap, Step 6.3 below)
13. Workload cluster acceptance        (reuse phase-08-workload-cluster-
                                         acceptance.md, per DC)
    -> GATE (this doc) -> Stage 7 exit; repeat 0-13 for the other DC if not
       already done

Command-label convention

Every command block below is bracketed by bold labels, so a command line is never mistaken for surrounding prose (these render in GitBucket and read clearly in a raw editor):

  • RUN -- LOC -- the block CHANGES state; run it at LOC (e.g. jumphost, vault/0, jumphost -> magnum/0).
  • CHECK (read-only) -- LOC -- a read-only verification; safe to re-run.
  • GATE: -- a hard stop; do NOT proceed past the block unless the stated condition holds.
  • Expect: -- what a passing result looks like.
  • > CAUTION: -- marks a destructive, secret-handling, or irreversible step.

Step 0 -- Entry gate + DC context (READ-ONLY)

MEASURE, never assume: which DC you are working, which Juju controller/model fronts it, and which admin credentials scope to it. Stage 4 (docs/dc-dc- buildout-design.md Phase 4) is where each DC's controller and Keystone came into existence; this runbook does not know that stage's naming choices in advance (they are not yet made this session) -- confirm them from that stage's own as-built record before substituting $DC, $DC_CONTROLLER, $DC_MODEL below.

CHECK (read-only) -- jumphost / Office1 operator seat

# Set once per DC-pass through this runbook -- MEASURE from Stage 4's as-built
# record, never guess a naming scheme:
#   DC=dc1|dc2
#   DC_CONTROLLER=<this DC's own Juju controller name, from Stage 4>
#   DC_MODEL=<this DC's OpenStack model name, from Stage 4 -- may or may not be
#             literally "openstack"; VR0/DC0 used "openstack" but VR1 per-DC
#             naming was not decided in this session>
juju controllers                 # confirm DC_CONTROLLER is listed and reachable
juju switch "${DC_CONTROLLER}:${DC_MODEL}"
juju status -m "${DC_MODEL}" --format=short | grep -vE 'active|idle' || echo "all active/idle"

GATE: the DC's controller is reachable, its model shows all charms active/idle (Stage 4/5/6 already delivered this), and you have confirmed which physical DC you are about to mutate. Do not proceed on an ambiguous target.

CHECK (read-only)

# Confirm Stage 6 (DR wiring / failover drill) closed for this DC -- this
# runbook's stated entry condition. Reference the Stage 6 runbook's own EXIT
# GATE for the exact check; it is not reproduced here (sibling-stage content).

GATE: Stage 6's own EXIT GATE has passed for this DC.

Step 1 -- bundle.yaml edit: reactivate Designate (REPO CHANGE, gated behind D-106)

1.1 -- What is currently in the repo (quoted, not paraphrased)

bundle.yaml's own description block states, at the line documenting decision D-019:

    D-019 (supersedes D-008) Designate deferred to v2

runbooks/phase-01-bundle-deploy.md states the dry-run/deploy expectation explicitly:

relations include
`octavia:certificates - vault:certificates`, `vault:shared-db - vault-mysql-router`,
`mysql-innodb-cluster:certificates - vault:certificates`; NO `vault:ha`, NO designate
(D-019).

And bundle.yaml itself carries the deferred application/subordinate as comments, not live entries:

  # designate-hacluster stays deferred (D-019).
  ...
  # v2-deferred (D-019): designate-hacluster: { charm: hacluster, channel: 2.4/stable }

There is no designate: or designate-bind: application block anywhere in bundle.yaml today. This is the concrete state D-106 reverses.

1.2 -- The governing decision, quoted

D-106 (Status: ADOPTED 2026-07-09) states:

Designate is deployed in-bundle in each DC, backend designate-bind (bind9, the Charmed-OpenStack-native backend, lowest delta).

and the reactivated bootstrap order, quoted verbatim (this is D-008's original text, reactivated by D-106 without change):

static /etc/hosts bootstrap -> os-public-hostname per API charm -> Vault issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron default_dns_domain and dns_servers -> tenant subnets with --dns-nameserver.

D-106 also carries a naming instantiation, not a new invention -- this is the one place a "zone name" appears in this runbook, and it is cited from the decision text itself, not invented:

Naming per D-008 instantiated with the VR1 labels: <service>.omega.dc1.vr1.cloud.neumatrix.local and ...dc2.vr1.... Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA records (AAAA for the v6-enabled planes per D-101).

1.3 -- The reversal is bigger than "add one application" -- flag this honestly

D-106's bootstrap order includes os-public-hostname per API charm as its SECOND step. bundle.yaml's current header records the opposite posture, under its own B5 label (not a separate D-NNN -- a 2026-06-01 bundle-cleanup item):

# Endpoints:        IP-ONLY -- os-public-hostname dropped on all API charms; the dual VIPs ARE the
#                   catalog endpoints (public 10.12.4.N / internal+admin 10.12.8.N). Vault issues
#                   per-VIP IP-SAN certs. No control-plane DNS dependency. (B5)

D-106's own text explicitly names os-public-hostname per API charm as part of the order it reactivates, so re-adding it is NOT an inferred scope addition -- it is in the ratified decision text itself. But it IS a second, larger surface than "add designate": every API charm's os-public-hostname option goes from unset back to a real FQDN, and Vault's cert issuance moves from per-VIP IP-SAN back to FQDN-SAN. Treat this as two coupled changes under one D-106 umbrella, not one:

  1. Add designate + designate-bind (+ designate-hacluster) applications and their relations.
  2. Re-add os-public-hostname: <fqdn> to every clustered API charm's options: block (the same charm list D-020 already dual-VIPs: keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, octavia, barbican, magnum, openstack-dashboard, ceph-radosgw).

If, on review, the operator wants item 2 scoped or sequenced differently from item 1 (e.g. reactivate DNS-serving without yet flipping every charm's hostname), that is a real design fork -- log it as a proposal against D-106 rather than deciding it inside this runbook.

1.4 -- Keep bundle.yaml DC-agnostic; push per-DC literals to an overlay

bundle.yaml is ONE shared file used to deploy both DC1 and DC2 (D-101: DC1 inherits DC0's v4 layout unchanged; DC2 gets its own NetBox-assigned supernet, still pending -- gap #3 in docs/dc-dc-deployment-workflow.md's tooling gap register). The new os-public-hostname values are NOT DC-agnostic -- they embed dc1 or dc2 per D-106's naming convention. Do not hardcode a DC-specific FQDN into the shared bundle.yaml. Follow the existing overlay pattern already established for secrets (overlays/octavia-pki.yaml, Step 1.0 of phase-01-bundle-deploy.md) and add one per-DC hostnames overlay: overlays/dc1-hostnames.yaml, overlays/dc2-hostnames.yaml, each setting only the os-public-hostname options for that DC, applied at deploy time with an additional --overlay alongside whichever overlays that DC's deploy already uses. This is a proposed pattern, not yet reviewed -- present it at the same review gate as the rest of Step 1, do not silently adopt it.

1.5 -- The application/relation shape (template -- CONFIRM before writing it live)

The block below is a STARTING TEMPLATE only. The exact channel and the exact relation/binding names are NOT verified against this repo's pinned Charmhub revisions this session -- confirm both via scripts/preflight.sh's Charmhub channel assert (the same mechanism that already gates every other charm in this bundle) before this goes into a real commit:

  designate:
    charm: designate
    channel: 2024.1/stable        # CONFIRM against Charmhub -- matches this
                                   # bundle's D-002 channel matrix pattern for
                                   # every other Caracal API charm; not yet
                                   # independently verified for designate.
    num_units: 1
    options:
      # os-public-hostname etc. as needed once the DC-specific overlay (1.4)
      # is drafted -- do not hardcode a zone/fqdn literal here.
    bindings:
      '': metal-admin
      identity-service: metal-internal
      shared-db: metal-internal
      dns-backend: metal-internal
      public: provider-public
      # CONFIRM these binding/relation endpoint names against the designate
      # charm's actual metadata.yaml -- written here from the standard
      # Charmed OpenStack pattern (identity-service / shared-db / dns-backend),
      # not from a live read of this repo's pinned revision.

  designate-bind:
    charm: designate-bind
    channel: 2024.1/stable        # CONFIRM, as above
    num_units: 1
    bindings:
      '': metal-admin
      dns-backend: metal-internal

  designate-mysql-router:
    charm: mysql-router
    channel: 8.0/stable            # matches every other *-mysql-router in this bundle
    bindings:
      '': metal-admin
      db-router: metal-internal
      shared-db: metal-internal

  # un-comment and activate (currently deferred at bundle.yaml's HA-subordinates block):
  designate-hacluster:
    charm: hacluster
    channel: 2.4/stable
    options: {cluster_count: 1}
    bindings: {'': metal-admin, ha: metal-internal, hanode: metal-internal,
               pacemaker-remote: metal-internal, peer-availability: metal-internal}

Relations to add (mirroring the existing pattern for every other clustered API charm in this bundle -- keystone:identity-service, the charm's own :shared-db to its mysql-router, :certificates to vault, :ha to its hacluster subordinate, plus designate:dns-backend - designate-bind:dns-backend). Do not copy these blindly -- diff against an actual juju deploy --dry-run on a real DC before treating this as final.

CHECK (read-only) -- jumphost, once bundle.yaml + overlays are edited

bash scripts/repo-lint.sh          # expect 0 fail
bash scripts/preflight.sh          # expect PREFLIGHT: PASS (incl. Charmhub channel assert)

GATE: repo-lint clean; preflight PASS. Only after this passes does Step 5 (the live per-DC apply) become eligible. Deliver this edit with a changelog entry (what changed, why -- D-106, how to revert) per this repo's standing delivery rule, same as any other script/runbook/bundle change.

Step 2 -- Static /etc/hosts bootstrap (D-008 step 1, per DC)

MEASURE this DC's own hostnames and IPs from its live juju status / maas admin machines read output -- do not carry over DC0's /etc/hosts entries or VR0's IP ranges. Per D-101, DC1's v4 layout equals DC0's (deliberate, ratified reuse -- not an assumption), so DC1's measured values may look identical to DC0's on paper; measure them live anyway. DC2's values are, per gap #3, not yet assigned in NetBox as of this writing -- if DC2's literals still are not populated by the time you reach this step, STOP and resolve that gap first (do not invent a DC2 supernet here).

CHECK (read-only) -- this DC's jumphost-equivalent seat

juju status -m "${DC_MODEL}" --format=yaml | grep -E 'public-address|hostname'

RUN -- every host + LXD container in this DC's model (mirrors phase-00's Pattern A hosts, D-008 step 1's "jumphost + all openstack hosts + all LXD containers" scope, but for THIS DC)

# Populate /etc/hosts on each unit with THIS DC's own measured
# <fqdn> <ip> pairs, one line per API charm, using D-106's naming:
#   <service>.omega.${DC}.vr1.cloud.neumatrix.local
# Do not template a value you have not just measured.

GATE: every host/LXD container in this DC resolves every other API charm's FQDN via /etc/hosts (the D-008 bootstrap floor, before Designate itself exists).

Step 3 -- os-public-hostname per API charm (D-008 step 2, per DC)

RUN -- jumphost, per charm (repeat for every clustered API charm listed in Step 1.3)

juju config -m "${DC_MODEL}" keystone os-public-hostname="keystone.omega.${DC}.vr1.cloud.neumatrix.local"
# ... repeat per charm: glance, neutron-api, nova-cloud-controller, placement,
#     cinder, octavia, barbican, magnum, openstack-dashboard, ceph-radosgw

CHECK (read-only) -- jumphost

juju status -m "${DC_MODEL}" --format=short | grep -vE 'active|idle' || echo "all active/idle"

GATE: all charms return to active/idle after the config change settles (a hostname-only config change is non-disruptive on this charm family, but confirm rather than assume for this DC).

Step 4 -- Vault issues FQDN-SAN certs (D-008 step 3, per DC)

This DC's Vault is an INDEPENDENT root (D-109) issuing its own certs -- do not reuse a cert or CA from the other DC or from VR0 DC0. Trigger the certificates relation to re-issue now that os-public-hostname is set; the charm-native cert cascade (Vault's certificates interface) picks up the new SAN automatically once the relation re-fires -- no manual CSR step is expected on this charm family (confirm this is still true for the designate charm's own certificate once Step 5 relates it).

CHECK (read-only) -- jumphost, per charm

juju ssh -m "${DC_MODEL}" keystone/leader \
  "sudo openssl x509 -in /etc/apache2/ssl/keystone_1/cert -noout -text | grep -A1 'Subject Alternative Name'" </dev/null
# expect the FQDN now present in the SAN list, alongside the existing VIP IP-SANs
# (D-020 dual-VIP posture is not being reversed -- IP-SANs stay, FQDN-SAN is added)

GATE: every clustered API charm's live cert carries the DC's own FQDN in its SAN. Do not proceed to Designate zone population (Step 6) until this holds -- Designate resolving a name to a VIP whose cert does not cover that name reproduces the exact D-019/D-021 failure mode this stage exists to close.

Step 5 -- Deploy designate + designate-bind into the LIVE per-DC model

This DC's OpenStack cloud is already running (Stage 4 delivered it). Adding Designate here is a bundle-diff INCREMENTAL apply against the live model, not a fresh juju deploy from empty -- Juju's bundle handling adds only the new applications/relations the diff introduces; it does not touch already-running apps whose config is unchanged in the bundle.

CHECK (read-only) -- jumphost -- dry-run against the live model first

juju deploy ./bundle.yaml \
  --overlay overlays/octavia-pki.yaml \
  --overlay "overlays/${DC}-hostnames.yaml" \
  -m "${DC_MODEL}" --dry-run

GATE: the diff shows ONLY the new designate/designate-bind/designate- mysql-router/designate-hacluster applications and their relations being added -- zero unexpected changes to any already-running application. If the diff shows anything else changing (e.g. an existing charm's binding), STOP and reconcile before applying.

RUN -- jumphost

juju deploy ./bundle.yaml \
  --overlay overlays/octavia-pki.yaml \
  --overlay "overlays/${DC}-hostnames.yaml" \
  -m "${DC_MODEL}"

GATE: designate, designate-bind, designate-mysql-router, designate-hacluster all reach active/idle; designate:dns-backend - designate-bind:dns-backend and designate:identity-service - keystone:identity-service relations up.

Step 6 -- Designate zone + A/AAAA records (D-008 step 4, per DC)

Per-DC INDEPENDENT zone (D-106). The zone name itself is the one place this runbook cites a literal beyond a live measurement, and it is cited from D-106's own ratified naming text, not invented: omega.${DC}.vr1.cloud.neumatrix.local. The A/AAAA record VALUES are NOT cited from any decision text -- they are this DC's real, live, measured VIP addresses. Populate them from what Step 3/4 just proved live, never from a planning number.

RUN -- jumphost (designate-scoped or admin, per this DC's own admin-openrc)

openstack zone create --email hostmaster@neumatrix.local \
  "omega.${DC}.vr1.cloud.neumatrix.local."

# per API charm, MEASURE the live provider VIP (A) and, if this DC's GUA plane
# is up (D-101), the live provider-public GUA address (AAAA) -- do not invent
# either:
KEYSTONE_VIP_V4=$(openstack ... )   # MEASURE: this DC's live keystone provider VIP
openstack recordset create "omega.${DC}.vr1.cloud.neumatrix.local." \
  keystone --type A --records "$KEYSTONE_VIP_V4"

# AAAA only if D-101's v6-enabled planes are confirmed up for this DC -- gate
# on that confirmation, do not assume:
KEYSTONE_VIP_V6=$(openstack ... )   # MEASURE: this DC's live GUA provider address
openstack recordset create "omega.${DC}.vr1.cloud.neumatrix.local." \
  keystone --type AAAA --records "$KEYSTONE_VIP_V6"
# repeat per API charm

GATE: every API charm has an A record resolving to its own live provider VIP; AAAA records exist wherever D-101's GUA/dual-stack planes are confirmed up for this DC (not before -- an AAAA record pointing at an address that is not yet live is worse than no record).

Step 7 -- Neutron default_dns_domain / dns_servers -> Designate (D-008 step 5, per DC)

RUN -- jumphost

DESIGNATE_VIP=$( ... )   # MEASURE: this DC's live designate-bind / designate API VIP
juju config -m "${DC_MODEL}" neutron-api \
  dns-domain="omega.${DC}.vr1.cloud.neumatrix.local." \
  dns-servers="${DESIGNATE_VIP}"

GATE: neutron-api returns to active/idle; `openstack network show

` (or the relevant config surface) reflects the new dns_domain.

Step 8 -- Tenant subnets --dns-nameserver (D-008 step 6, per DC)

Every existing tenant subnet in this DC created BEFORE Designate existed (most notably capi-mgmt-subnet from phase-06 Step 6.1, which set --dns-nameserver 1.1.1.1 --dns-nameserver 1.0.0.1 under the D-019 "no cloud DNS, tenants use public resolvers" posture) needs updating, and every subnet created AFTER this point should use the Designate VIP from the start.

RUN -- jumphost, per existing tenant subnet

DESIGNATE_VIP=$( ... )   # MEASURE: same value as Step 7
openstack subnet set --dns-nameserver "$DESIGNATE_VIP" capi-mgmt-subnet
# clear the old public-resolver entries only if the subnet update replaces
# rather than appends -- confirm openstack-client's actual semantics for
# --dns-nameserver on `subnet set` live, do not assume append vs. replace

GATE: openstack subnet show capi-mgmt-subnet shows the Designate VIP as the (or a) DNS nameserver.

Step 9 -- Verify the D-008/D-106 bootstrap end to end (read-only)

CHECK (read-only) -- jumphost -> a tenant VM in this DC (e.g. capi-mgmt-v2)

source ~/capi-mgmt-net.env   # this DC's own env file, per phase-06 Step 6.2 -- confirm it is THIS DC's, not the other DC's stale file
ssh ... ubuntu@"$MGMT_FIP" bash -s <<'REOF'
getent hosts keystone.omega.DC.vr1.cloud.neumatrix.local   # substitute the real DC label
dig +short AAAA keystone.omega.DC.vr1.cloud.neumatrix.local  # if AAAA populated
curl -sk https://keystone.omega.DC.vr1.cloud.neumatrix.local:5000/v3 -o /dev/null -w '%{http_code}\n'
REOF

GATE: the FQDN resolves (A, and AAAA where populated); the HTTPS call against the FQDN succeeds with a cert the client trusts (Vault's CA is this DC's own root, per D-109 -- the tenant VM/OS trust store must carry it, same as any other charm's client already does). This is the concrete proof the D-019/D-021 root cause (metal-only charms pulling a public FQDN endpoint that does not resolve/route, behind a cert with no FQDN SAN) is closed for this DC.

Step 10 -- COS observability (DECISION POINT + MUTATION, per DC)

No D-NNN or design doc in this repo specifies the COS deployment mechanism -- only its scope (D-105: per-DC, in scope, no Office1 roll-up). Do not pick one silently. Present the standard options to the operator at execution time:

  • Option A -- cos-lite bundle in a dedicated per-DC cos Juju model (Canonical's standard COS Lite: Prometheus, Grafana, Loki, Alertmanager, Traefik ingress), related to this DC's openstack model applications via Juju cross-model relations (juju offer / juju consume + cos-configuration-k8s or the grafana-agent subordinate pattern for metrics/log forwarding). Needs a Kubernetes substrate for the cos model (microk8s or equivalent) -- confirm this DC has one before picking this option.
  • Option B -- machine-charm-native observability (prometheus2 / grafana / telegraf charms in the SAME model as the OpenStack bundle, no k8s substrate needed) -- lower delta if no k8s host is available in this DC for a cos model, but is not "COS" in the strict Canonical-bundle sense; confirm this is an acceptable substitution for the D-105 "COS" scope before treating it as satisfying the gate.

CHECK (read-only) -- confirm before choosing

# Does this DC have a k8s substrate available for a cos-lite model?
# (microk8s, or another already-enrolled k8s -- MEASURE, do not assume)

GATE (decision, log it): record which option was chosen and why in the session ledger / a changelog entry -- this is a real architectural choice, not a detail to bury in a command block.

Once chosen, the deployment steps are option-specific and are NOT detailed further here (this is genuinely new tooling territory for this repo -- author the concrete steps as their own change when the option is picked, with its own harness per this repo's delivery rule, rather than improvising them live).

GATE (Stage 7's COS criterion): COS scraping -- at minimum, this DC's OpenStack API charms' metrics/logs and the CAPI mgmt cluster + workload cluster (Steps 11-13) are visible in whichever COS surface was chosen.

Step 11 -- CAPI management cluster (reuse phase-06-incloud-mgmt-cluster.md, per DC)

Run runbooks/phase-06-incloud-mgmt-cluster.md Steps 6.0-BOOT through 6.6f IN FULL, once per DC, against this DC's own project/Keystone/model. Do not duplicate its content here -- point at it by filename. DC-parameterization notes (found by tracing what that runbook's canonical scripts actually resolve dynamically vs. hardcode):

  • scripts/phase-06-k8s-bootstrap.sh sources scripts/lib-net.sh and reads KEYSTONE_VIP_DEFAULT from it (DOCFIX-138 centralized this). Before running it for a DC other than DC0, call lib_net_select_dc "$DC" (DOCFIX-151) FIRST: for dc1 this is currently a documented no-op (D-101: DC1 inherits DC0's v4 layout unchanged -- a ratified reuse, not a stale default), for dc2 it currently FAILS LOUDLY (lib-net.sh's dc2 literals are not yet assigned -- tooling gap register #3). If dc2's literals are still unassigned by the time you reach this stage, STOP; do not bypass the fail-loud guard.
  • scripts/phase-06-kubeconfig-gate.sh -- same lib-net.sh dependency; same lib_net_select_dc "$DC" call required first.
  • Every other literal in phase-06 (ENV(project), ENV(ext-net), ENV(image), ENV(mgmt-fip), project/image/network UUIDs) is ALREADY written as "resolve by name" / "resolve dynamically" in that runbook -- it re-discovers these fresh on every run, so it needs no further DC- parameterization beyond pointing it at this DC's own admin-openrc and Keystone. Confirm you have sourced THIS DC's own admin-openrc (naming convention for per-DC openrc files was not established this session -- confirm it from Stage 4's as-built record, do not assume a filename).
  • ENV(mgmt-cidr) 10.20.0.0/24 is a non-IPAM tenant overlay CIDR -- safe to reuse identically in both DCs (they are independent clouds; no collision risk). Do not route this through NetBox.
  • scripts/lib-hosts.sh's lib_hosts_select_dc() is NOT used by phase-06 (it governs MAAS host enrollment, a Stage 3 concern, not the tenant-VM layer phase-06 operates at) -- no action needed here, noted so it is not mistakenly assumed required.

GATE (this runbook, restating phase-06's EXIT GATE): per DC, GATE 1 (VIP-OK) and GATE 2 (agnhost pod-egress Completed) both pass; the CAPI provider stack is up with no crash-looping CAPO.

Step 12 -- Magnum conductor graft (reuse phase-07-conductor-graft.md, per DC)

Run runbooks/phase-07-conductor-graft.md in full, once per DC, against this DC's own magnum/0 and this DC's phase-06 kubeconfig/FIP. Point at it by filename; do not duplicate its content.

!!! D-046 GATE -- REPRODUCED HERE BECAUSE IT IS THE MOST-MISSED STEP IN THIS DECISION-DERIVED CHECKLIST, restated verbatim from D-105/D-046:

The magnum charm action domain-setup is MANUAL, not automatic; magnum reports active / "Unit is ready" regardless of whether it has run.

Concretely, this means: for THIS DC, run phase-07's Step 7.0 (juju run magnum/leader domain-setup, then assert openstack domain show magnum and openstack user show magnum_domain_admin --domain magnum both succeed, then gate openstack coe service list with no 403) EVERY time this DC's magnum unit is torn down and redeployed -- including if this whole Stage 7 pass is itself a redo after an earlier Stage 4-6 rebuild of this DC. Do not treat a green juju status for magnum as evidence this step already happened.

DC-parameterization: phase-07's ENV(conductor-unit), ENV(mgmt-fip), ENV(magnum-ns) are all already written as "resolve by name" / per-rebuild -- re-resolve them fresh against THIS DC's model; do not carry over the other DC's or DC0's recorded values (the as-built reference blocks in that runbook are VR0/DC0 audit trail, not this DC's expected values).

GATE (this runbook, restating phase-07's EXIT GATE): Step 7.0's domain- setup assertion passes; conductor reaches the mgmt apiserver; magnum-capi-helm 1.4.0 (or whatever the then-current contract-coherent pin is -- confirm against D-042/D-034 rather than assuming 1.4.0 is still current) installed; health_status gate discharged per that runbook's FRESH DEPLOY routing (i.e. deferred to Step 13 if no cluster exists yet in this DC).

Step 13 -- Workload-cluster acceptance (reuse phase-08-workload-cluster-acceptance.md, per DC)

Run runbooks/phase-08-workload-cluster-acceptance.md in full, once per DC. Point at it by filename; do not duplicate its content. One correction to apply when running it under D-106 (rather than the D-019-amended acceptance bar that runbook currently documents): D-011 item 8 ("Designate zones + tenant hostname resolution") is NO LONGER deferred for VR1 -- Step 9 above is what discharges it. Do not mark it DEFERRED per the old D-019 wording when running this under D-106; record it PASS/FAIL against Step 9's actual result instead.

DC-parameterization: same shape as Steps 11-12 -- every ENV(...) in that runbook already resolves by name; point it at this DC's own project/model, re-seed this DC's own workload image via stage-and-verify (D-048) rather than assuming the image already exists in a fresh DC.

GATE (this runbook, restating phase-08's EXIT GATE, D-106-corrected): capi-test-1-equivalent cluster reaches CREATE_COMPLETE; 3 (or as-templated) nodes Ready; CNI/CCM/CSI/CoreDNS Running; D-011 items 1-6 PASS; item 7 (snapshot) per this DC's own baseline policy; item 8 PASS (not deferred) via Step 9's resolution proof.

GATE -- Stage 7 exit condition (per DC; both DC1 and DC2 must pass independently)

Restating docs/dc-dc-buildout-design.md Phase 6 / docs/dc-dc-deployment- workflow.md Stage 7's gate:

  • Designate resolving A + AAAA for this DC's own zone (Step 6/9; AAAA only where D-101's v6 planes are confirmed up).
  • FQDN-SAN certs valid (Step 4/9) -- the D-019/D-021 root cause (metal-only charms pulling an unresolvable/unroutable public FQDN behind an IP-only cert) is CLOSED for this DC.
  • COS scraping this DC's OpenStack + CAPI mgmt + workload cluster (Step 10).
  • A CAPI workload cluster comes up in this DC (Step 13, CREATE_COMPLETE).

Do not mark Stage 7 complete for the DC-DC buildout as a whole until BOTH DCs independently clear all four bullets -- one DC passing does not imply the other DC's independent Keystone/Vault/Designate zone/CAPI stack also passes.

Delivery checklist

  • bundle.yaml + new overlays/dc1-hostnames.yaml / `overlays/dc2- 
    hostnames.yaml` edited per Step 1; `scripts/repo-lint.sh` 0 fail;
`scripts/preflight.sh` PASS (incl. Charmhub channel assert on the new
designate/designate-bind charms).
  • Changelog entry for the bundle.yaml + overlay change: what changed 
    (designate reactivation + os-public-hostname re-add), why (D-106,
superseding D-019), how to revert (re-comment the designate blocks,
drop the hostnames overlays, re-run preflight to confirm the revert is
clean).
  • docs/session-ledger.md updated with this runbook's authoring and, 
    once executed, each DC's Stage 7 pass/fail state.
  • docs/dc-dc-deployment-workflow.md Stage 7 row's **State:** line 
    updated from NOT STARTED once execution begins/completes.
  • Per DC: Steps 0-9 (Designate reactivation) executed and Step 9's gate 
    passed BEFORE Step 10 (COS) is attempted -- do not build COS scraping
against a DNS layer that has not yet proven itself.
  • Per DC: Step 10's COS mechanism choice recorded (which option, why).
  • Per DC: phase-06/07/08 run to their own EXIT GATES with THIS DC's own 
    resolved values (never a copy-pasted literal from DC0 or the other DC).
  • D-046 domain-setup re-run + assertion done for THIS pass, not assumed 
    from a prior one.
  • This runbook's own GATE section cleared for BOTH DCs before Stage 7 is 
    marked complete in the deployment workflow tracker.

Next

Once both DCs clear this runbook's GATE, the DC-DC buildout's per-DC service set is complete. Remaining work is whatever the deployment-workflow doc's "Where things stand today" table still lists as outstanding at that point (consult it fresh -- it is a living tracker, not reproduced here).