# DC-DC Phase 6 -- Designate, COS, Magnum/CAPI (per DC) (Stage 7) Complete the per-DC service set: reactivate Designate (cloud-internal DNS), stand up COS observability, and bring up the Magnum/CAPI management + workload cluster path -- once per DC, against that DC's OWN Juju model and Keystone. This is the LAST per-DC build stage before the buildout is functionally complete in both DCs; DR wiring (Stage 6 / Phase 5) is a peer stage this runbook does not re-verify beyond confirming its gate passed. This runbook is written PREP-ONLY: no DC1/DC2 cloud exists or is reachable this session. Every command below is the procedure to run LATER, once each DC has a live Charmed OpenStack deployment and its own Juju controller (Stage 4 / Phase 4 in `docs/dc-dc-buildout-design.md`). Every literal that depends on that future live state is marked `MEASURE` and must be resolved from real output at execution time -- nothing here is carried over from the VR0 DC0 as-built values quoted for reference. **Entry condition:** Stage 6's gate (DR wiring / failover drill) has passed for the DC you are about to work on. This runbook does not re-verify Stage 6; it only requires that gate closed before starting. Decisions: **D-106** (Designate reactivation -- supersedes D-019, reactivates D-008), **D-008** (original DNS architecture / bootstrap order, now reactivated verbatim), **D-019** (the v1 Designate deferral D-106 supersedes; its root cause is what this stage closes), **D-021** (the amphora-pipeline constraint whose root cause is the D-019 no-DNS posture -- resolved here), **D-105** (CAPI mgmt-cluster placement + COS/Magnum scope, per DC, carries D-035 and D-046), **D-046** (Magnum trustee domain-setup re-run trap), **D-101** (IPv6 family matrix -- gates whether AAAA records are populated), **D-110** (numbering; no new D-NNN is assigned by this runbook). Troubleshooting: `runbooks/appendix-A-troubleshooting.md` for anything charm/ driver-specific inherited from phase-06/07/08 below; this runbook does not duplicate that index. --- !!! D-046 TRAP -- READ BEFORE TOUCHING MAGNUM, EVERY TIME, PER DC: "the magnum charm action `domain-setup` is MANUAL, not automatic; magnum reports active / 'Unit is ready' regardless of whether it has run" (D-046). Per D-105: "The Magnum `domain-setup` action is re-run after every teardown / redeploy per DC (D-046 trap: magnum reports 'Unit is ready' regardless)." Magnum's unit status is NOT evidence the trustee domain exists. Re-run and re-assert Step 7.0 of `phase-07-conductor-graft.md` (reproduced as Step 6.3 below) after ANY teardown/redeploy of either DC's magnum unit -- do not skip it because the unit looks healthy. !!! BUNDLE.YAML IS A BUILT SURFACE -- this runbook's Step 1 changes it. Per this repo's standing discipline ("grep design-decisions for the governing D-NNN before ANY change to a built surface"): D-106 is the governing decision and its Status line reads ADOPTED (2026-07-09 operator ruling), so the edit below IS authorized in principle. That is NOT a license to wave the edit through unreviewed -- it is still a verify-before-mutate, individually-gated repo change like any other built-surface edit (repo-lint clean, `preflight.sh` PASS before any live apply, changelog entry with a revert). Do not fold it into a "just add designate" one-liner; read Step 1 in full before editing. !!! COS MECHANISM IS UNDETERMINED -- do not invent it. D-105 rules the SCOPE (COS in-scope, per-DC, no Office1 roll-up) but no D-NNN or design doc in this repo specifies the COS deployment MECHANISM (bundle vs. separate Juju model, exact channel, cross-model-relation topology). Step 10 below presents the standard options for the operator to pick from at execution time -- it does not pre-select one. (Corrected 2026-07-29: this read "Step 5", which is the Designate step, not the COS step.) --- ## Sequence (run ONCE PER DC -- DC1 first, then DC2, independently; do not ## assume DC2 repeats DC1's literals -- each DC has its own Keystone) ``` 0. Entry gate + DC context (read-only: confirm Stage 6 gate + which DC's controller/model you are targeting) 1. bundle.yaml edit (Designate) [REPO CHANGE: reviewed once, applies to both DCs -- gated behind D-106] 2. Static /etc/hosts bootstrap (D-008 step 1, per DC) 3. os-public-hostname per API charm [MUTATION: juju config, per DC] 4. Vault issues FQDN-SAN certs [MUTATION: cert cascade, per DC] 5. Verify designate present + active, (read-only assertion, then juju config then CONFIGURE it only -- NOT a deploy; designate ships in the bundle and Stage 5 installs it. D-106 ruling note 2026-07-27) 6. Designate zone + A/AAAA records [MUTATION: per DC, from measured values] 7. Neutron default_dns_domain/ [MUTATION: per DC] dns_servers -> Designate 8. Tenant subnets --dns-nameserver [MUTATION: per DC, incl. capi-mgmt-subnet] 9. Verify D-008/D-106 bootstrap (read-only: resolution + cert validity) 10. COS observability [DECISION POINT + MUTATION, per DC] 11. CAPI mgmt cluster (reuse phase-06-incloud-mgmt-cluster.md, per DC, DC-parameterized) 12. Magnum conductor graft (reuse phase-07-conductor-graft.md, per DC -- D-046 trap, Step 6.3 below) 13. Workload cluster acceptance (reuse phase-08-workload-cluster- acceptance.md, per DC) -> GATE (this doc) -> Stage 7 exit; repeat 0-13 for the other DC if not already done ``` --- ## Command-label convention Every command block below is bracketed by bold labels, so a command line is never mistaken for surrounding prose (these render in GitBucket and read clearly in a raw editor): - **RUN -- LOC** -- the block CHANGES state; run it at LOC (e.g. `jumphost`, `vault/0`, `jumphost -> magnum/0`). - **CHECK (read-only) -- LOC** -- a read-only verification; safe to re-run. - **GATE:** -- a hard stop; do NOT proceed past the block unless the stated condition holds. - **Expect:** -- what a passing result looks like. - `> CAUTION:` -- marks a destructive, secret-handling, or irreversible step. --- ## Step 0 -- Entry gate + DC context (READ-ONLY) MEASURE, never assume: which DC you are working, which Juju controller/model fronts it, and which admin credentials scope to it. Stage 4 (`docs/dc-dc- buildout-design.md` Phase 4) is where each DC's controller and Keystone came into existence; this runbook does not know that stage's naming choices in advance (they are not yet made this session) -- confirm them from that stage's own as-built record before substituting `$DC`, `$DC_CONTROLLER`, `$DC_MODEL` below. **CHECK (read-only) -- jumphost / Office1 operator seat** ```bash # Set once per DC-pass through this runbook -- MEASURE from Stage 4's as-built # record, never guess a naming scheme: # DC=vr1-dc0|vr1-dc1 # the D-119 region-qualified selector. Bare # # dcN is REJECTED loudly by lib-net/lib-hosts. # DC_CONTROLLER= # DC_MODEL= # DERIVE the two D-008 FQDN labels from the selector -- NEVER interpolate $DC # itself into a name (DOCFIX-205). The D-119 SELECTOR token and the D-008 # LABEL are different strings: "omega.${DC}.vr1..." expands to # "omega.vr1-dc0.vr1..." and carries the region TWICE. Same shell parameter # expansions octavia-pki.sh uses; verified for vr1-dc0, vr1-dc1 and vr0-dc0. DC_REGION="${DC%%-*}" # vr1-dc0 -> vr1 (D-008 ) DC_DNS_LABEL="${DC#*-}" # vr1-dc0 -> dc0 (D-008 , 0-indexed per D-117) # label + base domain from the ONE place they live: a rebuild that renames the # estate edits scripts/lib-identity.sh only (2026-07-30). source "$REPO/scripts/lib-identity.sh" || { echo "FAIL: cannot read scripts/lib-identity.sh"; exit 1; } DC_ZONE="${CLOUD_NAME}.${DC_DNS_LABEL}.${DC_REGION}.${CLOUD_DOMAIN}" echo "$DC_ZONE" # EXPECT omega.dc0.vr1.cloud.neumatrix.local (vr1-dc0) juju controllers # confirm DC_CONTROLLER is listed and reachable juju switch "${DC_CONTROLLER}:${DC_MODEL}" juju status -m "${DC_MODEL}" --format=short | grep -vE 'active|idle' || echo "all active/idle" ``` **GATE:** the DC's controller is reachable, its model shows all charms `active/idle` (Stage 4/5/6 already delivered this), and you have confirmed which physical DC you are about to mutate. Do not proceed on an ambiguous target. **CHECK (read-only)** ```bash # Confirm Stage 6 (DR wiring / failover drill) closed for this DC -- this # runbook's stated entry condition. Reference the Stage 6 runbook's own EXIT # GATE for the exact check; it is not reproduced here (sibling-stage content). ``` **GATE:** Stage 6's own EXIT GATE has passed for this DC. --- ## Step 1 -- bundle.yaml edit: reactivate Designate (REPO CHANGE, gated behind D-106) ### 1.1 -- What is currently in the repo (measured, not paraphrased) **CORRECTED 2026-07-29 per the D-106 RULING NOTE of 2026-07-27.** This section previously ended "There is no `designate:` or `designate-bind:` application block anywhere in `bundle.yaml` today", which CONTRADICTED section 1.2 below and the bundle itself. The ruling names this defect directly: ":172 states 'There is no `designate:` or `designate-bind:` application block anywhere in ...' while :178 states 'Designate is deployed in-bundle in each DC' ... Both cannot be true; the bundle settles it, and DOCFIX-167 put designate there on 2026-07-10." **MEASURED STATE OF `bundle.yaml` TODAY -- designate IS in the bundle.** Four application blocks plus eight relations, all of whose peers Stage 5 creates: | Application | Charm / channel | Placement | Where | |---|---|---|---| | `designate` | `designate` `2024.1/stable`, `num_units: 1` | `to: [lxd:0]` | `bundle.yaml:853` | | `designate-mysql-router` | `mysql-router` `8.0/stable` | subordinate | `bundle.yaml:872` | | `designate-bind` | `designate-bind` `2024.1/stable`, `num_units: 1` | `to: [lxd:2]` | `bundle.yaml:881` | | `designate-hacluster` | `hacluster` `2.4/stable`, `cluster_count: 1` | subordinate | `bundle.yaml:963` | (Line numbers are as of 2026-07-29; grep the app names, not the numbers, if the bundle has moved since.) Note `designate`'s provides-side binding is `dnsaas: provider-public`, not `public` (see 1.5) -- which is why it does not join provider-bundle-check's `EXPECT_PUBLIC_VIP` count of 11. **HISTORICAL -- the pre-D-106 state, retained because it is what the decision reverses.** Before DOCFIX-167 (2026-07-10), `bundle.yaml` recorded the D-019 deferral in its description block (`D-019 (supersedes D-008) Designate deferred to v2`) and carried the subordinate as a comment rather than a live entry (`# v2-deferred (D-019): designate-hacluster: ...`); `runbooks/phase-01-bundle-deploy.md` still states the VR0 dry-run expectation `NO vault:ha, NO designate (D-019)`. That phase-01 line is VR0/testcloud scope and is NOT a claim about this bundle's current VR1 content -- do not read it as one. **CONSEQUENCE FOR THIS RUNBOOK, RULED:** designate lands as INERT INFRASTRUCTURE at Stage 5 -- it deploys with the bundle and does nothing, because `os-public-hostname` is set in no deploy artifact. Stage 7 (this runbook) does not install it. Step 5 below is a CONFIGURE step, not a deploy step. ### 1.2 -- The governing decision, quoted D-106 (Status: ADOPTED 2026-07-09) states: > Designate is deployed in-bundle in each DC, backend `designate-bind` (bind9, > the Charmed-OpenStack-native backend, lowest delta). and the reactivated bootstrap order, quoted verbatim (this is D-008's original text, reactivated by D-106 without change): > static /etc/hosts bootstrap -> `os-public-hostname` per API charm -> Vault > issues FQDN-SAN certs -> Designate zone plus A/AAAA -> Neutron > `default_dns_domain` and `dns_servers` -> tenant subnets with > `--dns-nameserver`. D-106 also carries a naming instantiation, not a new invention -- this is the one place a "zone name" appears in this runbook, and it is cited from the decision text itself, not invented: > Naming per D-008 instantiated with the VR1 labels: > `.omega.dc1.vr1.cloud.neumatrix.local` and `...dc2.vr1...`. > Per-DC INDEPENDENT zones (independent Keystones), served with A and AAAA > records (AAAA for the v6-enabled planes per D-101). **THE QUOTED LABELS ABOVE ARE SUPERSEDED -- do not paste them (DOCFIX-205).** The quote is D-106's ratified 2026-07-09 text and is reproduced unchanged, but D-117 (ADOPTED 2026-07-13) retired the repo's `dc1`/`dc2` labels in favour of `dc0`/`dc1`, naming this supersession in its own Status line. The VR1 zones are therefore `omega.dc0.vr1.cloud.neumatrix.local` (substrate `vr1-dc0`) and `omega.dc1.vr1.cloud.neumatrix.local` (substrate `vr1-dc1`). Use the derived `$DC_ZONE` set in Step 1 rather than typing either. D-106 now carries a D-117 annotation recording this. ### 1.3 -- The reversal is bigger than "add one application" -- flag this honestly D-106's bootstrap order includes `os-public-hostname` per API charm as its SECOND step. `bundle.yaml`'s current header records the opposite posture, under its own B5 label (not a separate D-NNN -- a 2026-06-01 bundle-cleanup item): ``` # Endpoints: IP-ONLY -- os-public-hostname dropped on all API charms; the dual VIPs ARE the # catalog endpoints (public 10.12.4.N / internal+admin 10.12.8.N). Vault issues # per-VIP IP-SAN certs. No control-plane DNS dependency. (B5) ``` D-106's own text explicitly names `os-public-hostname` per API charm as part of the order it reactivates, so re-adding it is NOT an inferred scope addition -- it is in the ratified decision text itself. But it IS a second, larger surface than "add designate": every API charm's `os-public-hostname` option goes from unset back to a real FQDN, and Vault's cert issuance moves from per-VIP IP-SAN back to FQDN-SAN. Treat this as two coupled changes under one D-106 umbrella, not one: 1. Add `designate` + `designate-bind` (+ `designate-hacluster`) applications and their relations. 2. Re-add `os-public-hostname: ` to every clustered API charm's `options:` block (the same charm list D-020 already dual-VIPs: keystone, glance, neutron-api, nova-cloud-controller, placement, cinder, octavia, barbican, magnum, openstack-dashboard, ceph-radosgw). If, on review, the operator wants item 2 scoped or sequenced differently from item 1 (e.g. reactivate DNS-serving without yet flipping every charm's hostname), that is a real design fork -- log it as a proposal against D-106 rather than deciding it inside this runbook. ### 1.4 -- Keep bundle.yaml DC-agnostic; push per-DC literals to an overlay `bundle.yaml` is ONE shared file used to deploy both DC1 and DC2 (D-101: DC1 inherits DC0's v4 layout unchanged; DC2 gets its own NetBox-assigned supernet, still pending -- gap #3 in `docs/dc-dc-deployment-workflow.md`'s tooling gap register). The new `os-public-hostname` values are NOT DC-agnostic -- they embed `dc1` or `dc2` per D-106's naming convention. Do not hardcode a DC-specific FQDN into the shared `bundle.yaml`. Follow the existing overlay pattern already established for secrets (`overlays/${DC}-octavia-pki.yaml` -- itself made PER-DC by the D-109 amendment, gitignored by glob; Step 1.0 of `phase-01-bundle-deploy.md`) and add one per-DC hostnames overlay: `overlays/vr1-dc0-hostnames.yaml`, `overlays/vr1-dc1-hostnames.yaml` (region- qualified: the `dc1`/`dc2` spelling is retired -- item 3.1 measured "DC1" meaning `vr1-dc0` on one surface and `vr1-dc1` on another), each setting only the `os-public-hostname` options for that DC, applied at deploy time with an additional `--overlay` alongside whichever overlays that DC's deploy already uses. This is a proposed pattern, not yet reviewed -- present it at the same review gate as the rest of Step 1, do not silently adopt it. ### 1.5 -- The application/relation shape (done 2026-07-10 -- bundle.yaml already carries this) **This step is CLOSED, not a template anymore.** `bundle.yaml` now carries real `designate`/`designate-bind`/`designate-mysql-router`/`designate- hacluster` application blocks + 8 relations, researched directly against `charm-designate`'s real `metadata.yaml` (not the standard-pattern guess this runbook originally sketched here). Two things this runbook's first draft got wrong were caught and fixed during that research, worth knowing before you run `--dry-run`: - Designate's real provides-side endpoint is `dnsaas` (interface `designate`), **not** `public` like the other API charms in this bundle. - `dns-backend` (interface `bind-rndc`), `ha` (interface `hacluster`), and `coordinator-memcached` (interface `memcache`, relating to the existing `memcached` principal) were all confirmed from the charm's own source. `shared-db`/`identity-service`/`amqp`/`certificates` still follow the standard pattern (very high confidence, matches 10 other charms) but were NOT individually re-fetched for designate -- confirm with `juju deploy --dry-run` before treating this as fully final. See `docs/archive/changelogs/changelog-20260709-designate-cinderbackup-rbdmirror.md` for the full record. What's still genuinely open: the `os-public-hostname` overlay (Step 1.4 above) -- `bundle.yaml` deliberately stays DC-agnostic, so that overlay is still this runbook's own job at real execution time, not pre-built. **CHECK (read-only) -- jumphost, once bundle.yaml + overlays are edited** ```bash bash scripts/repo-lint.sh # expect 0 fail bash scripts/preflight.sh # expect PREFLIGHT: PASS (incl. Charmhub channel assert) ``` **GATE:** repo-lint clean; preflight PASS. Only after this passes do the live per-DC steps become eligible. (Corrected 2026-07-29: this read "does Step 5 (the live per-DC apply) become eligible" -- Step 5 is no longer an apply, see the D-106 ruling note; the applies live in Steps 3/6/7/8.) Deliver this edit with a changelog entry (what changed, why -- D-106, how to revert) per this repo's standing delivery rule, same as any other script/runbook/bundle change. --- ## Step 2 -- Static /etc/hosts bootstrap (D-008 step 1, per DC) MEASURE this DC's own hostnames and IPs from its live `juju status` / `maas admin machines read` output -- do not carry over DC0's `/etc/hosts` entries or VR0's IP ranges. Per D-101, DC1's v4 layout equals DC0's (deliberate, ratified reuse -- not an assumption), so DC1's measured values may look identical to DC0's on paper; measure them live anyway. DC2's values are, per gap #3, not yet assigned in NetBox as of this writing -- if DC2's literals still are not populated by the time you reach this step, STOP and resolve that gap first (do not invent a DC2 supernet here). **CHECK (read-only) -- this DC's jumphost-equivalent seat** ```bash juju status -m "${DC_MODEL}" --format=yaml | grep -E 'public-address|hostname' ``` **RUN -- every host + LXD container in this DC's model (mirrors phase-00's Pattern A hosts, D-008 step 1's "jumphost + all openstack hosts + all LXD containers" scope, but for THIS DC)** ```bash # Populate /etc/hosts on each unit with THIS DC's own measured # pairs, one line per API charm, using D-106's naming: # .${DC_ZONE} -- derived in Step 1; NEVER "omega.${DC}.vr1..." # Do not template a value you have not just measured. ``` **GATE:** every host/LXD container in this DC resolves every other API charm's FQDN via `/etc/hosts` (the D-008 bootstrap floor, before Designate itself exists). --- ## Step 3 -- os-public-hostname per API charm (D-008 step 2, per DC) **RUN -- jumphost, per charm (repeat for every clustered API charm listed in Step 1.3)** ```bash juju config -m "${DC_MODEL}" keystone os-public-hostname="keystone.${DC_ZONE}" # ... repeat per charm: glance, neutron-api, nova-cloud-controller, placement, # cinder, octavia, barbican, magnum, openstack-dashboard, ceph-radosgw ``` **CHECK (read-only) -- jumphost** ```bash juju status -m "${DC_MODEL}" --format=short | grep -vE 'active|idle' || echo "all active/idle" ``` **GATE:** all charms return to `active/idle` after the config change settles (a hostname-only config change is non-disruptive on this charm family, but confirm rather than assume for this DC). --- ## Step 4 -- Vault issues FQDN-SAN certs (D-008 step 3, per DC) This DC's Vault is an INDEPENDENT root (D-109) issuing its own certs -- do not reuse a cert or CA from the other DC or from VR0 DC0. Trigger the certificates relation to re-issue now that `os-public-hostname` is set; the charm-native cert cascade (Vault's `certificates` interface) picks up the new SAN automatically once the relation re-fires -- no manual CSR step is expected on this charm family (confirm this is still true for the designate charm's own certificate -- its `certificates` relation ships in the bundle and is already up by the time you reach here, so there is nothing to relate at Step 5; corrected 2026-07-29, this read "once Step 5 relates it"). **CHECK (read-only) -- jumphost, per charm** ```bash juju ssh -m "${DC_MODEL}" keystone/leader \ "sudo openssl x509 -in /etc/apache2/ssl/keystone_1/cert -noout -text | grep -A1 'Subject Alternative Name'" /tmp/designate-presence.yaml # 1. presence: grep -E '^\s+(designate|designate-bind|designate-mysql-router|designate-hacluster):' \ /tmp/designate-presence.yaml # 2. workload/agent state, human-readable: juju status -m "${DC_MODEL}" designate designate-bind # 3. the in-bundle relations, designate side: grep -i -A3 designate /tmp/designate-presence.yaml | grep -i 'relation\|endpoint' ``` **GATE (presence AND state -- both, not either):** 1. `designate`, `designate-bind`, `designate-mysql-router` and `designate-hacluster` are all PRESENT in the model. If any is MISSING, that is a Stage-5 deploy defect, not work for this step -- STOP and reconcile against Stage 5's own gate; do not deploy it from here. 2. All four are `active/idle`. 3. `designate:dns-backend - designate-bind:dns-backend` and `designate:identity-service - keystone:identity-service` are up. An `active/idle` designate at this point means the CHARM is healthy. It does NOT mean DNS works -- the feature is inert until the configuration below and Steps 6/7/8 land. Do not read unit status as evidence of function (same class of trap as the D-046 magnum `domain-setup` warning at the top of this runbook). ### 5.2 -- Configure designate Steps 2/3/4 have by now set `os-public-hostname` per API charm and re-issued Vault FQDN-SAN certs; Steps 6/7/8 own the zone + A/AAAA records, Neutron's `default_dns_domain`/`dns_servers`, and tenant subnet `--dns-nameserver`. What remains to THIS step is designate's own charm configuration -- the settings that make the already-running charm serve this DC's zone. **MEASURE THE OPTION NAMES, DO NOT CITE THEM FROM HERE.** This runbook deliberately names no designate charm config key: no such key is recorded in any ratified decision or measured capture in this repo, and 1.5 already records that several designate endpoints were confirmed only by reading the charm's own source. Read the real option set at execution time and configure from that: ```bash # THE authoritative read: for a deployed application this returns the full option # set -- real key names, current values, defaults and descriptions. It is # sufficient on its own; nothing else is needed to drive this step. juju config -m "${DC_MODEL}" designate ``` If you want the charm's raw `config.yaml` as well, the unit-local path is `/var/lib/juju/agents/unit-designate-*/charm/config.yaml` -- UNVERIFIED, recorded as a fallback only. If it is not there, do not go hunting: `juju config` above is the supported surface and already answers the question. **GATE:** designate returns to `active/idle` after any `juju config` change, and every key you set was read from the charm's own `config.yaml` in this session -- not carried from memory or from another cloud. If a value you need is a per-DC literal (a nameserver address, an FQDN), MEASURE it live per this runbook's standing rule; the zone NAME is the one literal cited from decision text (D-106), and it is cited in Step 6 where the zone is created. If nothing in the charm's config set needs to change for this DC, say so explicitly in the as-executed log and proceed -- an empty configure is a valid outcome of this step and is not a reason to invent one. --- ## Step 6 -- Designate zone + A/AAAA records (D-008 step 4, per DC) Per-DC INDEPENDENT zone (D-106). The zone name itself is the one place this runbook cites a literal beyond a live measurement, and it is cited from D-106's own ratified naming text as corrected by D-117, not invented: `$DC_ZONE` (derived in Step 1 = `omega.dc0.vr1.cloud.neumatrix.local` for `vr1-dc0`). The A/AAAA record VALUES are NOT cited from any decision text -- they are this DC's real, live, measured VIP addresses. Populate them from what Step 3/4 just proved live, never from a planning number. **RUN -- jumphost (designate-scoped or admin, per this DC's own admin-openrc)** ```bash openstack zone create --email hostmaster@neumatrix.local \ "${DC_ZONE}." # per API charm, MEASURE the live provider VIP (A) and, if this DC's GUA plane # is up (D-101), the live provider-public GUA address (AAAA) -- do not invent # either: KEYSTONE_VIP_V4=$(openstack ... ) # MEASURE: this DC's live keystone provider VIP openstack recordset create "${DC_ZONE}." \ keystone --type A --records "$KEYSTONE_VIP_V4" # AAAA only if D-101's v6-enabled planes are confirmed up for this DC -- gate # on that confirmation, do not assume: KEYSTONE_VIP_V6=$(openstack ... ) # MEASURE: this DC's live GUA provider address openstack recordset create "${DC_ZONE}." \ keystone --type AAAA --records "$KEYSTONE_VIP_V6" # repeat per API charm ``` **GATE:** every API charm has an A record resolving to its own live provider VIP; AAAA records exist wherever D-101's GUA/dual-stack planes are confirmed up for this DC (not before -- an AAAA record pointing at an address that is not yet live is worse than no record). --- ## Step 7 -- Neutron default_dns_domain / dns_servers -> Designate (D-008 step 5, per DC) **RUN -- jumphost** ```bash DESIGNATE_VIP=$( ... ) # MEASURE: this DC's live designate-bind / designate API VIP juju config -m "${DC_MODEL}" neutron-api \ dns-domain="${DC_ZONE}." \ dns-servers="${DESIGNATE_VIP}" ``` **GATE:** `neutron-api` returns to `active/idle`; `openstack network show ` (or the relevant config surface) reflects the new dns_domain. --- ## Step 8 -- Tenant subnets --dns-nameserver (D-008 step 6, per DC) Every existing tenant subnet in this DC created BEFORE Designate existed (most notably `capi-mgmt-subnet` from phase-06 Step 6.1, which set `--dns-nameserver 1.1.1.1 --dns-nameserver 1.0.0.1` under the D-019 "no cloud DNS, tenants use public resolvers" posture) needs updating, and every subnet created AFTER this point should use the Designate VIP from the start. **RUN -- jumphost, per existing tenant subnet** ```bash DESIGNATE_VIP=$( ... ) # MEASURE: same value as Step 7 openstack subnet set --dns-nameserver "$DESIGNATE_VIP" capi-mgmt-subnet # clear the old public-resolver entries only if the subnet update replaces # rather than appends -- confirm openstack-client's actual semantics for # --dns-nameserver on `subnet set` live, do not assume append vs. replace ``` **GATE:** `openstack subnet show capi-mgmt-subnet` shows the Designate VIP as the (or a) DNS nameserver. --- ## Step 9 -- Verify the D-008/D-106 bootstrap end to end (read-only) **CHECK (read-only) -- jumphost -> a tenant VM in this DC (e.g. capi-mgmt-v2)** ```bash source ~/capi-mgmt-net.env # this DC's own env file, per phase-06 Step 6.2 -- confirm it is THIS DC's, not the other DC's stale file ssh ... ubuntu@"$MGMT_FIP" bash -s <<'REOF' getent hosts keystone.${DC_ZONE} # substitute the real DC label dig +short AAAA keystone.${DC_ZONE} # if AAAA populated curl -sk https://keystone.${DC_ZONE}:5000/v3 -o /dev/null -w '%{http_code}\n' REOF ``` **GATE:** the FQDN resolves (A, and AAAA where populated); the HTTPS call against the FQDN succeeds with a cert the client trusts (Vault's CA is this DC's own root, per D-109 -- the tenant VM/OS trust store must carry it, same as any other charm's client already does). This is the concrete proof the D-019/D-021 root cause (metal-only charms pulling a public FQDN endpoint that does not resolve/route, behind a cert with no FQDN SAN) is closed for this DC. --- ## Step 10 -- COS observability (DECISION POINT + MUTATION, per DC) No D-NNN or design doc in this repo specifies the COS deployment mechanism -- only its scope (D-105: per-DC, in scope, no Office1 roll-up). Do not pick one silently. Present the standard options to the operator at execution time: !!! **Edge scrape is a SEPARATE scope decision (D-129, pinned 2026-07-18).** D-105 scopes COS to the cloud (per-DC, no Office1 roll-up). The DC OPNsense **edge sits OUTSIDE** that failure domain, so "should COS scrape the boundary edge?" is its own sub-decision -- NOT ruled by D-105. IF the operator rules the edge in-scope, install **`os-node_exporter`** on the DC edge (D-113 firmware API) and add it as a Prometheus scrape target; the edge LAN faces provider-public (D-100), so reachability from the in-cloud COS is a segmentation call to make explicitly. If out-of-scope, do nothing here. - **Option A -- `cos-lite` bundle in a dedicated per-DC `cos` Juju model** (Canonical's standard COS Lite: Prometheus, Grafana, Loki, Alertmanager, Traefik ingress), related to this DC's `openstack` model applications via Juju cross-model relations (`juju offer` / `juju consume` + `cos-configuration-k8s` or the `grafana-agent` subordinate pattern for metrics/log forwarding). Needs a Kubernetes substrate for the `cos` model (microk8s or equivalent) -- confirm this DC has one before picking this option. - **Option B -- machine-charm-native observability** (`prometheus2` / `grafana` / `telegraf` charms in the SAME model as the OpenStack bundle, no k8s substrate needed) -- lower delta if no k8s host is available in this DC for a `cos` model, but is not "COS" in the strict Canonical-bundle sense; confirm this is an acceptable substitution for the D-105 "COS" scope before treating it as satisfying the gate. **CHECK (read-only) -- confirm before choosing** ```bash # Does this DC have a k8s substrate available for a cos-lite model? # (microk8s, or another already-enrolled k8s -- MEASURE, do not assume) ``` **GATE (decision, log it):** record which option was chosen and why in the session ledger / a changelog entry -- this is a real architectural choice, not a detail to bury in a command block. Once chosen, the deployment steps are option-specific and are NOT detailed further here (this is genuinely new tooling territory for this repo -- author the concrete steps as their own change when the option is picked, with its own harness per this repo's delivery rule, rather than improvising them live). **GATE (Stage 7's COS criterion):** COS scraping -- at minimum, this DC's OpenStack API charms' metrics/logs and the CAPI mgmt cluster + workload cluster (Steps 11-13) are visible in whichever COS surface was chosen. --- ## Step 11 -- CAPI management cluster (reuse `phase-06-incloud-mgmt-cluster.md`, per DC) Run `runbooks/phase-06-incloud-mgmt-cluster.md` Steps 6.0-BOOT through 6.6f IN FULL, once per DC, against this DC's own project/Keystone/model. Do not duplicate its content here -- point at it by filename. DC-parameterization notes (found by tracing what that runbook's canonical scripts actually resolve dynamically vs. hardcode): - `scripts/phase-06-k8s-bootstrap.sh` sources `scripts/lib-net.sh` and reads `KEYSTONE_VIP_DEFAULT` from it (DOCFIX-138 centralized this). Before running it for a DC other than DC0, call `lib_net_select_dc "$DC"` (DOCFIX-151) FIRST: for `dc1` this is currently a documented no-op (D-101: DC1 inherits DC0's v4 layout unchanged -- a ratified reuse, not a stale default), for `dc2` it currently FAILS LOUDLY (`lib-net.sh`'s dc2 literals are not yet assigned -- tooling gap register #3). If dc2's literals are still unassigned by the time you reach this stage, STOP; do not bypass the fail-loud guard. - `scripts/phase-06-kubeconfig-gate.sh` -- same `lib-net.sh` dependency; same `lib_net_select_dc "$DC"` call required first. - Every other literal in phase-06 (`ENV(project)`, `ENV(ext-net)`, `ENV(image)`, `ENV(mgmt-fip)`, project/image/network UUIDs) is ALREADY written as "resolve by name" / "resolve dynamically" in that runbook -- it re-discovers these fresh on every run, so it needs no further DC- parameterization beyond pointing it at this DC's own `admin-openrc` and Keystone. Confirm you have sourced THIS DC's own admin-openrc (naming convention for per-DC openrc files was not established this session -- confirm it from Stage 4's as-built record, do not assume a filename). - `ENV(mgmt-cidr) 10.20.0.0/24` is a non-IPAM tenant overlay CIDR -- safe to reuse identically in both DCs (they are independent clouds; no collision risk). Do not route this through NetBox. - `scripts/lib-hosts.sh`'s `lib_hosts_select_dc()` is NOT used by phase-06 (it governs MAAS host enrollment, a Stage 3 concern, not the tenant-VM layer phase-06 operates at) -- no action needed here, noted so it is not mistakenly assumed required. **GATE (this runbook, restating phase-06's EXIT GATE):** per DC, GATE 1 (VIP-OK) and GATE 2 (agnhost pod-egress `Completed`) both pass; the CAPI provider stack is up with no crash-looping CAPO. --- ## Step 12 -- Magnum conductor graft (reuse `phase-07-conductor-graft.md`, per DC) Run `runbooks/phase-07-conductor-graft.md` in full, once per DC, against this DC's own `magnum/0` and this DC's phase-06 kubeconfig/FIP. Point at it by filename; do not duplicate its content. **!!! D-046 GATE -- REPRODUCED HERE BECAUSE IT IS THE MOST-MISSED STEP IN THIS DECISION-DERIVED CHECKLIST, restated verbatim from D-105/D-046:** > The magnum charm action `domain-setup` is MANUAL, not automatic; magnum > reports active / "Unit is ready" regardless of whether it has run. Concretely, this means: for THIS DC, run phase-07's Step 7.0 (`juju run magnum/leader domain-setup`, then assert `openstack domain show magnum` and `openstack user show magnum_domain_admin --domain magnum` both succeed, then gate `openstack coe service list` with no 403) EVERY time this DC's magnum unit is torn down and redeployed -- including if this whole Stage 7 pass is itself a redo after an earlier Stage 4-6 rebuild of this DC. Do not treat a green `juju status` for magnum as evidence this step already happened. DC-parameterization: phase-07's `ENV(conductor-unit)`, `ENV(mgmt-fip)`, `ENV(magnum-ns)` are all already written as "resolve by name" / per-rebuild -- re-resolve them fresh against THIS DC's model; do not carry over the other DC's or DC0's recorded values (the as-built reference blocks in that runbook are VR0/DC0 audit trail, not this DC's expected values). **GATE (this runbook, restating phase-07's EXIT GATE):** Step 7.0's domain- setup assertion passes; conductor reaches the mgmt apiserver; magnum-capi-helm 1.4.0 (or whatever the then-current contract-coherent pin is -- confirm against D-042/D-034 rather than assuming 1.4.0 is still current) installed; `health_status` gate discharged per that runbook's FRESH DEPLOY routing (i.e. deferred to Step 13 if no cluster exists yet in this DC). --- ## Step 13 -- Workload-cluster acceptance (reuse `phase-08-workload-cluster-acceptance.md`, per DC) Run `runbooks/phase-08-workload-cluster-acceptance.md` in full, once per DC. Point at it by filename; do not duplicate its content. One correction to apply when running it under D-106 (rather than the D-019-amended acceptance bar that runbook currently documents): D-011 item 8 ("Designate zones + tenant hostname resolution") is NO LONGER deferred for VR1 -- Step 9 above is what discharges it. Do not mark it DEFERRED per the old D-019 wording when running this under D-106; record it PASS/FAIL against Step 9's actual result instead. DC-parameterization: same shape as Steps 11-12 -- every `ENV(...)` in that runbook already resolves by name; point it at this DC's own project/model, re-seed this DC's own workload image via stage-and-verify (D-048) rather than assuming the image already exists in a fresh DC. **GATE (this runbook, restating phase-08's EXIT GATE, D-106-corrected):** `capi-test-1`-equivalent cluster reaches `CREATE_COMPLETE`; 3 (or as-templated) nodes Ready; CNI/CCM/CSI/CoreDNS Running; D-011 items 1-6 PASS; item 7 (snapshot) per this DC's own baseline policy; item 8 PASS (not deferred) via Step 9's resolution proof. --- ## GATE -- Stage 7 exit condition (per DC; both DC1 and DC2 must pass independently) Restating `docs/dc-dc-buildout-design.md` Phase 6 / `docs/dc-dc-deployment- workflow.md` Stage 7's gate: - Designate resolving A + AAAA for this DC's own zone (Step 6/9; AAAA only where D-101's v6 planes are confirmed up). - FQDN-SAN certs valid (Step 4/9) -- the D-019/D-021 root cause (metal-only charms pulling an unresolvable/unroutable public FQDN behind an IP-only cert) is CLOSED for this DC. - COS scraping this DC's OpenStack + CAPI mgmt + workload cluster (Step 10). - A CAPI workload cluster comes up in this DC (Step 13, `CREATE_COMPLETE`). Do not mark Stage 7 complete for the DC-DC buildout as a whole until BOTH DCs independently clear all four bullets -- one DC passing does not imply the other DC's independent Keystone/Vault/Designate zone/CAPI stack also passes. --- ## Delivery checklist - [ ] NOTE: the designate half of Step 1 is ALREADY DONE in the repo (DOCFIX-167, 2026-07-10) -- `bundle.yaml` carries all four designate applications and their eight relations, and Stage 5 deploys them. Do NOT re-add them. The only Step 1 repo work still owed is the per-DC `os-public-hostname` overlay proposed in 1.4, which is NOT AUTHORED (`overlays/vr1-dc0-hostnames.yaml` / `overlays/vr1-dc1-hostnames.yaml` do not exist -- and neither does the `overlays/${DC}-hostnames.yaml` spelling other surfaces used to reference; never pass a name from this bullet to `--overlay` before authoring the file, juju errors out on a missing overlay). It is a PROPOSED pattern per 1.4: present it at the review gate, do not silently adopt it. - [ ] Once that overlay is authored: `scripts/repo-lint.sh` 0 fail; `scripts/preflight.sh` PASS (incl. Charmhub channel assert). - [ ] Changelog entry for the overlay change: what changed (os-public-hostname re-add per DC), why (D-106, superseding D-019), how to revert (drop the hostnames overlay, re-run preflight to confirm the revert is clean). Reverting designate itself is NOT part of this stage -- it is Stage 5 bundle content under the D-106 ruling note. - [ ] `docs/session-ledger.md` updated with this runbook's authoring and, once executed, each DC's Stage 7 pass/fail state. - [ ] `docs/dc-dc-deployment-workflow.md` Stage 7 row's `**State:**` line updated from NOT STARTED once execution begins/completes. - [ ] Per DC: Steps 0-9 (Designate reactivation) executed and Step 9's gate passed BEFORE Step 10 (COS) is attempted -- do not build COS scraping against a DNS layer that has not yet proven itself. - [ ] Per DC: Step 10's COS mechanism choice recorded (which option, why). - [ ] Per DC: phase-06/07/08 run to their own EXIT GATES with THIS DC's own resolved values (never a copy-pasted literal from DC0 or the other DC). - [ ] D-046 domain-setup re-run + assertion done for THIS pass, not assumed from a prior one. - [ ] This runbook's own GATE section cleared for BOTH DCs before Stage 7 is marked complete in the deployment workflow tracker. ## Next Once both DCs clear this runbook's GATE, the DC-DC buildout's per-DC service set is complete. Remaining work is whatever the deployment-workflow doc's "Where things stand today" table still lists as outstanding at that point (consult it fresh -- it is a living tracker, not reproduced here).