STAGE 5 -- dc0 bundle deploy ATTEMPT 3 (2026-07-31): SUCCEEDED ============================================================== >>> VERDICT: `Deploy of bundle completed.` EXIT 0 <<< COMMAND (from the dc0 rack, D-138 client host, cwd ~/repo-stage): juju deploy ./bundle.yaml --overlay ./overlays/vr1-dc0-vips.yaml \ --overlay ./overlays/vr1-dc0-machines.yaml \ --overlay ./overlays/vr1-dc0-octavia-pki.yaml -m vr1-dc0 MEASURED IMMEDIATELY AFTER juju 56 applications, 9 machines, 33 units machine juju-status : 9 x pending machine instance-stat: 9 x allocating MAAS 10 machines: 9 Deploying, 1 Deployed (the juju controller, already up) WHAT MADE THIS ATTEMPT DIFFERENT -- three defects closed, in the order they were hit ------------------------------------------------------------------------------------ attempt 1 `unknown option "prefer-ipv6"` on barbican. Aborted at VALIDATION, model left empty. CLOSED by the D-101 RULING NOTE 2026-07-31: the six charms that do not declare the option no longer receive it, and every v6 VIP leg is retained -- the option was never what made HAProxy bind `:::port`. attempt 2a `file for resource "policyd-override": stat .../policies/overrides.zip: no such file or directory`. Reached EXECUTION and left 23 application definitions. CLOSED by staging `policies/` on the rack, sha256-verified. attempt 2b `barbican: downgrades are not currently supported: deployed revision 265 is newer than requested revision 261`. CLOSED by RULING 2026-07-31 (option D): an explicit `base: ubuntu@22.04/stable` on all 56 applications, plus clearing the model so the deploy runs the path that is MEASURED to work. THE PRE-DEPLOY SEQUENCE, all of it verified rather than assumed --------------------------------------------------------------- 1. bundle fixed: 56/56 applications carry an explicit base; the edit was cross-checked against `yaml.safe_load` (a first pass silently covered 44 of 56 -- the twelve `-hacluster` apps are single-line FLOW MAPPINGS) and asserted on the ARTIFACT (the parsed structures with `base` stripped are IDENTICAL to the original). 2. gated: `provider-bundle-check` invariant 12 is NEW and mutation-proven five ways, so the 56 lines cannot be silently simplified away. It caught a real gap on its FIRST run -- `overlays/dc-ha-scaleup.yaml` DEFINES `vault-hacluster`, an application in no other file, and it had no base. That is a LATER deploy step, so the trap would have re-opened after this deploy succeeded. 3. proven: the fully-based bundle `--dry-run` against the LIVE 23-app model exited 0, where the unbased one errored. The one-app test before it was an INFERENCE; this is the measurement. 4. deploy input staged on the rack and ALL FIVE files sha256-compared to repo HEAD: bundle.yaml 4c8a7852, vr1-dc0-vips daa2919d, vr1-dc0-machines b70e4eed, policies/overrides.zip 02fe1fd7, and the gitignored vr1-dc0-octavia-pki 5fc117f1 (0600, SEC-029 custody untouched). 5. model CLEARED: all 23 application definitions removed INDIVIDUALLY, never batched (hard rule 3), each read back; precondition re-verified immediately before the first removal (0 units, 0 machines, nothing provisioned). Read-back after: `Model "vr1-dc0" is empty`, 0 applications / 0 machines. 6. Step-3.5 model state confirmed to have SURVIVED the clear: `apt-mirror http://10.12.8.4/ubuntu` still set, all six spaces still bound with their v4 and v6 subnets. 7. preflight P1-P4 + P7 PASS; P5 RED with 11 findings, ALL ELEVEN ruled-accepted (six 2026-07-30, five 2026-07-31), enumerated in docs/audit/stage5-preflight-dc0-20260731.txt. NOTES CARRIED FORWARD --------------------- - `juju remove-application` PROMPTS by default and aborts on a non-interactive stdin (`ERROR application removal: aborted`, exit 1). `--no-prompt` is required from a non-interactive session. Recorded because the first removal read as a failure. - `ceph-osd`'s `tags=openstack` constraint (bundle.yaml) is still the known exposure and is still NOT a problem for this deploy: placement is by explicit machine id (`ceph-osd/0..3` -> machines 5,6,7,8). A later UNPLACED `juju add-unit ceph-osd` would match no machine. LOGGED, unchanged. - **G17's dc0 half is now genuinely armable.** The existing capture was taken on the CONTROLLER VM, not a role node, and said so. The nine role nodes are booting for the first time as this file is written -- that is G17's window. WHAT IS NOT CLAIMED ------------------- `Deploy of bundle completed.` means juju ACCEPTED and queued the bundle. It does NOT mean the cloud is up. 9 machines are allocating, 33 units are pending, and the settle to phase-01's documented pre-vault-init end state takes hours. Nothing here asserts unit health, relation settling, or any service verdict. ================================================================================ POST-DEPLOY: THREE ARTIFACT/CONFIG DEFECTS, IN THE ORDER THEY UNMASKED ================================================================================ Each was hidden by the one before it. `apt-get update --error-on=any` fails on the FIRST bad source, so only one is ever visible at a time. D1 -- `jammy-backports` 404 on the dc0 mirror. RULED + FIXED 2026-07-31 (D-135 amendment, operator: "Sync the backports into the mirror"). Fixed at source in scripts/dc-mirror.sh, T10 re-pointed, applied live, sync OK, verified on CONTENT (Release fields + a real .deb fetch: elfutils_0.188-1~bpo22.04.1_amd64.deb, 200, 502468 B). Mirror 951G -> 952G. D2 -- the UPSTREAM UCA is unreachable from nodes. MEASURED FROM A NODE (not the rack, which holds a default route nodes lack): node -> http://ubuntu-cloud.archive.canonical.com/... 000 node -> http://10.12.8.4/cloud-archive/... 200, real content The node's source line points UPSTREAM. juju's `apt-mirror` model-config rewrites only the Ubuntu archive, not the UCA source the charm adds. SIX apps set it explicitly across TWO option names -- `openstack-origin` on barbican/magnum/octavia and `source` on ceph-mon/ceph-osd/ceph-radosgw -- and every other OpenStack charm defaults to `openstack-origin: caracal`, which resolves to the same upstream pocket. The UCA signing key IS already on the node (/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg), so a raw `deb` line pointing at the mirrored UCA will verify with no `|key` suffix. EXPERIMENT RESULT WORTH KEEPING: this asymmetry is a genuine D-135 finding -- the FULL-MIRROR DC must rewrite every non-Ubuntu source; the PROXY DC needs none, because apt-cacher-ng forwards whatever URL it is handed. dc1 will not hit this. D3 -- >>> `prefer-ipv6: true` IS FATAL ON THE SEVEN CHARMS THAT DECLARE IT. <<< MEASURED 2026-07-31 19:08, and this is the exact risk this session flagged as "never observed live" when it was deliberately left OUT of the D-101 ruling note: unit-keystone-0 ... Exception: Interface 'eth0' does not have a scope global non-temporary ipv6 address. Cause, and it matches the charm source read earlier today EXACTLY: `get_relation_ip()` (charmhelpers/contrib/network/ip.py:617) returns EARLY with `get_ipv6_addr(dynamic_only=False)[0]` whenever `prefer-ipv6` is true -- so the unit tries to advertise a v6 address for every relation. MEASURED on the container: keystone/0 eth0 -> fe80::216:3eff:feb4:5518/64 (LINK-LOCAL ONLY, no global v6) The NODES are dual-stacked; the LXD CONTAINERS the API charms run in are NOT. Live config confirms the split: keystone, cinder, glance, neutron-api, nova-cloud-controller, openstack-dashboard, ceph-radosgw all read `prefer-ipv6=true` and ALL SEVEN are in `error`. ceph-mon and mysql-innodb-cluster read `false` (charm default) and fail for other reasons; ovn-central and vault do not declare the option at all. CONSEQUENCE: the 2026-07-31 ruling removed the option from the SIX charms that never declared it, which was correct and necessary. The SEVEN that DO declare it were explicitly NOT covered by that ruling -- and they are now the blocker. NOT TAKEN HERE: R2 / D-101 are ruled surfaces and this needs its own GA-R5 exchange. WHY DROPPING IT IS EXPECTED TO BE SAFE, from the same measurement chain: the option is a UNIT ADDRESS-FAMILY switch, not a listener switch. HAProxy's `:::port` bind is gated on `ipv6_enabled = not is_ipv6_disabled()` (the kernel sysctl), and the container carries a link-local v6, so v6 is NOT disabled and the bind still emits. Pacemaker assigns the v6 VIP by address-family detection, independent of the option. OPEN QUESTION that belongs to the operator, stated not glossed: whether a v6 VIP placed on a container whose eth0 has no global v6 is ROUTABLE. That is a different question from whether the charm installs, and it is not answered here.