STAGE 5 -- dc0 bundle deploy ATTEMPT 3 (2026-07-31): SUCCEEDED
==============================================================
>>> VERDICT: `Deploy of bundle completed.` EXIT 0 <<<
COMMAND (from the dc0 rack, D-138 client host, cwd ~/repo-stage):
juju deploy ./bundle.yaml --overlay ./overlays/vr1-dc0-vips.yaml \
--overlay ./overlays/vr1-dc0-machines.yaml \
--overlay ./overlays/vr1-dc0-octavia-pki.yaml -m vr1-dc0
MEASURED IMMEDIATELY AFTER
juju 56 applications, 9 machines, 33 units
machine juju-status : 9 x pending
machine instance-stat: 9 x allocating
MAAS 10 machines: 9 Deploying, 1 Deployed (the juju controller, already up)
WHAT MADE THIS ATTEMPT DIFFERENT -- three defects closed, in the order they were hit
------------------------------------------------------------------------------------
attempt 1 `unknown option "prefer-ipv6"` on barbican. Aborted at VALIDATION, model
left empty. CLOSED by the D-101 RULING NOTE 2026-07-31: the six charms that
do not declare the option no longer receive it, and every v6 VIP leg is
retained -- the option was never what made HAProxy bind `:::port`.
attempt 2a `file for resource "policyd-override": stat .../policies/overrides.zip: no
such file or directory`. Reached EXECUTION and left 23 application
definitions. CLOSED by staging `policies/` on the rack, sha256-verified.
attempt 2b `barbican: downgrades are not currently supported: deployed revision 265 is
newer than requested revision 261`. CLOSED by RULING 2026-07-31 (option D):
an explicit `base: ubuntu@22.04/stable` on all 56 applications, plus
clearing the model so the deploy runs the path that is MEASURED to work.
THE PRE-DEPLOY SEQUENCE, all of it verified rather than assumed
---------------------------------------------------------------
1. bundle fixed: 56/56 applications carry an explicit base; the edit was cross-checked
against `yaml.safe_load` (a first pass silently covered 44 of 56 -- the twelve
`-hacluster` apps are single-line FLOW MAPPINGS) and asserted on the ARTIFACT (the
parsed structures with `base` stripped are IDENTICAL to the original).
2. gated: `provider-bundle-check` invariant 12 is NEW and mutation-proven five ways, so
the 56 lines cannot be silently simplified away. It caught a real gap on its FIRST
run -- `overlays/dc-ha-scaleup.yaml` DEFINES `vault-hacluster`, an application in no
other file, and it had no base. That is a LATER deploy step, so the trap would have
re-opened after this deploy succeeded.
3. proven: the fully-based bundle `--dry-run` against the LIVE 23-app model exited 0,
where the unbased one errored. The one-app test before it was an INFERENCE; this is
the measurement.
4. deploy input staged on the rack and ALL FIVE files sha256-compared to repo HEAD:
bundle.yaml 4c8a7852, vr1-dc0-vips daa2919d, vr1-dc0-machines b70e4eed,
policies/overrides.zip 02fe1fd7, and the gitignored vr1-dc0-octavia-pki 5fc117f1
(0600, SEC-029 custody untouched).
5. model CLEARED: all 23 application definitions removed INDIVIDUALLY, never batched
(hard rule 3), each read back; precondition re-verified immediately before the first
removal (0 units, 0 machines, nothing provisioned). Read-back after: `Model
"vr1-dc0" is empty`, 0 applications / 0 machines.
6. Step-3.5 model state confirmed to have SURVIVED the clear: `apt-mirror
http://10.12.8.4/ubuntu` still set, all six spaces still bound with their v4 and v6
subnets.
7. preflight P1-P4 + P7 PASS; P5 RED with 11 findings, ALL ELEVEN ruled-accepted
(six 2026-07-30, five 2026-07-31), enumerated in
docs/audit/stage5-preflight-dc0-20260731.txt.
NOTES CARRIED FORWARD
---------------------
- `juju remove-application` PROMPTS by default and aborts on a non-interactive stdin
(`ERROR application removal: aborted`, exit 1). `--no-prompt` is required from a
non-interactive session. Recorded because the first removal read as a failure.
- `ceph-osd`'s `tags=openstack` constraint (bundle.yaml) is still the known exposure and
is still NOT a problem for this deploy: placement is by explicit machine id
(`ceph-osd/0..3` -> machines 5,6,7,8). A later UNPLACED `juju add-unit ceph-osd` would
match no machine. LOGGED, unchanged.
- **G17's dc0 half is now genuinely armable.** The existing capture was taken on the
CONTROLLER VM, not a role node, and said so. The nine role nodes are booting for the
first time as this file is written -- that is G17's window.
WHAT IS NOT CLAIMED
-------------------
`Deploy of bundle completed.` means juju ACCEPTED and queued the bundle. It does NOT mean
the cloud is up. 9 machines are allocating, 33 units are pending, and the settle to
phase-01's documented pre-vault-init end state takes hours. Nothing here asserts unit
health, relation settling, or any service verdict.
================================================================================
POST-DEPLOY: THREE ARTIFACT/CONFIG DEFECTS, IN THE ORDER THEY UNMASKED
================================================================================
Each was hidden by the one before it. `apt-get update --error-on=any` fails on the
FIRST bad source, so only one is ever visible at a time.
D1 -- `jammy-backports` 404 on the dc0 mirror. RULED + FIXED 2026-07-31 (D-135
amendment, operator: "Sync the backports into the mirror"). Fixed at source in
scripts/dc-mirror.sh, T10 re-pointed, applied live, sync OK, verified on CONTENT
(Release fields + a real .deb fetch: elfutils_0.188-1~bpo22.04.1_amd64.deb, 200,
502468 B). Mirror 951G -> 952G.
D2 -- the UPSTREAM UCA is unreachable from nodes. MEASURED FROM A NODE (not the rack,
which holds a default route nodes lack):
node -> http://ubuntu-cloud.archive.canonical.com/... 000
node -> http://10.12.8.4/cloud-archive/... 200, real content
The node's source line points UPSTREAM. juju's `apt-mirror` model-config rewrites
only the Ubuntu archive, not the UCA source the charm adds. SIX apps set it
explicitly across TWO option names -- `openstack-origin` on barbican/magnum/octavia
and `source` on ceph-mon/ceph-osd/ceph-radosgw -- and every other OpenStack charm
defaults to `openstack-origin: caracal`, which resolves to the same upstream pocket.
The UCA signing key IS already on the node
(/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg), so a raw `deb` line
pointing at the mirrored UCA will verify with no `|key` suffix.
EXPERIMENT RESULT WORTH KEEPING: this asymmetry is a genuine D-135 finding -- the
FULL-MIRROR DC must rewrite every non-Ubuntu source; the PROXY DC needs none,
because apt-cacher-ng forwards whatever URL it is handed. dc1 will not hit this.
D3 -- >>> `prefer-ipv6: true` IS FATAL ON THE SEVEN CHARMS THAT DECLARE IT. <<<
MEASURED 2026-07-31 19:08, and this is the exact risk this session flagged as
"never observed live" when it was deliberately left OUT of the D-101 ruling note:
unit-keystone-0 ... Exception: Interface 'eth0' does not have a scope global
non-temporary ipv6 address.
Cause, and it matches the charm source read earlier today EXACTLY:
`get_relation_ip()` (charmhelpers/contrib/network/ip.py:617) returns EARLY with
`get_ipv6_addr(dynamic_only=False)[0]` whenever `prefer-ipv6` is true -- so the unit
tries to advertise a v6 address for every relation. MEASURED on the container:
keystone/0 eth0 -> fe80::216:3eff:feb4:5518/64 (LINK-LOCAL ONLY, no global v6)
The NODES are dual-stacked; the LXD CONTAINERS the API charms run in are NOT.
Live config confirms the split: keystone, cinder, glance, neutron-api,
nova-cloud-controller, openstack-dashboard, ceph-radosgw all read
`prefer-ipv6=true` and ALL SEVEN are in `error`. ceph-mon and mysql-innodb-cluster
read `false` (charm default) and fail for other reasons; ovn-central and vault do
not declare the option at all.
CONSEQUENCE: the 2026-07-31 ruling removed the option from the SIX charms that
never declared it, which was correct and necessary. The SEVEN that DO declare it
were explicitly NOT covered by that ruling -- and they are now the blocker.
NOT TAKEN HERE: R2 / D-101 are ruled surfaces and this needs its own GA-R5 exchange.
WHY DROPPING IT IS EXPECTED TO BE SAFE, from the same measurement chain: the option
is a UNIT ADDRESS-FAMILY switch, not a listener switch. HAProxy's `:::port` bind is
gated on `ipv6_enabled = not is_ipv6_disabled()` (the kernel sysctl), and the
container carries a link-local v6, so v6 is NOT disabled and the bind still emits.
Pacemaker assigns the v6 VIP by address-family detection, independent of the option.
OPEN QUESTION that belongs to the operator, stated not glossed: whether a v6 VIP
placed on a container whose eth0 has no global v6 is ROUTABLE. That is a different
question from whether the charm installs, and it is not answered here.
