diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 2d50653..936f9c8 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1722,6 +1722,59 @@ overlay ONLY, so it now grades a narrower input than either P2 or the deploy. Not deploy-blocking -- it is a teardown-time check -- and it is DOCFIX material for the same fold. + **>>> THE CORRECTED dc0 DEPLOY WAS DRY-RUN 2026-08-02 AND EXITS 0 -- AND THE RUN PROVED + THREE OF STEP 4.2's OWN GATE ITEMS CANNOT BE GRADED BY THE COMMAND THE GATE NAMES. <<<** + Capture: `docs/audit/stage5-dryrun-dc0-20260802.txt` (400 lines, both verbosities). Run + FROM THE dc0 RACK (D-138), model `vr1-dc0` on controller `vr1-dc0-controller`, juju + `3.6.27-genericlinux-amd64`; the model was EMPTY (0 machines) at run time. The staged + deploy input was sha256-verified against repo HEAD IMMEDIATELY BEFORE the run rather + than inherited from the prior close -- all five tracked files match + (`bundle.yaml 4c8a7852`, `vr1-dc0-vips 80d86156`, `vr1-dc0-machines e3be85e4`, + `domain-manager-policy 64f99b63`, `overrides.zip 02fe1fd7`), and the gitignored PKI + overlay is unchanged at `5fc117f1` mode `0600` (hashed, never read). + **GRADED PASS:** nine machines `0`-`8` (not VR0's four); all 9 constraint lines read + `arch=amd64 tags=openstack-vr1-dc0,`, split 3 control / 2 compute / 4 storage per + D-121 Option C, none naming the other DC; EXIT 0 with all three overlays consumed. + **`ceph-osd`'s `tags=openstack` exposure, which this document deferred to "decided at + 4.2": the plan places `ceph-osd/0..3` on machines 5-8 by explicit placement exactly as + reasoned, and the bare tag never appears. The exposure remains a LATER unplaced + `juju add-unit ceph-osd`; this deploy does not perform one. Nothing to decide today.** + **>>> THE INSTRUMENT FINDING, WHICH IS THE DURABLE HALF. <<<** MEASURED at both + verbosities: `juju deploy --dry-run` on a bundle prints ONLY located charms, + upload/deploy application lines, BARE machine ids, unit/container placement, and + relations. **It prints NO application options and NO VIPs, at any verbosity** -- greps + over the `--debug` capture return ZERO for `bridge-interface-mappings`, `physnet1`, + `openstack-origin`, `cloud-archive` and `prefer-chassis`; the only `10.12.4.` hits are + the controller's own API addresses and the only `vip` hit is the echoed command line. + `--debug` adds exactly one thing: the per-machine CONSTRAINT lines + (`bundlehandler.go:1035`). **CONSEQUENCES, all three now fixed in the same commit:** + (i) gate item 1 (tags) was ungradable by the command as written -- the runbook now + names `--dry-run --debug` and says why; (ii) the PRE-EXISTING gate item "every VIP in + the plan is on THIS DC's prefixes" **COULD NEVER HAVE FAILED, because no VIP is ever in + the plan** -- removed, with the property relocated in writing to preflight P2, which + already asserts the VIP bands over the same merged input; (iii) the two option-reading + items added EARLIER THE SAME SESSION had the same defect and were removed within hours + of being written. **OWNED PLAINLY: I wrote two assertions that could not fail into the + gate whose whole purpose was to catch a defect that had survived because nothing could + see it, and the live run -- not my re-reading -- caught them.** That is instance + fourteen of this repo's instrument-currency pattern, and the second time in two sessions + that the catching agent was the measurement rather than the review. + **NEW STEP 4.4 (`juju config`, read-only, run the moment `juju deploy` returns).** The + ovn-chassis options-map merge and the 15 mirror origins are provable ONLY from the + deployed application's config: `provider-bundle-check.py`'s `_deep_merge` merges + key-by-key BY CONSTRUCTION, so no offline check can distinguish a key-by-key merge from + a map REPLACE, and the dry-run cannot see options at all. Both overlays' VERIFY-LIVE + headers, which instructed the reader to assert this AT the dry-run, are corrected in + place and now point at 4.4. + **CONTRADICTION MEASURED, LOGGED NOT RESOLVED (hard rule 1):** the phase-4 runbook says + `bundle.yaml` ships "NO designate" for this stage (D-019 as superseded by D-106), while + the plan deploys `designate`, `designate-bind`, `designate-hacluster` and + `designate-mysql-router` and places `designate/0` on `0/lxd/5`. One of the two is wrong; + it is not resolvable from the runbook and is NOT deploy-blocking (designate deploying + early costs containers, not correctness). Annotated at the bullet so no session reads it + as a fact about the current bundle. + **THE DEPLOY ITSELF IS NOT RUN.** Step 4.3 is a gated MUTATION and awaits operator + approval; the dry-run is where this session stops. **>>> THE dc0 SNAP FORWARD PROXY IS INSTALLED, RUNNING AND GATE-VERIFIED 2026-07-31. THE LAST BUILD BETWEEN HERE AND `add-model` IS DONE. <<<** Capture diff --git a/docs/audit/stage5-dryrun-dc0-20260802.txt b/docs/audit/stage5-dryrun-dc0-20260802.txt new file mode 100644 index 0000000..83b19de --- /dev/null +++ b/docs/audit/stage5-dryrun-dc0-20260802.txt @@ -0,0 +1,400 @@ +stage5-dryrun-dc0-20260802.txt +============================== +Step 4.2 dry-run gate, vr1-dc0, run 2026-08-02 FROM THE dc0 RACK (D-138 puts the +juju client in the DC; voffice1 has no controller registered). Model vr1-dc0 on +controller vr1-dc0-controller, juju 3.6.27-genericlinux-amd64. + +Deploy input, staged at ~/repo-stage and sha256-verified against repo HEAD 1706f22 +IMMEDIATELY BEFORE this run (all five tracked files match; the gitignored PKI +overlay 5fc117f1... 0600, hashed never read): + + juju deploy ./bundle.yaml \ + --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-machines.yaml \ + --overlay overlays/vr1-dc0-octavia-pki.yaml \ + -m vr1-dc0 --dry-run + +This is the DOCFIX-208 command -- the machines overlay was missing from the +runbook's dc0 form until this session. + +>>> WHAT THIS CAPTURE PROVES, AND WHAT IT CANNOT. READ BEFORE CITING IT. <<< + +MEASURED: `juju deploy --dry-run` on a bundle prints ONLY located charms, +upload/deploy application lines, bare machine ids, unit/container placement, and +relations. It prints NO application options, NO VIPs, and NO machine constraints. +Adding --debug surfaces the per-machine CONSTRAINTS (bundlehandler.go:1035) and +nothing else -- option values are still absent at any verbosity. Grepped, both +forms: bridge-interface-mappings 0, physnet1 0, openstack-origin 0, +cloud-archive 0, prefer-chassis 0. The only "10.12.4." hits under --debug are the +controller's own API addresses; the only "vip" hit is the echoed command line. + +CONSEQUENCE: Step 4.2's gate items asserting VIP prefixes and application options +CANNOT be graded from this command. That was true of the pre-existing VIP item as +well as the two option items added earlier the same session. Fixed in the same +commit as this capture; the offline merged-input assertion (preflight P2) and a +post-deploy `juju config` assertion are where those properties actually get read. + +GRADED HERE: + item 1 (tags) PASS -- via --debug only: 9 constraint lines, ALL + "arch=amd64 tags=openstack-vr1-dc0,", split 3 control / 2 compute / + 4 storage, matching D-121 Option C. No machine names the other DC. + item 2 (nine machines "0".."8") PASS -- machines 0-8 present, exactly nine. + Not four: this is VR1's role-separated topology, not VR0's. + item 4 (no overlay path error) PASS -- EXIT 0, all three overlays consumed. + ceph-osd tags=openstack exposure (CURRENT-STATE, "decided at 4.2"): the plan + places ceph-osd/0..3 on machines 5-8 by explicit placement, as recorded. The + bare `openstack` tag is ABSENT from the region and is NOT visible in this + plan; the exposure remains a LATER unplaced `juju add-unit ceph-osd`, which + this deploy does not perform. + + NOT GRADED, because the command cannot show them: VIP prefixes, + ovn-chassis's merged options map, the 15 mirror origins. + +--- plain --dry-run, verbatim, EXIT 0 ------------------------------------------- +WARNING These fields + unmarshal document 0: yaml: unmarshal errors: + line 62: field name not found in type charm.bundleData + line 96: field variables not found in type charm.bundleData +will be ignored during deployment +Located charm "barbican" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "barbican-vault" in charm-hub, channel 2024.1/stable +Located charm "ceph-mon" in charm-hub, channel squid/stable +Located charm "ceph-osd" in charm-hub, channel squid/stable +Located charm "ceph-radosgw" in charm-hub, channel squid/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "ceph-rbd-mirror" in charm-hub, channel squid/stable +Located charm "cinder" in charm-hub, channel 2024.1/stable +Located charm "cinder-backup" in charm-hub, channel 2024.1/stable +Located charm "cinder-ceph" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "designate" in charm-hub, channel 2024.1/stable +Located charm "designate-bind" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "glance" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "glance-simplestreams-sync" in charm-hub, channel 2024.1/stable +Located charm "keystone" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "magnum" in charm-hub, channel 2024.1/stable +Located charm "magnum-dashboard" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "memcached" in charm-hub, channel latest/stable +Located charm "mysql-innodb-cluster" in charm-hub, channel 8.0/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "neutron-api" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "neutron-api-plugin-ovn" in charm-hub, channel 2024.1/stable +Located charm "nova-cloud-controller" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "nova-compute" in charm-hub, channel 2024.1/stable +Located charm "octavia" in charm-hub, channel 2024.1/stable +Located charm "octavia-dashboard" in charm-hub, channel 2024.1/stable +Located charm "octavia-diskimage-retrofit" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "openstack-dashboard" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "ovn-central" in charm-hub, channel 24.03/stable +Located charm "ovn-chassis" in charm-hub, channel 24.03/stable +Located charm "ovn-chassis" in charm-hub, channel 24.03/stable +Located charm "placement" in charm-hub, channel 2024.1/stable +Located charm "hacluster" in charm-hub, channel 2.4/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Located charm "rabbitmq-server" in charm-hub, channel 3.9/stable +Located charm "vault" in charm-hub, channel 1.8/stable +Located charm "mysql-router" in charm-hub, channel 8.0/stable +Changes to deploy bundle: +- upload charm barbican from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application barbican from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm hacluster from charm-hub for base ubuntu@22.04/stable from channel 2.4/stable with architecture=amd64 +- deploy application barbican-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- upload charm mysql-router from charm-hub for base ubuntu@22.04/stable from channel 8.0/stable with architecture=amd64 +- deploy application barbican-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm barbican-vault from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application barbican-vault from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm ceph-mon from charm-hub for base ubuntu@22.04/stable from channel squid/stable with architecture=amd64 +- deploy application ceph-mon from charm-hub on ubuntu@22.04/stable with squid/stable +- upload charm ceph-osd from charm-hub for base ubuntu@22.04/stable from channel squid/stable with architecture=amd64 +- deploy application ceph-osd from charm-hub on ubuntu@22.04/stable with squid/stable +- upload charm ceph-radosgw from charm-hub for base ubuntu@22.04/stable from channel squid/stable with architecture=amd64 +- deploy application ceph-radosgw from charm-hub on ubuntu@22.04/stable with squid/stable +- deploy application ceph-radosgw-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- upload charm ceph-rbd-mirror from charm-hub for base ubuntu@22.04/stable from channel squid/stable with architecture=amd64 +- deploy application ceph-rbd-mirror from charm-hub on ubuntu@22.04/stable with squid/stable +- upload charm cinder from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application cinder from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm cinder-backup from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application cinder-backup from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm cinder-ceph from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application cinder-ceph from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application cinder-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application cinder-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- deploy application dashboard-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm designate from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application designate from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm designate-bind from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application designate-bind from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application designate-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application designate-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm glance from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application glance from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application glance-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application glance-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm glance-simplestreams-sync from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application glance-simplestreams-sync from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm keystone from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application keystone from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application keystone-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application keystone-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm magnum from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application magnum from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm magnum-dashboard from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application magnum-dashboard from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application magnum-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application magnum-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm memcached from charm-hub for base ubuntu@22.04/stable from channel latest/stable with architecture=amd64 +- deploy application memcached from charm-hub on ubuntu@22.04/stable with latest/stable +- upload charm mysql-innodb-cluster from charm-hub for base ubuntu@22.04/stable from channel 8.0/stable with architecture=amd64 +- deploy application mysql-innodb-cluster from charm-hub on ubuntu@22.04/stable with 8.0/stable +- deploy application ncc-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm neutron-api from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application neutron-api from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application neutron-api-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application neutron-api-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm neutron-api-plugin-ovn from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application neutron-api-plugin-ovn from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm nova-cloud-controller from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application nova-cloud-controller from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application nova-cloud-controller-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- upload charm nova-compute from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application nova-compute from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm octavia from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application octavia from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm octavia-dashboard from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application octavia-dashboard from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- upload charm octavia-diskimage-retrofit from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application octavia-diskimage-retrofit from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application octavia-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application octavia-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm openstack-dashboard from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application openstack-dashboard from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application openstack-dashboard-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- upload charm ovn-central from charm-hub for base ubuntu@22.04/stable from channel 24.03/stable with architecture=amd64 +- deploy application ovn-central from charm-hub on ubuntu@22.04/stable with 24.03/stable +- upload charm ovn-chassis from charm-hub for base ubuntu@22.04/stable from channel 24.03/stable with architecture=amd64 +- deploy application ovn-chassis from charm-hub on ubuntu@22.04/stable with 24.03/stable +- deploy application ovn-chassis-octavia from charm-hub on ubuntu@22.04/stable with 24.03/stable using ovn-chassis +- upload charm placement from charm-hub for base ubuntu@22.04/stable from channel 2024.1/stable with architecture=amd64 +- deploy application placement from charm-hub on ubuntu@22.04/stable with 2024.1/stable +- deploy application placement-hacluster from charm-hub on ubuntu@22.04/stable with 2.4/stable using hacluster +- deploy application placement-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- upload charm rabbitmq-server from charm-hub for base ubuntu@22.04/stable from channel 3.9/stable with architecture=amd64 +- deploy application rabbitmq-server from charm-hub on ubuntu@22.04/stable with 3.9/stable +- upload charm vault from charm-hub for base ubuntu@22.04/stable from channel 1.8/stable with architecture=amd64 +- deploy application vault from charm-hub on ubuntu@22.04/stable with 1.8/stable +- deploy application vault-mysql-router from charm-hub on ubuntu@22.04/stable with 8.0/stable using mysql-router +- add new machine 0 +- add new machine 1 +- add new machine 2 +- add new machine 3 +- add new machine 4 +- add new machine 5 +- add new machine 6 +- add new machine 7 +- add new machine 8 +- add relation nova-cloud-controller:memcache - memcached:cache +- add relation vault-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation vault:shared-db - vault-mysql-router:shared-db +- add relation mysql-innodb-cluster:certificates - vault:certificates +- add relation keystone-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation keystone-mysql-router:shared-db - keystone:shared-db +- add relation keystone:certificates - vault:certificates +- add relation keystone:ha - keystone-hacluster:ha +- add relation glance-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation glance-mysql-router:shared-db - glance:shared-db +- add relation glance:identity-service - keystone:identity-service +- add relation glance:certificates - vault:certificates +- add relation glance:ha - glance-hacluster:ha +- add relation glance-simplestreams-sync:identity-service - keystone:identity-service +- add relation glance-simplestreams-sync:certificates - vault:certificates +- add relation ncc-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation ncc-mysql-router:shared-db - nova-cloud-controller:shared-db +- add relation nova-cloud-controller:identity-service - keystone:identity-service +- add relation nova-cloud-controller:amqp - rabbitmq-server:amqp +- add relation nova-cloud-controller:image-service - glance:image-service +- add relation nova-cloud-controller:neutron-api - neutron-api:neutron-api +- add relation nova-cloud-controller:cloud-compute - nova-compute:cloud-compute +- add relation nova-cloud-controller:cinder-volume-service - cinder:cinder-volume-service +- add relation nova-cloud-controller:certificates - vault:certificates +- add relation nova-cloud-controller:ha - nova-cloud-controller-hacluster:ha +- add relation nova-compute:amqp - rabbitmq-server:amqp +- add relation nova-compute:image-service - glance:image-service +- add relation placement-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation placement-mysql-router:shared-db - placement:shared-db +- add relation placement:identity-service - keystone:identity-service +- add relation placement:placement - nova-cloud-controller:placement +- add relation placement:certificates - vault:certificates +- add relation placement:ha - placement-hacluster:ha +- add relation neutron-api-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation neutron-api-mysql-router:shared-db - neutron-api:shared-db +- add relation neutron-api:identity-service - keystone:identity-service +- add relation neutron-api:amqp - rabbitmq-server:amqp +- add relation neutron-api:certificates - vault:certificates +- add relation neutron-api-plugin-ovn:neutron-plugin - neutron-api:neutron-plugin-api-subordinate +- add relation neutron-api-plugin-ovn:ovsdb-cms - ovn-central:ovsdb-cms +- add relation neutron-api-plugin-ovn:certificates - vault:certificates +- add relation ovn-central:certificates - vault:certificates +- add relation ovn-chassis:ovsdb - ovn-central:ovsdb +- add relation ovn-chassis:nova-compute - nova-compute:neutron-plugin +- add relation ovn-chassis:certificates - vault:certificates +- add relation neutron-api:ha - neutron-api-hacluster:ha +- add relation cinder-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation cinder-mysql-router:shared-db - cinder:shared-db +- add relation cinder:identity-service - keystone:identity-service +- add relation cinder:amqp - rabbitmq-server:amqp +- add relation cinder:image-service - glance:image-service +- add relation cinder:certificates - vault:certificates +- add relation cinder-ceph:storage-backend - cinder:storage-backend +- add relation cinder-ceph:ceph - ceph-mon:client +- add relation cinder-ceph:ceph-access - nova-compute:ceph-access +- add relation cinder:ha - cinder-hacluster:ha +- add relation cinder-backup:backup-backend - cinder:backup-backend +- add relation cinder-backup:ceph - ceph-mon:client +- add relation ceph-mon:osd - ceph-osd:mon +- add relation ceph-mon:client - nova-compute:ceph +- add relation ceph-mon:client - glance:ceph +- add relation ceph-radosgw:mon - ceph-mon:radosgw +- add relation ceph-radosgw:identity-service - keystone:identity-service +- add relation ceph-radosgw:certificates - vault:certificates +- add relation ceph-radosgw:ha - ceph-radosgw-hacluster:ha +- add relation ceph-rbd-mirror:ceph-local - ceph-mon:rbd-mirror +- add relation dashboard-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation dashboard-mysql-router:shared-db - openstack-dashboard:shared-db +- add relation openstack-dashboard:identity-service - keystone:identity-service +- add relation openstack-dashboard:certificates - vault:certificates +- add relation openstack-dashboard:ha - openstack-dashboard-hacluster:ha +- add relation octavia-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation octavia-mysql-router:shared-db - octavia:shared-db +- add relation octavia:identity-service - keystone:identity-service +- add relation octavia:amqp - rabbitmq-server:amqp +- add relation octavia:neutron-api - neutron-api:neutron-load-balancer +- add relation octavia:certificates - vault:certificates +- add relation octavia-dashboard:dashboard - openstack-dashboard:dashboard-plugin +- add relation ovn-chassis-octavia:ovsdb - ovn-central:ovsdb +- add relation ovn-chassis-octavia:ovsdb-subordinate - octavia:ovsdb-subordinate +- add relation ovn-chassis-octavia:certificates - vault:certificates +- add relation octavia-diskimage-retrofit:juju-info - glance-simplestreams-sync:juju-info +- add relation octavia-diskimage-retrofit:identity-credentials - keystone:identity-credentials +- add relation octavia:ha - octavia-hacluster:ha +- add relation barbican-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation barbican-mysql-router:shared-db - barbican:shared-db +- add relation barbican:identity-service - keystone:identity-service +- add relation barbican:amqp - rabbitmq-server:amqp +- add relation barbican:certificates - vault:certificates +- add relation barbican:secrets - barbican-vault:secrets +- add relation barbican-vault:certificates - vault:certificates +- add relation barbican-vault:secrets-storage - vault:secrets +- add relation barbican:ha - barbican-hacluster:ha +- add relation magnum-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation magnum:shared-db - magnum-mysql-router:shared-db +- add relation magnum:identity-service - keystone:identity-service +- add relation magnum:amqp - rabbitmq-server:amqp +- add relation magnum:certificates - vault:certificates +- add relation magnum-dashboard:dashboard - openstack-dashboard:dashboard-plugin +- add relation magnum:ha - magnum-hacluster:ha +- add relation designate-mysql-router:db-router - mysql-innodb-cluster:db-router +- add relation designate-mysql-router:shared-db - designate:shared-db +- add relation designate:identity-service - keystone:identity-service +- add relation designate:amqp - rabbitmq-server:amqp +- add relation designate:certificates - vault:certificates +- add relation designate:coordinator-memcached - memcached:cache +- add relation designate:dns-backend - designate-bind:dns-backend +- add relation designate:ha - designate-hacluster:ha +- add unit ceph-osd/0 to new machine 5 +- add unit ceph-osd/1 to new machine 6 +- add unit ceph-osd/2 to new machine 7 +- add unit ceph-osd/3 to new machine 8 +- add unit nova-compute/0 to new machine 3 +- add unit nova-compute/1 to new machine 4 +- add lxd container 0/lxd/0 on new machine 0 +- add lxd container 0/lxd/1 on new machine 0 +- add lxd container 1/lxd/0 on new machine 1 +- add lxd container 2/lxd/0 on new machine 2 +- add lxd container 0/lxd/2 on new machine 0 +- add lxd container 0/lxd/3 on new machine 0 +- add lxd container 0/lxd/4 on new machine 0 +- add lxd container 0/lxd/5 on new machine 0 +- add lxd container 2/lxd/1 on new machine 2 +- add lxd container 0/lxd/6 on new machine 0 +- add lxd container 1/lxd/1 on new machine 1 +- add lxd container 0/lxd/7 on new machine 0 +- add lxd container 0/lxd/8 on new machine 0 +- add lxd container 0/lxd/9 on new machine 0 +- add lxd container 0/lxd/10 on new machine 0 +- add lxd container 1/lxd/2 on new machine 1 +- add lxd container 2/lxd/2 on new machine 2 +- add lxd container 0/lxd/11 on new machine 0 +- add lxd container 0/lxd/12 on new machine 0 +- add lxd container 0/lxd/13 on new machine 0 +- add lxd container 0/lxd/14 on new machine 0 +- add lxd container 0/lxd/15 on new machine 0 +- add lxd container 1/lxd/3 on new machine 1 +- add lxd container 2/lxd/3 on new machine 2 +- add lxd container 0/lxd/16 on new machine 0 +- add lxd container 0/lxd/17 on new machine 0 +- add lxd container 0/lxd/18 on new machine 0 +- add unit barbican/0 to 0/lxd/0 +- add unit ceph-mon/0 to 0/lxd/1 +- add unit ceph-mon/1 to 1/lxd/0 +- add unit ceph-mon/2 to 2/lxd/0 +- add unit ceph-radosgw/0 to 0/lxd/2 +- add unit ceph-rbd-mirror/0 to 0/lxd/3 +- add unit cinder/0 to 0/lxd/4 +- add unit designate/0 to 0/lxd/5 +- add unit designate-bind/0 to 2/lxd/1 +- add unit glance/0 to 0/lxd/6 +- add unit glance-simplestreams-sync/0 to 1/lxd/1 +- add unit keystone/0 to 0/lxd/7 +- add unit magnum/0 to 0/lxd/8 +- add unit memcached/0 to 0/lxd/9 +- add unit mysql-innodb-cluster/0 to 0/lxd/10 +- add unit mysql-innodb-cluster/1 to 1/lxd/2 +- add unit mysql-innodb-cluster/2 to 2/lxd/2 +- add unit neutron-api/0 to 0/lxd/11 +- add unit nova-cloud-controller/0 to 0/lxd/12 +- add unit octavia/0 to 0/lxd/13 +- add unit openstack-dashboard/0 to 0/lxd/14 +- add unit ovn-central/0 to 0/lxd/15 +- add unit ovn-central/1 to 1/lxd/3 +- add unit ovn-central/2 to 2/lxd/3 +- add unit placement/0 to 0/lxd/16 +- add unit rabbitmq-server/0 to 0/lxd/17 +- add unit vault/0 to 0/lxd/18 +EXIT=0 + +--- --dry-run --debug: the 9 constraint lines, verbatim ------------------------ +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,control" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,control" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,control" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,compute" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,compute" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,storage" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,storage" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,storage" +19:32:21 INFO cmd bundlehandler.go:1035 with base "ubuntu@22.04/stable", with constraints "arch=amd64 tags=openstack-vr1-dc0,storage" + +--- --dry-run --debug: option-value greps, all zero ---------------------------- +bridge-interface-mappings 0 +physnet1 0 +openstack-origin 0 +cloud-archive 0 +prefer-chassis 0 diff --git a/docs/changelog-20260802-deploy-input.md b/docs/changelog-20260802-deploy-input.md index 8a68f58..14b73a2 100644 --- a/docs/changelog-20260802-deploy-input.md +++ b/docs/changelog-20260802-deploy-input.md @@ -114,19 +114,94 @@ DC's nodes can reach.** Names dc1's deliberate absence of the block as D-135's experiment RESULT, so a later reader does not "fix" the asymmetry. -- **Revert:** delete gate items 5 and 6. Not recommended -- they are the only - assertions in the chain that can see this overlay's payload. +**>>> ITEMS 5 AND 6 AS FIRST WRITTEN WERE THEMSELVES CHECKERS THAT COULD NOT +FAIL, AND WERE REMOVED WITHIN THE HOUR. SEE ITEM 3. <<<** Left recorded here +rather than edited out: the mistake is the point. -## Item 3 -- LOGGED, NOT FIXED (hard rule 1) +- **Revert:** n/a -- superseded by item 3 in this same session. -`runbooks/phase-00-teardown-maas-reset.md:233` runs `provider-bundle-check.py` with -the vips overlay ONLY, so it grades a narrower input than either preflight P2 or the -deploy command. Not deploy-blocking (teardown-time check), DOCFIX material for the -same fold. Recorded in `docs/CURRENT-STATE.md`, not actioned here. +## Item 3 -- the dry-run graded the gate, and three of its items failed + +**THE RUN.** `docs/audit/stage5-dryrun-dc0-20260802.txt` (400 lines). From the dc0 +rack (D-138), model `vr1-dc0`, juju 3.6.27, model EMPTY at run time. Deploy input +sha256-verified against HEAD immediately before the run, not inherited from the +prior close: all five tracked files match, PKI overlay unchanged at `5fc117f1` +mode `0600` (hashed, never read). **EXIT 0.** + +**PASSED:** nine machines `0`-`8`; all 9 constraint lines +`arch=amd64 tags=openstack-vr1-dc0,` in the 3/2/4 split, none naming the +other DC; all three overlays consumed without a path error. Also settled the +`ceph-osd tags=openstack` item CURRENT-STATE deferred to this gate: the plan +places `ceph-osd/0..3` on machines 5-8 by explicit placement and the bare tag +never appears -- the exposure remains a later UNPLACED `add-unit`, which this +deploy does not perform. + +**THE INSTRUMENT FINDING.** Measured at both verbosities: `juju deploy --dry-run` +on a bundle prints ONLY located charms, upload/deploy lines, BARE machine ids, +placement and relations. **No application options, no VIPs, at any verbosity.** +Greps over the `--debug` capture: `bridge-interface-mappings` 0, `physnet1` 0, +`openstack-origin` 0, `cloud-archive` 0, `prefer-chassis` 0. The only `10.12.4.` +hits are the controller's own API addresses; the only `vip` hit is the echoed +command line. `--debug` adds exactly one thing -- the per-machine constraint lines +(`bundlehandler.go:1035`). + +Three consequences, all fixed here: + +1. **Gate item 1 (tags) was ungradable by the command as written.** The named + command is now `--dry-run --debug`, with the measurement quoted as the reason + so nobody drops the flag as noise. +2. **The PRE-EXISTING VIP item could never have failed** -- "every VIP in the + plan is on THIS DC's prefixes", when no VIP is ever in the plan. REMOVED, and + the property relocated IN WRITING to preflight P2, which already asserts the + VIP bands over the same merged input. Per the standing rule, the assertion was + replaced with the true invariant and the move is stated, not silently dropped. +3. **My own items 5 and 6, written earlier this session, had the same defect.** + REMOVED. The properties are real, so they became **new Step 4.4** -- a + `juju config` read run the moment `juju deploy` returns, before the model + settles. That is the only thing that can prove the merge: `_deep_merge` in + `provider-bundle-check.py` merges key-by-key BY CONSTRUCTION, so it cannot + distinguish a key-by-key merge from a map REPLACE, and the dry-run sees no + options at all. + +**OWNED.** I added two assertions that could not fail to the very gate whose +purpose was to catch a defect that had survived precisely because nothing could +see it. They were caught by the live run, not by re-reading my own work -- +instance fourteen of this repo's instrument-currency pattern and the second +consecutive session where measurement, not review, was the catching agent. + +**ALSO CORRECTED: both machines overlays' VERIFY-LIVE headers**, which instructed +the reader to assert the options merge "at the live dry-run". Measured impossible. +Corrected in place (not deleted) and pointed at Step 4.4. dc1's header additionally +notes the split that dc0's does not have: dc1's machines BLOCK *is* visible in the +`--debug` plan and *is* gradable at 4.2; only its options half is not. + +- **Revert:** `git revert`. Restores a gate with three ungradable items and two + overlay headers naming an impossible check. + +## Item 4 -- LOGGED, NOT FIXED (hard rule 1) + +1. `runbooks/phase-00-teardown-maas-reset.md:233` runs `provider-bundle-check.py` + with the vips overlay ONLY, so it grades a narrower input than either preflight + P2 or the deploy command. Not deploy-blocking (teardown-time check), DOCFIX + material for the same fold. +2. **The runbook says `bundle.yaml` ships "NO designate" for this stage; the plan + deploys four designate applications** and places `designate/0` on `0/lxd/5`. + One of the two is wrong and it is not resolvable from the runbook. NOT + deploy-blocking -- designate deploying early costs containers, not correctness. + The bullet is annotated in place so no session reads it as a fact about the + current bundle; the underlying question (does D-106 leave designate in the + Stage-5 bundle?) is unanswered and left for the operator. + +Both recorded in `docs/CURRENT-STATE.md`, neither actioned. --- **Gates at delivery:** `bash scripts/repo-lint.sh` -- 0 fail, 1 warn (the legacy -D-001..018 ASCII carve-out), 658 files. No script changed, so no harness is +D-001..018 ASCII carve-out). `provider-bundle-check.py` over the dc0 merged input +PASS. Both machines overlays re-parsed after editing (dc0: `applications` only, +16 apps; dc1: `applications` + `machines`). No script changed, so no harness is implicated; the gauntlet's last full run stands at ALL GREEN (98) from the 2026-08-02 (part 2) close. + +**THE DEPLOY IS NOT RUN.** Step 4.3 is a gated mutation and awaits operator +approval. This session stops at the dry-run. diff --git a/overlays/vr1-dc0-machines.yaml b/overlays/vr1-dc0-machines.yaml index 07be79c..f7ff223 100644 --- a/overlays/vr1-dc0-machines.yaml +++ b/overlays/vr1-dc0-machines.yaml @@ -88,7 +88,16 @@ # depends on it -- those overlays set `options.vip` on apps carrying other base # options) but it is NOT PROVEN HERE: provider-bundle-check.py's `_deep_merge` merges # key-by-key by construction, so validating through the checker cannot tell the two -# behaviours apart. Only the live dry-run can. Assert it explicitly; do not assume. +# behaviours apart. +# >>> CORRECTED 2026-08-02 (DOCFIX-208): "Only the live dry-run can" IS WRONG, and a +# reader following it would have asserted nothing. MEASURED at both verbosities +# (docs/audit/stage5-dryrun-dc0-20260802.txt): `juju deploy --dry-run` prints NO +# application options AT ALL -- greps for bridge-interface-mappings, physnet1, +# openstack-origin, cloud-archive and prefer-chassis all return ZERO, and --debug adds +# only the per-machine constraint lines. The merge is provable ONLY after deploy, from +# the deployed application's own config. The assertion now lives at +# runbooks/dc-dc-phase4-juju-bundle-per-dc.md Step 4.4, run the moment `juju deploy` +# returns and before the model settles. Assert it explicitly; do not assume. <<< # # Applied as part of the vr1-dc0 per-DC set (R6: this set lands BEFORE # overlays/dc-ha-scaleup.yaml, never in the same command as it): diff --git a/overlays/vr1-dc1-machines.yaml b/overlays/vr1-dc1-machines.yaml index bb2a13a..4fb19bc 100644 --- a/overlays/vr1-dc1-machines.yaml +++ b/overlays/vr1-dc1-machines.yaml @@ -48,7 +48,17 @@ # on it -- those overlays set `options.vip` on apps carrying other base options) but # it is NOT PROVEN HERE: provider-bundle-check.py's `_deep_merge` merges key-by-key # by construction, so validating through the checker cannot tell the two behaviours -# apart. Only the live dry-run can. Assert it explicitly; do not assume. +# apart. +# >>> CORRECTED 2026-08-02 (DOCFIX-208): "Only the live dry-run can" IS WRONG, and a +# reader following it would have asserted nothing. MEASURED at both verbosities on the +# dc0 twin (docs/audit/stage5-dryrun-dc0-20260802.txt): `juju deploy --dry-run` prints +# NO application options AT ALL -- greps for bridge-interface-mappings, physnet1, +# openstack-origin, cloud-archive and prefer-chassis all return ZERO, and --debug adds +# only the per-machine constraint lines. NOTE the halves differ for dc1: the machines +# BLOCK (this file's tag retag) IS visible in the --debug plan and IS gradable at 4.2 +# item 1; the options half is not. The options assertion now lives at +# runbooks/dc-dc-phase4-juju-bundle-per-dc.md Step 4.4, run the moment `juju deploy` +# returns and before the model settles. Assert it explicitly; do not assume. <<< # # ---- ovn-chassis bridge-interface-mappings: MOVED HERE FROM bundle.yaml 2026-07-29 ---- # These exact two values previously sat in the BASE bundle (item 3.8), which is diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index 3b9c440..ef0c018 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -596,13 +596,21 @@ EXACTLY the 4.3 commands plus `--dry-run`; if the two ever differ, this gate is validating something you are not deploying. +**`--debug` IS NOT OPTIONAL HERE, AND IT IS THE ONLY REASON ITEM 1 CAN FAIL.** +MEASURED 2026-08-02 (`docs/audit/stage5-dryrun-dc0-20260802.txt`): a plain +`juju deploy --dry-run` on a bundle prints ONLY located charms, upload/deploy +application lines, BARE machine ids, unit/container placement, and relations. It +prints no constraints at all. `--debug` adds the per-machine constraint lines +(`bundlehandler.go:1035`) that item 1 reads, and nothing else. Run the block for +YOUR DC and keep `--debug` in it. + `vr1-dc0` (order: vips -> machines -> pki): ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc0-vips.yaml \ --overlay overlays/vr1-dc0-machines.yaml \ --overlay overlays/vr1-dc0-octavia-pki.yaml \ - -m "${DC_MODEL}" --dry-run + -m "${DC_MODEL}" --dry-run --debug ``` `vr1-dc1` (machines overlay REQUIRED, order: vips -> machines -> pki): ```bash @@ -610,7 +618,7 @@ --overlay overlays/vr1-dc1-vips.yaml \ --overlay overlays/vr1-dc1-machines.yaml \ --overlay overlays/vr1-dc1-octavia-pki.yaml \ - -m "${DC_MODEL}" --dry-run + -m "${DC_MODEL}" --dry-run --debug ``` **GATE -- assert on the CONTENT of the plan, not on the command exiting 0:** 1. **Every machine reads `tags=openstack-$DC,`.** If any machine still @@ -621,35 +629,37 @@ 2. **Nine machines, `"0"` through `"8"`**, matching `bundle.yaml`'s machines block (D-121 Option C: 3 control / 2 compute / 4 storage). A four-machine plan means you are looking at VR0's topology. -3. **Every VIP in the plan is on THIS DC's prefixes** -- `10.12.4/8/12` for - `vr1-dc0`, `10.12.64/68/72` for `vr1-dc1` (`scripts/lib-net.sh:54-56`, - `:146-154`). A VIP from the other DC means the wrong overlay merged. -4. If the plan reports an ERROR on any `--overlay` path, do NOT drop the overlay +3. If the plan reports an ERROR on any `--overlay` path, do NOT drop the overlay to make it run. A missing overlay is a missing input, not an optional one. -5. **`ovn-chassis` shows THREE options, not one.** Added DOCFIX-208, because - items 1-3 could not have caught the defect that fix corrects -- they read the - machines block and the VIPs, and the machines overlay's whole payload is - under `applications:`. The plan must show - `bridge-interface-mappings` (from `overlays/${DC}-machines.yaml`) **alongside - both base options that survive only if juju merges the `options` MAP - key-by-key rather than replacing it**: `ovn-bridge-mappings: physnet1:br-ex` - and `prefer-chassis-as-gw: true` (B2, gateway-chassis election). This is the - VERIFY-LIVE assertion both machines overlays' own headers demand and that no - offline check can make: `provider-bundle-check.py`'s `_deep_merge` merges - key-by-key BY CONSTRUCTION, so validating through it cannot tell the two - behaviours apart. If only `bridge-interface-mappings` appears, the map was - REPLACED -- physnet mapping and gateway election are both gone from the - deploy input and provider egress dies. STOP. - Exactly TWO MACs in the value; a third means a wrong node. -6. **Every `openstack-origin` / ceph `source` in the plan resolves to an address - this DC's nodes can actually reach.** dc0 = 16 apps carry the overlay's - delta, 15 of them pointing at the in-DC mirror - (`deb http://10.12.8.4/cloud-archive jammy-updates/caracal main`). Any app - still reading `cloud:jammy-caracal` or a bare `ubuntu-cloud.archive.canonical.com` - URL is MEASURED UNREACHABLE from a node under the D-107 airgap and will fail - at package install, not at deploy. dc1 correctly has NO such block -- its - apt-cacher-ng forwards whatever URL it is handed, which is D-135's experiment - RESULT and not an asymmetry to "fix". + +> **>>> WHAT THIS DRY-RUN CANNOT SEE. DO NOT ADD AN ASSERTION HERE FOR ANY OF IT. +> <<<** MEASURED 2026-08-02, both verbosities +> (`docs/audit/stage5-dryrun-dc0-20260802.txt`): the plan carries **NO +> application options and NO VIPs**, at any verbosity. Greps over the `--debug` +> capture return ZERO for `bridge-interface-mappings`, `physnet1`, +> `openstack-origin`, `cloud-archive` and `prefer-chassis`; the only `10.12.4.` +> hits are the controller's own API addresses and the only `vip` hit is the +> echoed command line. This gate previously carried a fourth item asserting +> "every VIP in the plan is on THIS DC's prefixes" -- **it could never have +> failed**, because no VIP is ever in the plan. Two more option-reading items +> were added and removed within the same session, 2026-08-02, for the same +> reason: the runbook is not exempt from the standing rule that a checker which +> cannot fail is not a gate. The properties are real and still checked -- below. + +**WHERE THE VIP AND OPTION PROPERTIES ARE ACTUALLY GRADED:** + +- **VIPs, and the merged input generally -- OFFLINE, BEFORE this step.** + `scripts/preflight.sh` P2 runs `provider-bundle-check.py` over `bundle.yaml` + plus this DC's SAME three overlays and asserts the VIP bands per DC + (`scripts/lib-net.sh:54-56`, `:146-154`). That gate reads the merged INPUT, + which is exactly the artifact the dry-run does not expose. A VIP from the + other DC fails there, not here. +- **`ovn-chassis`'s merged options map, and the 15 mirror origins -- LIVE, in + Step 4c, immediately after the deploy returns.** No offline check can prove + them: `provider-bundle-check.py`'s `_deep_merge` merges key-by-key BY + CONSTRUCTION, so it cannot distinguish juju merging an `options` map + key-by-key from juju REPLACING it. Only the deployed application's own config + can, and the dry-run is not it. Capture this output. It is the plan-count evidence the exit gate cites -- plan counts are quoted only from captured command output, never from this file's @@ -678,6 +688,15 @@ Open the Step 4b watch windows before this returns. +- **CONTRADICTION MEASURED 2026-08-02, LOGGED NOT RESOLVED (hard rule 1).** The + bullet below says `bundle.yaml` ships "NO designate". The dc0 dry-run plan + deploys `designate`, `designate-bind`, `designate-hacluster` and + `designate-mysql-router`, and places `designate/0` on `0/lxd/5` + (`docs/audit/stage5-dryrun-dc0-20260802.txt`). One of the two is wrong and it + is not resolvable from this runbook: either the bullet is stale prose, or the + bundle carries applications this stage is not meant to deploy. It is NOT + deploy-blocking either way -- designate deploying early costs containers, not + correctness -- but do not read the bullet as a fact about the current bundle. - `bundle.yaml` ships "NO designate" for THIS stage per phase-01's own gate text (D-019, as superseded by D-106): Designate reactivation is Stage 7's job, do not reactivate it mid-Stage-5. @@ -687,6 +706,57 @@ (the Step 5 trigger, not a fault); the TLS consumers waiting on vault certs; octavia BLOCKED "Awaiting configure-resources". +### 4.4 -- Assert the MERGED CONFIG live, before the model settles [read-only gate] + +**Run this as soon as `juju deploy` returns -- do not wait for convergence.** +Added DOCFIX-208. Both properties below decide whether this deploy can ever +work, and **NOTHING before this point can read either of them**: the dry-run +carries no options (4.2's note, measured), and no offline merger can prove +juju's merge behaviour. If either fails, the fix is a `juju config` correction +on a young model, not a diagnosis after hours of failed hook retries. + +**CHECK (read-only) -- from the DC's juju client** +```bash +juju config ovn-chassis -m "${DC_MODEL}" | grep -A2 -E \ + 'ovn-bridge-mappings|prefer-chassis-as-gw|bridge-interface-mappings' +for app in barbican cinder designate glance keystone magnum neutron-api \ + nova-cloud-controller nova-compute octavia openstack-dashboard placement; do + printf '%-24s %s\n' "$app" "$(juju config "$app" -m "${DC_MODEL}" --format=json \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["settings"]["openstack-origin"].get("value"))')" +done +for app in ceph-mon ceph-osd ceph-radosgw; do + printf '%-24s %s\n' "$app" "$(juju config "$app" -m "${DC_MODEL}" --format=json \ + | python3 -c 'import json,sys; print(json.load(sys.stdin)["settings"]["source"].get("value"))')" +done +``` + +**GATE:** +1. **`ovn-chassis` carries all THREE options.** `bridge-interface-mappings` + (from `overlays/${DC}-machines.yaml`) **alongside** `ovn-bridge-mappings: + physnet1:br-ex` and `prefer-chassis-as-gw: true` (B2, gateway-chassis + election). All three = juju merged the `options` map key-by-key, as the whole + `*-vips.yaml` architecture assumes. **Only `bridge-interface-mappings` = the + map was REPLACED**: the physnet mapping and the gateway election are gone + from the deployed config, and provider egress dies by a different route than + the missing-overlay defect DOCFIX-208 fixes. STOP and `juju config` them back + before anything relates. Exactly TWO MACs in the value; a third means a wrong + node. +2. **Every origin/source resolves to an address THIS DC's nodes can reach.** + `vr1-dc0`: all 15 read `deb http://10.12.8.4/cloud-archive + jammy-updates/caracal main`. Any app still reading `cloud:jammy-caracal` or a + bare `ubuntu-cloud.archive.canonical.com` URL is MEASURED UNREACHABLE from a + node (000) under the D-107 airgap and will fail at package install, not at + deploy. `vr1-dc1` correctly has NO such block -- its apt-cacher-ng forwards + whatever URL it is handed. That asymmetry is D-135's experiment RESULT, not a + defect to "fix" for symmetry. +3. **`ovn-central` is EXPECTED to read `source: zed`, not caracal** -- its charm + default, logged and deliberately left alone in + `overlays/vr1-dc0-machines.yaml`'s header because repointing it would change + its RELEASE, not just its URL. Do not "correct" it here. + +Capture this output; it is the first evidence in the chain that reads the +deployed configuration rather than an input to it. + --- ## Step 4b -- Mid-deploy watch + failure decision tree (read-only, runs alongside Step 4)