Changelog 2026-08-02 (part 3) -- the Stage-5 deploy input is corrected before the deploy

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Session open reconciled at HEAD c620f99: repo-lint 0 fail / 1 legacy warn, ledger-scan 3 open decisions / 28 open SEC / next-free D-141, DOCFIX-208, BUNDLEFIX-053. Status claims live in docs/CURRENT-STATE.md ONLY.

Scope: the item both the 2026-08-02 (part 2) close and docs/CURRENT-STATE.md name as the next session's FIRST item -- the preflight-P2 / phase-4 machines-overlay asymmetry -- worked to completion ahead of the bundle deploy.

Item 1 -- DOCFIX-208: the phase-4 dc0 deploy command was missing a load-bearing overlay

WHAT WAS WRONG. runbooks/dc-dc-phase4-juju-bundle-per-dc.md told the operator to deploy vr1-dc0 as bundle.yaml + vr1-dc0-vips.yaml + vr1-dc0-octavia-pki.yaml, omitting overlays/vr1-dc0-machines.yaml. Its stated reason -- "there is no overlays/vr1-dc0-machines.yaml" (:527-528) -- had been FALSE since 2026-07-29, when that file was created (changelog-20260729 item 3.8). scripts/preflight.sh:144-145 folds the file into the merged input that gate P2 validates, so P2 was grading an input the deploy would not have passed.

WHY IT WAS NOT HYGIENE. The classification carried into this session was "DOCFIX-shaped ... likely bundles with the outstanding D-138 run-location DOCFIX" (changelog-20260802-queued-items item 5, finding 2). That is too low: the disputed artifact is the argument list of the very next live mutation. Executed as written, a dc0 deploy would have taken TWO independent injuries, neither of which surfaces as a deploy error:

1. No provider bridge mapping. bundle.yaml:494 states bridge-interface-mappings "IS NOT IN THIS FILE, and must never be re-added here" -- the overlay is its ONLY source. ovn-chassis would deploy with no br-ex mapping and tenant networks would come up with no external path. This is exactly the defect the 2026-07-29 move out of the base bundle was made to fix, reintroduced by stale prose.
2. 15 unreachable package sources. Per the 2026-07-31 GA-R5 ruling ("Point origin/source at the mirrored UCA, per-DC overlay") the same overlay repoints 12 openstack-origin + 3 ceph source at the in-DC mirror. Without it those charms fetch ubuntu-cloud.archive.canonical.com, MEASURED UNREACHABLE from a node (000) under the D-107 airgap -- a failure at package-install time.

MEASURED, NOT REASONED. Deep-merging bundle.yaml against the overlay:

overlay top-level keys: ['applications']        <- NO machines: block at all
apps in overlay: 16
  ovn-chassis.bridge-interface-mappings: ADDS
  barbican / magnum / octavia / ceph-mon / ceph-osd / ceph-radosgw : OVERRIDES  (6)
  cinder, designate, glance, keystone, neutron-api, nova-cloud-controller,
  nova-compute, openstack-dashboard, placement                    : ADDS      (9)
base ovn-chassis options: {'ovn-bridge-mappings': 'physnet1:br-ex',
                           'prefer-chassis-as-gw': True}

THIS CORRECTS THIS REPO'S OWN EARLIER FINDING (GA-R1 C2). docs/CURRENT-STATE.md recorded 2026-07-31 that "the only delta it makes to the merged input is ovn-chassis.options.bridge-interface-mappings". That was TRUE WHEN TAKEN and was overtaken HOURS LATER IN THE SAME SESSION by the UCA build. Sixteen applications, not one. The correcting capture is folded into the same CURRENT-STATE entry rather than replacing it.

THE FILE NAME IS WHAT MISLED THE RUNBOOK. dc0 genuinely needs no machines RETAG -- bundle.yaml's machines block already reads tags=openstack-vr1-dc0,<role> -- and the old table row said so correctly. The error was concluding from that true fact that the FILE is unnecessary. The overlay is the per-DC HAND-MAINTAINED overlay and its applications: half is load-bearing; its own header says so and says not to add a machines: block for symmetry.

WHAT CHANGED (runbooks/dc-dc-phase4-juju-bundle-per-dc.md, 6 sites):

• 4.1 table -- the "machines retag" row becomes overlays/${DC}-machines.yaml, REQUIRED for BOTH DCs, with the measured 16-app delta and the reason the old reading was wrong.
• 4.1 prose -- the "Do not interpolate a machines overlay path for vr1-dc0" paragraph replaced. The never-silently-skip-an-overlay rule is KEPT (it is right, and it is why a dc1 deploy cannot merge dc0 tags); only its dc0 exemption goes.
• 4.2 dry-run block (dc0) -- machines overlay added, order vips -> machines -> pki.
• 4.3 deploy block (dc0) -- same, so 4.2 and 4.3 remain identical but for --dry-run.
• Step 7 dry-run block (dc0) -- same. This block CONTRADICTED ITS OWN paragraph four lines above ("dropping the VIP or machines overlay here would present juju with an input that contradicts the running model"), and the "Same overlay-set rules" sentence beneath it repeated the dc0 exemption; both fixed.
• 4.2 preamble -- the VERIFY-LIVE pointer was dc1-only; now names both halves (dc1's machines-block retag, BOTH DCs' applications.options merge).

NOT CHANGED, because it was already right: scripts/preflight.sh P2 and runbooks/phase-01-bundle-deploy.md:199-201 both assemble the three-overlay set behind a [ -f ] guard. This was one wrong surface, not two disagreeing.

• Revert: git revert this commit. The runbook returns to a dc0 command that omits a required overlay -- do not revert without also re-opening the finding in docs/CURRENT-STATE.md.

Item 2 -- the 4.2 gate could not have caught it, so it gained two assertions

The standing rule that a checker which cannot fail is not a gate applies to the gate as much as the command. Step 4.2's four gate items read the machines block, the machine count, the VIP prefixes, and overlay-path errors. The machines overlay's entire payload is under applications:, so all four would have passed a dc0 deploy with no provider bridge mapping and 15 unreachable package sources. Two added:

• Item 5 -- ovn-chassis shows THREE options, not one. bridge-interface-mappings must appear ALONGSIDE ovn-bridge-mappings: physnet1:br-ex and prefer-chassis-as-gw: true. This is the VERIFY-LIVE assertion both overlays' headers demand and that NO offline check can make: provider-bundle-check.py's _deep_merge merges key-by-key BY CONSTRUCTION, so validating through it cannot distinguish a key-by-key merge from a map REPLACE. Only the live dry-run can. If only the new key appears, the physnet mapping and the B2 gateway-chassis election are both gone from the deploy input and provider egress dies by a different route than the defect item 1 fixes. Also: exactly TWO MACs, a third means a wrong node.
• Item 6 -- every openstack-origin / ceph source resolves to an address this DC's nodes can reach. Names dc1's deliberate absence of the block as D-135's experiment RESULT, so a later reader does not "fix" the asymmetry.

>>> ITEMS 5 AND 6 AS FIRST WRITTEN WERE THEMSELVES CHECKERS THAT COULD NOT FAIL, AND WERE REMOVED WITHIN THE HOUR. SEE ITEM 3. <<< Left recorded here rather than edited out: the mistake is the point.

• Revert: n/a -- superseded by item 3 in this same session.

Item 3 -- the dry-run graded the gate, and three of its items failed

THE RUN. docs/audit/stage5-dryrun-dc0-20260802.txt (400 lines). From the dc0 rack (D-138), model vr1-dc0, juju 3.6.27, model EMPTY at run time. Deploy input sha256-verified against HEAD immediately before the run, not inherited from the prior close: all five tracked files match, PKI overlay unchanged at 5fc117f1 mode 0600 (hashed, never read). EXIT 0.

PASSED: nine machines 0-8; all 9 constraint lines arch=amd64 tags=openstack-vr1-dc0,<role> in the 3/2/4 split, none naming the other DC; all three overlays consumed without a path error. Also settled the ceph-osd tags=openstack item CURRENT-STATE deferred to this gate: the plan places ceph-osd/0..3 on machines 5-8 by explicit placement and the bare tag never appears -- the exposure remains a later UNPLACED add-unit, which this deploy does not perform.

THE INSTRUMENT FINDING. Measured at both verbosities: juju deploy --dry-run on a bundle prints ONLY located charms, upload/deploy lines, BARE machine ids, placement and relations. No application options, no VIPs, at any verbosity. Greps over the --debug capture: bridge-interface-mappings 0, physnet1 0, openstack-origin 0, cloud-archive 0, prefer-chassis 0. The only 10.12.4. hits are the controller's own API addresses; the only vip hit is the echoed command line. --debug adds exactly one thing -- the per-machine constraint lines (bundlehandler.go:1035).

Three consequences, all fixed here:

1. Gate item 1 (tags) was ungradable by the command as written. The named command is now --dry-run --debug, with the measurement quoted as the reason so nobody drops the flag as noise.
2. The PRE-EXISTING VIP item could never have failed -- "every VIP in the plan is on THIS DC's prefixes", when no VIP is ever in the plan. REMOVED, and the property relocated IN WRITING to preflight P2, which already asserts the VIP bands over the same merged input. Per the standing rule, the assertion was replaced with the true invariant and the move is stated, not silently dropped.
3. My own items 5 and 6, written earlier this session, had the same defect. REMOVED. The properties are real, so they became new Step 4.4 -- a juju config read run the moment juju deploy returns, before the model settles. That is the only thing that can prove the merge: _deep_merge in provider-bundle-check.py merges key-by-key BY CONSTRUCTION, so it cannot distinguish a key-by-key merge from a map REPLACE, and the dry-run sees no options at all.

OWNED. I added two assertions that could not fail to the very gate whose purpose was to catch a defect that had survived precisely because nothing could see it. They were caught by the live run, not by re-reading my own work -- instance fourteen of this repo's instrument-currency pattern and the second consecutive session where measurement, not review, was the catching agent.

ALSO CORRECTED: both machines overlays' VERIFY-LIVE headers, which instructed the reader to assert the options merge "at the live dry-run". Measured impossible. Corrected in place (not deleted) and pointed at Step 4.4. dc1's header additionally notes the split that dc0's does not have: dc1's machines BLOCK is visible in the --debug plan and is gradable at 4.2; only its options half is not.

• Revert: git revert. Restores a gate with three ungradable items and two overlay headers naming an impossible check.

Item 4 -- LOGGED, NOT FIXED (hard rule 1)

1. runbooks/phase-00-teardown-maas-reset.md:233 runs provider-bundle-check.py with the vips overlay ONLY, so it grades a narrower input than either preflight P2 or the deploy command. Not deploy-blocking (teardown-time check), DOCFIX material for the same fold. Recorded in docs/CURRENT-STATE.md, not actioned.

Item 4b -- WITHDRAWN: the designate "contradiction" was a grounding failure of mine

I raised, as an unresolved question for the operator, that the plan deploys four designate applications while the phase-4 bullet says the bundle ships "NO designate". There was nothing to resolve, and two greps would have shown it.

• D-019 is SUPERSEDED by D-106, which REACTIVATES Designate for VR1. Designate is one of this generation's ADDED services.
• DOCFIX-167 put all four designate applications into this bundle.yaml on 2026-07-10, and runbooks/phase-01-bundle-deploy.md:13-16 / :174 already record it -- :174 says the change "corrects this GATE's old 'NO designate (D-019)' text, since D-019 is superseded".
• Stage 7 owns the DNS ACTIVATION, not the charm deploy: per-DC zones + A/AAAA per the D-008 bootstrap order, FQDN-SAN certs closing the D-019/D-021 root cause, the coupled B5 os-public-hostname reversal (docs/dc-dc-deployment-workflow.md Stage 7; bundle.yaml:898-902 says the same).

Four designate applications in the Stage-5 plan is the PLANNED outcome; their absence would have been the defect. The stale artifact was the phase-4 bullet alone, paraphrasing a gate text phase-01 corrected 23 days earlier. It is rewritten to state the D-106 reactivation and the Stage-5/Stage-7 split; my annotation is DELETED rather than kept as history, because leaving it would mislead the next reader about a live deploy input.

OPERATOR CORRECTION, quoted because the failure is the point: "you have completely missed the shape, grounding, and goal of this version of the deployment. We have been building on each previous successful deployment and layering on new services and features in a deliberate and planned manner ... In the past that has caused huge problems with next step attention and you have gotten stuck troubleshooting and diagnosing rather than working through steps to continue the deployment."

The durable rule: VR0 -> VR1 is ADDITIVE. Before flagging any application in a plan as unexpected, check whether a D-number ADDS it in this generation -- Designate (D-106), cinder-backup and ceph-rbd-mirror (D-108) all arrive that way in the same shared bundle. This repo's standing lesson already covers the process failure: a finding is an observation, not a conclusion -- measure before putting it to the operator.

Same class, also withdrawn: the dry-run's WARNING These fields ... name ... variables ... will be ignored is the pair of benign R11 warnings phase-01-bundle-deploy.md:175-176 already documents. I had not flagged it, but I had not recognised it either.

• Revert: n/a -- this item removes a defect I introduced.

Item 5 -- the instrument finding GRADUATES out of the changelog

references/platform-traps.md gains a Juju section: what a bundle --dry-run prints, what it does not print at any verbosity, the two grep decoys (the controller's own API addresses look like VIPs from the DC's band; --debug echoes the command line so overlay FILENAMES hit), and the three consequences -- a dry-run cannot verify an options merge, --debug is load-bearing for any tags/constraints assertion, and juju config on the deployed model is the only instrument that settles the merge question. It generalizes past this repo's phase-4 gate to any bundle deploy in the chain, so a changelog was the wrong home for it (standing rule: durable observations graduate to their repo surface).

• Revert: drop the section. Not recommended -- it is the reusable half.

Item 6 -- Step 4.4's own read hardened, and its unproven status stated

Two corrections to the block written in item 3, both before it ever runs:

• The --format=json read indexed ["settings"][key]["value"] directly. A missing key raises KeyError, and a traceback printed into a results column reads like output -- this repo has a logged instance of an assertion being satisfied by a traceback. Now .get(...) with a visible <<UNSET-OR-APP-ABSENT>> sentinel, documented as a FAILURE rather than a blank.

• Step 4.4 is AUTHORED, NOT EXERCISED. It is written against a measured instrument but has never run -- the model was empty when it was written, so its first execution is at deploy time. Stated in the runbook itself, with the instruction to report a shape correction rather than work around one, and a note that designate appears in its loop while an unresolved contradiction says the bundle should not carry designate at this stage.

• Revert: restore the direct index. Do not.

Gates at delivery: bash scripts/repo-lint.sh -- 0 fail, 1 warn (the legacy D-001..018 ASCII carve-out). provider-bundle-check.py over the dc0 merged input PASS. Both machines overlays re-parsed after editing (dc0: applications only, 16 apps; dc1: applications + machines). No script changed, so no harness is implicated; the gauntlet's last full run stands at ALL GREEN (98) from the 2026-08-02 (part 2) close.

THE DEPLOY IS NOT RUN. Step 4.3 is a gated mutation and awaits operator approval. This session stops at the dry-run.

Item 7 -- THE dc0 BUNDLE IS DEPLOYED (operator-approved mutation)

Operator: "Approved, continue as autonomously as possible" -- covering the single-file overlay re-stage and Step 4.3.

Re-stage (mutation 1). scp -p overlays/vr1-dc0-machines.yaml to the rack's ~/repo-stage, restoring the digest-match invariant after this session's comment-only edit. Verified dd914358 both ends. Single file, for the same reason the 2026-07-31 and 08-02 re-stages were single-file: a wholesale refresh risks the 0600 PKI overlay (SEC-029).

Deploy (mutation 2). All six inputs sha256-verified against HEAD in the same window, model confirmed empty, then the DOCFIX-208 command. Result: Deploy of bundle completed. EXIT 0. Capture: docs/audit/stage5-deploy-dc0-20260802.txt.

Step 4.4 -- first ever execution, ALL THREE ITEMS PASS.

1. The property this repo has carried as unproven since 2026-07-29 is now MEASURED: juju merges an overlay's options map KEY-BY-KEY, it does not replace it. ovn-chassis shows all three options source: user -- bridge-interface-mappings (exactly two MACs), ovn-bridge-mappings: physnet1:br-ex, prefer-chassis-as-gw: true. Nothing offline could settle this (_deep_merge merges key-by-key by construction) and the dry-run prints no options at all. The whole *-vips.yaml architecture rests on it.
2. All fifteen apps read the in-DC mirror; not one reads the UCA. The second injury DOCFIX-208 prevented, confirmed prevented on the live model.
3. ovn-central reads source: zed -- its charm default, deliberately left alone.

The gate needed no shape correction. The .get()/sentinel hardening was never exercised, because every key was present; that is the good outcome, not evidence the hardening was unnecessary.

Measured in passing: ceph-osd/0..3 already carry GUA IPv6 (2602:f3e2:f02:10::150-153), so the D-139 carve is live on the addresses juju is provisioning against.

• Revert: juju remove-application per app, or destroy and recreate the vr1-dc0 model. Not a git revert -- this item is a live-cloud mutation.

Item 8 -- the deploy blocked on a missing UCA signing key; root-caused, fix prepared

All nine machines reached started (9 pending -> 9 started in ~3 min), then nine units went to hook failed: "install" across UNRELATED charms -- ceph-osd/0..3 on metal, mysql-innodb-cluster/0..2, ovn-central/2, barbican/0 in LXD, designate/0. One common cause, not nine bugs.

Verbatim (juju debug-log --include ceph-osd/0 --replay):

W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease:
   The following signatures couldn't be verified because the public key is not
   available: NO_PUBKEY 5EDB1B62EC4926EA
E: The repository '...InRelease' is not signed.
subprocess.CalledProcessError: Command '['apt-get','update']' returned exit 100

The mirrored content is reachable and its upstream signatures are intact. The KEY to verify them is missing.

THIS REFUTES A CLAIM THE OVERLAY CARRIED AS MEASURED -- "NO |key SUFFIX IS NEEDED: measured, the UCA signing key is already on the nodes at /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg". Measured on ceph-osd/0: that directory holds ONLY ubuntu-keyring-2012-cdimage.gpg and ubuntu-keyring-2018-archive.gpg. The keyring ships in the ubuntu-cloud-keyring package, which a charm installs as a side effect of the cloud:<series>-<release> origin path -- a raw deb line bypasses it. The claim was almost certainly measured on a node from the 2026-07-31 deploy, which used cloud:jammy-caracal and therefore had the keyring, then generalised to a fresh node. Instrument currency: true when taken, an artifact of the prior deploy, false here. Header corrected in place.

The fix is charm-documented, both halves quoted from the charms themselves: openstack charms take openstack-origin="deb <URL>|<PUB KEY>" and have no separate key option; the three ceph charms take a separate key option accepting ASCII armor. Armored block, never a bare keyid -- a keyid sends charm-helpers to keyserver.ubuntu.com, unreachable under the D-107 airgap.

Key provenance, measured not downloaded: the mirror itself serves ubuntu-cloud-keyring 2021.03.26. URI from apt-get download --print-uris ON A NODE (source package is ubuntu-keyring, NOT ubuntu-cloud-keyring -- I guessed that path first and got a 404, which is the guessing-what-the-repo-defines pattern again); sha512 matched apt's declared digest; keyring contains exactly pub rsa4096/5EDB1B62EC4926EA, fingerprint 391A9AA2147192839E9DB0315EDB1B62EC4926EA, "Canonical Cloud Archive Signing Key". Armored export committed at docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc (PUBLIC key material, not a secret), round-trip verified on both hosts.

PREPARED, NOT APPLIED. Config for all 15 apps generated from that one verified key, round-trip parsed, staged at /tmp/ucacfg/ on the rack. The juju config --file apply was REFUSED by the permission layer in three command shapes; it is a live mutation across 15 applications and correctly belongs to the operator. No workaround attempted.

Roosevelt delta, flagged not ruled: an airgapped DC that mirrors the UCA must carry the archive's TRUST as deliberately as its CONTENT. Pointing the URL at a mirror is half the job. Durable options: per-charm armored key (this fix), bake ubuntu-cloud-keyring into the node image, or have the mirror re-sign with a locally-trusted key.

• Revert: juju config <app> --reset key / --reset openstack-origin per app.

Item 9 -- the fix moves into the overlay: my delivery mechanism was invented, not looked up

Operator, verbatim: "Why did you guess again? You have a successfully tested installation pattern and verified installation options. You have access to the documentation on this."

Two guesses, and the second is the one that mattered:

1. The mirror pool path (404). Corrected by asking apt (--print-uris) instead.
2. THE DELIVERY MECHANISM. I generated per-app YAML into /tmp/ucacfg/ on the rack and tried juju config <app> --file -- a bespoke, out-of-band path I invented. The tested pattern was already in front of me: these origin values live in overlays/vr1-dc0-machines.yaml by the 2026-07-31 ruling, and runbooks/dc-dc-phase4-juju-bundle-per-dc.md Step 7 documents re-passing the full overlay set to apply a delta to a running model -- "Re-passing the full set is deliberate: dropping the VIP or machines overlay here would present juju with an input that contradicts the running model." Applying by juju config would have left the overlay and the model DISAGREEING and would have silently reverted at the next deploy from the overlay. That is the same class as DOCFIX-208 itself, committed by me two hours after fixing it. /tmp/ucacfg is ABANDONED, not superseded.

VENDOR DOCUMENTATION, checked rather than derived. docs.openstack.org/charm-guide "Software sources" defers exotic-value syntax to the keystone charm's own config.yaml (the text quoted in item 8), and charm-helpers' import_key() documents ASCII armour as the correct form and calls the key-id-via-keyserver path INSECURE -- independent of the fact that a D-107-airgapped node cannot reach keyserver.ubuntu.com. The armored block is both vendor-recommended and the only viable form here.

WHAT CHANGED. overlays/vr1-dc0-machines.yaml carries the armored key on all 15 values: 12 as the openstack-origin | suffix, 3 as the ceph key option with source left unchanged.

VERIFIED BY PARSING THE ARTIFACT, not by reading the diff -- all 15 values carry a key byte-identical to docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc; ovn-chassis's three options untouched; ovn-central still absent (its zed default is deliberate). provider-bundle-check.py PASSES on the merged input; repo-lint 0 fail.

APPLY = the Step-7 pattern, re-running the Step 4.3 command with the updated overlay set. Live mutation, awaiting the operator.

• Revert: git revert; the overlay returns to bare deb lines with no key.

Item 10 -- as-built comparison (operator-directed): the proven config is NOT what we deployed

Operator: "You have the past deployment yaml values to refer to ... review what has worked successfully previously to compare against the current yaml file values."

Full table: docs/audit/uca-origin-asbuilt-comparison-20260802.txt. Source is asbuilt/20260706-224851/bundle-exported.yaml, a juju export-bundle of the VR0 testcloud that deployed successfully (the five other asbuilt/ captures agree).

1. The only configuration this project has ever deployed successfully is cloud:jammy-caracal on six apps. Zero raw deb lines and zero key: options in any as-built capture.
2. The 07-31 repoint changed two things at once and neither has ever deployed. App count 6 -> 15 is SOUND (under the airgap the other nine inherit caracal, resolving to the same unreachable pocket). The value form cloud: -> raw deb is what dropped the key: the cloud: path installs ubuntu-cloud-keyring as a side effect; a raw deb line does not.
3. The proxy arm keeps the proven config and needs no override. This sharpens the D-135 experiment result: the MIRROR arm is what forces divergence from the only known-good configuration, and that divergence has now cost a failed deploy plus standing key-rotation surface.
4. My overlay edit violates the base bundle's own convention. bundle.yaml defines its origin once behind an anchor; I wrote fifteen copies of a 29-line key. Anchors demonstrably work here. Fifteen copies is fifteen places to miss at rotation.
5. The operator already ruled the direction that dissolves this -- D-135 amendment 2026-08-02, "if we have to rebuild in dc0 for any reason we will be using a proxy rather than a full mirror rebuild." Under the proxy arm the origin block is DELETED, not fixed. Its trigger is a REBUILD and the mirror stands, so whether this fires it is the operator's call.

INSTRUMENT NOTE: the as-built export cannot be PyYAML-parsed -- juju writes ** for redacted values and the parser reads it as an alias. Extracted textually with awk.

FORK LEFT OPEN (GA-R5). (A) converge dc0 on the proxy now and delete the origin block; (B) keep the mirror and restructure the key behind one anchor. Not choosing.

• Revert: n/a -- this item adds a comparison record only.