diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index f268a14..2d50653 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1685,6 +1685,43 @@ br-ex:52:54:00:50:48:88'`, the two dc0 compute provider MACs. Deploying dc0 exactly as the runbook reads would leave ovn-chassis with no provider bridge mapping, surfacing later as tenant networks with no external path rather than as a deploy error. + **>>> FIXED 2026-08-02 AS DOCFIX-208, AND THE BLAST RADIUS WAS SIXTEEN TIMES WHAT THIS + ENTRY RECORDED. <<<** The "only delta is `bridge-interface-mappings`" measurement above + was TRUE WHEN TAKEN and was overtaken HOURS LATER IN THE SAME SESSION by the 2026-07-31 + GA-R5 UCA ruling ("Point origin/source at the mirrored UCA, per-DC overlay"), which added + 15 more applications to the same file. **RE-MEASURED 2026-08-02 by deep-merging + `bundle.yaml` against the overlay: SIXTEEN applications carry a delta** -- `ovn-chassis` + (`bridge-interface-mappings`, ADDS) plus 15 mirror repoints (12 `openstack-origin`, + 3 ceph `source`; 6 OVERRIDE a base value, 9 ADD one). So the second failure mode is + package-install failure across the deploy -- those charms would fetch from + `ubuntu-cloud.archive.canonical.com`, measured UNREACHABLE from a node (000) under the + D-107 airgap -- not just dead provider egress. GA-R1 C2: this capture corrects the entry + above rather than replacing it. **The overlay has NO `machines:` block** (measured: its + only top-level key is `applications:`), which is why the file NAME misled the runbook and + why passing it cannot move a tag or a machine id. **THREE dc0 command blocks were wrong, + not one** -- 4.2 dry-run, 4.3 deploy, and Step 7's ipv6-family-matrix dry-run, the last + contradicting its own paragraph four lines above ("dropping the VIP or machines overlay + here would present juju with an input that contradicts the running model"). All corrected, + plus the 4.1 table row and the "Do not interpolate" paragraph whose stated reason -- "there + is no `overlays/vr1-dc0-machines.yaml`" -- had been false since 2026-07-29. + **THE GATE COULD NOT HAVE CAUGHT IT EITHER, so two assertions were added.** Step 4.2's + gate read the machines block, the machine count and the VIP prefixes; the overlay's entire + payload is under `applications:`, so all four items would have passed a dc0 deploy with no + provider bridge mapping and 15 unreachable package sources. New item 5 asserts `ovn-chassis` + shows THREE options -- `bridge-interface-mappings` ALONGSIDE `ovn-bridge-mappings` and + `prefer-chassis-as-gw` -- which is the VERIFY-LIVE proof both overlays' headers demand and + that no offline check can make (`provider-bundle-check.py`'s `_deep_merge` merges + key-by-key BY CONSTRUCTION, so it cannot distinguish a key-by-key merge from a map + REPLACE). New item 6 asserts every origin/source resolves to an address this DC's nodes can + reach. **`scripts/preflight.sh` P2 needed NO change: it was already right** -- it assembles + the same three-overlay set with a `[ -f ]` guard, as does + `runbooks/phase-01-bundle-deploy.md:199-201`. The asymmetry was one wrong surface, not a + disagreement between two. + **LOGGED, NOT FIXED (hard rule 1, different runbook and a different step):** + `runbooks/phase-00-teardown-maas-reset.md:233` validates `bundle.yaml` with the vips + overlay ONLY, so it now grades a narrower input than either P2 or the deploy. Not + deploy-blocking -- it is a teardown-time check -- and it is DOCFIX material for the same + fold. **>>> THE dc0 SNAP FORWARD PROXY IS INSTALLED, RUNNING AND GATE-VERIFIED 2026-07-31. THE LAST BUILD BETWEEN HERE AND `add-model` IS DONE. <<<** Capture diff --git a/docs/changelog-20260802-deploy-input.md b/docs/changelog-20260802-deploy-input.md new file mode 100644 index 0000000..8a68f58 --- /dev/null +++ b/docs/changelog-20260802-deploy-input.md @@ -0,0 +1,132 @@ +# Changelog 2026-08-02 (part 3) -- the Stage-5 deploy input is corrected before the deploy + +Session changelog (GA-R2/D1: ONE per session). Branch `dc-dc-stage5-preconditions`. +Session open reconciled at HEAD `c620f99`: repo-lint 0 fail / 1 legacy warn, +`ledger-scan` 3 open decisions / 28 open SEC / next-free D-141, DOCFIX-208, +BUNDLEFIX-053. Status claims live in `docs/CURRENT-STATE.md` ONLY. + +Scope: the item both the 2026-08-02 (part 2) close and `docs/CURRENT-STATE.md` +name as the next session's FIRST item -- the preflight-P2 / phase-4 +machines-overlay asymmetry -- worked to completion ahead of the bundle deploy. + +--- + +## Item 1 -- DOCFIX-208: the phase-4 dc0 deploy command was missing a load-bearing overlay + +**WHAT WAS WRONG.** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md` told the operator +to deploy `vr1-dc0` as `bundle.yaml` + `vr1-dc0-vips.yaml` + +`vr1-dc0-octavia-pki.yaml`, omitting `overlays/vr1-dc0-machines.yaml`. Its stated +reason -- "there is no `overlays/vr1-dc0-machines.yaml`" (`:527-528`) -- had been +FALSE since 2026-07-29, when that file was created (changelog-20260729 item 3.8). +`scripts/preflight.sh:144-145` folds the file into the merged input that gate P2 +validates, so **P2 was grading an input the deploy would not have passed.** + +**WHY IT WAS NOT HYGIENE.** The classification carried into this session was +"DOCFIX-shaped ... likely bundles with the outstanding D-138 run-location DOCFIX" +(changelog-20260802-queued-items item 5, finding 2). That is too low: the disputed +artifact is the argument list of the very next live mutation. Executed as written, +a dc0 deploy would have taken TWO independent injuries, neither of which surfaces +as a deploy error: + +1. **No provider bridge mapping.** `bundle.yaml:494` states + `bridge-interface-mappings` "IS NOT IN THIS FILE, and must never be re-added + here" -- the overlay is its ONLY source. `ovn-chassis` would deploy with no + `br-ex` mapping and tenant networks would come up with no external path. This + is exactly the defect the 2026-07-29 move out of the base bundle was made to + fix, reintroduced by stale prose. +2. **15 unreachable package sources.** Per the 2026-07-31 GA-R5 ruling ("Point + origin/source at the mirrored UCA, per-DC overlay") the same overlay repoints + 12 `openstack-origin` + 3 ceph `source` at the in-DC mirror. Without it those + charms fetch `ubuntu-cloud.archive.canonical.com`, MEASURED UNREACHABLE from a + node (000) under the D-107 airgap -- a failure at package-install time. + +**MEASURED, NOT REASONED.** Deep-merging `bundle.yaml` against the overlay: + +``` +overlay top-level keys: ['applications'] <- NO machines: block at all +apps in overlay: 16 + ovn-chassis.bridge-interface-mappings: ADDS + barbican / magnum / octavia / ceph-mon / ceph-osd / ceph-radosgw : OVERRIDES (6) + cinder, designate, glance, keystone, neutron-api, nova-cloud-controller, + nova-compute, openstack-dashboard, placement : ADDS (9) +base ovn-chassis options: {'ovn-bridge-mappings': 'physnet1:br-ex', + 'prefer-chassis-as-gw': True} +``` + +**THIS CORRECTS THIS REPO'S OWN EARLIER FINDING (GA-R1 C2).** +`docs/CURRENT-STATE.md` recorded 2026-07-31 that "the only delta it makes to the +merged input is `ovn-chassis.options.bridge-interface-mappings`". That was TRUE +WHEN TAKEN and was overtaken HOURS LATER IN THE SAME SESSION by the UCA build. +Sixteen applications, not one. The correcting capture is folded into the same +CURRENT-STATE entry rather than replacing it. + +**THE FILE NAME IS WHAT MISLED THE RUNBOOK.** dc0 genuinely needs no machines +RETAG -- `bundle.yaml`'s machines block already reads +`tags=openstack-vr1-dc0,` -- and the old table row said so correctly. The +error was concluding from that true fact that the FILE is unnecessary. The overlay +is the per-DC HAND-MAINTAINED overlay and its `applications:` half is load-bearing; +its own header says so and says not to add a `machines:` block for symmetry. + +**WHAT CHANGED** (`runbooks/dc-dc-phase4-juju-bundle-per-dc.md`, 6 sites): + +- **4.1 table** -- the "machines retag" row becomes `overlays/${DC}-machines.yaml`, + REQUIRED for BOTH DCs, with the measured 16-app delta and the reason the old + reading was wrong. +- **4.1 prose** -- the "Do not interpolate a machines overlay path for `vr1-dc0`" + paragraph replaced. The never-silently-skip-an-overlay rule is KEPT (it is right, + and it is why a dc1 deploy cannot merge dc0 tags); only its dc0 exemption goes. +- **4.2 dry-run block (dc0)** -- machines overlay added, order vips -> machines -> pki. +- **4.3 deploy block (dc0)** -- same, so 4.2 and 4.3 remain identical but for `--dry-run`. +- **Step 7 dry-run block (dc0)** -- same. This block CONTRADICTED ITS OWN paragraph + four lines above ("dropping the VIP or machines overlay here would present juju with + an input that contradicts the running model"), and the "Same overlay-set rules" + sentence beneath it repeated the dc0 exemption; both fixed. +- **4.2 preamble** -- the VERIFY-LIVE pointer was dc1-only; now names both halves + (dc1's machines-block retag, BOTH DCs' `applications.options` merge). + +**NOT CHANGED, because it was already right:** `scripts/preflight.sh` P2 and +`runbooks/phase-01-bundle-deploy.md:199-201` both assemble the three-overlay set +behind a `[ -f ]` guard. This was one wrong surface, not two disagreeing. + +- **Revert:** `git revert` this commit. The runbook returns to a dc0 command that + omits a required overlay -- do not revert without also re-opening the finding in + `docs/CURRENT-STATE.md`. + +## Item 2 -- the 4.2 gate could not have caught it, so it gained two assertions + +The standing rule that a checker which cannot fail is not a gate applies to the +gate as much as the command. Step 4.2's four gate items read the machines block, +the machine count, the VIP prefixes, and overlay-path errors. **The machines +overlay's entire payload is under `applications:`, so all four would have passed a +dc0 deploy with no provider bridge mapping and 15 unreachable package sources.** +Two added: + +- **Item 5 -- `ovn-chassis` shows THREE options, not one.** `bridge-interface-mappings` + must appear ALONGSIDE `ovn-bridge-mappings: physnet1:br-ex` and + `prefer-chassis-as-gw: true`. This is the VERIFY-LIVE assertion both overlays' + headers demand and that NO offline check can make: `provider-bundle-check.py`'s + `_deep_merge` merges key-by-key BY CONSTRUCTION, so validating through it cannot + distinguish a key-by-key merge from a map REPLACE. Only the live dry-run can. If + only the new key appears, the physnet mapping and the B2 gateway-chassis election + are both gone from the deploy input and provider egress dies by a different route + than the defect item 1 fixes. Also: exactly TWO MACs, a third means a wrong node. +- **Item 6 -- every `openstack-origin` / ceph `source` resolves to an address this + DC's nodes can reach.** Names dc1's deliberate absence of the block as D-135's + experiment RESULT, so a later reader does not "fix" the asymmetry. + +- **Revert:** delete gate items 5 and 6. Not recommended -- they are the only + assertions in the chain that can see this overlay's payload. + +## Item 3 -- LOGGED, NOT FIXED (hard rule 1) + +`runbooks/phase-00-teardown-maas-reset.md:233` runs `provider-bundle-check.py` with +the vips overlay ONLY, so it grades a narrower input than either preflight P2 or the +deploy command. Not deploy-blocking (teardown-time check), DOCFIX material for the +same fold. Recorded in `docs/CURRENT-STATE.md`, not actioned here. + +--- + +**Gates at delivery:** `bash scripts/repo-lint.sh` -- 0 fail, 1 warn (the legacy +D-001..018 ASCII carve-out), 658 files. No script changed, so no harness is +implicated; the gauntlet's last full run stands at ALL GREEN (98) from the +2026-08-02 (part 2) close. diff --git a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md index 948763c..3b9c440 100644 --- a/runbooks/dc-dc-phase4-juju-bundle-per-dc.md +++ b/runbooks/dc-dc-phase4-juju-bundle-per-dc.md @@ -521,15 +521,47 @@ | Overlay | `vr1-dc0` | `vr1-dc1` | |---|---|---| | `overlays/${DC}-vips.yaml` | REQUIRED (`vr1-dc0-vips.yaml`) | REQUIRED (`vr1-dc1-vips.yaml`) | -| machines retag | **none -- and that is correct**: the base bundle's machines block already carries `tags=openstack-vr1-dc0,` | REQUIRED: `overlays/vr1-dc1-machines.yaml` | +| `overlays/${DC}-machines.yaml` | **REQUIRED** (`vr1-dc0-machines.yaml`) | **REQUIRED** (`vr1-dc1-machines.yaml`) | | `overlays/${DC}-octavia-pki.yaml` | REQUIRED (gitignored secret) | REQUIRED (gitignored secret) | -**Do not interpolate a machines overlay path for `vr1-dc0`** -- there is no -`overlays/vr1-dc0-machines.yaml`, and juju errors out on a missing `--overlay` -path. The asymmetry is deliberate; the gate that catches BOTH arms is the merged -tag assertion in 4.2, not the presence of a file. Never "guard" the argument -with a silent skip: an overlay that quietly disappears is exactly how a -`vr1-dc1` deploy would merge `vr1-dc0`'s tags. +**BOTH DCs take a machines overlay. DOCFIX-208, 2026-08-02.** This row and the +`vr1-dc0` command blocks below previously read "none -- and that is correct" and +"there is no `overlays/vr1-dc0-machines.yaml`". That file HAS EXISTED since +2026-07-29 and the dc0 deploy REQUIRES it. What made the old text wrong is that +it reasoned from the FILE NAME: the overlay's job is no longer a machines retag, +and dc0 still needs none -- `bundle.yaml`'s machines block already carries +`tags=openstack-vr1-dc0,`, and `overlays/vr1-dc0-machines.yaml` has NO +`machines:` block at all (measured: its only top-level key is `applications:`, +so passing it cannot move a tag or a machine id). It is the per-DC +HAND-MAINTAINED overlay, and its `applications:` half is load-bearing. + +**MEASURED 2026-08-02 -- the dc0 overlay's delta to the merged input is SIXTEEN +applications, not one:** + +- `ovn-chassis.bridge-interface-mappings` (ADDS) -- the two dc0 compute + provider MACs. `bundle.yaml:494` states this key "IS NOT IN THIS FILE, and + must never be re-added here", so the overlay is its ONLY source. Deploy dc0 + without it and ovn-chassis builds no `br-ex` mapping: tenant networks come up + with no external path, surfacing later as broken egress rather than as a + deploy error. This is the exact defect the 2026-07-29 move out of the base + bundle was made to fix. +- **15 apps repointed at the in-DC mirror** (12 `openstack-origin`, 3 ceph + `source`; 6 OVERRIDE a base value, 9 ADD one) per the 2026-07-31 GA-R5 ruling + "Point origin/source at the mirrored UCA, per-DC overlay". Without it those + charms fetch from `ubuntu-cloud.archive.canonical.com`, MEASURED UNREACHABLE + from a node (000) under the D-107 airgap. + +Supersedes the narrower 2026-07-31 finding in `docs/CURRENT-STATE.md` that the +overlay's "only delta" is `bridge-interface-mappings` -- true when measured, and +overtaken hours later by the UCA build in the same session (GA-R1 C2: +measurement wins, and this is the correcting capture). + +The `[ -f ]` guard in `scripts/preflight.sh:144-145` and +`runbooks/phase-01-bundle-deploy.md:199-201` assembles this same three-overlay +set, so P2 was validating the right input and this runbook was the wrong +surface. Never "guard" a `--overlay` argument with a silent skip in a DEPLOY +command: an overlay that quietly disappears is exactly how a `vr1-dc1` deploy +would merge `vr1-dc0`'s tags, and how a dc0 deploy lost its provider mapping. Make NO hand edit to `bundle.yaml` or to either generated VIP overlay (ruling 3, 2026-07-25; both `*-vips.yaml` are RENDERED -- edit `render/values/${DC}-vips.yaml` @@ -555,18 +587,20 @@ > **Run this BEFORE 4.3. It is the only thing standing between a typo in the > overlay set and nine wrongly-allocated MAAS machines.** There is no offline -> juju merger that can prove the machines-block override merges (see -> `overlays/vr1-dc1-machines.yaml`'s own VERIFY-LIVE note), so the dry-run IS -> the proof. +> juju merger that can prove either half of `overlays/${DC}-machines.yaml` +> merges -- dc1's machines-block retag, or BOTH DCs' `applications.options` +> key-by-key merge (see each overlay's own VERIFY-LIVE note) -- so the dry-run +> IS the proof. Gate items 5 and 6 below are that proof for the options half. **CHECK (read-only) -- voffice1 -- run the block for YOUR DC.** These are EXACTLY the 4.3 commands plus `--dry-run`; if the two ever differ, this gate is validating something you are not deploying. -`vr1-dc0` (no machines overlay -- the base bundle already carries its tags): +`vr1-dc0` (order: vips -> machines -> pki): ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-machines.yaml \ --overlay overlays/vr1-dc0-octavia-pki.yaml \ -m "${DC_MODEL}" --dry-run ``` @@ -592,6 +626,30 @@ `:146-154`). A VIP from the other DC means the wrong overlay merged. 4. If the plan reports an ERROR on any `--overlay` path, do NOT drop the overlay to make it run. A missing overlay is a missing input, not an optional one. +5. **`ovn-chassis` shows THREE options, not one.** Added DOCFIX-208, because + items 1-3 could not have caught the defect that fix corrects -- they read the + machines block and the VIPs, and the machines overlay's whole payload is + under `applications:`. The plan must show + `bridge-interface-mappings` (from `overlays/${DC}-machines.yaml`) **alongside + both base options that survive only if juju merges the `options` MAP + key-by-key rather than replacing it**: `ovn-bridge-mappings: physnet1:br-ex` + and `prefer-chassis-as-gw: true` (B2, gateway-chassis election). This is the + VERIFY-LIVE assertion both machines overlays' own headers demand and that no + offline check can make: `provider-bundle-check.py`'s `_deep_merge` merges + key-by-key BY CONSTRUCTION, so validating through it cannot tell the two + behaviours apart. If only `bridge-interface-mappings` appears, the map was + REPLACED -- physnet mapping and gateway election are both gone from the + deploy input and provider egress dies. STOP. + Exactly TWO MACs in the value; a third means a wrong node. +6. **Every `openstack-origin` / ceph `source` in the plan resolves to an address + this DC's nodes can actually reach.** dc0 = 16 apps carry the overlay's + delta, 15 of them pointing at the in-DC mirror + (`deb http://10.12.8.4/cloud-archive jammy-updates/caracal main`). Any app + still reading `cloud:jammy-caracal` or a bare `ubuntu-cloud.archive.canonical.com` + URL is MEASURED UNREACHABLE from a node under the D-107 airgap and will fail + at package install, not at deploy. dc1 correctly has NO such block -- its + apt-cacher-ng forwards whatever URL it is handed, which is D-135's experiment + RESULT and not an asymmetry to "fix". Capture this output. It is the plan-count evidence the exit gate cites -- plan counts are quoted only from captured command output, never from this file's @@ -605,6 +663,7 @@ ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-machines.yaml \ --overlay overlays/vr1-dc0-octavia-pki.yaml \ -m "${DC_MODEL}" ``` @@ -775,6 +834,7 @@ ```bash juju deploy ./bundle.yaml \ --overlay overlays/vr1-dc0-vips.yaml \ + --overlay overlays/vr1-dc0-machines.yaml \ --overlay overlays/vr1-dc0-octavia-pki.yaml \ --overlay overlays/dc-dc-ipv6-family-matrix.yaml \ -m "${DC_MODEL}" --dry-run @@ -788,9 +848,11 @@ --overlay overlays/dc-dc-ipv6-family-matrix.yaml \ -m "${DC_MODEL}" --dry-run ``` -Same overlay-set rules as Step 4.1: `vr1-dc0` takes no machines overlay because -the base bundle already carries its tags, and no overlay is ever dropped to make -the command run. +Same overlay-set rules as Step 4.1: BOTH DCs pass their `${DC}-machines.yaml`, +and no overlay is ever dropped to make the command run. (DOCFIX-208: this +sentence previously exempted `vr1-dc0`, contradicting its own paragraph four +lines above -- "dropping the VIP or machines overlay here would present juju +with an input that contradicts the running model".) Review the diff before applying for real, same discipline as every other overlay in this repo. **GATE:** the diff changes ONLY the address-family options