Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260802-deploy-input.md
@JANeumatrix JANeumatrix 1 day ago 12 KB DOCFIX-208 part 2: the dry-run graded the gate, and three items could not fail
Raw Blame History

Changelog 2026-08-02 (part 3) -- the Stage-5 deploy input is corrected before the deploy

Session changelog (GA-R2/D1: ONE per session). Branch dc-dc-stage5-preconditions. Session open reconciled at HEAD c620f99: repo-lint 0 fail / 1 legacy warn, ledger-scan 3 open decisions / 28 open SEC / next-free D-141, DOCFIX-208, BUNDLEFIX-053. Status claims live in docs/CURRENT-STATE.md ONLY.

Scope: the item both the 2026-08-02 (part 2) close and docs/CURRENT-STATE.md name as the next session's FIRST item -- the preflight-P2 / phase-4 machines-overlay asymmetry -- worked to completion ahead of the bundle deploy.

Item 1 -- DOCFIX-208: the phase-4 dc0 deploy command was missing a load-bearing overlay

WHAT WAS WRONG. runbooks/dc-dc-phase4-juju-bundle-per-dc.md told the operator to deploy vr1-dc0 as bundle.yaml + vr1-dc0-vips.yaml + vr1-dc0-octavia-pki.yaml, omitting overlays/vr1-dc0-machines.yaml. Its stated reason -- "there is no overlays/vr1-dc0-machines.yaml" (:527-528) -- had been FALSE since 2026-07-29, when that file was created (changelog-20260729 item 3.8). scripts/preflight.sh:144-145 folds the file into the merged input that gate P2 validates, so P2 was grading an input the deploy would not have passed.

WHY IT WAS NOT HYGIENE. The classification carried into this session was "DOCFIX-shaped ... likely bundles with the outstanding D-138 run-location DOCFIX" (changelog-20260802-queued-items item 5, finding 2). That is too low: the disputed artifact is the argument list of the very next live mutation. Executed as written, a dc0 deploy would have taken TWO independent injuries, neither of which surfaces as a deploy error:

  1. No provider bridge mapping. bundle.yaml:494 states bridge-interface-mappings "IS NOT IN THIS FILE, and must never be re-added here" -- the overlay is its ONLY source. ovn-chassis would deploy with no br-ex mapping and tenant networks would come up with no external path. This is exactly the defect the 2026-07-29 move out of the base bundle was made to fix, reintroduced by stale prose.
  2. 15 unreachable package sources. Per the 2026-07-31 GA-R5 ruling ("Point origin/source at the mirrored UCA, per-DC overlay") the same overlay repoints 12 openstack-origin + 3 ceph source at the in-DC mirror. Without it those charms fetch ubuntu-cloud.archive.canonical.com, MEASURED UNREACHABLE from a node (000) under the D-107 airgap -- a failure at package-install time.

MEASURED, NOT REASONED. Deep-merging bundle.yaml against the overlay:

overlay top-level keys: ['applications']        <- NO machines: block at all
apps in overlay: 16
  ovn-chassis.bridge-interface-mappings: ADDS
  barbican / magnum / octavia / ceph-mon / ceph-osd / ceph-radosgw : OVERRIDES  (6)
  cinder, designate, glance, keystone, neutron-api, nova-cloud-controller,
  nova-compute, openstack-dashboard, placement                    : ADDS      (9)
base ovn-chassis options: {'ovn-bridge-mappings': 'physnet1:br-ex',
                           'prefer-chassis-as-gw': True}

THIS CORRECTS THIS REPO'S OWN EARLIER FINDING (GA-R1 C2). docs/CURRENT-STATE.md recorded 2026-07-31 that "the only delta it makes to the merged input is ovn-chassis.options.bridge-interface-mappings". That was TRUE WHEN TAKEN and was overtaken HOURS LATER IN THE SAME SESSION by the UCA build. Sixteen applications, not one. The correcting capture is folded into the same CURRENT-STATE entry rather than replacing it.

THE FILE NAME IS WHAT MISLED THE RUNBOOK. dc0 genuinely needs no machines RETAG -- bundle.yaml's machines block already reads tags=openstack-vr1-dc0,<role> -- and the old table row said so correctly. The error was concluding from that true fact that the FILE is unnecessary. The overlay is the per-DC HAND-MAINTAINED overlay and its applications: half is load-bearing; its own header says so and says not to add a machines: block for symmetry.

WHAT CHANGED (runbooks/dc-dc-phase4-juju-bundle-per-dc.md, 6 sites):

  • 4.1 table -- the "machines retag" row becomes overlays/${DC}-machines.yaml, REQUIRED for BOTH DCs, with the measured 16-app delta and the reason the old reading was wrong.
  • 4.1 prose -- the "Do not interpolate a machines overlay path for vr1-dc0" paragraph replaced. The never-silently-skip-an-overlay rule is KEPT (it is right, and it is why a dc1 deploy cannot merge dc0 tags); only its dc0 exemption goes.
  • 4.2 dry-run block (dc0) -- machines overlay added, order vips -> machines -> pki.
  • 4.3 deploy block (dc0) -- same, so 4.2 and 4.3 remain identical but for --dry-run.
  • Step 7 dry-run block (dc0) -- same. This block CONTRADICTED ITS OWN paragraph four lines above ("dropping the VIP or machines overlay here would present juju with an input that contradicts the running model"), and the "Same overlay-set rules" sentence beneath it repeated the dc0 exemption; both fixed.
  • 4.2 preamble -- the VERIFY-LIVE pointer was dc1-only; now names both halves (dc1's machines-block retag, BOTH DCs' applications.options merge).

NOT CHANGED, because it was already right: scripts/preflight.sh P2 and runbooks/phase-01-bundle-deploy.md:199-201 both assemble the three-overlay set behind a [ -f ] guard. This was one wrong surface, not two disagreeing.

  • Revert: git revert this commit. The runbook returns to a dc0 command that omits a required overlay -- do not revert without also re-opening the finding in docs/CURRENT-STATE.md.

Item 2 -- the 4.2 gate could not have caught it, so it gained two assertions

The standing rule that a checker which cannot fail is not a gate applies to the gate as much as the command. Step 4.2's four gate items read the machines block, the machine count, the VIP prefixes, and overlay-path errors. The machines overlay's entire payload is under applications:, so all four would have passed a dc0 deploy with no provider bridge mapping and 15 unreachable package sources. Two added:

  • Item 5 -- ovn-chassis shows THREE options, not one. bridge-interface-mappings must appear ALONGSIDE ovn-bridge-mappings: physnet1:br-ex and prefer-chassis-as-gw: true. This is the VERIFY-LIVE assertion both overlays' headers demand and that NO offline check can make: provider-bundle-check.py's _deep_merge merges key-by-key BY CONSTRUCTION, so validating through it cannot distinguish a key-by-key merge from a map REPLACE. Only the live dry-run can. If only the new key appears, the physnet mapping and the B2 gateway-chassis election are both gone from the deploy input and provider egress dies by a different route than the defect item 1 fixes. Also: exactly TWO MACs, a third means a wrong node.
  • Item 6 -- every openstack-origin / ceph source resolves to an address this DC's nodes can reach. Names dc1's deliberate absence of the block as D-135's experiment RESULT, so a later reader does not "fix" the asymmetry.

>>> ITEMS 5 AND 6 AS FIRST WRITTEN WERE THEMSELVES CHECKERS THAT COULD NOT FAIL, AND WERE REMOVED WITHIN THE HOUR. SEE ITEM 3. <<< Left recorded here rather than edited out: the mistake is the point.

  • Revert: n/a -- superseded by item 3 in this same session.

Item 3 -- the dry-run graded the gate, and three of its items failed

THE RUN. docs/audit/stage5-dryrun-dc0-20260802.txt (400 lines). From the dc0 rack (D-138), model vr1-dc0, juju 3.6.27, model EMPTY at run time. Deploy input sha256-verified against HEAD immediately before the run, not inherited from the prior close: all five tracked files match, PKI overlay unchanged at 5fc117f1 mode 0600 (hashed, never read). EXIT 0.

PASSED: nine machines 0-8; all 9 constraint lines arch=amd64 tags=openstack-vr1-dc0,<role> in the 3/2/4 split, none naming the other DC; all three overlays consumed without a path error. Also settled the ceph-osd tags=openstack item CURRENT-STATE deferred to this gate: the plan places ceph-osd/0..3 on machines 5-8 by explicit placement and the bare tag never appears -- the exposure remains a later UNPLACED add-unit, which this deploy does not perform.

THE INSTRUMENT FINDING. Measured at both verbosities: juju deploy --dry-run on a bundle prints ONLY located charms, upload/deploy lines, BARE machine ids, placement and relations. No application options, no VIPs, at any verbosity. Greps over the --debug capture: bridge-interface-mappings 0, physnet1 0, openstack-origin 0, cloud-archive 0, prefer-chassis 0. The only 10.12.4. hits are the controller's own API addresses; the only vip hit is the echoed command line. --debug adds exactly one thing -- the per-machine constraint lines (bundlehandler.go:1035).

Three consequences, all fixed here:

  1. Gate item 1 (tags) was ungradable by the command as written. The named command is now --dry-run --debug, with the measurement quoted as the reason so nobody drops the flag as noise.
  2. The PRE-EXISTING VIP item could never have failed -- "every VIP in the plan is on THIS DC's prefixes", when no VIP is ever in the plan. REMOVED, and the property relocated IN WRITING to preflight P2, which already asserts the VIP bands over the same merged input. Per the standing rule, the assertion was replaced with the true invariant and the move is stated, not silently dropped.
  3. My own items 5 and 6, written earlier this session, had the same defect. REMOVED. The properties are real, so they became new Step 4.4 -- a juju config read run the moment juju deploy returns, before the model settles. That is the only thing that can prove the merge: _deep_merge in provider-bundle-check.py merges key-by-key BY CONSTRUCTION, so it cannot distinguish a key-by-key merge from a map REPLACE, and the dry-run sees no options at all.

OWNED. I added two assertions that could not fail to the very gate whose purpose was to catch a defect that had survived precisely because nothing could see it. They were caught by the live run, not by re-reading my own work -- instance fourteen of this repo's instrument-currency pattern and the second consecutive session where measurement, not review, was the catching agent.

ALSO CORRECTED: both machines overlays' VERIFY-LIVE headers, which instructed the reader to assert the options merge "at the live dry-run". Measured impossible. Corrected in place (not deleted) and pointed at Step 4.4. dc1's header additionally notes the split that dc0's does not have: dc1's machines BLOCK is visible in the --debug plan and is gradable at 4.2; only its options half is not.

  • Revert: git revert. Restores a gate with three ungradable items and two overlay headers naming an impossible check.

Item 4 -- LOGGED, NOT FIXED (hard rule 1)

  1. runbooks/phase-00-teardown-maas-reset.md:233 runs provider-bundle-check.py with the vips overlay ONLY, so it grades a narrower input than either preflight P2 or the deploy command. Not deploy-blocking (teardown-time check), DOCFIX material for the same fold.
  2. The runbook says bundle.yaml ships "NO designate" for this stage; the plan deploys four designate applications and places designate/0 on 0/lxd/5. One of the two is wrong and it is not resolvable from the runbook. NOT deploy-blocking -- designate deploying early costs containers, not correctness. The bullet is annotated in place so no session reads it as a fact about the current bundle; the underlying question (does D-106 leave designate in the Stage-5 bundle?) is unanswered and left for the operator.

Both recorded in docs/CURRENT-STATE.md, neither actioned.

Gates at delivery: bash scripts/repo-lint.sh -- 0 fail, 1 warn (the legacy D-001..018 ASCII carve-out). provider-bundle-check.py over the dc0 merged input PASS. Both machines overlays re-parsed after editing (dc0: applications only, 16 apps; dc1: applications + machines). No script changed, so no harness is implicated; the gauntlet's last full run stands at ALL GREEN (98) from the 2026-08-02 (part 2) close.

THE DEPLOY IS NOT RUN. Step 4.3 is a gated mutation and awaits operator approval. This session stops at the dry-run.