STAGE 5 -- dc0 bundle deploy ATTEMPT 2 (2026-07-31): TWO NEW DEFECTS, AND THE MODEL IS
PARTIALLY POPULATED FOR THE FIRST TIME
=======================================================================================

PRECONDITIONS, all met before the command was run:
  - the prefer-ipv6 defect that killed attempt 1 is RULED and BUILT; the six no longer
    carry the option and every v6 VIP leg is retained
  - both DCs' deploy inputs sweep CLEAN on option NAMES and VALUE TYPES (75/20/0/0/0)
  - the dc0 rack's ~/repo-stage was refreshed and all four files hash-verified against
    repo HEAD
  - preflight P1-P4 + P7 PASS; P5 RED with 11 findings, all ELEVEN ruled-accepted
    (six on 2026-07-30, five on 2026-07-31)
  - `juju status -m vr1-dc0` -> "Model \"vr1-dc0\" is empty", 0 applications, 0 machines

COMMAND (from the dc0 rack, D-138 client host):
  juju deploy ./bundle.yaml --overlay ./overlays/vr1-dc0-vips.yaml \
    --overlay ./overlays/vr1-dc0-machines.yaml \
    --overlay ./overlays/vr1-dc0-octavia-pki.yaml -m vr1-dc0

>>> RUN 2a VERDICT:
      ERROR cannot deploy bundle: file for resource "policyd-override":
      stat /home/jessea123/repo-stage/policies/overrides.zip: no such file or directory

>>> THIS TIME IT WAS **NOT** ATOMIC. MEASURED IMMEDIATELY AFTER:
      applications: 23   machines: 0   total units: 0
    So 23 application definitions exist and NOTHING is provisioned -- no machine was
    allocated, no unit created, no hardware touched. Attempt 1 aborted during
    VALIDATION and left the model empty; this one got past validation into EXECUTION,
    which is why it left residue.

DEFECT 1 -- THE DEPLOY INPUT ON THE CLIENT HOST IS A PARTIAL COPY OF THE REPO
-----------------------------------------------------------------------------
`~/repo-stage` on the rack holds `bundle.yaml`, `overlays/` and three scripts. It does
NOT hold `policies/`. `bundle.yaml:214` declares

    resources:
      policyd-override: ./policies/overrides.zip

which is the ONLY local-file reference in the entire deploy input (measured: one hit
across bundle.yaml and all three overlays). juju resolves it RELATIVE TO THE BUNDLE and
uploads it as a charm resource, so the file must exist ON THE HOST THAT DEPLOYS.

WHY NOTHING CAUGHT IT:
  - `juju deploy --dry-run` does not upload resources, so it exits 0 without ever
    touching the path. This is the SECOND blind spot found in --dry-run in two days;
    the first was config option NAMES (attempt 1).
  - attempt 1 aborted at validation, BEFORE resource upload, so the gap was masked.
  - `preflight.sh` runs on voffice1, where `policies/` DOES exist -- so P1-P7 green says
    nothing about the rack. Same wrong-host instrument class as the `dc-mirror.sh` and
    egress-probe errors already recorded, and the SECOND time in this session (the stale
    `~/repo-stage` overlay was the first).

FIX-FORWARD APPLIED (gated): `policies/overrides.zip` + `policies/domain-manager-policy.yaml`
copied to `~/repo-stage/policies/`, both sha256-verified equal to repo HEAD
(`02fe1fd7...`, `64f99b63...`).

>>> RUN 2b VERDICT, after the fix:
      ERROR cannot deploy bundle: application "barbican": downgrades are not currently
      supported: deployed revision 265 is newer than requested revision 261

DEFECT 2 -- A BUNDLE THAT RELIES ON `default-base` CANNOT BE SAFELY RE-RUN
--------------------------------------------------------------------------
MEASURED, both directions, with `juju download` on the rack:

    barbican 2024.1/stable  --base ubuntu@22.04 --arch amd64  ->  revision 265
    barbican 2024.1/stable  --base ubuntu@24.04 --arch amd64  ->  revision 261
    barbican 2024.1/stable  (no base given)                   ->  revision 261

The 23 applications created by run 2a are ALL on base 22.04 (measured from
`juju status --format json`) -- run 2a honoured `bundle.yaml:85 default-base:
ubuntu@22.04/stable`. **On the RE-RUN, for an application that ALREADY EXISTS, juju
resolves the charm WITHOUT that default and lands on the 24.04 revision**, then refuses
the resulting downgrade. So the bundle is deployable exactly once and is NOT re-runnable
after a partial failure.

TWO CANDIDATE FIXES TESTED READ-ONLY. `--dry-run` REPRODUCES the error against the
now-populated model, which makes it a safe test harness:

  (a) `juju model-config -m vr1-dc0 default-base=ubuntu@22.04` -- SET and READ BACK, then
      re-ran: **NO EFFECT**, identical error. The model-level default is not consulted on
      this path either. The setting was RESET afterwards, so model config is back to its
      documented Step-3.5 state.
  (b) an explicit per-application `base: ubuntu@22.04/stable` -- **WORKS.** Added to
      `barbican` alone in a THROWAWAY copy, the dry-run's error moved to the next
      application:
          ERROR ... application "barbican-vault": deployed revision 99 is newer than
          requested revision 97
      i.e. barbican resolved correctly at 265 and the same defect simply surfaced on the
      next app. Fixing it for all would mean an explicit base on all 56 applications.

INSTRUMENT NOTE, recorded because it cost two runs: the first two attempts at test (b)
staged the throwaway bundle under `/tmp` and both returned
`ERROR no charm was found at "./bundle.yaml"`. **That is the juju SNAP's PRIVATE /tmp** --
already recorded in this repo and in the phase-4 runbook, and walked into anyway. Staged
under a plain `$HOME/bundletest` it worked immediately. The error names neither snap nor
confinement.

STATE AT THE POINT THIS CAPTURE WAS WRITTEN
-------------------------------------------
  applications  23   (definitions only)
  machines       0
  units          0
  model-config  default-base RESET to unset; `apt-mirror` still http://10.12.8.4/ubuntu
  throwaway test artifacts (~/bundletest, /tmp/bt) REMOVED
  the dc0 controller model is untouched and healthy

NOTHING IS PROVISIONED. No MAAS machine moved out of `Ready`; no disk was written. The
residue is 23 application rows in the juju model database.

THE RECOVERY DECISION IS THE OPERATOR'S AND IS PUT SEPARATELY
-------------------------------------------------------------
The repo's rollback decision tree (`runbooks/dc-dc-teardown-rollback.md:586`) is written
for `tofu apply` and defaults to FIX-FORWARD. **Fix-forward by re-running the bundle is
MEASURED not to work here** (defect 2), so the tree does not reach this case; a partial
`juju deploy` is a failure mode the repo does not yet cover. That gap is itself a
finding.

Options as put:
  (A) REMOVE the 23 application definitions, returning the model to empty, then ONE
      clean deploy from the UNCHANGED bundle. Blast radius is measured zero -- 0 units,
      0 machines. Cost: 23 destructive operations, which hard rule 3 says are never
      batched.
  (B) ADD an explicit `base: ubuntu@22.04/stable` to all 56 applications in
      `bundle.yaml` and re-run incrementally, keeping the 23. Fixes the re-runnability
      defect properly and has a real Roosevelt delta -- but it is a 56-line change to a
      ruled surface MADE MID-DEPLOY, which is what hard rule 1 forbids.
  (C) (A) now, and LOG (B) as the durable fix for a later step -- hard rule 1's own
      shape: findings are logged, not executed mid-step.
