diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index c91d6c7..8f8ae0c 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1188,6 +1188,50 @@ placement is by explicit machine id exactly as reasoned, so the absent tag never has to match. The residual is unchanged and still LOGGED NOT ACTIONED -- a later UNPLACED `juju add-unit ceph-osd` would match no machine. That is a measurement, not a decision. + **>>> BUNDLE DEPLOY ATTEMPT 2 RUN 2026-07-31: THE prefer-ipv6 DEFECT IS GONE, TWO NEW ONES + FOUND, AND FOR THE FIRST TIME THE MODEL IS PARTIALLY POPULATED. <<<** Capture + `docs/audit/stage5-dc0-deploy-attempt2-20260731.txt`. **MEASURED IMMEDIATELY AFTER, not + assumed: 23 applications, 0 machines, 0 units.** Attempt 1 aborted during VALIDATION and + left the model empty; this one got past validation into EXECUTION, so it left residue -- + 23 application DEFINITIONS and **nothing provisioned**: no MAAS machine left `Ready`, no + disk written. **DEFECT 1: `ERROR ... file for resource "policyd-override": stat + /home/jessea123/repo-stage/policies/overrides.zip: no such file or directory`.** The rack's + `~/repo-stage` is a PARTIAL COPY of the repo and has no `policies/`; `bundle.yaml:214` is + the ONLY local-file reference in the whole deploy input and juju resolves it relative to + the bundle, so the file must exist ON THE HOST THAT DEPLOYS. **Nothing caught it because + `--dry-run` does not upload resources** -- the SECOND `--dry-run` blind spot found in two + days, after config option NAMES -- attempt 1 aborted before resource upload so the gap was + masked, and `preflight.sh` runs on voffice1 where `policies/` DOES exist. **Same wrong-host + instrument class as the `dc-mirror.sh` and egress-probe errors, and the SECOND instance in + this session.** FIXED FORWARD (gated): both policy files copied to the rack and + sha256-verified equal to repo HEAD. **DEFECT 2, found by the re-run: `ERROR ... application + "barbican": downgrades are not currently supported: deployed revision 265 is newer than + requested revision 261`. A BUNDLE THAT RELIES ON `default-base` IS DEPLOYABLE EXACTLY ONCE + AND IS NOT RE-RUNNABLE.** Measured with `juju download` on the rack: barbican + `2024.1/stable` is rev **265** at `--base ubuntu@22.04` and rev **261** at + `--base ubuntu@24.04` or with no base given. The 23 apps created are all on 22.04 (measured + from `juju status --format json`), so run 2a honoured `bundle.yaml:85`; **on the RE-RUN, for + an application that ALREADY EXISTS, juju resolves the charm WITHOUT that default and lands + on the 24.04 revision**, then refuses the downgrade. TWO FIXES TESTED READ-ONLY (`--dry-run` + REPRODUCES the error against the populated model, which makes it a safe harness): **(a) + `juju model-config default-base=ubuntu@22.04` set and read back -- NO EFFECT, identical + error; RESET afterwards, so model config is back to its documented Step-3.5 state. (b) an + explicit per-application `base: ubuntu@22.04/stable` -- WORKS**: added to `barbican` alone + in a throwaway, the error moved to `barbican-vault` (99 vs 97), i.e. barbican resolved + correctly. Fixing all would mean an explicit base on all 56 applications. **INSTRUMENT NOTE, + it cost two runs: the first two attempts at test (b) staged the throwaway under `/tmp` and + returned `ERROR no charm was found at "./bundle.yaml"` -- the juju SNAP's PRIVATE /tmp, + already recorded in this repo AND in the phase-4 runbook, and walked into anyway.** Under + `$HOME` it worked at once; the error names neither snap nor confinement. + **THE ROLLBACK DECISION TREE DOES NOT REACH THIS CASE** -- + `runbooks/dc-dc-teardown-rollback.md:586` is written for `tofu apply` and defaults to + FIX-FORWARD, but fix-forward by re-running the bundle is MEASURED not to work. A partial + `juju deploy` is a failure mode the repo does not cover; that gap is itself a finding. + **STAGE 5 IS BLOCKED ON AN OPERATOR RECOVERY DECISION**, options enumerated in the capture: + (A) remove the 23 definitions and deploy once cleanly from the unchanged bundle (blast + radius measured zero; 23 destructive ops, never batched); (B) add an explicit per-app + `base:` to all 56 and re-run incrementally (fixes re-runnability, real Roosevelt delta, but + a 56-line mid-deploy change to a ruled surface); (C) (A) now and LOG (B) for a later step. **RUNBOOK DEFECT FOUND IN PASSING, LOGGED NOT FIXED (DOCFIX material):** `runbooks/dc-dc-phase4-juju-bundle-per-dc.md:553-557` gives the dc0 deploy WITHOUT `overlays/vr1-dc0-machines.yaml`, while the dc1 block three lines below includes its diff --git a/docs/audit/stage5-dc0-deploy-attempt2-20260731.txt b/docs/audit/stage5-dc0-deploy-attempt2-20260731.txt new file mode 100644 index 0000000..7ee0015 --- /dev/null +++ b/docs/audit/stage5-dc0-deploy-attempt2-20260731.txt @@ -0,0 +1,128 @@ +STAGE 5 -- dc0 bundle deploy ATTEMPT 2 (2026-07-31): TWO NEW DEFECTS, AND THE MODEL IS +PARTIALLY POPULATED FOR THE FIRST TIME +======================================================================================= + +PRECONDITIONS, all met before the command was run: + - the prefer-ipv6 defect that killed attempt 1 is RULED and BUILT; the six no longer + carry the option and every v6 VIP leg is retained + - both DCs' deploy inputs sweep CLEAN on option NAMES and VALUE TYPES (75/20/0/0/0) + - the dc0 rack's ~/repo-stage was refreshed and all four files hash-verified against + repo HEAD + - preflight P1-P4 + P7 PASS; P5 RED with 11 findings, all ELEVEN ruled-accepted + (six on 2026-07-30, five on 2026-07-31) + - `juju status -m vr1-dc0` -> "Model \"vr1-dc0\" is empty", 0 applications, 0 machines + +COMMAND (from the dc0 rack, D-138 client host): + juju deploy ./bundle.yaml --overlay ./overlays/vr1-dc0-vips.yaml \ + --overlay ./overlays/vr1-dc0-machines.yaml \ + --overlay ./overlays/vr1-dc0-octavia-pki.yaml -m vr1-dc0 + +>>> RUN 2a VERDICT: + ERROR cannot deploy bundle: file for resource "policyd-override": + stat /home/jessea123/repo-stage/policies/overrides.zip: no such file or directory + +>>> THIS TIME IT WAS **NOT** ATOMIC. MEASURED IMMEDIATELY AFTER: + applications: 23 machines: 0 total units: 0 + So 23 application definitions exist and NOTHING is provisioned -- no machine was + allocated, no unit created, no hardware touched. Attempt 1 aborted during + VALIDATION and left the model empty; this one got past validation into EXECUTION, + which is why it left residue. + +DEFECT 1 -- THE DEPLOY INPUT ON THE CLIENT HOST IS A PARTIAL COPY OF THE REPO +----------------------------------------------------------------------------- +`~/repo-stage` on the rack holds `bundle.yaml`, `overlays/` and three scripts. It does +NOT hold `policies/`. `bundle.yaml:214` declares + + resources: + policyd-override: ./policies/overrides.zip + +which is the ONLY local-file reference in the entire deploy input (measured: one hit +across bundle.yaml and all three overlays). juju resolves it RELATIVE TO THE BUNDLE and +uploads it as a charm resource, so the file must exist ON THE HOST THAT DEPLOYS. + +WHY NOTHING CAUGHT IT: + - `juju deploy --dry-run` does not upload resources, so it exits 0 without ever + touching the path. This is the SECOND blind spot found in --dry-run in two days; + the first was config option NAMES (attempt 1). + - attempt 1 aborted at validation, BEFORE resource upload, so the gap was masked. + - `preflight.sh` runs on voffice1, where `policies/` DOES exist -- so P1-P7 green says + nothing about the rack. Same wrong-host instrument class as the `dc-mirror.sh` and + egress-probe errors already recorded, and the SECOND time in this session (the stale + `~/repo-stage` overlay was the first). + +FIX-FORWARD APPLIED (gated): `policies/overrides.zip` + `policies/domain-manager-policy.yaml` +copied to `~/repo-stage/policies/`, both sha256-verified equal to repo HEAD +(`02fe1fd7...`, `64f99b63...`). + +>>> RUN 2b VERDICT, after the fix: + ERROR cannot deploy bundle: application "barbican": downgrades are not currently + supported: deployed revision 265 is newer than requested revision 261 + +DEFECT 2 -- A BUNDLE THAT RELIES ON `default-base` CANNOT BE SAFELY RE-RUN +-------------------------------------------------------------------------- +MEASURED, both directions, with `juju download` on the rack: + + barbican 2024.1/stable --base ubuntu@22.04 --arch amd64 -> revision 265 + barbican 2024.1/stable --base ubuntu@24.04 --arch amd64 -> revision 261 + barbican 2024.1/stable (no base given) -> revision 261 + +The 23 applications created by run 2a are ALL on base 22.04 (measured from +`juju status --format json`) -- run 2a honoured `bundle.yaml:85 default-base: +ubuntu@22.04/stable`. **On the RE-RUN, for an application that ALREADY EXISTS, juju +resolves the charm WITHOUT that default and lands on the 24.04 revision**, then refuses +the resulting downgrade. So the bundle is deployable exactly once and is NOT re-runnable +after a partial failure. + +TWO CANDIDATE FIXES TESTED READ-ONLY. `--dry-run` REPRODUCES the error against the +now-populated model, which makes it a safe test harness: + + (a) `juju model-config -m vr1-dc0 default-base=ubuntu@22.04` -- SET and READ BACK, then + re-ran: **NO EFFECT**, identical error. The model-level default is not consulted on + this path either. The setting was RESET afterwards, so model config is back to its + documented Step-3.5 state. + (b) an explicit per-application `base: ubuntu@22.04/stable` -- **WORKS.** Added to + `barbican` alone in a THROWAWAY copy, the dry-run's error moved to the next + application: + ERROR ... application "barbican-vault": deployed revision 99 is newer than + requested revision 97 + i.e. barbican resolved correctly at 265 and the same defect simply surfaced on the + next app. Fixing it for all would mean an explicit base on all 56 applications. + +INSTRUMENT NOTE, recorded because it cost two runs: the first two attempts at test (b) +staged the throwaway bundle under `/tmp` and both returned +`ERROR no charm was found at "./bundle.yaml"`. **That is the juju SNAP's PRIVATE /tmp** -- +already recorded in this repo and in the phase-4 runbook, and walked into anyway. Staged +under a plain `$HOME/bundletest` it worked immediately. The error names neither snap nor +confinement. + +STATE AT THE POINT THIS CAPTURE WAS WRITTEN +------------------------------------------- + applications 23 (definitions only) + machines 0 + units 0 + model-config default-base RESET to unset; `apt-mirror` still http://10.12.8.4/ubuntu + throwaway test artifacts (~/bundletest, /tmp/bt) REMOVED + the dc0 controller model is untouched and healthy + +NOTHING IS PROVISIONED. No MAAS machine moved out of `Ready`; no disk was written. The +residue is 23 application rows in the juju model database. + +THE RECOVERY DECISION IS THE OPERATOR'S AND IS PUT SEPARATELY +------------------------------------------------------------- +The repo's rollback decision tree (`runbooks/dc-dc-teardown-rollback.md:586`) is written +for `tofu apply` and defaults to FIX-FORWARD. **Fix-forward by re-running the bundle is +MEASURED not to work here** (defect 2), so the tree does not reach this case; a partial +`juju deploy` is a failure mode the repo does not yet cover. That gap is itself a +finding. + +Options as put: + (A) REMOVE the 23 application definitions, returning the model to empty, then ONE + clean deploy from the UNCHANGED bundle. Blast radius is measured zero -- 0 units, + 0 machines. Cost: 23 destructive operations, which hard rule 3 says are never + batched. + (B) ADD an explicit `base: ubuntu@22.04/stable` to all 56 applications in + `bundle.yaml` and re-run incrementally, keeping the 23. Fixes the re-runnability + defect properly and has a real Roosevelt delta -- but it is a 56-line change to a + ruled surface MADE MID-DEPLOY, which is what hard rule 1 forbids. + (C) (A) now, and LOG (B) as the durable fix for a later step -- hard rule 1's own + shape: findings are logged, not executed mid-step. diff --git a/docs/changelog-20260731-prefer-ipv6-research.md b/docs/changelog-20260731-prefer-ipv6-research.md index 76b3dcc..547a4fa 100644 --- a/docs/changelog-20260731-prefer-ipv6-research.md +++ b/docs/changelog-20260731-prefer-ipv6-research.md @@ -300,3 +300,49 @@ **Revert.** A ruling is a record, not a change; there is nothing to revert. Withdrawing it would require a new GA-R5 exchange. + +## Item 10 -- BUNDLE DEPLOY ATTEMPT 2: the prefer-ipv6 defect is gone; two new ones found + +**What happened.** The deploy was run from the dc0 rack against the fixed, hash-verified +input. The `prefer-ipv6` failure did not recur. Two new defects did, and for the first +time the model is PARTIALLY POPULATED: **23 applications, 0 machines, 0 units** -- +measured immediately, not assumed. Attempt 1 aborted during VALIDATION and left the model +empty; this one got past validation into EXECUTION, which is why it left residue. +**Nothing is provisioned**: no MAAS machine left `Ready`, no disk was written. + +**Defect 1 -- the deploy input on the client host is a PARTIAL COPY of the repo.** +`~/repo-stage` has no `policies/`, and `bundle.yaml:214` is the only local-file reference +in the whole deploy input (`policyd-override: ./policies/overrides.zip`), resolved +relative to the bundle and uploaded as a charm resource -- so it must exist ON THE HOST +THAT DEPLOYS. Nothing caught it: `--dry-run` does not upload resources (the SECOND +`--dry-run` blind spot in two days), attempt 1 aborted before resource upload, and +`preflight.sh` runs on voffice1 where `policies/` DOES exist. **Same wrong-host instrument +class as the `dc-mirror.sh` and egress-probe errors, and the second instance this +session.** Fixed forward under gating: both policy files copied and sha256-verified. + +**Defect 2 -- a bundle that relies on `default-base` is deployable exactly once.** The +re-run failed with `barbican: downgrades are not currently supported: deployed revision +265 is newer than requested revision 261`. Measured: barbican `2024.1/stable` is rev 265 +at `--base ubuntu@22.04` and 261 at `--base ubuntu@24.04` or with no base. The 23 +existing apps are all on 22.04, so run 2a honoured `bundle.yaml:85` -- but on the re-run, +for an application that ALREADY EXISTS, juju resolves without that default. + +**Two fixes tested READ-ONLY** (`--dry-run` reproduces the error against the populated +model, making it a safe harness): a model-level `default-base=ubuntu@22.04` has NO EFFECT +(set, read back, re-run identical, then RESET -- model config is back to its documented +Step-3.5 state); an explicit per-application `base:` WORKS (added to barbican alone, the +error moved to barbican-vault). + +**OWNED -- I walked into a trap this repo has already recorded twice.** The first two +attempts at the second test staged a throwaway bundle under `/tmp` and returned `ERROR no +charm was found at "./bundle.yaml"`. That is the juju SNAP's PRIVATE `/tmp`, documented in +CURRENT-STATE and in the phase-4 runbook. Under `$HOME` it worked immediately. + +**The rollback decision tree does not reach this case.** +`runbooks/dc-dc-teardown-rollback.md:586` is written for `tofu apply` and defaults to +fix-forward, which is measured not to work here. A partial `juju deploy` is a failure +mode the repo does not cover -- logged as a finding, not fixed here. + +**Revert.** The repo change is the capture and the CURRENT-STATE entry. The rack-side +policy copy is reversed with `rm -rf ~/repo-stage/policies`. The model residue is the +subject of the operator recovery decision and is deliberately untouched.