Stage 5 dc0: RETIRE + archive dc-ha-scaleup.yaml (operator-ruled; R6 superseded) [DOCFIX-211] ...

Operator ruling 2026-08-06, exact utterance "Retire the redundancy and archive." The
overlay was wholly redundant -- BUNDLEFIX-053 folded its entire content into bundle.yaml,
so applying it was an all-keys no-op deep-merge (Task-#1 post-wave review, measured).

- Archived: git mv overlays/dc-ha-scaleup.yaml -> docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml
  + a do-not-deploy banner. docs/archive/ chosen explicitly (no prior overlay-archive precedent).
- design-decisions.md: GA-R5 SUPERSESSION note on D-121's R6 (original 2026-07-27 record retained
  as history); fixed :503 and :4563 "overlay encodes v-a" refs -> bundle.yaml/BUNDLEFIX-053.
- Harness (tests/provider-bundle-check): T17 REMOVED (== T18 once vault-hacluster is in base);
  T17b/T32/T33/T34 RE-POINTED onto good.yaml (base HA chain). T33/T34 now use the deterministic
  mutate() helper, not sed-on-overlay (no silent no-op fixture; a wrong key path KeyErrors). Each
  FAIL case proven to fire against the real checker message. 58->57 cases; provider-bundle-check 57/0.
- Runbook dc-dc-phase4 Step-4 note + Step-12.3(a) gate: the two-phase "base at cluster_count:1 then
  apply the overlay to reach 3" model RETIRED -- base deploys 3-unit HA directly, so cluster_count:3
  is the expected post-Step-4 value.
- Rendered vips overlays: edited the render SOURCE (render/values/vr1-dc{0,1}-vips.yaml, D-136) and
  re-rendered (not hand-edited); overlay diff = the two comment lines only; render-drift 4/0.
  Hand-maintained vr1-dc{0,1}-machines.yaml deploy-command comments + provider-bundle-check.py +
  cloud-assert.sh comments + dc-dc-deployment-workflow.md item 22 reconciled.
- CURRENT-STATE: no edit needed -- active status already omits the HA overlay from the deploy input;
  remaining mentions are dated historical narrative (retained as history, GA-R1).
- Also finalizes Item 3/5 changelog notes (F9 approved+staged; Task #1 recommendation ruled).

Gauntlet ALL GREEN (99 harnesses); repo-lint 0 fail (1 pre-existing legacy L1 warn).
Staging: dc0 rack vr1-dc0-vips.yaml now stale-by-one-comment vs the re-render (functionally
identical); re-syncs + sha-verifies at the next deploy per D-138 (logged, no rack write for a comment).

Revert: git mv the overlay back (strip banner) + git revert this commit + re-render vips from
reverted values.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728

converge dc0: proxy installed + wired, origin override DELETED from the overlay ...

Executes D-135 amendment (b), operator ruling "Option A, converge".

dc-cache-proxy.sh install dc0: before-state FAIL exit 1 (and NOT uniformly --
the .4 alias and edge route read OK, so a real red, not the wrong-host trap),
after-state PASS exit 0, 12/12, re-verified INDEPENDENTLY of the installer's own
embedded check. Both decisive assertions green: the proxy serves archive.ubuntu.com
AND ubuntu-cloud.archive.canonical.com Release at 200 -- which is what makes
cloud:jammy-caracal work again.

apt-cacher-ng install touched ONLY apt-cacher-ng + 4 deps (apt history verified).
The needrestart warning about outdated qemu on the running guests traces to the
2026-08-01 unattended-upgrade of libssl3t64/openssl, not to this command --
pre-existing, deferred, and nothing was restarted. Logged: unattended-upgrades
is active on the rack that hosts every dc0 VM.

Model wired per the scripts' own documented lines: apt-http-proxy=10.12.8.4:3142,
snap-http-proxy/snap-https-proxy=10.12.8.4:3129, all read back on content.
The snap keys were RULED on 07-31 (RULING 2) and had NEVER been set on this model
-- ruled but not wired, found by measuring rather than assuming.

Overlay origin override DELETED, not repaired: 15 apps return to bundle.yaml's
cloud:jammy-caracal anchors. Overlay now carries ONLY ovn-chassis
bridge-interface-mappings, verified by parsing. provider-bundle-check PASS,
repo-lint 0 fail.

Logged not fixed: phase4:451 tells the operator to run
'dc-cache-proxy.sh check vr1-dc1', but SITE is taken verbatim and the case accepts
only dc0|dc1 -- that line exits 2 'unknown site'.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

UCA key goes in the OVERLAY, the tested pattern -- not the mechanism I invented ...

Operator: "Why did you guess again? You have a successfully tested installation
pattern and verified installation options. You have access to the documentation
on this."

Two guesses. The pool path 404 was the visible one; the delivery mechanism was
the one that mattered. I generated per-app YAML into /tmp/ucacfg on the rack and
tried juju config --file -- bespoke, out-of-band, invented. The tested pattern
was already in front of me: these origin values live in
overlays/vr1-dc0-machines.yaml by the 07-31 ruling, and phase-4 Step 7 documents
re-passing the FULL overlay set to apply a delta to a running model. Applying by
juju config would have left the overlay and the model disagreeing and would have
reverted at the next deploy from the overlay -- the same class as DOCFIX-208,
committed two hours after fixing it. /tmp/ucacfg is abandoned.

Vendor docs checked rather than derived: the charm-guide defers exotic-value
syntax to keystone's own config.yaml, and charm-helpers import_key() documents
ASCII armour as correct while calling key-id-via-keyserver insecure --
independent of the airgap making it unreachable.

The overlay now carries the armored key on all 15 values: 12 as the
openstack-origin | suffix, 3 as the ceph key option with source unchanged.
Verified by PARSING the overlay, not reading the diff -- all 15 byte-identical
to docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc, ovn-chassis untouched,
ovn-central still absent. provider-bundle-check PASSES; repo-lint 0 fail.

Apply is the Step-7 pattern and awaits the operator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

dc0 deploy blocked: mirrored UCA has no signing key on fresh nodes; root-caused ...

All 9 machines reached started. Nine units then failed the install hook across
unrelated charms -- one common cause:

  W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease:
     NO_PUBKEY 5EDB1B62EC4926EA
  E: The repository is not signed.
  apt-get update returned exit 100

The mirrored content is reachable and its upstream signatures are intact; the
key to verify them is absent.

This refutes a claim overlays/vr1-dc0-machines.yaml carried as MEASURED -- that
the UCA key is already on the nodes. Measured on ceph-osd/0: trusted.gpg.d holds
only the cdimage and 2018-archive keyrings. The cloud-archive keyring ships in
the ubuntu-cloud-keyring package, installed as a side effect of the
cloud:<series>-<release> origin path; a raw deb line bypasses it. The claim was
almost certainly measured on a node from the 07-31 deploy, which used
cloud:jammy-caracal and had the keyring. True when taken, false on a fresh node.

Fix is charm-documented, quoted from the charms: openstack charms take
openstack-origin "deb <URL>|<PUB KEY>"; the three ceph charms take a separate
key option accepting ASCII armor. Armored block, never a keyid -- a keyid sends
charm-helpers to keyserver.ubuntu.com, unreachable under the D-107 airgap.

Key provenance measured, not downloaded: the mirror serves ubuntu-cloud-keyring
2021.03.26; URI from apt-get download --print-uris on a node (source package is
ubuntu-keyring, not ubuntu-cloud-keyring -- I guessed and got a 404); sha512
matched apt's declared digest; keyring holds exactly 5EDB1B62EC4926EA. Armored
export committed as public key material.

PREPARED, NOT APPLIED: config for all 15 apps generated from that one verified
key and staged on the rack. juju config --file was refused by the permission
layer in three shapes; it is a live mutation across 15 apps and belongs to the
operator. No workaround attempted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

DOCFIX-208 part 2: the dry-run graded the gate, and three items could not fail ...

The corrected dc0 command dry-ran EXIT 0 from the dc0 rack against an empty
vr1-dc0 model, staged input sha256-verified against HEAD first. Nine machines
0-8, all nine constraints tags=openstack-vr1-dc0,<role> in the 3/2/4 split,
all three overlays consumed. Capture: docs/audit/stage5-dryrun-dc0-20260802.txt.

The run also measured what the command can and cannot show. juju deploy
--dry-run on a bundle prints NO application options and NO VIPs at any
verbosity; --debug adds only the per-machine constraint lines. Greps over the
--debug capture: bridge-interface-mappings 0, physnet1 0, openstack-origin 0,
cloud-archive 0, prefer-chassis 0.

So three Step-4.2 gate items could not be graded by the command the gate names:
item 1 (tags) needed --debug, which is now in the named command; the
PRE-EXISTING VIP item could never have failed because no VIP is ever in the
plan, and its property is relocated in writing to preflight P2; and the two
option-reading items I added earlier this session had the same defect and are
removed within the hour. Their properties become new Step 4.4 -- a juju config
read run the moment deploy returns, which is the only thing that can prove
juju merged the options map rather than replacing it.

Both machines overlays' VERIFY-LIVE headers said to assert this "at the live
dry-run". Measured impossible; corrected in place, pointed at 4.4.

Also settles the deferred ceph-osd tags=openstack item (placed by explicit
placement, bare tag absent from the plan) and logs a contradiction not resolved
here: the runbook says the bundle ships NO designate, the plan deploys four
designate applications.

The deploy itself is NOT run -- 4.3 is a gated mutation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Build ruling 1: UCA points at the in-DC mirror for 15 apps (scope derived, not guessed) ...

overlays/vr1-dc0-machines.yaml sets
  deb http://10.12.8.4/cloud-archive jammy-updates/caracal main
on 15 apps. SCOPE DERIVED FROM THE CHARM SCHEMAS rather than hand-listed: every
app whose charm accepts an origin key AND resolves to a UCA pocket -- 12
openstack-origin (explicit cloud:jammy-caracal or the charm default 'caracal')
plus 3 ceph 'source'. Everything else defaults to 'distro' (Ubuntu archive only)
and needs nothing, which is why the mysql-router subordinates, ceph-rbd-mirror,
mysql-innodb-cluster, glance-simplestreams-sync and rabbitmq-server are absent.

CORRECTION: the earlier in-session figure of 'six apps' was the count that set it
EXPLICITLY and was never the scope. Measured on the containers, both keystone/0
and ceph-mon/1 carry the upstream UCA source.

Also measured: keystone CLEARED install anyway, so the unreachable UCA is not
universally fatal -- it is fatal where a charm's apt_update uses --error-on=any
(the ceph charms), and a CORRECTNESS problem everywhere else, since a node that
cannot reach the Caracal pocket silently gets jammy's own OpenStack instead.

Per-DC by necessity: the mirror address differs per DC, so this cannot live in
bundle.yaml. dc1 gets no equivalent block -- its apt-cacher-ng forwards the
upstream URL transparently, and that asymmetry is D-135's experiment result.

No |key suffix needed: measured, the UCA signing key is already on the nodes and
debmirror preserved the upstream Release/InRelease signatures.

LOGGED NOT CHANGED: ovn-central's charm default is source: zed, not caracal -- it
points at a pocket the dc0 mirror does not carry, and repointing would change its
RELEASE rather than its URL. Separate question.

Gauntlet ALL GREEN (93); repo-lint 0 fail; provider-bundle-check PASS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

3.8 fixed per-DC -- and the fix exposed two repo records naming the wrong NIC ...

bridge-interface-mappings is out of bundle.yaml and per-DC in overlays/vr1-dc0-machines.yaml
(new) and vr1-dc1-machines.yaml. The machines overlay was chosen over the vips overlay for a
measured reason: both *-vips.yaml are marked GENERATED / do-not-hand-edit, so a key added
there is clobbered on the next render, and preflight.sh already assembles ${DC}-machines.yaml.
Creating the dc0 file also closes NEW-3 from the other side -- preflight's [ -f ] guard was
silently skipping a file the deploy needs.

THE WORSE DEFECT THE FIX EXPOSED. dc0's compute provider-public MACs are 52:54:00:8c:2a:8c /
52:54:00:50:48:88 -- position [1] in each node's macs list. Audit register row U5 and the
D-124 correction both name the [0] values, which are the PXE/boot MACs, and D-124 further
tells the reader to source them "from lib-hosts.sh" -- verified independently: lib-hosts.sh
declares only HOST_BOOT_MAC and cannot supply a provider MAC at all. Following either record
would have bridged br-ex onto the boot NIC: not a missing mapping that fails loudly, but a
wrong one that comes up and misbehaves. Confirmed three ways -- substrate main.tf:109-116 list
order, the macpin plan capture, and live MAAS showing br-ex over enp2s0. The correct values
and the [0]-vs-[1] trap are now recorded in the overlay, next to the data.

NEW-1: 10.12.12.0/22 was named as the geneve/DATA plane; that is metal-internal, data-tenant
is 10.12.16.0/22. Deliberately not swept repo-wide -- the 10.12.12.5x VIP legs are correct
metal-internal values.

NEW-2 phantom vr0-dc0-testcloud.yaml removed. NEW-4/6 stale octavia-pki name and the phantom
${DC}-hostnames.yaml removed (juju errors on a missing overlay). NEW-7 scale-up removed from
the VIP deploy command per R6, ${DC1_MODEL} -> ${DC_MODEL}.

R5/D-106 BUILT: phase-6 Step 5 rewritten configure-not-deploy, the unsatisfiable diff gate
replaced by a presence-and-state assertion, and the "no designate block exists" claim
corrected against bundle.yaml. os-public-hostname stays unset, per the ruling's refused
option (c).

Verified independently rather than from reported output: provider-bundle-check PASSes for
both DCs in preflight P2's own overlay order. repo-lint 0 fail / 617 files.

Logged not fixed: provider-bundle-check treats an absent bridge-interface-mappings as a silent
skip and never checks the MAC prefix against --dc, so it could not fail on either variant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf