| 2026-08-02 |

converge dc0: proxy installed + wired, origin override DELETED from the overlay
...
Executes D-135 amendment (b), operator ruling "Option A, converge".
dc-cache-proxy.sh install dc0: before-state FAIL exit 1 (and NOT uniformly --
the .4 alias and edge route read OK, so a real red, not the wrong-host trap),
after-state PASS exit 0, 12/12, re-verified INDEPENDENTLY of the installer's own
embedded check. Both decisive assertions green: the proxy serves archive.ubuntu.com
AND ubuntu-cloud.archive.canonical.com Release at 200 -- which is what makes
cloud:jammy-caracal work again.
apt-cacher-ng install touched ONLY apt-cacher-ng + 4 deps (apt history verified).
The needrestart warning about outdated qemu on the running guests traces to the
2026-08-01 unattended-upgrade of libssl3t64/openssl, not to this command --
pre-existing, deferred, and nothing was restarted. Logged: unattended-upgrades
is active on the rack that hosts every dc0 VM.
Model wired per the scripts' own documented lines: apt-http-proxy=10.12.8.4:3142,
snap-http-proxy/snap-https-proxy=10.12.8.4:3129, all read back on content.
The snap keys were RULED on 07-31 (RULING 2) and had NEVER been set on this model
-- ruled but not wired, found by measuring rather than assuming.
Overlay origin override DELETED, not repaired: 15 apps return to bundle.yaml's
cloud:jammy-caracal anchors. Overlay now carries ONLY ovn-chassis
bridge-interface-mappings, verified by parsing. provider-bundle-check PASS,
repo-lint 0 fail.
Logged not fixed: phase4:451 tells the operator to run
'dc-cache-proxy.sh check vr1-dc1', but SITE is taken verbatim and the case accepts
only dc0|dc1 -- that line exits 2 'unknown site'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

UCA key goes in the OVERLAY, the tested pattern -- not the mechanism I invented
...
Operator: "Why did you guess again? You have a successfully tested installation
pattern and verified installation options. You have access to the documentation
on this."
Two guesses. The pool path 404 was the visible one; the delivery mechanism was
the one that mattered. I generated per-app YAML into /tmp/ucacfg on the rack and
tried juju config --file -- bespoke, out-of-band, invented. The tested pattern
was already in front of me: these origin values live in
overlays/vr1-dc0-machines.yaml by the 07-31 ruling, and phase-4 Step 7 documents
re-passing the FULL overlay set to apply a delta to a running model. Applying by
juju config would have left the overlay and the model disagreeing and would have
reverted at the next deploy from the overlay -- the same class as DOCFIX-208,
committed two hours after fixing it. /tmp/ucacfg is abandoned.
Vendor docs checked rather than derived: the charm-guide defers exotic-value
syntax to keystone's own config.yaml, and charm-helpers import_key() documents
ASCII armour as correct while calling key-id-via-keyserver insecure --
independent of the airgap making it unreachable.
The overlay now carries the armored key on all 15 values: 12 as the
openstack-origin | suffix, 3 as the ceph key option with source unchanged.
Verified by PARSING the overlay, not reading the diff -- all 15 byte-identical
to docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc, ovn-chassis untouched,
ovn-central still absent. provider-bundle-check PASSES; repo-lint 0 fail.
Apply is the Step-7 pattern and awaits the operator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

dc0 deploy blocked: mirrored UCA has no signing key on fresh nodes; root-caused
...
All 9 machines reached started. Nine units then failed the install hook across
unrelated charms -- one common cause:
W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease:
NO_PUBKEY 5EDB1B62EC4926EA
E: The repository is not signed.
apt-get update returned exit 100
The mirrored content is reachable and its upstream signatures are intact; the
key to verify them is absent.
This refutes a claim overlays/vr1-dc0-machines.yaml carried as MEASURED -- that
the UCA key is already on the nodes. Measured on ceph-osd/0: trusted.gpg.d holds
only the cdimage and 2018-archive keyrings. The cloud-archive keyring ships in
the ubuntu-cloud-keyring package, installed as a side effect of the
cloud:<series>-<release> origin path; a raw deb line bypasses it. The claim was
almost certainly measured on a node from the 07-31 deploy, which used
cloud:jammy-caracal and had the keyring. True when taken, false on a fresh node.
Fix is charm-documented, quoted from the charms: openstack charms take
openstack-origin "deb <URL>|<PUB KEY>"; the three ceph charms take a separate
key option accepting ASCII armor. Armored block, never a keyid -- a keyid sends
charm-helpers to keyserver.ubuntu.com, unreachable under the D-107 airgap.
Key provenance measured, not downloaded: the mirror serves ubuntu-cloud-keyring
2021.03.26; URI from apt-get download --print-uris on a node (source package is
ubuntu-keyring, not ubuntu-cloud-keyring -- I guessed and got a 404); sha512
matched apt's declared digest; keyring holds exactly 5EDB1B62EC4926EA. Armored
export committed as public key material.
PREPARED, NOT APPLIED: config for all 15 apps generated from that one verified
key and staged on the rack. juju config --file was refused by the permission
layer in three shapes; it is a live mutation across 15 apps and belongs to the
operator. No workaround attempted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

DOCFIX-208 part 2: the dry-run graded the gate, and three items could not fail
...
The corrected dc0 command dry-ran EXIT 0 from the dc0 rack against an empty
vr1-dc0 model, staged input sha256-verified against HEAD first. Nine machines
0-8, all nine constraints tags=openstack-vr1-dc0,<role> in the 3/2/4 split,
all three overlays consumed. Capture: docs/audit/stage5-dryrun-dc0-20260802.txt.
The run also measured what the command can and cannot show. juju deploy
--dry-run on a bundle prints NO application options and NO VIPs at any
verbosity; --debug adds only the per-machine constraint lines. Greps over the
--debug capture: bridge-interface-mappings 0, physnet1 0, openstack-origin 0,
cloud-archive 0, prefer-chassis 0.
So three Step-4.2 gate items could not be graded by the command the gate names:
item 1 (tags) needed --debug, which is now in the named command; the
PRE-EXISTING VIP item could never have failed because no VIP is ever in the
plan, and its property is relocated in writing to preflight P2; and the two
option-reading items I added earlier this session had the same defect and are
removed within the hour. Their properties become new Step 4.4 -- a juju config
read run the moment deploy returns, which is the only thing that can prove
juju merged the options map rather than replacing it.
Both machines overlays' VERIFY-LIVE headers said to assert this "at the live
dry-run". Measured impossible; corrected in place, pointed at 4.4.
Also settles the deferred ceph-osd tags=openstack item (placed by explicit
placement, bare tag absent from the plan) and logs a contradiction not resolved
here: the runbook says the bundle ships NO designate, the plan deploys four
designate applications.
The deploy itself is NOT run -- 4.3 is a gated mutation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-31 |

Build ruling 1: UCA points at the in-DC mirror for 15 apps (scope derived, not guessed)
...
overlays/vr1-dc0-machines.yaml sets
deb http://10.12.8.4/cloud-archive jammy-updates/caracal main
on 15 apps. SCOPE DERIVED FROM THE CHARM SCHEMAS rather than hand-listed: every
app whose charm accepts an origin key AND resolves to a UCA pocket -- 12
openstack-origin (explicit cloud:jammy-caracal or the charm default 'caracal')
plus 3 ceph 'source'. Everything else defaults to 'distro' (Ubuntu archive only)
and needs nothing, which is why the mysql-router subordinates, ceph-rbd-mirror,
mysql-innodb-cluster, glance-simplestreams-sync and rabbitmq-server are absent.
CORRECTION: the earlier in-session figure of 'six apps' was the count that set it
EXPLICITLY and was never the scope. Measured on the containers, both keystone/0
and ceph-mon/1 carry the upstream UCA source.
Also measured: keystone CLEARED install anyway, so the unreachable UCA is not
universally fatal -- it is fatal where a charm's apt_update uses --error-on=any
(the ceph charms), and a CORRECTNESS problem everywhere else, since a node that
cannot reach the Caracal pocket silently gets jammy's own OpenStack instead.
Per-DC by necessity: the mirror address differs per DC, so this cannot live in
bundle.yaml. dc1 gets no equivalent block -- its apt-cacher-ng forwards the
upstream URL transparently, and that asymmetry is D-135's experiment result.
No |key suffix needed: measured, the UCA signing key is already on the nodes and
debmirror preserved the upstream Release/InRelease signatures.
LOGGED NOT CHANGED: ovn-central's charm default is source: zed, not caracal -- it
points at a pocket the dc0 mirror does not carry, and repointing would change its
RELEASE rather than its URL. Separate question.
Gauntlet ALL GREEN (93); repo-lint 0 fail; provider-bundle-check PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |

3.8 fixed per-DC -- and the fix exposed two repo records naming the wrong NIC
...
bridge-interface-mappings is out of bundle.yaml and per-DC in overlays/vr1-dc0-machines.yaml
(new) and vr1-dc1-machines.yaml. The machines overlay was chosen over the vips overlay for a
measured reason: both *-vips.yaml are marked GENERATED / do-not-hand-edit, so a key added
there is clobbered on the next render, and preflight.sh already assembles ${DC}-machines.yaml.
Creating the dc0 file also closes NEW-3 from the other side -- preflight's [ -f ] guard was
silently skipping a file the deploy needs.
THE WORSE DEFECT THE FIX EXPOSED. dc0's compute provider-public MACs are 52:54:00:8c:2a:8c /
52:54:00:50:48:88 -- position [1] in each node's macs list. Audit register row U5 and the
D-124 correction both name the [0] values, which are the PXE/boot MACs, and D-124 further
tells the reader to source them "from lib-hosts.sh" -- verified independently: lib-hosts.sh
declares only HOST_BOOT_MAC and cannot supply a provider MAC at all. Following either record
would have bridged br-ex onto the boot NIC: not a missing mapping that fails loudly, but a
wrong one that comes up and misbehaves. Confirmed three ways -- substrate main.tf:109-116 list
order, the macpin plan capture, and live MAAS showing br-ex over enp2s0. The correct values
and the [0]-vs-[1] trap are now recorded in the overlay, next to the data.
NEW-1: 10.12.12.0/22 was named as the geneve/DATA plane; that is metal-internal, data-tenant
is 10.12.16.0/22. Deliberately not swept repo-wide -- the 10.12.12.5x VIP legs are correct
metal-internal values.
NEW-2 phantom vr0-dc0-testcloud.yaml removed. NEW-4/6 stale octavia-pki name and the phantom
${DC}-hostnames.yaml removed (juju errors on a missing overlay). NEW-7 scale-up removed from
the VIP deploy command per R6, ${DC1_MODEL} -> ${DC_MODEL}.
R5/D-106 BUILT: phase-6 Step 5 rewritten configure-not-deploy, the unsatisfiable diff gate
replaced by a presence-and-state assertion, and the "no designate block exists" claim
corrected against bundle.yaml. os-public-hostname stays unset, per the ruling's refused
option (c).
Verified independently rather than from reported output: provider-bundle-check PASSes for
both DCs in preflight P2's own overlay order. repo-lint 0 fail / 617 files.
Logged not fixed: provider-bundle-check treats an absent bridge-interface-mappings as a silent
skip and never checks the MAC prefix against --dc, so it could not fail on either variant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|