| 2026-08-02 |
as-built comparison: the proven origin config is cloud:jammy-caracal on 6 apps
...
Operator-directed review of past deployment YAML against current values.
Source: asbuilt/20260706-224851/bundle-exported.yaml, a juju export-bundle of
the VR0 testcloud that deployed successfully; five sibling captures agree.
The only configuration this project has ever deployed successfully is
cloud:jammy-caracal on six apps -- barbican/magnum/octavia openstack-origin,
ceph-mon/osd/radosgw source. Zero raw deb lines, zero key: options, anywhere in
any as-built capture. bundle.yaml encodes exactly that and defines the value
ONCE behind YAML anchors.
The 07-31 repoint changed two things at once and neither has ever deployed: app
count 6->15 (sound -- the other nine inherit caracal, same unreachable pocket)
and the value form cloud: -> raw deb. The form change is what dropped the key,
because the cloud: path installs ubuntu-cloud-keyring as a side effect.
Sharpens the D-135 experiment result: the MIRROR arm is what forces divergence
from the only known-good config, at the cost of a failed deploy and standing
key-rotation surface. The PROXY arm needs no override at all.
My own overlay edit violates the base bundle's convention -- it defines origin
once behind an anchor, I wrote fifteen copies of a 29-line key.
Fork left open for a GA-R5 ruling: converge dc0 on the proxy and delete the
origin block, or keep the mirror and restructure behind one anchor. Not choosing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
UCA key goes in the OVERLAY, the tested pattern -- not the mechanism I invented
...
Operator: "Why did you guess again? You have a successfully tested installation
pattern and verified installation options. You have access to the documentation
on this."
Two guesses. The pool path 404 was the visible one; the delivery mechanism was
the one that mattered. I generated per-app YAML into /tmp/ucacfg on the rack and
tried juju config --file -- bespoke, out-of-band, invented. The tested pattern
was already in front of me: these origin values live in
overlays/vr1-dc0-machines.yaml by the 07-31 ruling, and phase-4 Step 7 documents
re-passing the FULL overlay set to apply a delta to a running model. Applying by
juju config would have left the overlay and the model disagreeing and would have
reverted at the next deploy from the overlay -- the same class as DOCFIX-208,
committed two hours after fixing it. /tmp/ucacfg is abandoned.
Vendor docs checked rather than derived: the charm-guide defers exotic-value
syntax to keystone's own config.yaml, and charm-helpers import_key() documents
ASCII armour as correct while calling key-id-via-keyserver insecure --
independent of the airgap making it unreachable.
The overlay now carries the armored key on all 15 values: 12 as the
openstack-origin | suffix, 3 as the ceph key option with source unchanged.
Verified by PARSING the overlay, not reading the diff -- all 15 byte-identical
to docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc, ovn-chassis untouched,
ovn-central still absent. provider-bundle-check PASSES; repo-lint 0 fail.
Apply is the Step-7 pattern and awaits the operator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 deploy blocked: mirrored UCA has no signing key on fresh nodes; root-caused
...
All 9 machines reached started. Nine units then failed the install hook across
unrelated charms -- one common cause:
W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease:
NO_PUBKEY 5EDB1B62EC4926EA
E: The repository is not signed.
apt-get update returned exit 100
The mirrored content is reachable and its upstream signatures are intact; the
key to verify them is absent.
This refutes a claim overlays/vr1-dc0-machines.yaml carried as MEASURED -- that
the UCA key is already on the nodes. Measured on ceph-osd/0: trusted.gpg.d holds
only the cdimage and 2018-archive keyrings. The cloud-archive keyring ships in
the ubuntu-cloud-keyring package, installed as a side effect of the
cloud:<series>-<release> origin path; a raw deb line bypasses it. The claim was
almost certainly measured on a node from the 07-31 deploy, which used
cloud:jammy-caracal and had the keyring. True when taken, false on a fresh node.
Fix is charm-documented, quoted from the charms: openstack charms take
openstack-origin "deb <URL>|<PUB KEY>"; the three ceph charms take a separate
key option accepting ASCII armor. Armored block, never a keyid -- a keyid sends
charm-helpers to keyserver.ubuntu.com, unreachable under the D-107 airgap.
Key provenance measured, not downloaded: the mirror serves ubuntu-cloud-keyring
2021.03.26; URI from apt-get download --print-uris on a node (source package is
ubuntu-keyring, not ubuntu-cloud-keyring -- I guessed and got a 404); sha512
matched apt's declared digest; keyring holds exactly 5EDB1B62EC4926EA. Armored
export committed as public key material.
PREPARED, NOT APPLIED: config for all 15 apps generated from that one verified
key and staged on the rack. juju config --file was refused by the permission
layer in three shapes; it is a live mutation across 15 apps and belongs to the
operator. No workaround attempted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
STAGE 5 dc0: BUNDLE DEPLOYED, exit 0; Step 4.4 passes and proves the options merge
...
Operator: "Approved, continue as autonomously as possible" -- covering the
single-file overlay re-stage and Step 4.3.
Re-stage restored the digest-match invariant (dd914358 both ends). All six
deploy inputs sha256-verified against HEAD in the same window, model confirmed
empty, then the DOCFIX-208 command: "Deploy of bundle completed." EXIT 0.
Step 4.4 ran at its first ever execution and passed all three items. Item 1 is
the consequential one: juju config ovn-chassis shows ALL THREE options with
source: user -- bridge-interface-mappings (exactly two MACs), ovn-bridge-mappings
physnet1:br-ex, prefer-chassis-as-gw true. JUJU MERGES AN OVERLAY'S options MAP
KEY-BY-KEY RATHER THAN REPLACING IT. Both machines overlays have carried that as
"near-certain but NOT PROVEN HERE" since 2026-07-29; no offline check could
settle it and the dry-run prints no options at all. The whole *-vips.yaml
architecture rests on it and it is now measured.
Item 2: all fifteen apps read the in-DC mirror, none the UCA -- the second
injury DOCFIX-208 prevented, confirmed prevented live. Item 3: ovn-central reads
source: zed, its charm default, deliberately left alone.
The gate needed no shape correction; every key was present, so the .get()
sentinel hardening was never exercised.
Also measured: ceph-osd/0..3 already carry GUA IPv6 2602:f3e2:f02:10::150-153,
so the D-139 carve is live on the addresses juju provisions against.
Convergence in progress; target is phase-01's PRE-vault-init end state.
Capture: docs/audit/stage5-deploy-dc0-20260802.txt. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
WITHDRAWN: the designate "contradiction" was mine, not the bundle's
...
I raised, as an unresolved question for the operator, that the Stage-5 plan
deploys four designate applications while a phase-4 bullet says the bundle
ships "NO designate". There was nothing to resolve.
D-019 (v1 ships no cloud DNS) is SUPERSEDED by D-106, which REACTIVATES
Designate for VR1. DOCFIX-167 put all four designate applications into this
bundle.yaml on 2026-07-10, and phase-01-bundle-deploy.md:174 already says the
change "corrects this GATE's old 'NO designate (D-019)' text, since D-019 is
superseded". Stage 7 owns the DNS ACTIVATION -- per-DC zones + A/AAAA per the
D-008 bootstrap order, FQDN-SAN certs, the B5 os-public-hostname reversal --
explicitly not the charm deploy. Four designate applications in the plan is the
planned outcome; their absence would have been the defect.
The stale artifact was the phase-4 bullet alone, paraphrasing a gate text
phase-01 corrected 23 days earlier. Rewritten to state the D-106 reactivation
and the Stage-5/Stage-7 split. My annotation is deleted, not kept as history --
leaving it would mislead the next reader about a live deploy input. Step 4.4's
speculative designate caveat removed for the same reason.
Operator correction quoted in CURRENT-STATE and the changelog: VR0 -> VR1 is
ADDITIVE, and a manufactured contradiction costs next-step attention, which is
this project's named failure mode. Also recognised late: the dry-run's ignored
"name"/"variables" field warnings are the benign R11 pair phase-01 already
documents.
repo-lint 0 fail / 1 legacy warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-208 part 3: graduate the dry-run instrument finding; harden Step 4.4
...
The juju finding belongs above a changelog -- it generalizes to any bundle
deploy, not just phase-4's gate. platform-traps.md gains a Juju section: what
--dry-run prints, what it never prints at any verbosity, the two grep decoys
(controller API addresses look like DC-band VIPs; --debug echoes the command
line so overlay filenames hit), and the three consequences.
Step 4.4, written earlier this session and never run, gets two corrections
before its first execution: the --format=json read used a direct index, so a
missing key would print a traceback into a results column -- this repo has a
logged instance of an assertion satisfied by a traceback -- now .get() with a
visible <<UNSET-OR-APP-ABSENT>> sentinel documented as a failure. And the block
now states plainly that it is authored against a measured instrument but has
never been exercised, with the instruction to report a shape correction rather
than work around one.
repo-lint 0 fail / 1 legacy warn; gauntlet ALL GREEN (98).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-208 part 2: the dry-run graded the gate, and three items could not fail
...
The corrected dc0 command dry-ran EXIT 0 from the dc0 rack against an empty
vr1-dc0 model, staged input sha256-verified against HEAD first. Nine machines
0-8, all nine constraints tags=openstack-vr1-dc0,<role> in the 3/2/4 split,
all three overlays consumed. Capture: docs/audit/stage5-dryrun-dc0-20260802.txt.
The run also measured what the command can and cannot show. juju deploy
--dry-run on a bundle prints NO application options and NO VIPs at any
verbosity; --debug adds only the per-machine constraint lines. Greps over the
--debug capture: bridge-interface-mappings 0, physnet1 0, openstack-origin 0,
cloud-archive 0, prefer-chassis 0.
So three Step-4.2 gate items could not be graded by the command the gate names:
item 1 (tags) needed --debug, which is now in the named command; the
PRE-EXISTING VIP item could never have failed because no VIP is ever in the
plan, and its property is relocated in writing to preflight P2; and the two
option-reading items I added earlier this session had the same defect and are
removed within the hour. Their properties become new Step 4.4 -- a juju config
read run the moment deploy returns, which is the only thing that can prove
juju merged the options map rather than replacing it.
Both machines overlays' VERIFY-LIVE headers said to assert this "at the live
dry-run". Measured impossible; corrected in place, pointed at 4.4.
Also settles the deferred ceph-osd tags=openstack item (placed by explicit
placement, bare tag absent from the plan) and logs a contradiction not resolved
here: the runbook says the bundle ships NO designate, the plan deploys four
designate applications.
The deploy itself is NOT run -- 4.3 is a gated mutation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-208: the dc0 deploy command was missing a load-bearing overlay
...
The phase-4 runbook told the operator to deploy vr1-dc0 WITHOUT
overlays/vr1-dc0-machines.yaml, on the stated grounds that the file does
not exist. It has existed since 2026-07-29. preflight.sh P2 folds it into
the merged input it validates, so P2 was grading an input the deploy would
not have passed.
Re-measured by deep-merging bundle.yaml against the overlay: SIXTEEN
applications carry a delta, not the one this repo recorded on 2026-07-31.
ovn-chassis.bridge-interface-mappings (whose ONLY source is this overlay --
bundle.yaml:494 forbids re-adding it there) plus 15 mirror repoints from
the 2026-07-31 UCA ruling. Deploying as written would have taken two
injuries, neither surfacing as a deploy error: no br-ex mapping (dead
provider egress) and 15 charms pointed at a UCA measured unreachable from
a node under the D-107 airgap.
Corrected at 6 sites (4.1 table + prose, 4.2 dry-run, 4.3 deploy, Step 7
dry-run + its rules sentence, 4.2 VERIFY-LIVE pointer). Step 4.2's gate
could not have caught any of it -- its four items read the machines block,
machine count and VIPs, while the overlay's whole payload is under
applications: -- so gate items 5 and 6 were added to assert the merged
ovn-chassis options map and the reachability of every origin/source.
preflight.sh needed no change: it was already right.
CURRENT-STATE corrects its own narrower 2026-07-31 finding in place
(GA-R1 C2). repo-lint 0 fail / 1 legacy warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
