| 2026-08-01 |
The "juju client blocker" is NOT one -- it is D-138 working correctly
...
CORRECTS this repo's own 2026-07-31 finding 9d, which called it a blocker.
Stage 5 can reach `add-model` TODAY, from the dc0 rack. No key movement needed.
VERIFIED INDEPENDENTLY from the main session, not accepted on the agent's word:
juju 3.6.27-genericlinux-amd64 at /snap/bin/juju on the rack; controller
vr1-dc0-controller* admin/superuser on cloud vr1-maas; the controller model reads
"Last connection: just now"; client credentials list EXACTLY ONE entry,
vr1-maas -> vr1-dc0-cred.
ROOT CAUSE OF THE ssh REFUSAL, MEASURED ON THE MACHINE: the alias is correct in
every part -- hop chain, host, port, per-hop identity, host key. subtle-grouse's
authorized_keys holds exactly two lines, both Juju's, so the office1_svc key the
alias offers is genuinely absent and the refusal is right. The obvious hypothesis
was REFUTED rather than assumed: the machine booted AFTER the 2026-07-30 key
import and still got only Juju keys, because deploy-time cloud-config carries
Juju:juju-client-key alone. `juju ssh -m controller 0` already works.
SEC-026 control (1) DISCHARGED BY MEASUREMENT on both sides -- rack lists one
credential, voffice1 lists two, so the forbidden whole-store copy did not happen.
No new credential residency; no new security-ledger row owed.
ONLY HYGIENE OWED, NOT EXECUTED (hard rule 3): a dangling current-model pointing
at the model destroyed 2026-07-31 makes bare `juju status` error. Measured NOT to
be an add-model precondition.
FOUR GAPS LOGGED NOT FIXED, in consequence order:
G4 the `openstack` CLI is MISSING ON THE RACK -- D-138 definition-of-done gap
blocking phase-03+. SAME item as F1 from the 2026-07-30 sweep, now confirmed
on the D-138 host. It has survived two sweeps.
G3 preflight CANNOT PASS on vcloud by construction: the octavia-pki overlay is
gitignored PKI material absent from the vcloud tree, and
pre-flight-checks.sh:161 hard-fails on it. The host-dependence class again.
G2 the D-138 client host's deploy input has NO PROVENANCE -- ~/repo-stage on the
rack is a hand-staged 9-file copy with no .git (8/9 byte-identical by sha256).
A third copy beside the two clones verified at c58bf95.
G1 phase-4 runbook :379,398 still say "voffice1" for add-model/spaces, stale vs
D-138; voffice1 has the binary but no registered controller, so the runbook as
written fails. DOCFIX owed.
DECLARED UNMEASURED with the reason: whether MAAS user juju-vr1-dc0 has an
imported ssh key -- no usable maas CLI profile exists anywhere (voffice1's
~/.maascli.db is ZERO BYTES). Does not change the fix; the proximate cause was
measured directly. The PreToolUse guard refused a command that would print the
MAAS API key (DOCFIX-016) and it was NOT retried in an altered shape.
Staged by explicit path; agent files still in flight are deliberately excluded.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 ruling B RECONSIDERED and CONFIRMED; refuted rationale STRUCK in place
...
Operator, exact utterance: "Deciding to hold to gua does not cost anything
operationally and the case for ULA over gua is not very strong. Let's stay with
gua". Recorded as RULING NOTE 2026-08-01 -- D-139.
RULED: ruling B STANDS, unchanged in substance. The GUA carve table is confirmed,
the ULA /48 stays RETIRED for VR1, the D-101/D-111 amendments hold, and Phase-2
execution of D-139 is UNBLOCKED.
WHAT CHANGED IS THE RATIONALE OF RECORD. The RFC 6724 precedence argument is
STRUCK IN PLACE in D-139 -- quoted, with the measurement that kills it and a
pointer to the capture, marked DO NOT CITE IT. Struck rather than deleted on
purpose: deleting it would leave a future session free to re-derive an argument
that has already been considered and refuted. The block also warns against the
near-miss rescue -- glibc 2.35 DOES carry fc00::/7 in default_labels[] (label 6),
which drives SOURCE-selection rules 5/6, a different mechanism from destination
precedence; finding it does not resurrect the struck claim.
Ruling B now rests on two project-constraint arguments -- conformance (Willamette
and VR0 DC0 are already full GUA; VR1 was the outlier) and MINIMIZE DELTA TO
ROOSEVELT -- plus the operator's recorded reasoning that GUA costs nothing
operationally. MADE EXPLICIT because it was not visible before: on glibc 2.35 GUA
buys NO address-selection advantage on the two dual-stack planes; ULA would have
won equally at 40. Anyone later reasoning "GUA was chosen so v6 would beat v4" is
reasoning from the struck argument.
NEW STANDING RULE, recorded on D-139 because it generalises: the repo's citation
rule (open it, check STATUS and DATES) was FOLLOWED here and was NOT ENOUGH. The
citation was real, current, correctly quoted and correctly understood, and still
wrong, because nobody checked whether the IMPLEMENTATION follows the standard.
For a STANDARDS citation, add a third check -- confirm the deployed software
implements it, at the deployed version. An RFC is not a description of your system.
Staged by explicit path; an agent's in-flight files (netbox/d139-gua-carve.py,
tests/d139-gua-carve/, docs/audit/d139-carve-dryrun-20260801.txt) are deliberately
NOT included. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 ruling B's deciding reason REFUTED by measurement; ruling FLAGGED
...
Closing the unverified premise D-139 named at ruling time inverted its own
argument. Recorded, flagged for operator reconsideration, NOT amended.
MEASURED, two steps:
1. jammy libc-bin 2.35-0ubuntu3 ships /etc/gai.conf with ZERO active lines --
every line commented. The node image applies NO override, so glibc's
compiled-in table governs.
2. glibc 2.35 default_precedence[], fetched in full from sourceware at tag
glibc-2.35 and read rather than recalled, carries glibc's OWN comment
"See RFC 3484 for the details" and is:
::1/128 50 | 2002::/16 30 | ::/96 20 | ::ffff:0:0/96 10 | ::/0 40
THERE IS NO fc00::/7 ENTRY IN THE PRECEDENCE TABLE.
CONSEQUENCE: ULA falls through to ::/0 = 40, identical to GUA; IPv4-mapped is
10. So on the node image ULA = GUA = 40 and BOTH outrank IPv4. Ruling B was
declared on "ULA loses to IPv4 (3 vs 35), GUA wins (40 vs 35)" -- the RFC 6724
table. RFC 6724 obsoletes RFC 3484 and does define those values, but glibc 2.35
DOES NOT IMPLEMENT IT. Right about the RFC, wrong about the software this cloud
runs. The fc00::/7 entry that does exist is in default_labels[] (label 6), which
drives SOURCE selection rules 5/6 -- a different mechanism, recorded so the two
are not later conflated into "the argument stands after all".
A NEW VARIANT OF THE CITATION CLASS, worth naming: the citation was real,
current and correctly quoted, and still wrong here because nobody checked
whether the IMPLEMENTATION follows the RFC. Checking a source's status and dates
is not sufficient for a standards citation.
SURVIVING REASONS (independent of the refuted one): conformance with Willamette
(a real site) and VR0 DC0, both full GUA; and MINIMIZE DELTA TO ROOSEVELT, whose
2602:f3e2:103::/48 is GUA with no ULA and no plane carve yet. The ruling may well
stand on these -- but under GA-R5 that is the operator's call, since they
answered a question whose stated deciding reason no longer holds.
D-139's text is UNCHANGED. Nothing of D-139 is executed, so nothing is half-built
in either direction. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 ADOPTED: VR1 goes IPv6-only east-west, whole carve moves to GUA
...
TWO GA-R5 rulings, taken in SEPARATE exchanges, recorded separately.
A -- family matrix. Operator: "IPv6 on all planes except for metal-admin and
provider-public which will remain dual stack". metal-internal, data-tenant,
storage, replication and lb-mgmt become IPv6-ONLY. Amends D-101's matrix twice:
metal-internal's "datastore east-west stays v4-bound" is superseded, and lb-mgmt
(ruled v6-only by D-101 but NEVER carved anywhere) becomes a first-class plane.
B -- addressing model. Operator: "Full GUA on every plane (Recommended)". Every
plane carves from its DC's GUA /48 on the :10/:11/:20/:21/:30/:40/:50/:80 octet
map VR0 DC0 and the Willamette site ALREADY use -- conforming to an existing org
standard, not inventing one. The ULA /48 fd50:840e:74e2::/48 is RETIRED for VR1.
Amends D-101 and D-111. Deciding reason is measurable: RFC 6724 ranks IPv4-mapped
at precedence 35 and ULA at 3 while GUA falls under ::/0 at 40, so on a dual-stack
plane a ULA leg LOSES address selection to IPv4 and is decorative. SEC-010, D-052,
D-125 and D-107 unchanged -- containment lives at the forwarding layer.
THE RECORDED ROOT CAUSE OF THE v4-ONLY CONTAINERS WAS WRONG AND THIS REPO CARRIED
IT (GA-R1 C2 -- measurement corrects the document). CURRENT-STATE said MAAS "has
nothing to give for v6". Measured, and re-verified independently: subnet
statistics on fd50:840e:74e2:220::/64 returns available_string "100%",
num_available 18446744069414584320. Zero ipranges rows means zero RESTRICTIONS,
not zero availability. dc-plane-ipam.sh:368-372 has carried the correct behaviour
since 2026-07-27, in the tree, contradicting the authoritative doc the whole time.
The real mechanism is juju-side and there is NO knob: EthernetDeviceForBridge
(tag v3.6.27) takes addrs[0] from an UNSORTED query and derives one CIDR -> one
LinkSubnetArgs. gomaasapi documents "Any number of STATIC links can exist on an
interface", so MAAS would accept both families; the limit is juju's. LP #1723240
is this exact symptom, Triaged/Low since 2017, and the theory this repo recorded
is the one its reporter rebutted in-thread. Consequence: on a container plane
DUAL-STACK IS NOT EXPRESSIBLE while v6-only IS -- ruling A is the achievable
configuration, not merely the desired one.
SNAP PROXY: dc0 INSTALLED, RUNNING, check dc0 PASS exit 0 (16 assertions,
re-verified independently). A REAL SNAP PAYLOAD WAS FETCHED THROUGH IT -- HTTP 206,
first bytes hsqs. No longer fixture-green. BUG-3 confirmed by measurement, no
assertion changed. BUG-4 fixed at BOTH sites (the review named one); T20b/T41b NEW.
OWNED: my BUG-1 fix was WRONG. ip route get 10.12.8.4 -> src 10.12.8.2; .4 is a
SECONDARY and Linux never auto-selects a secondary as a source, so the rule is
still dead. Not fixed now -- editing the generator turns the LIVE gate red until
install re-runs. T23b annotated so nobody "fixes" the test to a wrong value.
Generalisable: any ACL keyed to a service ALIAS (the whole D-134 utility .4 band)
misses that host's own traffic -- same exposure in dc-mirror.sh, dc-cache-proxy.sh.
OWED, unverified: the deployed jammy /etc/gai.conf, which can override the RFC 6724
default table that ruling B rests on. Nothing of D-139 is executed.
repo-lint 0 fail; tests/dc-snap-proxy 55/55; 4 mutations killed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-31 |
SESSION CLOSE 2026-07-31 (GA-R4 bookend): Stage 5 dc0 deploy + teardown
...
Bounded ledger summary (10 lines, cap 15), rotation, sweep, memory review.
DURABILITY: 0 uncommitted / 0 unpushed, 45 commits today. dc0 rack repo-stage
digests all MATCH. HAZARD, not loss: the voffice1 clone is 26 commits behind --
pull it forward before it is used.
GATES: repo-lint 0 fail / 1 standing warn / 641 files. Gauntlet ALL GREEN (94).
ledger-scan: 3 open decisions, SEC 26, D 139 / DOCFIX 207 / BUNDLEFIX 053.
Ledger 296 lines after rotating the oldest summary out to
docs/archive/session-ledger-rotated-20260731.md.
SWEEP: docs/audit/queued-findings-20260731-stage5-deploy.txt -- ELEVEN FIRST
SURFACE items that existed only in the transcript. Highest-consequence is F1:
MAAS's own squid ALREADY CONNECT-proxies api.snapcraft.io and would unblock the
deploy today with nothing built, measured from the failing container -- RULED
AGAINST, with the operator's verbatim utterance and the deciding downsides
recorded, precisely because a future session will rediscover it and reach for
it. Also first surface: containers source from METAL-ADMIN not the address juju
status displays; ports 3128 and 8000 already held by MAAS's squid on both racks;
the four reviewed bugs in dc-snap-proxy.sh; the teardown instrument error; and
that a MAAS hostname rename on a Deployed machine is record-only.
Method note: a phrase-exact grep produced false negatives on items recorded under
different wording. Every candidate was re-grepped on distinctive tokens before
being classified -- the same instrument-error class this sweep documents.
GA-R7 MEMORY REVIEW -- one real violation found and corrected: the
multi-workstation memory asserted 'never additions to allow for mutations', an
operator-posture claim memory may not hold, contradicted by the recorded
2026-07-30 ruling 'Add it to allow'. Re-pointed to an observation with the
contradiction recorded. instrument-currency memory gained this session's two
misreads.
OWNED: a red-lint push (| tail masked the exit code); a git add -A that swept an
agent's files into an unrelated commit; two instrument misreads (a juju models
summary read as progress, stale unit logs read as current); and a wrong
'immutable system_id' claim the operator corrected.
NEXT: apply the dc-snap-proxy review cuts, RE-RUN the mutation pass (S8
invalidates M32-M39/M42-M47/M49-M52 as evidence), confirm BUG-3's dstdomain deny
shape once at apply time, install the proxy, then add-model + spaces + deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Snap proxy tool BUILT and REVIEWED (not yet applied); 4 real bugs logged
...
Ruling 2's repo side. squid 6.14 as a dedicated instance on each DC rack at the
D-134 utility .4:3129, CONNECT-only, restricted to Canonical's documented store
and CDN hosts, consumed via juju snap-http-proxy/snap-https-proxy. Design cited
to vendor sources, not preference: every snap endpoint is :443 so a CONNECT
proxy is mandatory and CACHING IS IMPOSSIBLE (squid cannot read a CONNECT
tunnel), which removes the apt-style caching argument entirely; squid is in
main, tinyproxy is universe with six unevaluated CVEs and defaults to allowing
all CONNECT ports.
MEASUREMENTS THAT CHANGED THE DESIGN:
- the failing apps are LXD containers sourcing from METAL-ADMIN (10.12.8.122),
NOT the 10.12.12.116 juju status displays. An ACL on the displayed address
would have denied every client the proxy exists for.
- ports 3128 AND 8000 are already held wildcard-bound by MAAS's own squid on
both racks -- hence 3129 and a dedicated instance.
- no new D-134 octet needed: .4 is already aliased on both racks.
- LOGGED NOT ADOPTED: MAAS's squid ALREADY CONNECT-proxies api.snapcraft.io,
measured working from the failing container. snap-https-proxy pointed at
10.12.8.6:8000 may unblock the deploy today with nothing built. Rejected as
the shipped design (per-revision path, NO destination restriction, the
hidden-coupling shape D-135 already has a scar from) but it is a real
operator option, not a dismissal.
QUALITY: 53 mutations, 53 killed, 0 survived -- including the dc-mirror.sh
existence-vs-content false-green class reproduced and caught. One survivor was
a MALFORMED mutation that broke bash -n and therefore tested nothing; recorded
as malformed and redone rather than counted. Three of the harness's own cases
were found too weak and were STRENGTHENED, not left green.
INDEPENDENT REVIEW: ruled design HONOURED on all four counts -- utility band, no
direct node egress (in fact stronger: destination-allowlisted, and that scope is
flagged as a build-time choice NOT a ruling), site-keyed both DCs with the dc1
limit stated honestly, and it OWNS its own network prerequisites rather than
borrowing another service's -- with the coupling anti-pattern now under test
(M09/M10) rather than only under comment.
FOUR REAL BUGS LOGGED, NOT FIXED (hard rule 1 -- the file was frozen for review):
BUG-1 dead snap_probe ACL; the probes are actually admitted by snap_clients,
and the comment misdescribes its own mechanism
BUG-2 | grep -q under pipefail -- INHERITED from dc-mirror.sh and
dc-cache-proxy.sh alike; a three-script sweep, not a local fix
BUG-3 the deny probe's 403 shape was measured from a PORT deny, never a
DSTDOMAIN deny. If squid answers differently the gate REFUSES
PERMANENTLY -- and a gate that cannot PASS blocks a GA-R6 close as hard
as one that cannot fail. Requires a one-time apply-time confirmation.
BUG-4 dpkg -s reports a deinstall-ok-config-files package as present
SIZE: 636 lines / 320 code vs dc-mirror 385/227 and dc-cache-proxy 396/200. The
reviewer's landing zone is ~535/~250 and it RECOMMENDS AGAINST forcing 385 --
two extra generators (75 lines) are measured-forced because the packaged squid
cannot be repointed, and further cuts would delete mutation-proven assertions.
~50 lines of the excess is header prose, which carries zero bug risk.
TWO BLOCKERS BEFORE THE PLAN IS APPLIED: the S8 consolidation invalidates
M32-M39/M42-M47/M49-M52 as evidence, so the mutation pass MUST be re-run; and
BUG-3 must be confirmed once against the built proxy before check is cited to
close anything.
NOTHING IS PROVEN END TO END -- no proxy installed, no snap fetched through one.
The harness green is FIXTURE green and says so.
repo-lint 0 fail; gauntlet ALL GREEN (94).
OWNED: my earlier teardown commit e57ad09 used 'git add -A' while the agent was
writing and swept its audit capture and CURRENT-STATE paragraph into a commit
about the model teardown. This commit is staged EXPLICITLY by path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Teardown: stalled on stopped agents, forced clear, no nodes stranded
...
Plain destroy-model STALLED and would not self-resolve -- 'attempt 30 ... model
not empty, found 26 machines, 37 applications', flat ~19 min with the app set
byte-identical across a 12-minute name-level diff.
MECHANISM MEASURED, and it makes the stall terminal rather than slow: ALL 26
machine/container agents were 'stopped', so NO hook could execute. Units already
in error from the snap failures could never run their teardown hooks.
MAAS was NOT the bottleneck -- the six nodes juju released went to Ready/owner
None in minutes; juju never issued a release for the other three.
destroy-model --force --no-wait cleared it (18 -> 5 -> 2 machines, then 'Model
destroyed.'), and NO NODES WERE STRANDED: all nine read back Ready/owner=None,
so no maas machine release was needed or run. subtle-grouse correctly stays
Deployed -- it is the D-104 controller VM in the controller model.
INSTRUMENT ERROR, OWNED: I reported '0 machines / 0 units' from the juju models
SUMMARY COLUMNS while juju status -m read 26 machines / 37 applications. The
summary zeroes during 'destroying' and is not a progress signal. The real tell --
three control nodes stuck Deployed -- was visible and I explained it away.
Standing lesson: during a teardown, juju status -m <model> is the instrument;
juju models counts are not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Set apt-mirror as a controller model-default; record the durability gap
...
Operator: 'Yes, set the defaults'.
The teardown demonstrated the gap: destroy-model takes the model config with it,
so Step 3.5's apt-mirror vanished and NOTHING in the repo would have caught it --
the next deploy would have failed on package fetches and read like a mirror fault
rather than a missing model setting. That is the 'prose cannot close a stage'
class: Step 3.5 is runbook prose with no gate.
MEASURED: juju model-defaults carries apt-mirror, snap-https-proxy,
snap-store-proxy and snap-store-proxy-url, all previously unset at Controller
level. Set and read back: apt-mirror -> Controller http://10.12.8.4/ubuntu.
Defaults are inherited by NEW models only, which suits the pending add-model.
Per-DC by construction: D-104 gives each DC its own Juju controller, and these
values are per-DC, so per-controller defaults map onto per-DC values with no
cross-DC coupling.
THE SNAP KEY IS DELIBERATELY NOT SET YET -- the proxy does not exist, and
pointing snap-https-proxy at a dead address would make snap installs fail WORSE
than they do now. It lands with the proxy.
OWED: a site-keyed dc-model-defaults.sh with a check, so the values are VERIFIED
at every DC standup rather than remembered. Roosevelt shape: fold into D-136's
per-DC render pipeline. DC1 lesson: set model-defaults at controller bootstrap,
before its first add-model.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Node-naming canary: a MAAS rename alone is cosmetic-only
...
Operator asked for the juju status Inst id values to become role-based.
CORRECTION to an earlier claim this session: that column shows the MAAS
HOSTNAME, not the system_id. My first reading came from the JSON instance-id
field (677cta) and was wrong -- the displayed value is the hostname and it IS
renameable.
Canary: civil-bug resolved BY PINNED BOOT MAC (52:54:00:2b:ed:ab) to the ruled
vr1-dc0-storage-04, then renamed.
MEASURED: MAAS ACCEPTED the rename on a Deployed machine (fqdn
vr1-dc0-storage-04.maas), but juju status still shows civil-bug AND the running
OS still answers civil-bug (hostname and hostnamectl --static). MAAS's record
moves and nothing else does -- juju captured the name at provisioning, and the
OS hostname is applied by cloud-init at DEPLOY time.
CONSEQUENCE: renaming the other eight buys nothing where the operator is looking
and leaves a three-way divergence. Nothing functional rides on it -- the bundle
places by TAG and every gate resolves by PINNED BOOT MAC.
THE CLEAN POINT IS ENLISTMENT: folds into the DC1 standup as a DoD item (set the
ruled hostname BEFORE commissioning so MAAS, the OS and the ruled name agree),
and reaches DC0 at its next node redeploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Ruling 1 applied live: only the snap class remains
...
Overlay re-staged hash-verified (e3be85e4); the in-DC UCA origin set on all 15
apps and read back on a sample of each kind.
MEASURED AFTER: units in error are EXACTLY the three snap-dependent applications
-- mysql-innodb-cluster (x3), ovn-central (x3), vault -- 7 units. Every ceph and
OpenStack API app has cleared. State: 14 waiting, 5 maintenance, 4 blocked,
3 active, 7 error.
The deploy is now blocked on exactly ONE unbuilt thing: the snap path (ruling 2),
the D-135 items 2-3 gap.
Session progression, all measured: 22 error -> 17 (backports) -> 4 (prefer-ipv6)
-> 7 units in the single snap class. The count rose at the last step because more
units REACHED the snap stage, not because more broke.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Build ruling 1: UCA points at the in-DC mirror for 15 apps (scope derived, not guessed)
...
overlays/vr1-dc0-machines.yaml sets
deb http://10.12.8.4/cloud-archive jammy-updates/caracal main
on 15 apps. SCOPE DERIVED FROM THE CHARM SCHEMAS rather than hand-listed: every
app whose charm accepts an origin key AND resolves to a UCA pocket -- 12
openstack-origin (explicit cloud:jammy-caracal or the charm default 'caracal')
plus 3 ceph 'source'. Everything else defaults to 'distro' (Ubuntu archive only)
and needs nothing, which is why the mysql-router subordinates, ceph-rbd-mirror,
mysql-innodb-cluster, glance-simplestreams-sync and rabbitmq-server are absent.
CORRECTION: the earlier in-session figure of 'six apps' was the count that set it
EXPLICITLY and was never the scope. Measured on the containers, both keystone/0
and ceph-mon/1 carry the upstream UCA source.
Also measured: keystone CLEARED install anyway, so the unreachable UCA is not
universally fatal -- it is fatal where a charm's apt_update uses --error-on=any
(the ceph charms), and a CORRECTNESS problem everywhere else, since a node that
cannot reach the Caracal pocket silently gets jammy's own OpenStack instead.
Per-DC by necessity: the mirror address differs per DC, so this cannot live in
bundle.yaml. dc1 gets no equivalent block -- its apt-cacher-ng forwards the
upstream URL transparently, and that asymmetry is D-135's experiment result.
No |key suffix needed: measured, the UCA signing key is already on the nodes and
debmirror preserved the upstream Release/InRelease signatures.
LOGGED NOT CHANGED: ovn-central's charm default is source: zed, not caracal -- it
points at a pocket the dc0 mirror does not carry, and repointing would change its
RELEASE rather than its URL. Separate question.
Gauntlet ALL GREEN (93); repo-lint 0 fail; provider-bundle-check PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
RULED 2026-07-31 (GA-R5) x2: UCA points in-DC; snaps get an in-DC forward proxy
...
Presented together at operator direction ('Yes, both') and answered SEPARATELY,
so neither is a batch adoption. Both OPS under GA-R3; D-107 UNAMENDED.
RULING 1 -- UCA. Operator utterance: 'Point origin/source at the mirrored UCA,
per-DC overlay (Recommended)'. dc0 gets an explicit deb line at
http://10.12.8.4/cloud-archive in the per-DC hand-maintained overlay; the mirror
address is per-DC so it cannot live in bundle.yaml. dc1 UNCHANGED -- its
apt-cacher-ng forwards the upstream URL transparently, and that asymmetry is
D-135's experiment RESULT, not a defect. Measured precondition: the UCA signing
key is already on the nodes, so a raw deb line verifies with no |key suffix.
RULING 2 -- SNAPS. Operator utterance: 'HTTP(S) forward proxy in the DC utility
band + juju snap-https-proxy (Recommended)'. D-107's core statement stays TRUE --
nodes reach an in-DC proxy, not the internet. Closes the D-135 items 2-3 gap for
BOTH DCs with one mechanism rather than widening the mirror-vs-proxy asymmetry.
Placement is build-time engineering; if it takes its own VM the D-134 octet map
needs a ruled octet first, since that map is a standing cross-DC standard.
Committed and pushed BEFORE the dependent work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Ruling (b) applied live: error 17 -> 4; remaining blocker is snap access
...
Applied live after the repo build: overlay re-staged hash-verified, the seven set
to false and each read back, every erroring unit resolved. MEASURED: units in
error went 17 -> 4 and all seven prefer-ipv6 apps CLEARED.
THE REMAINING BLOCKER IS ONE CLASS -- snap installs from an airgapped node, and
it is the already-recorded D-135 items 2-3 gap hitting for real:
mysql-innodb-cluster snap install mysql-shell (api.snapcraft.io unreachable)
ovn-central snap install prometheus-ovn-exporter
vault snap install core
This document already records the shape: three artifact classes, only ONE local
-- apt to 10.12.8.4 (local), MAAS boot images to images.maas.io (not local),
juju agent stream and snaps to streams.canonical.com / api.snapcraft.io (not
local). It will hit dc1 EQUALLY, since apt-cacher-ng proxies apt and not snaps,
so unlike the UCA finding this is NOT a full-mirror-only asymmetry.
LOGGED NOT FIXED: a D-107 (airgap) / D-135 (items 2-3) decision, not an
engineering choice.
D2 (the upstream UCA) is still unfixed and still owed -- it stopped being the
visible error only because those units now block earlier on snaps.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Fix the red-lint push: D-101 heading collision + note placement
...
OWNED: the previous commit was pushed with repo-lint RED. 'repo-lint | tail -2 &&'
masks the lint exit code with tail's, so the && proceeded on a FAIL -- the same
silenced-pipeline trap already recorded in this repo (| grep -q under pipefail;
git pull -q &&).
Two defects, both fixed here:
- the note used a '### D-NNN --' heading, which L5 reads as a second DEFINITION
of D-101 (collision). Re-titled to the established
'### RULING NOTE <date> -- D-101:' form the other four notes use.
- it was appended after D-136 instead of beside the other D-101 notes. Moved to
sit immediately after note (a).
Also recorded: repo-lint exits 2 on WARN and 0 only when fully clean, so 'exit 0'
is the WRONG success predicate for this repo -- its standing state carries one
warn (the legacy D-001..018 non-ASCII carve-out). Gate on '0 fail'.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
RULED 2026-07-31 (GA-R5): prefer-ipv6 set on NO application until IPv6 is operational
...
Operator utterance, verbatim: 'Set it false on the seven, keep every v6 VIP leg
(Recommended)'. Standing context from the same exchange: 'We have DC1 to stand
up with the IPv6 configuration changes. Lets continue with the IPv4/6 stand up
on DC0. We will fold in all lessons learned from the DC0 stand up into the DC1
stand up.'
Recorded as a D-101 RULING NOTE 2026-07-31 (b). OPS under GA-R3, no D-number;
D-101's matrix is UNAMENDED and every dual-family VIP is retained. It supersedes
the emission half of note (a) and now covers all thirteen apps.
Measured cause: get_relation_ip() returns early with get_ipv6_addr()[0] when the
option is true, and the LXD containers hold ONLY a link-local v6 while the HOSTS
are fully dual-stacked.
Why this is not abandoning v6: the option is a unit ADDRESS-FAMILY switch, not a
listener switch. HAProxy's :::port bind is gated on the kernel sysctl and
pacemaker selects IPv6addr by family detection. Neither consults it.
HONEST RESIDUAL, recorded not glossed: pacemaker will place a v6 VIP on a
container whose eth0 has no global v6, and whether that leg is ROUTABLE is
UNVERIFIED. Not a regression -- true today -- but it is what the v6 completion
work must close.
GATE CONSEQUENCE: invariant 9b coupled the option to the v6 legs for declaring
charms and would now FAIL. It is RE-POINTED to the new invariant and re-proven,
never deleted; when IPv6 becomes operational it returns to its coupling form and
this note records why it left.
Committed and pushed BEFORE the dependent work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
v6 sweep: the IPv6 half is carved as addresses but never made operational
...
Operator: 'The dual stack was only a safety net instead of jumping straight into
a ipv6 only deployment but it appears that more items were not configured with
IPv6 like they should have been.' So IPv6 is the target and v4 the fallback,
matching D-101's own 'v6 wherever possible, v4 only where forced'. A systematic
dc0 sweep was run rather than fixing the container layer alone.
BUILT on v6: node statics (54 links), the six v6 plane subnets, juju's per-plane
host bridges (all six carry global v6), spaces in both families, the dual-family
VIPs, Octavia PKI v6 IP SANs.
ABSENT on v6, all measured:
(i) ALL SIX v6 plane subnets carry ZERO ip ranges, against every v4 plane
holding its D-134 reserved bands and metal-admin also holding dynamic
.201-.254. Uniform, not a metal-admin quirk.
(ii) THE RACK HAS NO GLOBAL v6 AT ALL -- so the mirror, the node-DNS forwarder
and the MAAS rack agent are v4-only by construction. F2 from 2026-07-30,
now measured as the whole rack rather than one plane.
(iii) The mirror does not answer over v6 (000), following from (ii).
(iv) Nodes have no v6 default route -- no RA, no gateway.
CONSEQUENCE: completing IPv6 to the charm layer is a PROJECT, not a fix. A range
alone would give containers v6 addresses with no v6 default route and no
v6-reachable services, so prefer-ipv6 would still not produce a working cloud.
The safety net is doing what it was put there for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D3 root cause: dual-stack is RULED but was never built to the container layer
...
Operator: "The dual stack configuration was supposed to have included charms. We
have had conversations and decisions were made to approval the dual stack
configuration all the way down." CHECKED AGAINST THE RECORD, not accepted: the
operator is CORRECT. D-101's 2026-07-25 note carries "Dual stack deployment for
DC0 and DC1" verbatim and R2 re-confirmed it. A grep for any decision text
covering the LXD CONTAINER layer returns NOTHING.
MEASURED MECHANISM: the host is fully dual-stacked -- juju built a bridge per
plane on machine 0, every one with a global v6. What is missing is anything to
ALLOCATE: 10.12.8.0/22 has a dynamic range .201-.254 plus the D-134 reserved
bands; fd50:840e:74e2:220::/64 has NO IP RANGES AT ALL. juju asks MAAS for a
container address on the bound space, gets v4, and there is nothing for v6 -- so
every container is v4-only on a dual-stacked host.
THIS INVERTS THE FIX DIRECTION AND MY EARLIER FRAMING WAS WRONG. Dropping
prefer-ipv6 from the seven would make the deploy green by ABANDONING a ruled
posture at the charm layer, and would leave the dual-family v6 VIPs on
containers with no v6 leg.
STATED AND UNVERIFIED: whether juju REQUESTS a v6 address for a container when
an allocatable v6 range exists. Adding a range is necessary; it is NOT proven
sufficient, and that must be measured before any range is created.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Three post-deploy defects; memcached scaled; ceph-rbd-mirror PINNED (item 22)
...
D1 jammy-backports: RULED + FIXED. Verified on CONTENT (Release fields + a real
.deb fetch), mirror 951G -> 952G. Units moved 22 error -> 17 with 15 apps past
install (was 1), so the fix works and auto-retry is running.
D2 the UPSTREAM UCA is unreachable from nodes -- LOGGED NOT FIXED. Measured FROM
A NODE, not the rack: upstream 000, mirrored 200 with real content. SIX apps set
it across TWO option names (openstack-origin on barbican/magnum/octavia, source
on ceph-mon/ceph-osd/ceph-radosgw); every other charm defaults to
openstack-origin: caracal, same upstream pocket. The UCA signing key is already
on the node, so a raw deb line verifies with no |key suffix. Genuine D-135
experiment result: the full-mirror DC must rewrite every non-Ubuntu source; the
proxy DC needs none. dc1 will not hit this.
D3 prefer-ipv6: true is FATAL on the seven charms that declare it, and is now
the blocker. This is the exact risk flagged earlier today as "never observed
live" and deliberately left OUT of the D-101 ruling note. keystone/0:
"Interface 'eth0' does not have a scope global non-temporary ipv6 address" --
get_relation_ip() returns early with get_ipv6_addr() when the option is true.
Measured on the container: eth0 holds only fe80::/64. The NODES are dual-stacked;
the LXD CONTAINERS are not. All seven true-valued apps are erroring. NOT TAKEN --
R2/D-101 are ruled surfaces and this needs its own GA-R5 exchange. One question
is stated and NOT answered: whether a v6 VIP on a container with no global v6 is
routable.
memcached scaled 1 -> 3 in dc-ha-scaleup (operator-directed); its exclusion
comment re-pointed rather than left stale. ceph-rbd-mirror PINNED as gap
register item 22 with four options and a recommendation -- (d) add the missing
liveness detection now, (b) scale to 2 active/standby at Roosevelt -- and the
overlay now forbids a silent scale without a D-108 amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Build the D-135 amendment: jammy-backports at source, T10 re-pointed
...
scripts/dc-mirror.sh's generated sync script gains jammy-backports in --dist=,
with a comment at the generator recording why it is not optional: the MAAS
jammy image ships backports in its sources.list and juju's apt-mirror
model-config rewrites every suite to this mirror, so a mirror without it 404s
and every charm install hook fails. That cost 22 units today.
FIXED AT SOURCE, not only on the live rack -- a live-only edit would leave the
generator regenerating the old scope at the next install or at any future DC
standup.
tests/dc-mirror T10 asserted the old dist string verbatim. RE-POINTED to the
new invariant with the reason recorded in-file, never deleted to go green.
Mutation-proven: reverting the generator's dist line turns T10 red (23/1);
restored byte-identical, 24/24.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
RULED 2026-07-31 (GA-R5): sync jammy-backports into the dc0 mirror (D-135 amendment)
...
Operator utterance, verbatim: "Sync the backports into the mirror".
Question as presented: 22 of 33 units failed their install hook on one measured
cause -- the mirror serves jammy/jammy-security/jammy-updates 200 and
jammy-backports 404, exactly dc-mirror.sh:179's "jammy triple" scope, while
every node's sources.list carries jammy-backports because the Step-3.5
apt-mirror model-config rewrites EVERY suite to the DC mirror. Options: (a) add
the suite and re-sync, (b) drop backports from the nodes' sources.
Recorded as a D-135 AMENDMENT. OPS under GA-R3, no new D-number; D-135's per-DC
strategy split (dc0 full mirror / dc1 caching proxy) is unchanged.
Cost measured BEFORE the ruling was put: ~1.00 GiB / 461 packages against a
951 GB mirror with 1.8 T free -- ~0.1%.
Recorded as owed, not built: dc-mirror.sh check verifies sync STATUS, not SUITE
COVERAGE against what the deployed image's sources.list requests, which is why
it read PASS throughout. Same class the mirror gate was already fixed for once.
Committed and pushed BEFORE the dependent work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Defer Tailscale decisions to end of step; measure jammy-backports at 1.0 GiB
...
OPERATOR DIRECTION, verbatim: "To make sure we don't muddle the in process
deployment, queue all the tailscale decisions to the end of the step. When you
bring them back up give me decisions choices with your recommendation for
each." All four sub-decisions in gap register item 21 are deferred; when
re-raised they come back as choices, each with a recommendation.
MEASURED, at the exact scope dc-mirror.sh:177-181 uses (main,restricted,
universe,multiverse / amd64 / nosource), summing Size: across each component's
binary-amd64 Packages.gz:
main 859,001,998 338 pkgs
universe 211,685,404 123 pkgs
restricted 0 (empty in backports)
multiverse 0 (empty in backports)
TOTAL 1,070,687,402 = ~1.00 GiB, 461 packages
Context: the existing dc0 mirror is 951 GB with 1.8 T free, so backports is
~0.1% of it -- capacity is a non-issue.
Sanity-checked rather than trusted: the five largest entries were listed to
confirm the Size: field was read correctly, and a separate binary-all index
probes 404, so arch:all is already inside binary-amd64 (301 of main's 338) and
is neither double-counted nor missed.
Worth knowing before choosing: ~462 MB of main's 859 MB is LibreOffice -- about
46% of the suite is a desktop package set these nodes will never install.
Transfer estimate stated as the weak measurement it is: one single-stream fetch
measured 932 KB/s, putting 1.07 GB at roughly 19 minutes. Treat as an upper
bound; debmirror parallelises.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Deploy blocker: mirror lacks jammy-backports. Queue per-DC Tailscale (item 21)
...
BLOCKER, logged not fixed (D-135 scope is a ruled surface). 22 of 33 units in
error, all hook failed: "install", all one cause:
E: The repository 'http://10.12.8.4/ubuntu jammy-backports Release'
does not have a Release file.
Measured: the mirror serves jammy/jammy-security/jammy-updates 200 and
jammy-backports 404 -- exactly D-135's "jammy triple" scope -- while the node's
sources.list carries jammy-backports, because the Step-3.5 apt-mirror
model-config rewrites EVERY suite to the DC mirror. Machines are all
started/running; this is purely the artifact layer and units retry.
dc-mirror.sh check dc0 PASSES while this is true -- it verifies sync STATUS,
not suite COVERAGE against what the deployed image asks for.
QUEUED BY OPERATOR DIRECTION, not built: a per-DC Tailscale subnet router is
now a standing DC-standup requirement, dc0 first. Home of record is the tooling
gap register item 21 (new), with a per-DC definition-of-done, the
vendor-documented build constraints, and four sub-decisions needing GA-R5
rulings first: the utility-band OCTET (D-134's map is a standing cross-DC
standard), the admin-reachability model (star vs mesh -- the real architecture
question at region-region scale), HA count, SNAT on/off. This is EXECUTION of
the already-ruled D-129(iii) shape, not a new decision.
Measured while scoping: SEC-010 does NOT need relaxing (the forwarding to
permit is on the router VM, not the transit leg); voffice1 holds no route to
10.12.x at all, so the tailnet stops at Office1 by construction; the existing
Office1 node is UNTAGGED and carries a 180-day key-expiry clock on the
operator's only tailnet path; and D-129(iii) cites D-107 as governing Office1
while D-107 rules nothing about Tailscale -- the second instance of the F7
miscitation class, this time inside a ruling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
>>> BUNDLE DEPLOY SUCCEEDED: "Deploy of bundle completed." exit 0 <<<
...
Option D half 2. The 23 application definitions were removed INDIVIDUALLY,
never batched (hard rule 3), each read back, with the precondition re-verified
immediately before the first removal and the model read back empty afterwards.
Step-3.5 state survived the clear -- apt-mirror still set, all six spaces still
bound with both address families, checked not assumed.
MEASURED IMMEDIATELY AFTER: juju 56 applications / 9 machines / 33 units,
machines pending+allocating; MAAS 10 machines -- 9 Deploying, 1 Deployed (the
controller).
Three defects closed in the order they were hit: attempt 1's unknown option
prefer-ipv6 (D-101 ruling note); attempt 2a's missing policies/overrides.zip on
the client host; attempt 2b's barbican 265-vs-261 downgrade refusal (explicit
base on all 56 AND a clean model).
Operational note, because the first removal read as a failure: juju
remove-application PROMPTS by default and aborts on non-interactive stdin.
--no-prompt is required.
WHAT IS NOT CLAIMED: this means juju ACCEPTED and QUEUED the bundle. It does
NOT mean the cloud is up. 9 machines are allocating, 33 units pending; the
settle takes hours and nothing here asserts unit health or any service verdict.
G17's dc0 half is now genuinely armable -- the existing capture was taken on
the CONTROLLER VM and says so. The nine role nodes are booting for the first
time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Option D half 1: explicit base on all 56 apps, gated by new invariant 12
...
PROVEN AGAINST THE REAL MODEL, NOT INFERRED. The earlier one-app test only
showed the error MOVING to the next app. The fully-based bundle was staged as a
throwaway on the rack and --dry-run against the LIVE 23-application model --
the same command that errored on barbican 265-vs-261 -- exits 0.
Subordinates are in scope, measured: mysql-router is 1154 at 22.04 and 1178 at
24.04. hacluster sharing rev 166 across both bases is not a property to depend
on.
TWO SILENT UNDER-MATCHES, both caught by cross-checking the edit against the
parsed file rather than by review:
(1) a first pass keyed on the block form inserted 44 of 56 -- the twelve
-hacluster apps are single-line FLOW MAPPINGS. The inserter asserts its
app list equals what yaml.safe_load sees, and refuses otherwise.
(2) overlays/dc-ha-scaleup.yaml DEFINES vault-hacluster, an application that
exists nowhere in bundle.yaml, and it had no base. That overlay is a LATER
deploy step, so this would have re-opened the trap AFTER the deploy
succeeded. Found by the new gate on its FIRST run, in a file the fix was
not looking at -- the same later-step-overlay class as the ceph-mon miss
earlier today.
INVARIANT 12 (NEW): every application carries an explicit base equal to the
bundle's OWN default-base -- keyed on that value, not a literal. A bundle with
no default-base REFUSES rather than passing vacuously. Without this, deleting
the 56 lines is a byte-identical PASS and the trap returns at the next partial
failure, at any DC.
Five cases, each mutation-proven individually (harness 50 -> 55/55): neutering
nobase kills T49/T50/T52, wrongbase kills T51, the refuse branch kills T53.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
RULED 2026-07-31 (GA-R5): option D -- fix the bundle AND clear the model
...
Operator utterance, verbatim: "I want to do option D: Fix the bundle by
defining the base 22.04 on each app and then clearing the modeling and
deploying from a clear model. It is very cheap to clear and redeploy."
I offered three options; the operator composed a fourth that is strictly
stronger than any of them. The durable fix lands (explicit per-app base, so
the bundle survives a partial failure), AND the deploy runs the path that is
MEASURED to work -- a clean deploy onto an empty model, exactly what run 2a
did before it hit the missing resource file -- rather than juju's incremental
re-run path, which is the one carrying the defect.
OPS under GA-R3, no D-number: an explicit base per application states what
bundle.yaml:85's default-base already means. No ruled surface changes.
The invariant it establishes is skill material at stage close: a bundle whose
applications rely solely on default-base is deployable exactly ONCE, and a
production bundle that cannot survive a partial failure is a trap.
Committed and pushed BEFORE the dependent work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Deploy attempt 2: prefer-ipv6 fixed, two new defects, model partially populated
...
The prefer-ipv6 failure did NOT recur. Two new ones did, and for the first time
the model is not empty: MEASURED 23 applications, 0 machines, 0 units.
Attempt 1 aborted during VALIDATION; this one reached EXECUTION, which is why
it left residue. NOTHING IS PROVISIONED -- no MAAS machine left Ready, no disk
written.
DEFECT 1: the rack's ~/repo-stage is a PARTIAL COPY of the repo with no
policies/, and bundle.yaml:214 is the only local-file reference in the deploy
input. juju resolves it relative to the bundle, so it must exist ON THE HOST
THAT DEPLOYS. Nothing caught it: --dry-run does not upload resources (the
SECOND --dry-run blind spot in two days), attempt 1 aborted before resource
upload, and preflight runs on voffice1 where policies/ DOES exist -- the same
wrong-host instrument class already recorded twice, and the second instance
this session. Fixed forward, both files sha256-verified.
DEFECT 2: a bundle relying on default-base is deployable exactly ONCE. The
re-run failed on barbican 265-vs-261. Measured: 265 is the ubuntu@22.04
revision, 261 the ubuntu@24.04 one. The 23 existing apps are all 22.04, so run
2a honoured bundle.yaml:85 -- but on a re-run, for an app that ALREADY EXISTS,
juju resolves without that default. A model-level default-base has NO EFFECT
(tested, then RESET). An explicit per-application base WORKS (tested read-only
on a throwaway; the error moved to the next app).
OWNED: two of those read-only tests were staged under /tmp and failed with
"no charm was found" -- the juju snap's PRIVATE /tmp, a trap this repo has
already recorded in CURRENT-STATE and the phase-4 runbook.
The rollback decision tree is written for tofu apply and defaults to
fix-forward, which is measured not to work here. A partial juju deploy is a
failure mode the repo does not cover. Recovery is an operator decision;
options are enumerated in the capture and the model residue is untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
RULED 2026-07-31 (GA-R5): the five new P5 findings accepted; dc0 deploy is clear
...
Operator utterance, verbatim: "Accept the five and proceed to the dc0 deploy".
The 2026-07-30 acceptance covered SIX findings and says it covers "these six,
enumerated, and nothing else". This preflight run reports eleven: all six still
present, plus FIVE new -- maas-region-db-password, maas-region-admin-password,
maas-region-api-key.txt (SEC-027) and maas-juju-api-key.txt,
maas-juju-user-password (SEC-028), all vr1-dc1. Every one is the D-137 forward
register correctly reporting that dc1's MAAS region is authored but not applied.
Carried as known, by-design absences on their existing SEC rows. preflight keeps
exiting FAIL on P5 for the rest of this stage; that RED is ruled-accepted and
must NOT be made green by deleting or weakening a matrix row. Like its
predecessor this covers these FIVE, enumerated, and nothing else -- total
accepted at P5 is now ELEVEN across the two rulings.
Also corrects this changelog's header, which claimed the session was read-only
throughout. True for items 1-5; after the ruling landed it edited the renderer,
the gate, both overlays and both values files. No cloud state changed at any
point.
Committed and pushed BEFORE the deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-deploy: rack input refreshed + hash-verified, preflight captured, dry-run green
...
(a) The dc0 rack's ~/repo-stage is a COPY, not a clone -- nothing updates it, and
it still held the pre-ruling vips overlay. Exactly one of four files differed;
copied, then ALL FOUR hash-compared against repo HEAD. The gitignored
octavia-pki overlay was deliberately not touched (SEC-029) and verifies
byte-identical at 5fc117f1, still 0600. A sweep is evidence about the deploy
only if the swept bytes ARE the deployed bytes.
(b) Preflight captured (242 lines, exit 1), run with
MAAS_PROFILE=vr1-dc0-region -- without it preflight is REGION-BLIND and emits 19
false negatives. Instrument proven current first: maas-profile-assert.sh
vr1-dc0-region hot-kid exits 0. P1-P4 and P7 PASS, incl. aligned VIPs 13 OK / 0
bad against the RE-RENDERED overlay. P5 FAIL, 11 findings.
(c) The P5 delta is ENUMERATED. The 2026-07-30 acceptance covered six findings
and says it covers "these six, enumerated, and nothing else". All six are still
present; FIVE NEW are all vr1-dc1 S2 rows (SEC-027/-028) -- the D-137 forward
register correctly reporting that dc1's region is not built. Diff appended to
the capture. Needs its own GA-R5 exchange before the deploy.
(d) Step 4.2 --dry-run against the fixed input: exit 0, 56 apps / 108 relations
/ 33 unit placements. The ceph-osd tags=openstack exposure is now MEASURED, not
reasoned: the plan places ceph-osd/0..3 on machines 5-8 by explicit id, so the
absent tag never has to match. The residual (a later UNPLACED add-unit) stands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Fix a defect in my own build: invariant 9a was VIP-scoped and the charm list was short
...
Found by a repo-wide grep, not by any gate. Two faults, one cause -- I
measured and gated the population the QUESTION named (the 13 VIP charms)
rather than the population the INVARIANT covers (every application juju
validates).
(1) PREFER_IPV6_CHARMS was WRONG. Re-measured across ALL 33 charms in
bundle.yaml: TWELVE declare the option, not seven. The five missing are
ceph-mon 491, ceph-osd 953 (squid/stable), nova-compute 894, hacluster 166
(2.4/stable) and mysql-innodb-cluster 164 (8.0/stable).
(2) 9a was written INSIDE the VIP loop, so an application with no vip never
reached it -- and the repo ALREADY has that case:
overlays/dc-dc-ipv6-family-matrix.yaml sets prefer-ipv6 on ceph-mon, which
carries no VIP. That overlay is a LATER deploy step, so the miss would have
passed attempt 2 cleanly and surfaced at the step after, looking like a new
fault.
9a now runs over EVERY application, before the VIP loop; the VIP loop skips an
app 9a already flagged so the diagnosis is not doubled. ceph-mon's own in-file
claim of the option is now INDEPENDENTLY VERIFIED from the artifact rather
than taken from the comment.
Harness 48 -> 50/50. T47 (non-VIP app whose charm lacks the option FAILS) and
T48 (ceph-mon PASSES) both mutation-proven: re-scoping 9a back to the VIP loop
kills T47; dropping ceph-mon from the list kills T48.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Sweep both deploy inputs for option NAMES and VALUE TYPES: clean, and proven able to fail
...
The two extensions owed after the ruling, now run:
dc0 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
dc1 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
Both EXIT 0. dc1 introduces no new charm@channel pair -- checked, not assumed.
PROOF OF TEETH, because a clean sweep across two inputs is exactly the
implausibly-uniform result this repo has been burned by. Three defects planted
in a THROWAWAY copy, all three caught, EXIT 1:
[FAIL name] barbican does not declare option 'prefer-ipv6' (attempt-1 defect)
[FAIL type] keystone 'prefer-ipv6': declared boolean, got str ('true')
[note] glance 'vip': declared string, value is unquoted float
Type judgement is deliberately conservative: a mismatch is reported only where
juju's own coercion cannot save it. A scalar where a STRING is declared is a
NOTE, not a failure -- but it is still surfaced, because an unquoted value is
how a version string silently becomes a float. An UNRECOGNISED declared type
REFUSES rather than passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
