| 2026-08-01 |
D-139 ruling A NOT achievable at these charm revisions; apex carve tool built
...
TWO agent results, both recorded; nothing actioned, nothing pushed to NetBox.
>>> RULING A (v6-only on five planes) IS BLOCKED BY UPSTREAM CHARM DEFECTS <<<
RULING B (full GUA) IS ORTHOGONAL AND STANDS -- GUA-vs-ULA is independent of
whether a plane keeps v4, so the carve tool is unaffected and a GUA DUAL-STACK
carve is achievable today. This needs an operator ruling.
RE-VERIFIED INDEPENDENTLY against the shipped ceph-osd rev 953 artifact, because
it decides the ruling. hooks/utils.py:203-217 get_host_ip() returns
get_ipv6_addr()[0] when prefer-ipv6 is set, else socket.inet_aton(hostname) and
on failure dns.resolver.query(hostname, 'A') -- an IPv4-ONLY lookup -- under the
charm's OWN comment "...just let it kill the hook". So ceph-osd fails on a
v6-only plane in BOTH directions: false -> inet_aton raises on a v6 literal then
NXDOMAINs an IN A lookup (LP #2109798, New); true -> bare get_ipv6_addr() (also
ceph_hooks.py:547, WITHOUT dynamic_only=False) returns DYNAMIC/SLAAC only while
every VR1 node holds a MAAS STATIC v6 (LP #2061836 -- Fix Committed on ceph-mon,
New and never fixed on ceph-osd, confirmed absent from rev 953).
Reported from the capture and NOT re-verified here, recorded as such: mysql /
mysql-router bare user:pw@addr URIs invalid for an unbracketed v6 literal;
hacluster 2.4/stable rev 166 (released 2026-06-29) still hard-writes
ip_version: ipv4 though LP #2111852 is Fix Committed -- FIX COMMITTED IS NOT FIX
RELEASED; OVN documenting encap as "The IPv4 address of the encapsulation tunnel
endpoint" with zero v6 encap values in OVN 24.03's test suite.
Counted locally: metal-internal appears 273 times in bundle.yaml across ~44 apps;
rabbitmq-server is DEPLOYED by the bundle but ABSENT from PREFER_IPV6_CHARMS --
the only mismatch across all 33 bundle pairs, where invariant 9a would emit a
factually false message.
APEX CARVE TOOL (netbox/d139-gua-carve.py, 232 lines) + tests/d139-gua-carve
(65 cases, offline stub apex); manifest 94->95. Live dry run: CREATE 22 |
EXISTS 6 untouched | RETIRE-REPORT 18, and it surfaced 52 DEPENDENT ip-addresses
inside the retiring ULA /64s that a later delete would orphan. Proof no write
occurred: apex prefix count 139 before and after. 5 mutations, each turning a
NAMED test red, tool restored sha256-identical.
OWED: (a) the foundational measurement is STILL not taken -- does network-get
return v6 on a v6-only bound space? (b) D-139's "remove v4 LAST, after each is
proven" is UNSATISFIABLE as written; per-plane conversion is atomic.
(c) the rabbitmq-server mismatch.
PROCESS FAILURE, OWNED: the research was written to /tmp and would have been LOST
-- I scoped the agent read-only and gave it no repo path. Rescued, sha256-verified.
STANDING CONSEQUENCE: every agent mandate must NAME A REPO PATH for its findings;
"read-only" constrains what it may CHANGE, not where it may RECORD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
The "juju client blocker" is NOT one -- it is D-138 working correctly
...
CORRECTS this repo's own 2026-07-31 finding 9d, which called it a blocker.
Stage 5 can reach `add-model` TODAY, from the dc0 rack. No key movement needed.
VERIFIED INDEPENDENTLY from the main session, not accepted on the agent's word:
juju 3.6.27-genericlinux-amd64 at /snap/bin/juju on the rack; controller
vr1-dc0-controller* admin/superuser on cloud vr1-maas; the controller model reads
"Last connection: just now"; client credentials list EXACTLY ONE entry,
vr1-maas -> vr1-dc0-cred.
ROOT CAUSE OF THE ssh REFUSAL, MEASURED ON THE MACHINE: the alias is correct in
every part -- hop chain, host, port, per-hop identity, host key. subtle-grouse's
authorized_keys holds exactly two lines, both Juju's, so the office1_svc key the
alias offers is genuinely absent and the refusal is right. The obvious hypothesis
was REFUTED rather than assumed: the machine booted AFTER the 2026-07-30 key
import and still got only Juju keys, because deploy-time cloud-config carries
Juju:juju-client-key alone. `juju ssh -m controller 0` already works.
SEC-026 control (1) DISCHARGED BY MEASUREMENT on both sides -- rack lists one
credential, voffice1 lists two, so the forbidden whole-store copy did not happen.
No new credential residency; no new security-ledger row owed.
ONLY HYGIENE OWED, NOT EXECUTED (hard rule 3): a dangling current-model pointing
at the model destroyed 2026-07-31 makes bare `juju status` error. Measured NOT to
be an add-model precondition.
FOUR GAPS LOGGED NOT FIXED, in consequence order:
G4 the `openstack` CLI is MISSING ON THE RACK -- D-138 definition-of-done gap
blocking phase-03+. SAME item as F1 from the 2026-07-30 sweep, now confirmed
on the D-138 host. It has survived two sweeps.
G3 preflight CANNOT PASS on vcloud by construction: the octavia-pki overlay is
gitignored PKI material absent from the vcloud tree, and
pre-flight-checks.sh:161 hard-fails on it. The host-dependence class again.
G2 the D-138 client host's deploy input has NO PROVENANCE -- ~/repo-stage on the
rack is a hand-staged 9-file copy with no .git (8/9 byte-identical by sha256).
A third copy beside the two clones verified at c58bf95.
G1 phase-4 runbook :379,398 still say "voffice1" for add-model/spaces, stale vs
D-138; voffice1 has the binary but no registered controller, so the runbook as
written fails. DOCFIX owed.
DECLARED UNMEASURED with the reason: whether MAAS user juju-vr1-dc0 has an
imported ssh key -- no usable maas CLI profile exists anywhere (voffice1's
~/.maascli.db is ZERO BYTES). Does not change the fix; the proximate cause was
measured directly. The PreToolUse guard refused a command that would print the
MAAS API key (DOCFIX-016) and it was NOT retried in an altered shape.
Staged by explicit path; agent files still in flight are deliberately excluded.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 ruling B's deciding reason REFUTED by measurement; ruling FLAGGED
...
Closing the unverified premise D-139 named at ruling time inverted its own
argument. Recorded, flagged for operator reconsideration, NOT amended.
MEASURED, two steps:
1. jammy libc-bin 2.35-0ubuntu3 ships /etc/gai.conf with ZERO active lines --
every line commented. The node image applies NO override, so glibc's
compiled-in table governs.
2. glibc 2.35 default_precedence[], fetched in full from sourceware at tag
glibc-2.35 and read rather than recalled, carries glibc's OWN comment
"See RFC 3484 for the details" and is:
::1/128 50 | 2002::/16 30 | ::/96 20 | ::ffff:0:0/96 10 | ::/0 40
THERE IS NO fc00::/7 ENTRY IN THE PRECEDENCE TABLE.
CONSEQUENCE: ULA falls through to ::/0 = 40, identical to GUA; IPv4-mapped is
10. So on the node image ULA = GUA = 40 and BOTH outrank IPv4. Ruling B was
declared on "ULA loses to IPv4 (3 vs 35), GUA wins (40 vs 35)" -- the RFC 6724
table. RFC 6724 obsoletes RFC 3484 and does define those values, but glibc 2.35
DOES NOT IMPLEMENT IT. Right about the RFC, wrong about the software this cloud
runs. The fc00::/7 entry that does exist is in default_labels[] (label 6), which
drives SOURCE selection rules 5/6 -- a different mechanism, recorded so the two
are not later conflated into "the argument stands after all".
A NEW VARIANT OF THE CITATION CLASS, worth naming: the citation was real,
current and correctly quoted, and still wrong here because nobody checked
whether the IMPLEMENTATION follows the RFC. Checking a source's status and dates
is not sufficient for a standards citation.
SURVIVING REASONS (independent of the refuted one): conformance with Willamette
(a real site) and VR0 DC0, both full GUA; and MINIMIZE DELTA TO ROOSEVELT, whose
2602:f3e2:103::/48 is GUA with no ULA and no plane carve yet. The ruling may well
stand on these -- but under GA-R5 that is the operator's call, since they
answered a question whose stated deciding reason no longer holds.
D-139's text is UNCHANGED. Nothing of D-139 is executed, so nothing is half-built
in either direction. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 ADOPTED: VR1 goes IPv6-only east-west, whole carve moves to GUA
...
TWO GA-R5 rulings, taken in SEPARATE exchanges, recorded separately.
A -- family matrix. Operator: "IPv6 on all planes except for metal-admin and
provider-public which will remain dual stack". metal-internal, data-tenant,
storage, replication and lb-mgmt become IPv6-ONLY. Amends D-101's matrix twice:
metal-internal's "datastore east-west stays v4-bound" is superseded, and lb-mgmt
(ruled v6-only by D-101 but NEVER carved anywhere) becomes a first-class plane.
B -- addressing model. Operator: "Full GUA on every plane (Recommended)". Every
plane carves from its DC's GUA /48 on the :10/:11/:20/:21/:30/:40/:50/:80 octet
map VR0 DC0 and the Willamette site ALREADY use -- conforming to an existing org
standard, not inventing one. The ULA /48 fd50:840e:74e2::/48 is RETIRED for VR1.
Amends D-101 and D-111. Deciding reason is measurable: RFC 6724 ranks IPv4-mapped
at precedence 35 and ULA at 3 while GUA falls under ::/0 at 40, so on a dual-stack
plane a ULA leg LOSES address selection to IPv4 and is decorative. SEC-010, D-052,
D-125 and D-107 unchanged -- containment lives at the forwarding layer.
THE RECORDED ROOT CAUSE OF THE v4-ONLY CONTAINERS WAS WRONG AND THIS REPO CARRIED
IT (GA-R1 C2 -- measurement corrects the document). CURRENT-STATE said MAAS "has
nothing to give for v6". Measured, and re-verified independently: subnet
statistics on fd50:840e:74e2:220::/64 returns available_string "100%",
num_available 18446744069414584320. Zero ipranges rows means zero RESTRICTIONS,
not zero availability. dc-plane-ipam.sh:368-372 has carried the correct behaviour
since 2026-07-27, in the tree, contradicting the authoritative doc the whole time.
The real mechanism is juju-side and there is NO knob: EthernetDeviceForBridge
(tag v3.6.27) takes addrs[0] from an UNSORTED query and derives one CIDR -> one
LinkSubnetArgs. gomaasapi documents "Any number of STATIC links can exist on an
interface", so MAAS would accept both families; the limit is juju's. LP #1723240
is this exact symptom, Triaged/Low since 2017, and the theory this repo recorded
is the one its reporter rebutted in-thread. Consequence: on a container plane
DUAL-STACK IS NOT EXPRESSIBLE while v6-only IS -- ruling A is the achievable
configuration, not merely the desired one.
SNAP PROXY: dc0 INSTALLED, RUNNING, check dc0 PASS exit 0 (16 assertions,
re-verified independently). A REAL SNAP PAYLOAD WAS FETCHED THROUGH IT -- HTTP 206,
first bytes hsqs. No longer fixture-green. BUG-3 confirmed by measurement, no
assertion changed. BUG-4 fixed at BOTH sites (the review named one); T20b/T41b NEW.
OWNED: my BUG-1 fix was WRONG. ip route get 10.12.8.4 -> src 10.12.8.2; .4 is a
SECONDARY and Linux never auto-selects a secondary as a source, so the rule is
still dead. Not fixed now -- editing the generator turns the LIVE gate red until
install re-runs. T23b annotated so nobody "fixes" the test to a wrong value.
Generalisable: any ACL keyed to a service ALIAS (the whole D-134 utility .4 band)
misses that host's own traffic -- same exposure in dc-mirror.sh, dc-cache-proxy.sh.
OWED, unverified: the deployed jammy /etc/gai.conf, which can override the RFC 6724
default table that ruling B rests on. Nothing of D-139 is executed.
repo-lint 0 fail; tests/dc-snap-proxy 55/55; 4 mutations killed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-31 |
SESSION CLOSE 2026-07-31 (GA-R4 bookend): Stage 5 dc0 deploy + teardown
...
Bounded ledger summary (10 lines, cap 15), rotation, sweep, memory review.
DURABILITY: 0 uncommitted / 0 unpushed, 45 commits today. dc0 rack repo-stage
digests all MATCH. HAZARD, not loss: the voffice1 clone is 26 commits behind --
pull it forward before it is used.
GATES: repo-lint 0 fail / 1 standing warn / 641 files. Gauntlet ALL GREEN (94).
ledger-scan: 3 open decisions, SEC 26, D 139 / DOCFIX 207 / BUNDLEFIX 053.
Ledger 296 lines after rotating the oldest summary out to
docs/archive/session-ledger-rotated-20260731.md.
SWEEP: docs/audit/queued-findings-20260731-stage5-deploy.txt -- ELEVEN FIRST
SURFACE items that existed only in the transcript. Highest-consequence is F1:
MAAS's own squid ALREADY CONNECT-proxies api.snapcraft.io and would unblock the
deploy today with nothing built, measured from the failing container -- RULED
AGAINST, with the operator's verbatim utterance and the deciding downsides
recorded, precisely because a future session will rediscover it and reach for
it. Also first surface: containers source from METAL-ADMIN not the address juju
status displays; ports 3128 and 8000 already held by MAAS's squid on both racks;
the four reviewed bugs in dc-snap-proxy.sh; the teardown instrument error; and
that a MAAS hostname rename on a Deployed machine is record-only.
Method note: a phrase-exact grep produced false negatives on items recorded under
different wording. Every candidate was re-grepped on distinctive tokens before
being classified -- the same instrument-error class this sweep documents.
GA-R7 MEMORY REVIEW -- one real violation found and corrected: the
multi-workstation memory asserted 'never additions to allow for mutations', an
operator-posture claim memory may not hold, contradicted by the recorded
2026-07-30 ruling 'Add it to allow'. Re-pointed to an observation with the
contradiction recorded. instrument-currency memory gained this session's two
misreads.
OWNED: a red-lint push (| tail masked the exit code); a git add -A that swept an
agent's files into an unrelated commit; two instrument misreads (a juju models
summary read as progress, stale unit logs read as current); and a wrong
'immutable system_id' claim the operator corrected.
NEXT: apply the dc-snap-proxy review cuts, RE-RUN the mutation pass (S8
invalidates M32-M39/M42-M47/M49-M52 as evidence), confirm BUG-3's dstdomain deny
shape once at apply time, install the proxy, then add-model + spaces + deploy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Snap proxy tool BUILT and REVIEWED (not yet applied); 4 real bugs logged
...
Ruling 2's repo side. squid 6.14 as a dedicated instance on each DC rack at the
D-134 utility .4:3129, CONNECT-only, restricted to Canonical's documented store
and CDN hosts, consumed via juju snap-http-proxy/snap-https-proxy. Design cited
to vendor sources, not preference: every snap endpoint is :443 so a CONNECT
proxy is mandatory and CACHING IS IMPOSSIBLE (squid cannot read a CONNECT
tunnel), which removes the apt-style caching argument entirely; squid is in
main, tinyproxy is universe with six unevaluated CVEs and defaults to allowing
all CONNECT ports.
MEASUREMENTS THAT CHANGED THE DESIGN:
- the failing apps are LXD containers sourcing from METAL-ADMIN (10.12.8.122),
NOT the 10.12.12.116 juju status displays. An ACL on the displayed address
would have denied every client the proxy exists for.
- ports 3128 AND 8000 are already held wildcard-bound by MAAS's own squid on
both racks -- hence 3129 and a dedicated instance.
- no new D-134 octet needed: .4 is already aliased on both racks.
- LOGGED NOT ADOPTED: MAAS's squid ALREADY CONNECT-proxies api.snapcraft.io,
measured working from the failing container. snap-https-proxy pointed at
10.12.8.6:8000 may unblock the deploy today with nothing built. Rejected as
the shipped design (per-revision path, NO destination restriction, the
hidden-coupling shape D-135 already has a scar from) but it is a real
operator option, not a dismissal.
QUALITY: 53 mutations, 53 killed, 0 survived -- including the dc-mirror.sh
existence-vs-content false-green class reproduced and caught. One survivor was
a MALFORMED mutation that broke bash -n and therefore tested nothing; recorded
as malformed and redone rather than counted. Three of the harness's own cases
were found too weak and were STRENGTHENED, not left green.
INDEPENDENT REVIEW: ruled design HONOURED on all four counts -- utility band, no
direct node egress (in fact stronger: destination-allowlisted, and that scope is
flagged as a build-time choice NOT a ruling), site-keyed both DCs with the dc1
limit stated honestly, and it OWNS its own network prerequisites rather than
borrowing another service's -- with the coupling anti-pattern now under test
(M09/M10) rather than only under comment.
FOUR REAL BUGS LOGGED, NOT FIXED (hard rule 1 -- the file was frozen for review):
BUG-1 dead snap_probe ACL; the probes are actually admitted by snap_clients,
and the comment misdescribes its own mechanism
BUG-2 | grep -q under pipefail -- INHERITED from dc-mirror.sh and
dc-cache-proxy.sh alike; a three-script sweep, not a local fix
BUG-3 the deny probe's 403 shape was measured from a PORT deny, never a
DSTDOMAIN deny. If squid answers differently the gate REFUSES
PERMANENTLY -- and a gate that cannot PASS blocks a GA-R6 close as hard
as one that cannot fail. Requires a one-time apply-time confirmation.
BUG-4 dpkg -s reports a deinstall-ok-config-files package as present
SIZE: 636 lines / 320 code vs dc-mirror 385/227 and dc-cache-proxy 396/200. The
reviewer's landing zone is ~535/~250 and it RECOMMENDS AGAINST forcing 385 --
two extra generators (75 lines) are measured-forced because the packaged squid
cannot be repointed, and further cuts would delete mutation-proven assertions.
~50 lines of the excess is header prose, which carries zero bug risk.
TWO BLOCKERS BEFORE THE PLAN IS APPLIED: the S8 consolidation invalidates
M32-M39/M42-M47/M49-M52 as evidence, so the mutation pass MUST be re-run; and
BUG-3 must be confirmed once against the built proxy before check is cited to
close anything.
NOTHING IS PROVEN END TO END -- no proxy installed, no snap fetched through one.
The harness green is FIXTURE green and says so.
repo-lint 0 fail; gauntlet ALL GREEN (94).
OWNED: my earlier teardown commit e57ad09 used 'git add -A' while the agent was
writing and swept its audit capture and CURRENT-STATE paragraph into a commit
about the model teardown. This commit is staged EXPLICITLY by path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Teardown: stalled on stopped agents, forced clear, no nodes stranded
...
Plain destroy-model STALLED and would not self-resolve -- 'attempt 30 ... model
not empty, found 26 machines, 37 applications', flat ~19 min with the app set
byte-identical across a 12-minute name-level diff.
MECHANISM MEASURED, and it makes the stall terminal rather than slow: ALL 26
machine/container agents were 'stopped', so NO hook could execute. Units already
in error from the snap failures could never run their teardown hooks.
MAAS was NOT the bottleneck -- the six nodes juju released went to Ready/owner
None in minutes; juju never issued a release for the other three.
destroy-model --force --no-wait cleared it (18 -> 5 -> 2 machines, then 'Model
destroyed.'), and NO NODES WERE STRANDED: all nine read back Ready/owner=None,
so no maas machine release was needed or run. subtle-grouse correctly stays
Deployed -- it is the D-104 controller VM in the controller model.
INSTRUMENT ERROR, OWNED: I reported '0 machines / 0 units' from the juju models
SUMMARY COLUMNS while juju status -m read 26 machines / 37 applications. The
summary zeroes during 'destroying' and is not a progress signal. The real tell --
three control nodes stuck Deployed -- was visible and I explained it away.
Standing lesson: during a teardown, juju status -m <model> is the instrument;
juju models counts are not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Three post-deploy defects; memcached scaled; ceph-rbd-mirror PINNED (item 22)
...
D1 jammy-backports: RULED + FIXED. Verified on CONTENT (Release fields + a real
.deb fetch), mirror 951G -> 952G. Units moved 22 error -> 17 with 15 apps past
install (was 1), so the fix works and auto-retry is running.
D2 the UPSTREAM UCA is unreachable from nodes -- LOGGED NOT FIXED. Measured FROM
A NODE, not the rack: upstream 000, mirrored 200 with real content. SIX apps set
it across TWO option names (openstack-origin on barbican/magnum/octavia, source
on ceph-mon/ceph-osd/ceph-radosgw); every other charm defaults to
openstack-origin: caracal, same upstream pocket. The UCA signing key is already
on the node, so a raw deb line verifies with no |key suffix. Genuine D-135
experiment result: the full-mirror DC must rewrite every non-Ubuntu source; the
proxy DC needs none. dc1 will not hit this.
D3 prefer-ipv6: true is FATAL on the seven charms that declare it, and is now
the blocker. This is the exact risk flagged earlier today as "never observed
live" and deliberately left OUT of the D-101 ruling note. keystone/0:
"Interface 'eth0' does not have a scope global non-temporary ipv6 address" --
get_relation_ip() returns early with get_ipv6_addr() when the option is true.
Measured on the container: eth0 holds only fe80::/64. The NODES are dual-stacked;
the LXD CONTAINERS are not. All seven true-valued apps are erroring. NOT TAKEN --
R2/D-101 are ruled surfaces and this needs its own GA-R5 exchange. One question
is stated and NOT answered: whether a v6 VIP on a container with no global v6 is
routable.
memcached scaled 1 -> 3 in dc-ha-scaleup (operator-directed); its exclusion
comment re-pointed rather than left stale. ceph-rbd-mirror PINNED as gap
register item 22 with four options and a recommendation -- (d) add the missing
liveness detection now, (b) scale to 2 active/standby at Roosevelt -- and the
overlay now forbids a silent scale without a D-108 amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
>>> BUNDLE DEPLOY SUCCEEDED: "Deploy of bundle completed." exit 0 <<<
...
Option D half 2. The 23 application definitions were removed INDIVIDUALLY,
never batched (hard rule 3), each read back, with the precondition re-verified
immediately before the first removal and the model read back empty afterwards.
Step-3.5 state survived the clear -- apt-mirror still set, all six spaces still
bound with both address families, checked not assumed.
MEASURED IMMEDIATELY AFTER: juju 56 applications / 9 machines / 33 units,
machines pending+allocating; MAAS 10 machines -- 9 Deploying, 1 Deployed (the
controller).
Three defects closed in the order they were hit: attempt 1's unknown option
prefer-ipv6 (D-101 ruling note); attempt 2a's missing policies/overrides.zip on
the client host; attempt 2b's barbican 265-vs-261 downgrade refusal (explicit
base on all 56 AND a clean model).
Operational note, because the first removal read as a failure: juju
remove-application PROMPTS by default and aborts on non-interactive stdin.
--no-prompt is required.
WHAT IS NOT CLAIMED: this means juju ACCEPTED and QUEUED the bundle. It does
NOT mean the cloud is up. 9 machines are allocating, 33 units pending; the
settle takes hours and nothing here asserts unit health or any service verdict.
G17's dc0 half is now genuinely armable -- the existing capture was taken on
the CONTROLLER VM and says so. The nine role nodes are booting for the first
time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Deploy attempt 2: prefer-ipv6 fixed, two new defects, model partially populated
...
The prefer-ipv6 failure did NOT recur. Two new ones did, and for the first time
the model is not empty: MEASURED 23 applications, 0 machines, 0 units.
Attempt 1 aborted during VALIDATION; this one reached EXECUTION, which is why
it left residue. NOTHING IS PROVISIONED -- no MAAS machine left Ready, no disk
written.
DEFECT 1: the rack's ~/repo-stage is a PARTIAL COPY of the repo with no
policies/, and bundle.yaml:214 is the only local-file reference in the deploy
input. juju resolves it relative to the bundle, so it must exist ON THE HOST
THAT DEPLOYS. Nothing caught it: --dry-run does not upload resources (the
SECOND --dry-run blind spot in two days), attempt 1 aborted before resource
upload, and preflight runs on voffice1 where policies/ DOES exist -- the same
wrong-host instrument class already recorded twice, and the second instance
this session. Fixed forward, both files sha256-verified.
DEFECT 2: a bundle relying on default-base is deployable exactly ONCE. The
re-run failed on barbican 265-vs-261. Measured: 265 is the ubuntu@22.04
revision, 261 the ubuntu@24.04 one. The 23 existing apps are all 22.04, so run
2a honoured bundle.yaml:85 -- but on a re-run, for an app that ALREADY EXISTS,
juju resolves without that default. A model-level default-base has NO EFFECT
(tested, then RESET). An explicit per-application base WORKS (tested read-only
on a throwaway; the error moved to the next app).
OWNED: two of those read-only tests were staged under /tmp and failed with
"no charm was found" -- the juju snap's PRIVATE /tmp, a trap this repo has
already recorded in CURRENT-STATE and the phase-4 runbook.
The rollback decision tree is written for tofu apply and defaults to
fix-forward, which is measured not to work here. A partial juju deploy is a
failure mode the repo does not cover. Recovery is an operator decision;
options are enumerated in the capture and the model residue is untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-deploy: rack input refreshed + hash-verified, preflight captured, dry-run green
...
(a) The dc0 rack's ~/repo-stage is a COPY, not a clone -- nothing updates it, and
it still held the pre-ruling vips overlay. Exactly one of four files differed;
copied, then ALL FOUR hash-compared against repo HEAD. The gitignored
octavia-pki overlay was deliberately not touched (SEC-029) and verifies
byte-identical at 5fc117f1, still 0600. A sweep is evidence about the deploy
only if the swept bytes ARE the deployed bytes.
(b) Preflight captured (242 lines, exit 1), run with
MAAS_PROFILE=vr1-dc0-region -- without it preflight is REGION-BLIND and emits 19
false negatives. Instrument proven current first: maas-profile-assert.sh
vr1-dc0-region hot-kid exits 0. P1-P4 and P7 PASS, incl. aligned VIPs 13 OK / 0
bad against the RE-RENDERED overlay. P5 FAIL, 11 findings.
(c) The P5 delta is ENUMERATED. The 2026-07-30 acceptance covered six findings
and says it covers "these six, enumerated, and nothing else". All six are still
present; FIVE NEW are all vr1-dc1 S2 rows (SEC-027/-028) -- the D-137 forward
register correctly reporting that dc1's region is not built. Diff appended to
the capture. Needs its own GA-R5 exchange before the deploy.
(d) Step 4.2 --dry-run against the fixed input: exit 0, 56 apps / 108 relations
/ 33 unit placements. The ceph-osd tags=openstack exposure is now MEASURED, not
reasoned: the plan places ceph-osd/0..3 on machines 5-8 by explicit id, so the
absent tag never has to match. The residual (a later UNPLACED add-unit) stands.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Sweep both deploy inputs for option NAMES and VALUE TYPES: clean, and proven able to fail
...
The two extensions owed after the ruling, now run:
dc0 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
dc1 75 assignments / 20 schemas / 0 unknown name / 0 type mismatch / 0 note
Both EXIT 0. dc1 introduces no new charm@channel pair -- checked, not assumed.
PROOF OF TEETH, because a clean sweep across two inputs is exactly the
implausibly-uniform result this repo has been burned by. Three defects planted
in a THROWAWAY copy, all three caught, EXIT 1:
[FAIL name] barbican does not declare option 'prefer-ipv6' (attempt-1 defect)
[FAIL type] keystone 'prefer-ipv6': declared boolean, got str ('true')
[note] glance 'vip': declared string, value is unquoted float
Type judgement is deliberately conservative: a mismatch is reported only where
juju's own coercion cannot save it. A scalar where a STRING is declared is a
NOTE, not a failure -- but it is still surfaced, because an unquoted value is
how a version string silently becomes a float. An UNRECOGNISED declared type
REFUSES rather than passing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
prefer-ipv6 research: size option (b) and scope the sweep claim
...
Two corrections to my own framing, both measured:
(1) Option (b) is NOT a values-file edit. 'prefer-ipv6: true' is INJECTED
UNCONDITIONALLY by scripts/render-dc-overlays.py:243-246 whenever fam ==
"dual", and 'family: dual' is a single TOP-LEVEL key in
render/values/vr1-dc{0,1}-vips.yaml:3 -- not per-app data. Implementing (b)
means teaching the renderer which charms declare the option, re-pointing
tests/render-drift, and replacing invariant 9: a code change with a harness.
Recorded before the fork is put to the operator so the ruling is taken on the
real cost.
(2) The option-name sweep's claim is scoped inline. It checked NAMES, dc0
only. Option VALUE TYPES (the 'type:' field in every schema already
downloaded) and the dc1 input are both owed before the redeploy, and each
could cost an attempt the same way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
prefer-ipv6 research DONE: the option never drove the v6 HAProxy bind
...
Answers the 2026-07-31 ruled question ("Research what those 6 charms do with
v6 VIPs first"). Measured by downloading each charm at its pinned channel for
amd64/ubuntu@22.04 and reading its templates and code -- not Charmhub's
config-yaml, which answers only whether the knob is declared.
FINDING: in BOTH template families the ':::port' frontend bind is gated on
ipv6_enabled = not is_ipv6_disabled(), a read of the kernel
net.ipv6.conf.all.disable_ipv6 sysctl -- MEASURED 0 on a live MAAS-deployed
jammy node in this DC. The v6 VIP is assigned by pacemaker on per-address
family detection. vault ships no haproxy template at all; its listener is the
hardcoded literal [::]:8200. prefer-ipv6 is a UNIT ADDRESS-FAMILY switch, and
the one value it sets that sounds like a bind address (haproxy_host) is
consumed by no template in any charm downloaded.
So option (b) -- drop the option from the six, keep their v6 VIP legs -- is
what the measurement supports. NOT TAKEN: the ruling is the operator's, the
overlay is untouched and R2 is unamended.
Also recorded: provider-bundle-check invariant 9's mechanism claim is wrong
and must be REPLACED (never deleted) when a ruling lands; a new and separate
question about the SEVEN charms that do declare the option; a full
option-name sweep of the dc0 deploy input (56 apps / 81 assignments / 20
schemas / 0 unresolvable) showing prefer-ipv6 on the six is the ONLY unknown
option, so nothing else is waiting for attempt 2; and a runbook defect --
the dc0 deploy block omits vr1-dc0-machines.yaml, which is NOT a no-op (it
carries ovn-chassis bridge-interface-mappings).
Read-only session: no cloud mutation, no overlay edit, no ruling adopted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Bundle deploy attempt 1 FAILED: prefer-ipv6 unsupported on 6 of 13 VIP charms
...
ERROR cannot deploy application 'barbican': unknown option 'prefer-ipv6'.
Model measured EMPTY afterwards -- juju validated and aborted atomically, so
nothing is half-applied and Steps 1-3.5 remain good.
The dry-run passed and the deploy did not: --dry-run resolves charms and plans
relations but does NOT validate config option names. A green dry-run is not a
gate on option validity.
Measured against charmhub's own config-yaml per pinned channel: 7 of 13 VIP
charms declare prefer-ipv6, 6 do not. barbican never had it at any channel, so
it was not an obsoleted option -- R2's uniform application was never valid.
Not fixed: invariant 9 couples prefer-ipv6 to the v6 VIP legs, so this needs a
GA-R5 ruling, not a mid-deploy edit.
Owned: my first charmhub query used a wrong field and returned a uniform
'no config' across 13 charms -- implausible, which is what caught it.
Capture: docs/audit/stage5-dc0-deploy-attempt1-20260731.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
JUJU CONTROLLER BOOTSTRAPPED: vr1-dc0-controller live, active/idle
...
Fourth attempt, first success. Each earlier failure was a real defect one layer
deeper and all three are now closed: D-138 (client into the DC), the controller
under-carve fix (default route), and D-132 q1 (per-DC MAAS region).
controller/0 active/idle on juju-controller rev 311; machine 0 started
ubuntu@22.04 as subtle-grouse; MAAS reads arfr7p Deployed ubuntu/jammy.
Transcript evidence: agent binaries resolved on attempt 1 (339-retry class stays
closed); targeting took the tagged VM not a role node; juju offered the v6 leg
restored earlier this session before connecting on metal-admin.
Capture: docs/audit/stage5-bootstrap-dc0-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 v4 node carve COMPLETE: 10/10 machines, gate 134/0, diff vs pre-migration EMPTY
...
Canary-first then node by node, never batched. 11 + 13x8 + 4 mutations, every one
read back. Nine role nodes on six planes with OVS br-ex; the juju controller on
two planes raw with no br-ex and enp3-6 left on auto VLANs.
Verified by diffing against the Office1 pre-migration capture: every node's six
v4 legs match address for address. Compared by LEG SET, since system_id and
hostname are both re-minted at re-enlistment.
Instrument note: the first extraction read the wrong capture field and returned
'-' for every old address, which would have read as 'the old region had no
statics'. Caught by the implausible shape, not by the parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-30 |
Close sweep: queued findings F1-F10, memory review, skill invariants
...
Sweep capture docs/audit/queued-findings-20260730-dc0-region-migration.txt.
Three items lived ONLY in the transcript and would have been lost on a
clear:
F1 -- this session's permission rules exist only in the GITIGNORED
.claude/settings.local.json, so their verbatim text is now recorded,
including a broad Bash(ssh vr1-dc0-rack *) that an interactive approval
auto-added and which deserves review.
F2 -- the Office1 region STILL registers a rack controller on the dc0 rack
(vvr1-dc0/7chphy) and still holds the dc0 region VM's machine record. Not
harmful (Office1's DHCP for that VLAN is off, verified by process) but
dual-registration the operator should rule on.
F10 -- the five instrument errors of this session as one pattern, with the
rule that catches them.
F9 carries the next session's first commands and every measured input the
carve tool needs.
GA-R7 memory review: sharpened the classifier memory with the profile-token
lesson (a rule pinning 'maas admin' would have matched step 1 of the DHCP
cutover and silently missed step 3), and added an instrument-currency
memory. Both graduate to repo surfaces; neither is the only copy.
Skill swept: the per-DC region migration invariants folded in -- region
identity by rack, ids do not cross regions, the per-node ordering, batches
of three, zero SSH keys, the role-dependent carve, the role-tags gate that
passes on an incomplete tag set, and the two scripts that must not be
pointed at a new region.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Plane IPAM complete on the new region: 24/0 PASS, existing tooling only
...
dc-plane-ipam.sh check vr1-dc0 goes from pass 7 / fail 17 to pass 24 /
fail 0, exit 0. Two idempotent applies, dry-run first, every write read
back: 13 D-134 reserved bands plus the FIP pool, then the 5 missing v6
plane subnets each created on the same MAAS vlan as its v4 twin.
No new code was needed. The tool's forced sequencing also proved itself --
the reserve run skipped every v6 band with 'not in MAAS yet, run carve-v6
first' rather than failing or silently no-opping, so running the two out of
order is safe.
This is the step that justifies the tunnel architecture: carve-v6 derives
its prefixes from the NetBox apex, reachable only from voffice1, while the
target region is reachable only via the rack. Neither host alone could have
run it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
All ten dc0 machines Ready in the per-DC region; task 6 complete
...
Enlisted as canary + three batches of three. Shapes match D-121 Option C
exactly (3x16/64, 2x12/48, 4x8/24, 1x4/8), all with 6 interfaces, every one
matched to its libvirt domain by pinned boot MAC -- hostname and system_id
were both re-minted.
maas-role-tags.sh check: 0 tags missing, 0 nodes needing a tag, 0 not in
MAAS, exit 0.
But it reported that clean PASS while the machines were still missing the
openstack-vr1-dc0 placement tag, which is not in its ROLES set and has no
creator anywhere in the repo. bundle.yaml places on that tag and
dc-node-v6-carve.py dies without it, so both the deploy and the v6 carve
would have failed. Caught only by diffing against the pre-migration
snapshot. Applied to the nine role nodes and deliberately not to the
controller -- reproducing the Office1 shape row-for-row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Canary re-enrolment: the ordering a fresh region actually needs
...
Canary xqqwdq/moral-salmon = vr1-dc0-storage-04 (a storage node, least
central of the ten). Office1 23 -> 22; it re-enlisted into the new region
carrying boot MAC 52:54:00:2b:ed:ab -- the machine that arrived is the
machine that left. system_id AND hostname are both re-minted
(xqqwdq -> 6q4syf, moral-salmon -> civil-bug), so the boot MAC is the only
stable key; the pre-delete snapshot records it as the identity.
MEASURED ORDERING, not the obvious one: a freshly-enlisted machine has no
power config, so MAAS cannot power-cycle it and cannot drive commissioning.
delete -> power on BY HAND via virsh -> self-enlist -> SET POWER -> abort ->
commission. Skipping the power step wedges the machine in Commissioning
producing zero events, because the enlistment ephemeral powers the node off
and nothing can bring it back.
maas-node-power.sh already supports the split the migration needs: VIRSH_URI
enumerates from voffice1 over the transit /30 while the positional address
stores what the REGION dials over metal-admin. Result was a real
query-power-state, proving the region drives the rack's libvirt with the
SEC-012 key in its snap.
Found: the per-node serial log stopped capturing on 2026-07-21 while the
domain XML still configures append-mode logging. It contains a complete,
plausible boot that I read as current and reported as fact before checking
the mtime. An append log that silently stopped appending returns confident,
well-formed, wrong evidence. Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DHCP handover EXECUTED: dc0 metal-admin now served by the DC-local region
...
Ruled order, region identity proven first. Office1 vlan update 4 0
dhcp_on=false -> read back False; verified STOPPED BY PROCESS (pgrep -c
dhcpd = 0 on the rack, because MAAS self-report is not evidence -- the
2026-07-20 Temporal incident class); new region vlan update 0 0
dhcp_on=true primary_rack=c3aqh8 -> read back True on fabric-0 / space
metal-admin; verified RUNNING BY PROCESS on hot-kid with the rack
re-checked at 0. Exactly one DHCP server on the segment. All 10 machines
Ready and powered off throughout.
Recorded so it is not later read as a fault: the new region also starts
dhcpd6 where the Office1 rack ran v4 only. Its v6 metal-admin subnet has
NO dynamic range -- only rfc-4291-2.6.1 and assigned-ip/reserved, same as
the old region -- so it has nothing to hand out and nodes keep their ruled
statics.
The permission wall was a rule that FAILED TO MATCH, not a classifier
fault. Existing ask rules pin the double-quote form while the commands used
single quotes, and pin the maas admin profile while the new region needs
maas vr1-dc0-region. Operator-approved ask rules added to
settings.local.json (gitignored; committed team policy unchanged),
profile-agnostic so both regions match, plus read-only pgrep/ps allow
rules. The first rule shape I proposed pinned maas admin and would have
matched step 1 while silently missing step 3 -- stalling the cutover at
exactly the two-servers boundary.
Capture docs/audit/dc0-dhcp-handover-20260730.txt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Session record: dc0 region topology 40/0; cutover blocked with exact commands
...
Records the fabric-rename remediation (changelog item 12), the two
pre-cutover checks (hot-kid IS eligible as primary_rack for fabric-0 -- had
it not been, DHCP-off would have succeeded and DHCP-on failed, leaving the
segment with no DHCP and ten machines about to PXE), the measured NIC ->
plane order, and the exact unblock sequence with every parameter
re-resolved live.
hot-kid is c3aqh8 in the new region and tw7ptw in Office1 -- the same
machine, two ids.
Capture updated with the corrected 40 passed / 0 failed, exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 region plane topology BUILT and verified (39/0); DHCP cutover blocked
...
dc-region-topology.sh applied to the new dc0 region: 5 named plane fabrics,
6 spaces, 4 plane subnets, 10.12.4.0/22 MOVED off the auto fabric-1 onto
vr1-dc0-provider-public, 6 VLAN->space bindings, openstack-vr1-dc0 tag.
Named check now reads 39 passed / 0 failed, exit 0.
metal-admin service config set: dns_servers=10.12.8.6, allow_dns=false,
D-134 dynamic range .201-.254, subnet resolved by CIDR.
The DNS target was measured rather than carried over, and my first reading
was wrong: a +time=3 +tries=1 probe suggested the new region's BIND could
not resolve externally. That was a cold-cache timeout. At +time=5 +tries=3
it answers archive.ubuntu.com, streams.canonical.com and api.snapcraft.io
with flags qr rd ra. Pointing node DNS at the DC-local region is correct --
it is authoritative for the zone the migrated nodes will live in and drops
the cross-fiber dependency D-132 q1 exists to remove.
Measured in passing, and worth keeping: VLAN row id 5005 is dc0's
metal-admin VLAN in Office1 and vr1-dc0-data-tenant in the new region.
Same integer, different DC plane.
The DHCP cutover was REFUSED by the harness classifier and was not retried
in an altered shape. Verified read-only afterwards that nothing is
half-applied: Office1 still reads dhcp_on=True primary_rack=7chphy and the
rack still runs exactly one dhcpd on virbr2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Per-region power address, region power-key tool, and two red gates fixed
...
lib-hosts.sh: MAAS power ops originate from the REGION, so the reachable
rack address depends on which region dials. Measured from vr1-dc0-maas-01:
the rack's transit leg 172.31.0.2:22 is CLOSED, its metal-admin leg
10.12.8.2:22 is OPEN. Adds VIRSH_POWER_ADDRESS_FROM_{OFFICE1,DCREGION} for
both DCs (dc1's 10.12.68.2 measured on the dc1 rack, not inferred from
dc0's .2). VIRSH_POWER_ADDRESS still aliases the Office1 form so existing
callers are byte-for-byte unchanged. dc-selector 81 checks; 4 mutations
each killed tests.
New scripts/maas-region-power-key.sh (harness 44/44) installs and verifies
the per-DC MAAS->libvirt key inside a region's snap, ending on a real virsh
call rather than on config. Runbook prerequisite 5 carried this as prose,
and prose is not a gate -- it must also be re-asserted after every snap
refresh. LIVE at dc0: 9/9 exit 0, virsh enumerates 12 domains. The key is
the SEC-012 dc0 key already authorized on that rack, so per-DC isolation
holds.
Three defects its own testing found: ssh-keygen -lf reads an adjacent .pub
and reports ITS fingerprint; a snap's data root (/var/snap) and exec root
(/snap) are not interchangeable; the snap's virsh must run inside the snap
runtime or it fails on missing libraries.
tests/node-vm T8/T9 were RED at HEAD before any of this session's work
(proven by stashing and re-running): commits 086c827 + 447315f added the
region VMs, moving the dc0 root to 11 macs lists / 66 MAC literals against
assertions reading 10 / 60, and the gauntlet was not re-run. Re-pointed to
the new invariant, reason recorded in-file, re-proven able to fail.
HARNESS-MANIFEST re-recorded (91) for the two new harnesses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Region-resolution gate: maas-profile-assert.sh + named dc0 region profile
...
The wrong-REGION hazard is the migration's unrecoverable failure mode: 13
repo scripts default to MAAS_PROFILE=admin, which on voffice1 resolved to
the OFFICE1 region. Against the wrong profile a carve recreate is a silent
no-op that prints PASS, and machine delete destroys the real record.
New scripts/maas-profile-assert.sh asserts which region a profile resolves
to by rack-controller IDENTITY (a machine count is not proof -- two regions
can hold the same number). Harness 20/20; mutation pass found one of its own
assertions could not fail (nameless rack entries were caught by a different
branch) and it was made branch-specific.
Live both directions: admin -> 23 machines / office1 trio (exit 0);
vr1-dc0-region -> 0 machines / hot-kid (exit 0); admin asserted as hot-kid
(exit 1). Capture docs/audit/dc0-region-profile-assert-20260730.txt.
Tunnel voffice1 -> rack -> 10.12.8.6:5240 because no single host has repo +
CLI + new-region reach + NetBox apex reach. Application-layer and
rack-originated, so SEC-010 is not punctured; nothing was rerouted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 session close: Stage 5 opened, D-138 + D-132 ruled, dc0 region live
...
Sweep first: every in-session finding grepped against CURRENT-STATE and the
changelog. THREE lived only in the transcript and would have been lost -- the
absent openstack CLI on the DC client host (blocks Step 7+), the metal-admin
v6 rack-leg gap, and the plane-fabric mtu=1500 vs libvirt 9000 mismatch. All
now in docs/audit/queued-findings-20260730-stage5.txt (F1-F12), with six tool
traps collected as platform-traps candidates.
F6 recorded honestly: the as-executed log for this window is PARTIAL. The
classifier refused several `script -aqe ... -c` wrapped forms while the plain
ssh form passed, so those calls ran unwrapped. The index row says so -- a log
that silently looks complete is worse than one declaring its gap.
Bookend appended (14 lines, under the GA-R4 cap; ledger 250/300 so no
rotation owed). Counters: 3 decisions, SEC 23, D 139 / DOCFIX 206 /
BUNDLEFIX 053.
GA-R7 memory review: one entry added (as-executed vs classifier), indexed.
Nothing in memory claims operator policy.
Stage 5 remains OPEN. Nothing is half-applied -- Office1 still owns all 9 dc0
nodes, correctly carved and Ready.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Region images synced (jammy+noble); pre-migration carve captured
...
Both releases now Synced on the dc0 region, 6 rows each, matching Office1.
jammy is NOT seeded by `maas init` (noble only) and the OpenStack nodes
require it. The first import raced the selection's creation and returned
noble-only while reporting is-importing=false -- assert on Synced ROWS, never
on the import call returning.
Measured and recorded so the two are not conflated: MAAS boot images do NOT
come from the DC mirror. The region pulls images.maas.io; the mirror 404s on
every simplestreams index path because dc-mirror.sh mirrors debmirror APT
packages only. Three artifact classes, one local. Consequence: the
D-135/D-107 egress narrowing would BREAK image sync and juju bootstrap, not
just slow them, and nothing sequences it after them.
Pre-migration carve captured to docs/audit (179 lines): fabrics, VLANs,
spaces, subnets, D-134 bands, tags, per-node identity/power and the full
per-interface carve. MAAS cannot move machines between regions, so the
migration must recreate all of it.
The destructive half has NOT started -- the Office1 region still owns all 9
dc0 nodes, correctly carved and Ready. Nothing is half-applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
G17 dc0 CAPTURED (both assertions PASS); bootstrap root cause = under-carved controller
...
G17's one-shot first-boot window opened during the second bootstrap attempt and
was taken rather than lost. Capture docs/audit/g17-dc0-firstboot-20260730.txt,
exit 0: mirror asserted on FOUR body fields of a real package path (not the
autoindex root), and the node time source is the MAAS rack 10.12.8.2 chaining
to the region, not the DC edge. Scope stated: the node is the dc0 CONTROLLER
VM, not a role node; G17's dc1 half stays OPEN.
Two defects in G17's own text, both measured: it names `chronyc sources` and
chrony is NOT installed on the MAAS jammy image (node runs systemd-timesyncd),
so the gate as written can only REFUSE; and a first pass false-FAILED on
`printf | grep -q` under pipefail -- the same trap recorded earlier the same
day. Capture uses here-strings.
Root cause of both bootstrap failures is measured and is NOT architectural:
the controller VMs are UNDER-CARVED. D-138's path works -- the second attempt
reached "Connected to 10.12.8.5" -- then failed fetching the juju agent
because the node has NO DEFAULT ROUTE. Role nodes carry provider-public with
gw 10.12.4.1; the controllers carry metal-admin only, because they were added
after both carves and Stage 4's 90 NIC re-homes covered the 18 role nodes.
Owned: this session's egress re-probe was run FROM THE RACK and reported the
window open. The rack holds a default route nodes lack, so it did not
discharge the caveat for a node-side fetch. Same wrong-host class as the
dc-mirror instrument note, made after writing that note.
Blocked on a harness permission wall for four MAAS interface calls, not on a
decision. Not retried in altered shapes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 5 BLOCKED: no juju-client->node-plane path; D-134 .5 now BUILT both DCs
...
Bootstrap attempted and failed. Machine selection was correct (7n87bt, the
tagged VM) and MAAS deployed jammy end to end -- confirming the
--bootstrap-base ubuntu@22.04 pin against a live deploy. juju then could not
SSH the machine and released it.
Root cause: voffice1 has NO route to any DC node plane and two deliberate
controls forbid one -- SEC-010's transit FORWARD-drop on the rack, and
libvirt's blanket reject into the isolated plane bridges. The DC edge has no
metal-admin leg. Nothing regressed; this path never existed.
This is a contradiction between RULED surfaces: SEC-010/D-052 make
metal-admin DC-local and forbid the region routing to 10.12.8.0/22, while
D-100 says the fiber carries Juju traffic and D-128 puts the juju client on
voffice1. SEC-010's "pinning is free" justification was priced against MAAS,
which proxies at the app layer; juju dials the machine at L3 and was not in
scope. Needs a ruling, not a firewall edit. No reachability change made.
BUILT, both DCs (operator: "Fix now: static .5 + v6, re-bootstrap"): the
controller VMs held AUTO v4-only addresses; now static 10.12.8.5 +
fd50:840e:74e2:220::5 and 10.12.68.5 + fd50:840e:74e2:320::5. v6 prefixes
confirmed by VLAN pairing, not inferred. D-134's amendment was
ruled-but-not-built until now.
Capture: docs/audit/stage5-bootstrap-reachability-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
