| 2026-08-02 |
dc0 deploy blocked: mirrored UCA has no signing key on fresh nodes; root-caused
...
All 9 machines reached started. Nine units then failed the install hook across
unrelated charms -- one common cause:
W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease:
NO_PUBKEY 5EDB1B62EC4926EA
E: The repository is not signed.
apt-get update returned exit 100
The mirrored content is reachable and its upstream signatures are intact; the
key to verify them is absent.
This refutes a claim overlays/vr1-dc0-machines.yaml carried as MEASURED -- that
the UCA key is already on the nodes. Measured on ceph-osd/0: trusted.gpg.d holds
only the cdimage and 2018-archive keyrings. The cloud-archive keyring ships in
the ubuntu-cloud-keyring package, installed as a side effect of the
cloud:<series>-<release> origin path; a raw deb line bypasses it. The claim was
almost certainly measured on a node from the 07-31 deploy, which used
cloud:jammy-caracal and had the keyring. True when taken, false on a fresh node.
Fix is charm-documented, quoted from the charms: openstack charms take
openstack-origin "deb <URL>|<PUB KEY>"; the three ceph charms take a separate
key option accepting ASCII armor. Armored block, never a keyid -- a keyid sends
charm-helpers to keyserver.ubuntu.com, unreachable under the D-107 airgap.
Key provenance measured, not downloaded: the mirror serves ubuntu-cloud-keyring
2021.03.26; URI from apt-get download --print-uris on a node (source package is
ubuntu-keyring, not ubuntu-cloud-keyring -- I guessed and got a 404); sha512
matched apt's declared digest; keyring holds exactly 5EDB1B62EC4926EA. Armored
export committed as public key material.
PREPARED, NOT APPLIED: config for all 15 apps generated from that one verified
key and staged on the rack. juju config --file was refused by the permission
layer in three shapes; it is a live mutation across 15 apps and belongs to the
operator. No workaround attempted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
STAGE 5 dc0: BUNDLE DEPLOYED, exit 0; Step 4.4 passes and proves the options merge
...
Operator: "Approved, continue as autonomously as possible" -- covering the
single-file overlay re-stage and Step 4.3.
Re-stage restored the digest-match invariant (dd914358 both ends). All six
deploy inputs sha256-verified against HEAD in the same window, model confirmed
empty, then the DOCFIX-208 command: "Deploy of bundle completed." EXIT 0.
Step 4.4 ran at its first ever execution and passed all three items. Item 1 is
the consequential one: juju config ovn-chassis shows ALL THREE options with
source: user -- bridge-interface-mappings (exactly two MACs), ovn-bridge-mappings
physnet1:br-ex, prefer-chassis-as-gw true. JUJU MERGES AN OVERLAY'S options MAP
KEY-BY-KEY RATHER THAN REPLACING IT. Both machines overlays have carried that as
"near-certain but NOT PROVEN HERE" since 2026-07-29; no offline check could
settle it and the dry-run prints no options at all. The whole *-vips.yaml
architecture rests on it and it is now measured.
Item 2: all fifteen apps read the in-DC mirror, none the UCA -- the second
injury DOCFIX-208 prevented, confirmed prevented live. Item 3: ovn-central reads
source: zed, its charm default, deliberately left alone.
The gate needed no shape correction; every key was present, so the .get()
sentinel hardening was never exercised.
Also measured: ceph-osd/0..3 already carry GUA IPv6 2602:f3e2:f02:10::150-153,
so the D-139 carve is live on the addresses juju provisions against.
Convergence in progress; target is phase-01's PRE-vault-init end state.
Capture: docs/audit/stage5-deploy-dc0-20260802.txt. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
WITHDRAWN: the designate "contradiction" was mine, not the bundle's
...
I raised, as an unresolved question for the operator, that the Stage-5 plan
deploys four designate applications while a phase-4 bullet says the bundle
ships "NO designate". There was nothing to resolve.
D-019 (v1 ships no cloud DNS) is SUPERSEDED by D-106, which REACTIVATES
Designate for VR1. DOCFIX-167 put all four designate applications into this
bundle.yaml on 2026-07-10, and phase-01-bundle-deploy.md:174 already says the
change "corrects this GATE's old 'NO designate (D-019)' text, since D-019 is
superseded". Stage 7 owns the DNS ACTIVATION -- per-DC zones + A/AAAA per the
D-008 bootstrap order, FQDN-SAN certs, the B5 os-public-hostname reversal --
explicitly not the charm deploy. Four designate applications in the plan is the
planned outcome; their absence would have been the defect.
The stale artifact was the phase-4 bullet alone, paraphrasing a gate text
phase-01 corrected 23 days earlier. Rewritten to state the D-106 reactivation
and the Stage-5/Stage-7 split. My annotation is deleted, not kept as history --
leaving it would mislead the next reader about a live deploy input. Step 4.4's
speculative designate caveat removed for the same reason.
Operator correction quoted in CURRENT-STATE and the changelog: VR0 -> VR1 is
ADDITIVE, and a manufactured contradiction costs next-step attention, which is
this project's named failure mode. Also recognised late: the dry-run's ignored
"name"/"variables" field warnings are the benign R11 pair phase-01 already
documents.
repo-lint 0 fail / 1 legacy warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-208 part 3: graduate the dry-run instrument finding; harden Step 4.4
...
The juju finding belongs above a changelog -- it generalizes to any bundle
deploy, not just phase-4's gate. platform-traps.md gains a Juju section: what
--dry-run prints, what it never prints at any verbosity, the two grep decoys
(controller API addresses look like DC-band VIPs; --debug echoes the command
line so overlay filenames hit), and the three consequences.
Step 4.4, written earlier this session and never run, gets two corrections
before its first execution: the --format=json read used a direct index, so a
missing key would print a traceback into a results column -- this repo has a
logged instance of an assertion satisfied by a traceback -- now .get() with a
visible <<UNSET-OR-APP-ABSENT>> sentinel documented as a failure. And the block
now states plainly that it is authored against a measured instrument but has
never been exercised, with the instruction to report a shape correction rather
than work around one.
repo-lint 0 fail / 1 legacy warn; gauntlet ALL GREEN (98).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-208 part 2: the dry-run graded the gate, and three items could not fail
...
The corrected dc0 command dry-ran EXIT 0 from the dc0 rack against an empty
vr1-dc0 model, staged input sha256-verified against HEAD first. Nine machines
0-8, all nine constraints tags=openstack-vr1-dc0,<role> in the 3/2/4 split,
all three overlays consumed. Capture: docs/audit/stage5-dryrun-dc0-20260802.txt.
The run also measured what the command can and cannot show. juju deploy
--dry-run on a bundle prints NO application options and NO VIPs at any
verbosity; --debug adds only the per-machine constraint lines. Greps over the
--debug capture: bridge-interface-mappings 0, physnet1 0, openstack-origin 0,
cloud-archive 0, prefer-chassis 0.
So three Step-4.2 gate items could not be graded by the command the gate names:
item 1 (tags) needed --debug, which is now in the named command; the
PRE-EXISTING VIP item could never have failed because no VIP is ever in the
plan, and its property is relocated in writing to preflight P2; and the two
option-reading items I added earlier this session had the same defect and are
removed within the hour. Their properties become new Step 4.4 -- a juju config
read run the moment deploy returns, which is the only thing that can prove
juju merged the options map rather than replacing it.
Both machines overlays' VERIFY-LIVE headers said to assert this "at the live
dry-run". Measured impossible; corrected in place, pointed at 4.4.
Also settles the deferred ceph-osd tags=openstack item (placed by explicit
placement, bare tag absent from the plan) and logs a contradiction not resolved
here: the runbook says the bundle ships NO designate, the plan deploys four
designate applications.
The deploy itself is NOT run -- 4.3 is a gated mutation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-208: the dc0 deploy command was missing a load-bearing overlay
...
The phase-4 runbook told the operator to deploy vr1-dc0 WITHOUT
overlays/vr1-dc0-machines.yaml, on the stated grounds that the file does
not exist. It has existed since 2026-07-29. preflight.sh P2 folds it into
the merged input it validates, so P2 was grading an input the deploy would
not have passed.
Re-measured by deep-merging bundle.yaml against the overlay: SIXTEEN
applications carry a delta, not the one this repo recorded on 2026-07-31.
ovn-chassis.bridge-interface-mappings (whose ONLY source is this overlay --
bundle.yaml:494 forbids re-adding it there) plus 15 mirror repoints from
the 2026-07-31 UCA ruling. Deploying as written would have taken two
injuries, neither surfacing as a deploy error: no br-ex mapping (dead
provider egress) and 15 charms pointed at a UCA measured unreachable from
a node under the D-107 airgap.
Corrected at 6 sites (4.1 table + prose, 4.2 dry-run, 4.3 deploy, Step 7
dry-run + its rules sentence, 4.2 VERIFY-LIVE pointer). Step 4.2's gate
could not have caught any of it -- its four items read the machines block,
machine count and VIPs, while the overlay's whole payload is under
applications: -- so gate items 5 and 6 were added to assert the merged
ovn-chassis options map and the reachability of every origin/source.
preflight.sh needed no change: it was already right.
CURRENT-STATE corrects its own narrower 2026-07-31 finding in place
(GA-R1 C2). repo-lint 0 fail / 1 legacy warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
SESSION CLOSE 2026-08-02 (part 2, GA-R4 bookend): queued backlog cleared, mirror root-caused, D-139 step 6 EXECUTED
...
DURABILITY TRIAD, MEASURED: vcloud 0 uncommitted / 0 unpushed; voffice1 SYNCED (was 1
behind -- it is the D-128 Plane-2 host where preflight runs); dc0 rack ~/repo-stage all 13
tracked files MATCH the repo, so the D-138 deploy client is in sync. That last check is the
one that caught the stale VIP overlay at the previous close.
GATES, quoted: repo-lint 0 fail / 1 warn (legacy D-001..018 ASCII carve-out); run-tests-all
GAUNTLET: ALL GREEN (98 harnesses), 97 -> 98 for the one harness added with its manifest
recorded deliberately; ledger-scan 3 open decisions, 28 open SEC (none opened this session),
next-free D-141 / DOCFIX-208 / BUNDLEFIX-053. RECONCILED: DOCFIX 207 -> 208 matches the one
number assigned; SEC unchanged at 28 matches zero rows opened; manifest 97 -> 98 matches the
one harness added.
WHAT THE SESSION DID: closed sweep F1 (rack deploy input re-staged) and F6 (maasdb proven
uncorrupted by a full-table pg_dump); diagnosed then ROOT-CAUSED F2 -- one of nine
archive.ubuntu.com backends hangs on a single dep11 object while serving its siblings, the
resolver rotates, 11/12 fetches succeed, and apt is unaffected because it fetches the .xz;
graduated F3/F4/F5 to platform-traps and script-authoring; assigned DOCFIX-207; and EXECUTED
D-139 step 6 -- 26 GUA VIP addresses created, 26 ULA addresses + 9 prefixes deprecated,
nothing deleted, plus 4 of 5 MAAS ULA subnets removed with the fifth HELD because it carries
the juju controller's and the MAAS region's only recorded v6.
CLOSE SWEEP: docs/audit/queued-findings-20260802-step6-queued-items.txt -- SIX FIRST
SURFACE, each grep-proven absent from every repo surface first. Highest-consequence: a broad
Bash(ssh vr1-dc0-maas *) allow rule in gitignored state, and -- found by tracing consequence
-- four destructive MAAS deletes that matched NO ask rule, because the committed rules pin
the `maas admin` profile and the double-quoted form while this session used
`maas vr1-dc0-region` in single quotes. Consent was given separately, so nothing was
bypassed, but the GATE did not fire; same rule-fails-to-MATCH class as the 2026-07-30 DHCP
cutover, now RECURRING rather than one-off.
GA-R7 MEMORY REVIEW: CLEAN -- zero entries claiming operator policy, priority or posture.
One update: instrument-currency gains instance THIRTEEN, a new shape (a comparison that
could never match, returning a clean ZERO) plus the meta-lesson that writing the rule down
does not inoculate against it -- three checkers that could not fail shipped in the same
session that landed the rule, and none was caught by re-reading my own work.
LEDGER ROTATED (GA-R4 rule 3): 294 lines would have breached the 300 cap; the TWO oldest
closed summaries moved VERBATIM to docs/archive/session-ledger-rotated-20260802b.md; ledger
now 287. No orphaned session -- rule 7 is a no-op. NO STAGE OPENED OR CLOSED; Stage 5
remains OPEN and this is a session bookend.
NEXT SESSION'S FIRST ITEM: the preflight-P2 / phase4 machines-overlay asymmetry -- P2
validates a merged input the deploy never passes -- then the bundle deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
retire-v6-ula harness + the defect it found in my own by-hand check
...
Operator direction: "Proceed with 1 and 2". Item 2 (the repo tool for the MAAS half) is
delivered. Item 1 (the held ULA subnet) is NOT executed, and the tool is why.
THE TOOL'S FIRST LIVE RUN CAUGHT A DEFECT IN THE BY-HAND CHECK USED AN HOUR EARLIER, and
that is the more important finding here. It reported `interface links: subtle-grouse:enp1s0`
on subnet 2. My by-hand check had reported ZERO links for all five subnets, because the
subnet id was interpolated as sid="'$id'" -- the comparison ran against "'2'" and could
NEVER match. A checker that could not fail, written in the same session that landed exactly
that rule into script-authoring.md.
CONSEQUENCE: the earlier claim that the four deletes were "proven safe twice" was HALF
FALSE. The allocation count was sound; the link half was inert, so link status was UNKNOWN
at the time of those deletions.
THE OUTCOME WAS SAFE ANYWAY, MEASURED NOT ASSUMED: all nine role nodes read v4=6 v6=6, so
every v6 link they carry is GUA and the four deleted ULA subnets genuinely held none.
enp2s0 unlinked on all nine is EXPECTED (provider-public carries no L3 link; the static
lands on br-ex, D-133/D-100). subtle-grouse (the controller, Deployed) reads v4=2 v6=1, its
single v6 link being the ULA ::5 -- independently confirming the hold was correct. Right
outcome, unsound verification; recorded as two separate facts.
The mode lives in dc-plane-ipam.sh rather than a new script because dc-region-topology.sh's
header already assigns v6 plane subnets to it. It refuses on an allocation OR an interface
link, NAMES the holder, deletes one at a time (never looped, hard rule 3), and READS BACK
that the row is gone -- R6 proves that assertion has teeth by faking a delete that returns
success and leaves the row.
Harness 25 -> 32 (R1-R7), delta checked. R4 is the case that matters: an interface link
with NO allocation still refuses -- the exact blind spot the broken by-hand check had.
ITEM 1 NOT EXECUTED: retire-v6-ula REFUSES on fd50:840e:74e2:220::/64, naming ::5
(juju-vr1-dc0), ::6 (MAAS) and the link subtle-grouse:enp1s0. Deleting it needs the juju
controller's and region's v6 re-homed to GUA first -- re-addressing the DEPLOY CLIENT
immediately before the deploy, and this project has already lost three bootstrap attempts
to under-carved controller networking. Not covered by an approval given about four empty
subnets, and NOT deploy-blocking: an extra ULA subnet in MAAS harms nothing.
Gates: gauntlet ALL GREEN (98); repo-lint 0 fail; dc-plane-ipam 32/32.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-plane-ipam: add retire-v6-ula (D-139 step 6 MAAS half), refuse-on-non-empty
...
The MAAS half of step 6 was executed BY HAND for dc0 on 2026-08-02 -- four maas subnet
delete calls -- which is the 'no repo tool, mutations live only in a transcript' gap D-132
q1 already forced this repo to close once for the region carve. Without this the dc1
rebuild cannot reproduce the dc0 end state.
The guard is the point and it is not theoretical: on dc0 fd50:840e:74e2:220::/64 looked
like the other four but held two STICKY allocations -- ::5 user juju-vr1-dc0 (the deploy
client) and ::6 user MAAS (the region the delete is issued to), neither with a GUA
counterpart. So a holder or an interface link is a REFUSAL, and the refusal NAMES the
holder: '2 allocations' does not tell you it is the controller.
Both checks are required -- zero allocated IPs does not mean nothing references a subnet.
Deletes one at a time, never looped (hard rule 3), and READS BACK that each row is gone,
because a delete returning success is not evidence the row went away.
Harness update follows in the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 STEP 6 EXECUTED: apex applied; 4 of 5 MAAS ULA subnets deleted, 1 HELD on a hazard
...
Operator approval, exact utterance: "Queue up the MAAS half to run after you complete
netbox. I approve the MAAS deletes to clean up the data and config. Go ahead with both now".
APEX HALF APPLIED, exit 0: CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9,
both read-backs OK. Independently re-verified rather than taken on the tool's own word --
a re-run reports CREATE 0 | ALREADY 26 | DEPRECATE-ADDR 0 | DEPRECATE-PFX 0, fully
idempotent and converged.
Recorded so a later reader is not misled: d139-gua-carve.py STILL reports "dependent
objects ... 26 ip-address(es)". That is correct -- the ULA records still EXIST, deprecated
rather than deleted, per the ruling. The carve tool counts existence, not status.
VERIFY BEFORE MUTATE FOUND A REAL HAZARD AND THE FIFTH SUBNET WAS NOT DELETED. The approval
was given before anyone knew what fd50:840e:74e2:220::/64 held. Measured: ::5 STICKY user
juju-vr1-dc0 (the JUJU CONTROLLER) and ::6 STICKY user MAAS (the MAAS REGION VM ITSELF),
with NO GUA counterpart -- the GUA metal-admin subnet holds exactly the nine tagged role
nodes (::100-102, ::120-121, ::150-153). This matches the record that dc-node-v6-carve.py
walks only those nine and that the controller's v6 was "restored separately", on ULA and
never migrated. Deleting it would remove the only recorded v6 of the deploy client AND of
the region the delete is issued to. HELD for its own decision.
The four empty subnets were proven safe twice: zero allocations AND zero machine-interface
links across all five -- the stronger check, since "no allocated IP" does not mean "nothing
references it". Zero links also confirms D-139 step 3 succeeded. Deleted INDIVIDUALLY with
a fresh re-check before each, never looped (hard rule 3; this repo has a logged incident
from batching nine deletes in a loop): subnet 9 :221::/64 metal-internal, 8 :230::/64
data-tenant, 12 :240::/64 storage, 11 :250::/64 replication -- all rc=0.
Post-state measured: six GUA /64s remain, one ULA held; machines unchanged and healthy at
9 Ready + 1 Deployed, total 10.
Still open: the held subnet (migrate controller/region v6 to GUA then delete, or delete and
accept the loss, or leave). And NO REPO TOOL exists for the MAAS half -- these were
hand-issued maas subnet delete calls, so the dc1 rebuild cannot reproduce them. Owed.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 step 6 tool built + adversarially reviewed; 4 defects fixed. Apex NOT yet written.
...
Implements the two 2026-08-02 rulings ("Full step 6 first, then deploy" / "Deprecate both,
delete nothing"): CREATE 26 GUA VIP addresses -> read-back verify -> deprecate 26 ULA
addresses + 9 ULA prefixes. NO delete path anywhere, asserted against the artifact.
Dry run (live apex): CREATE 26 | ALREADY 0 | DEPRECATE-ADDR 26 | DEPRECATE-PFX 9. The 26
CREATE targets diff EXACTLY against the 26 GUA VIP legs in overlays/vr1-dc0-vips.yaml --
every address written is one the deploy configures. --commit has NOT been run.
AN ADVERSARIAL REVIEW RETURNED "FIX FIRST" AND WAS RIGHT ON ALL FOUR COUNTS
(docs/audit/d139-step6-tool-review-20260802.txt). Mapping logic was correct; the gaps were
preconditions and coverage.
DEF-1 CRITICAL -- --dc vr1-dc1 would ORPHAN-CREATE. Targets were computed arithmetically
and never checked to exist. Measured: dc1's GUA carve is incomplete (four provider-public
rows under 2602:f3e2:f03::/48, no :20::/64, no :21::/64), so dc1 planned 26 creates into
non-existent prefixes then deprecated dc1's only authoritative rows, rc=0, no warning. dc0
hid it because all sixteen of its targets happen to exist. Reachable via the other valid
value of a required flag. FIXED + verified live: dc1 refuses, dc0 unchanged at 26/26/9.
DEF-2 CRITICAL -- the apex-IDENTITY guard was gone. It lives in d139-gua-carve.py's main()
(:159-163) and importing a module never runs its main(), so subclassing C.NB inherited the
TRANSPORT and left the SAFETY POSTURE behind: netbox.baldurkeep.com (the v1 reference)
would have connected fine and taken writes. FIXED: identity checked before any network call.
DEF-3 HIGH -- silent under-count. One missing ULA /64 row gave CREATE=13/DEPA=13/DEPP=8 at
exit 0. FIXED: any ULA address claimed by no prefix row refuses. My first fix was itself
wrong and RUNNING it caught that -- it scanned the whole retired /48 and flagged dc1's 26
VIPs while planning dc0; the /48 is SHARED (dc0 :22x, dc1 :32x). A /60 parent deliberately
does not count as coverage: the reviewer's scenario was a missing /64 whose /60 survived.
DEF-4 HIGH -- main() had ZERO coverage; the reviewer hoisted the deprecate loops above the
create phase and the suite reported ALL PASS. FIXED: T16-T18 drive main() through a fake
client that records CALL ORDER, proven by re-running that exact mutation on a copy (T18
goes RED).
TWO OF MY ASSERTIONS COULD NOT FAIL and the review killed both. T13 asserted the ABSENCE of
a string, so a traceback satisfied it -- it passed against a tool file that did not parse;
it now requires a positive, well-formed, DIFFERENT target, and new T15 asserts the tool
parses. T14 grepped ONE file, so adding a delete to the IMPORTED d139-gua-carve.py left it
green; it now covers both.
Corrected in the ruling record (GA-R1 C2): the amendment said the GUA records would be
status=active. Measured: the live ULA VIP records are "reserved", and
dc-plane-apex-import.py:186,200 creates addresses reserved. Also corrected my own
docstring overclaim -- the 26+9 deprecations are reversible, the 26 CREATES are not.
OPEN SCOPE QUESTION, MEASURED, not a tool defect: D-139 says retire the ULA rows "in the
apex AND in MAAS"; this tool is apex-only, so step 6 is NOT complete when it finishes. MAAS
on the dc0 region holds five ULA /64s beside six GUA; four are empty but
fd50:840e:74e2:220::/64 still holds 2 allocated entries. MAAS has no deprecated status for
a subnet, so delete-or-leave is a separate operator decision.
Gates: harness 20/20 (was 14, delta = the 6 cases added); gauntlet ALL GREEN (98, manifest
recorded deliberately 97 -> 98); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R5: D-139 step 6 "retire" means DEPRECATE -- nothing is deleted (ruling before the work)
...
Taken in a SEPARATE exchange from the ordering ruling (GA-R5: one decision per exchange --
that one settled WHEN step 6 runs, this settles WHAT it does). Committed and pushed before
any dependent work.
Operator answer, exact utterance: "Deprecate both, delete nothing".
IT HAD TO BE ASKED RATHER THAN INFERRED. D-139's list says "retire the ULA rows" and this
repo has never defined that for an IPAM OBJECT: every netbox/*.py importer uses only
active/container/reserved, "deprecated" appears nowhere, and d139-gua-carve.py:4 says "NO
deletion ever". The word could equally have meant DELETE or MARK-UNUSABLE, and those differ
irreversibly in one direction. Status choices were verified against the LIVE API before the
options were put, not assumed from documentation:
prefixes -> ['container', 'active', 'reserved', 'deprecated']
ip-addresses -> ['active', 'reserved', 'deprecated', 'dhcp', 'slaac']
STEP 6 IS THEREFORE: (1) create 26 GUA VIP ip-addresses (f02:20::50-::62 metal-admin,
f02:21::50-::62 metal-internal) status=active; (2) deprecate the 26 ULA VIP addresses;
(3) deprecate the 9 ULA prefixes. CREATE FIRST, VERIFY, THEN DEPRECATE -- reversed, there
would be an interval in which the apex marks a live VIP's only record unusable.
THE COST THE ORDERING RULING FLAGGED IS WITHDRAWN. It called step 6 the largest new surface
before a deploy because it needed a DELETE path the repo has deliberately never had. There
is now no delete path: CREATE plus STATUS-UPDATE only, never-delete preserved repo-wide,
every step reversible by flipping a status back, migration stays visible in the apex.
Recorded for a later reader: "deprecated" is ADVISORY in NetBox, not enforcing. The
functional protection against handing out an in-use GUA address comes from the 26 GUA
records EXISTING, not from the deprecation.
Tool still NOT BUILT. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R5: D-139 step 6 runs IN FULL before the Stage-5 deploy (ruling recorded before the work)
...
Closes the exchange the 2026-08-01 ordering ruling explicitly left owed: "at least step 6 has
a deploy coupling that this ruling does not settle ... needs its own GA-R5 exchange before
Step 4". Committed and pushed BEFORE any dependent work, per GA-R5.
Operator answer, exact utterance: "Full step 6 first, then deploy".
THE 08-01 PREMISE IS NO LONGER TRUE, and the question was put on re-measurement rather than
on that text. It reasoned the VIP overlays "carry v6 VIPs in the ULA range"; measured
2026-08-02, overlays/vr1-dc0-vips.yaml has ZERO fd50: legs and 39 GUA legs, re-rendered onto
GUA on 08-02 with the rack's staged copy brought into line the same day. The deploy INPUT is
already correct; the residual coupling is in the APEX only.
MEASURED (d139-gua-carve.py --dc vr1-dc0, dry run, against the working apex office1-netbox
per DOCFIX-195): CREATE 0 | EXISTS 16 | RETIRE-REPORT 9, dependents 26 ip-addresses / 0
ip-ranges. Step 1 fully applied and idempotent. The 26 were ENUMERATED rather than inferred
from the count: all are VIP records, 13 in fd50:840e:74e2:220::/64 (metal-admin) and 13 in
:221::/64 (metal-internal), octets ::50-::62. The other three retiring /64s hold zero.
CONSEQUENCE: the Stage-5 deploy is blocked on step 6 and nothing else in the D-139 list.
Step 6 = create 26 GUA VIP addresses, delete 26 ULA VIP addresses, retire 9 ULA prefixes.
Steps 4, 5 and 7 remain sequenced after the deploy (7 necessarily -- its network-get
prerequisite needs a deployed unit).
COST, recorded because it is the argument against the option chosen: it needs a repo tool
with a DELETE path against the apex, which this repo has deliberately never had.
d139-gua-carve.py refuses to delete BY DESIGN and that refusal is why the step-1 push was
safe to run. The new tool is the largest new surface introduced immediately before a deploy
and must meet the step-1 standard: independently reviewed, dry-run first, assertions proven
able to FAIL, and CREATE strictly before DELETE so no window exists in which a live VIP is
recorded nowhere.
The tool is NOT YET BUILT. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
sweep F2 ROOT-CAUSED: one bad archive.ubuntu.com backend; "not transient" WITHDRAWN
...
Operator ruling, exact utterance: "Root-cause the curl/debmirror anomaly first" -- taken over
accepting the red gate, over recording error shape/count, and over --ignore-small-errors.
ROOT CAUSE IS UPSTREAM AND BACKEND-SPECIFIC, not a defect in this deployment.
archive.ubuntu.com is round-robin across NINE A records and ONE, 91.189.92.23, HANGS on
dists/jammy-backports/main/dep11/Components-amd64.yml.gz while serving its directory
siblings normally. The control is what makes it a finding rather than "a backend is down":
that same backend returns the .xz sibling 200 / 5776 B / 0.543s, and the other eight return
the .gz in 0.14-0.51s. The resolver ROTATES -- six consecutive lookups, six different
orderings -- so every fetch re-rolls the backend.
MEASURED RATE: 12 sequential fetches -> 11 x 200, 1 timeout (~8%), consistent with exactly
one bad record of nine.
THEREFORE THE PRIOR COMMIT'S "NOT TRANSIENT" IS WITHDRAWN (GA-R1 C2 -- measurement wins over
a document, including one written an hour earlier). The failure is probabilistic; two
consecutive debmirror failures were not the proof of persistence they appeared to be.
A hypothesis was tested and REFUTED, and is kept because it is the one a reader forms first:
debmirror sets keep_alive => 1 and curl does not reuse connections, so a stale persistent
connection looked like the answer. The test inverted it -- keep_alive=1 (debmirror's own
setting) returned 200 in 1s while keep_alive=0 timed out, same library, seconds apart.
ALSO CORRECTED: the "500" was never a server 500. LWP synthesises 500 for CLIENT-side
failures, so it is LWP reporting its own 300s timeout (debmirror:629). I had read it as an
archive-side error, which points the investigation the wrong way.
Two structural facts read from the vendor script, both bearing on options that were on the
table: there is NO dep11 exclusion option (dep11_from_release :1201 and get_dep11_files
:1384 are unconditional), so the mirror's scope cannot be narrowed without patching
debmirror; and --ignore-small-errors is NOT a bounded tolerance -- :2891 DISCARDS the error
count, so it would read clean on a wholly failed mirror. It stays refuted.
RESIDUAL, declared not explained: debmirror failed 2 of 2, which a naive 1-in-9 model
predicts at ~1.2%. The per-IP sweep tested each backend ONCE, so it establishes that
91.189.92.23 was bad, NOT that it is the only bad one.
SECOND, INDEPENDENT FINDING: the local Release expects 6349 B while every backend now serves
6361 B, and the object was republished DURING the investigation. A successful download would
now fail HASH verification against the stale local Release -- a different failure mode from
the timeout, not fixed by fixing the timeout.
scripts/dc-mirror.sh UNTOUCHED. repo-lint 0 fail; ledger-scan unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
sweep F2 re-trigger: failure is PERSISTENT, and measured not to affect the deploy
...
Operator ruling (operational, GA-R3 -- no architectural consequence, no D-number). Question
put: what should dc-mirror.sh's gate ATTEST, given debmirror's exit code cannot distinguish
"nothing mirrored" from "mirrored minus N transient files" from "debmirror died"? Exact
utterance: "Re-trigger the sync first, decide after". Run as a MEASUREMENT, not a remedy.
The attestation question stays OPEN.
NOT TRANSIENT. The re-run (10:26:05->10:31:20Z) failed identically -- same file, same error,
same 300s timeout, two hours on: jammy-backports/main/dep11/Components-amd64.yml.gz. This
CORRECTS the earlier "flaky, not stuck" reading as applied to this failure: the unit has a
history of transient failures AND now one persistent one, and the two were being conflated.
Waiting will not clear it.
The gap is now EXACT, replacing the agent's "at least one file missing": debmirror's own
"Files to download: 6349 B" equals that single file's size, so the entire outstanding delta
of the 952 GiB tree is this one file. Its .xz twin is present and size-correct per Release.
AND IT DOES NOT BREAK apt -- measured, not reasoned, which the prior agent had explicitly
declined to do. With a fully isolated apt state dir on the rack (nothing system-wide
touched, no package installed), apt-get update against the mirror's jammy-backports main
returns rc=0 and fetches Components [5748 B] -- the .xz. apt asks for the format that is
present and never asks for the missing .gz; apt-cache policy resolves the suite normally.
So the mirror is functionally complete for the deploy and the gate is red over a file no
consumer requests.
One anomaly left UNEXPLAINED rather than hand-waved: curl fetches that same .gz from the
same host in 0.44s (HTTP 200, 6349 B, real body read) while debmirror times out at 300s on
it twice. A proxy is ruled out -- unit env is HOME=/root only, and there is no proxy in
/etc/environment, apt.conf.d, profile.d or root's login shell. Not chased: it does not gate
the deploy, and guessing at an LWP-vs-curl difference would be the reasoning-instead-of-
measuring failure this repo keeps logging.
scripts/dc-mirror.sh UNTOUCHED. No tolerance relaxed, no assertion weakened.
repo-lint 0 fail; ledger-scan DOCFIX next-free 208 (unchanged, no new decoy token).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Fix a self-inflicted DOCFIX counter inflation in the entry that restated the rule
...
The 2026-08-02 queued-findings entry wrote the next-free DOCFIX number in hyphenated
identifier form. ledger-scan derives the high-water mark by matching that form anywhere in
prose, so it read highest=208 / next-free=209 when only 207 was ever assigned -- the exact
decoy-token defect the standing numbering rule (hardened in DOCFIX-174) exists to prevent,
committed one paragraph after restating that rule.
The first correction then reproduced the same token inside the sentence explaining the
mistake, and ledger-scan stayed at 209. Both are recorded in the entry rather than quietly
fixed: the rule is unenforceable by reading, because a decoy token looks exactly like
correct bookkeeping. Only re-running the scan catches it.
Verified: ledger-scan now reports DOCFIX highest=207 next-free=208, matching the single
number this session assigned. D 141 / BUNDLEFIX 053 unchanged. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Queued-findings backlog: sweep F1 + F6 CLOSED, F2 diagnosed, F3/F4/F5 graduated, DOCFIX-207
...
Works the 2026-08-02 close sweep (docs/audit/queued-findings-20260802-stage5-edge-fold.txt)
with three read-only agents in parallel. Cites the SWEEP register (F1-F6); the runbook fold
register has its own F1-F12 and is untouched.
sweep F1 CLOSED -- the dc0 rack's staged deploy input matches the repo again. One gated scp
of overlays/vr1-dc0-vips.yaml, ed19d989e80da8da -> 80d861560a6b3c52. A single-file copy was
provably sufficient because the WHOLE staging dir was enumerated first: 14 files, exactly 1
diverged, 0 missing from the Step-4 deploy closure (policies/overrides.zip present at the
repo digest), 0 orphans. All 14 re-verified against CURRENT HEAD after the copy. The 0600
octavia PKI overlay is untouched (same digest/mode/mtime) and was hashed, never read.
Repo-side correctness MEASURED not inherited: 0 ULA legs, 39 GUA, render-drift 4/4 naming
the file with a proof-of-teeth case.
sweep F6 CLOSED, PASS -- the dc0 MAAS region DB is proven uncorrupted. pg_dump read every
page of every table in maasdb: 23,878,796 bytes / 37,199 lines / exit 0 / empty stderr,
completion marker asserted separately. F6's own diagnosis was wrong: snap confinement does
not reproduce as ubuntu. The discriminators are ROLE (maas, not ubuntu) and TRANSPORT (over
TCP the role is password-challenged, over the unix socket it needs no credential). Both
identity values now measured -- maasdb had been prose. Dump streamed, nothing persisted.
sweep F2 DIAGNOSED, not fixed -- reading R1 is true, R2 refuted. debmirror prints "All done."
then exits non-zero; confirmed at vendor source and re-verified independently here
(debmirror 1:2.39ubuntu2, 1615 say("All done."), 1620 exit 1 if !$ignore_small_errors).
Cause: one 500 read timeout on a jammy-backports dep11 index. TWO corrections to the sweep:
its "the log says it succeeded" quotes all came from the PASSING UCA leg; and "stays RED"
overstates it -- measured 16 runs, 7 finished, 9 failed, with four fail-then-succeed pairs
hours apart. No remedy applied: debmirror's exit code conflates "nothing mirrored" with
"mirrored minus N transient files", so any tolerance change alters what the gate attests.
DOCFIX-207 -- preflight P6 quoted "50 apps / 97 relations"; measured is 56 / 108. The same
figures were corrected in phase-01's own gate on 2026-07-10 and this copy was missed.
sweep F3/F4/F5 graduated from the audit capture to durable homes: systemctl show fabricating
Result=success for a non-existent unit -> platform-traps 5c; assert the harness case count
moved, and two scripts probing one endpoint must share the probe definition -> script-authoring.
Logged NOT fixed (hard rule 1): preflight P2 validates a merged input including
vr1-dc0-machines.yaml while phase4:527-528 says the file does not exist and Step 4 does not
pass it -- a gate grading a different artifact than the deploy consumes, on the Step-4 path.
And security-ledger SEC-029(3) calls ~/repo-stage a nine-file copy; it is fourteen.
session-ledger machine-derived block re-seeded (was the 2026-07-27 seed: 21 SEC / D-138;
now 28 SEC / D-141 / DOCFIX-208, each added SEC row verified against the register).
Gates: repo-lint 0 fail / 1 warn (legacy carve-out); run-tests-all ALL GREEN (97 harnesses,
count unchanged, so nothing moved silently); tests/preflight 43/43; tests/render-drift 4/4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
SESSION CLOSE 2026-08-02 (GA-R4 bookend): dc0 edge rebuilt, D-139 steps 1-3, fold opened
...
Durability triad caught a live deploy hazard: the dc0 rack's ~/repo-stage VIP overlay is
STALE (ed19d989 vs repo 80d86156) because this session re-rendered it onto GUA and nothing
propagates to a directory with no git. Under D-138 that rack IS the deploy client, so a
deploy from it would place 26 VIP legs on the ULA prefixes D-139 retires. The 08-01 sweep
recorded these digests matching, so that note now misleads. Logged, not fixed -- first
item next session.
Gates: repo-lint 0 fail / 1 warn (654 files); gauntlet ALL GREEN (97); ledger-scan 3 open
decisions, 28 open SEC, next-free D-141 / DOCFIX-207 / BUNDLEFIX-053. Reconciled against
what the session claimed: D 140->141, SEC 26->28, manifest 96->97.
Mirror sync re-ran and the content IS current (both Release files rewritten today, UCA now
succeeds so egress is genuinely fixed) but records FAIL: the journal says "Everything OK...
All done." then systemd status=1/FAILURE. dc-mirror.sh check stays red on an exit code that
disagrees with the sync's own verdict. Not relaxed to clear it.
Sweep: docs/audit/queued-findings-20260802-stage5-edge-fold.txt, 6 FIRST SURFACE, each
grep-proven absent first. Memory review clean; instrument-currency gains instances ten
through twelve. Ledger rotated 283 -> 266, now 286; no orphaned session. No stage opened
or closed -- Stage 5 remains OPEN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-deploy items 1-3: preflight re-run, mirror sync running, dnsmasq settled on measurement
...
(1) Preflight for dc0 with the region profile: EXIT 1, exactly 11 [FAIL] lines, all the
ruled-accepted P5 set. Zero new findings, so no fresh GA-R5 exchange is owed. P8 clean
after this morning's guard fix; P9 warns correctly off-rack.
(2) dc0-mirror-sync re-triggered and running. The 08-02 failure was upstream-unreachable
during the edge outage; content was verified intact throughout. dc-mirror.sh check stays
red until the status file reads OK -- the attested-currency gate working as designed.
(3) The edge dnsmasq question settled by measurement, not reasoning: vtnet0 carries
inet 10.12.4.1/22 and inet6 LINK-LOCAL ONLY, no radvd. The v4 192.168.1.x range is not on
the interface subnet, and the v6 constructor:vtnet0,slaac range has no global prefix to
construct from. Inert today.
The residual risk is the v6 half, not the stale v4 one: provider-public has a GUA prefix
OpenStack will use, so if vtnet0 ever gains a global v6 the constructor range
self-activates and the edge advertises SLAAC on the provider segment. A constructor range
follows the interface -- it is not stale config. Logged, not changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 edge REBUILT, egress restored (8/8); SEC-032; two dc-egress-check defects fixed
...
Agent-executed rebuild. tofu -replace asserted on `tofu show -json`: exactly 2 non-no-op
changes, both delete,create, both naming vr1_dc0_opnsense, 0 not naming it -- and the
POSITIVE half asserted too, because a plan replacing only the DOMAIN would have reattached
the corrupt qcow2 and passed a "nothing extra" check. Artifact verified, not the log.
Convergence re-plan clean. pfctl -s nat explicitly verified to show real outbound NAT --
the exact defect dc1 is sitting in.
MY BRIEF CARRIED A WRONG BASE-IMAGE PATH. I cited the module default; d124-inner.auto.tfvars
overrides it to a path on voffice1. The apply destroys the volume before recreating, and
neither edge disk has a backingStore, so a wrong path would have left no disk and no
rollback. The agent verified before mutating. Same guess-instead-of-look failure the
operator called out earlier today, in a brief written to prevent exactly that.
dc-egress-check: A4's snap probe omitted `Snap-Device-Series: 16` (400 without it through
the proxy AND direct; 200 with) -- a false FAIL on a healthy proxy. A3 now names the
forwards-without-translating signature when A2 passed. Harness 14 -> 16; T15 asserts on
the CODE after a first cut passed on the comment explaining the header.
Three findings needing attention pre-Stage-5: the rebuilt edge runs dnsmasq on udp4 *:67
over vtnet0 with a stale 192.168.1.x range on the segment carrying juju .5 and MAAS .6
(measured; inertness reasoned, not measured); SEC-021(a) answered -- the off-jumphost copy
exists; and a rebuilt edge invalidates the rack's known_hosts, which no runbook mentions.
Gauntlet ALL GREEN (97); repo-lint 0 fail / 1 warn. Open SEC 28.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc1 edge: forwards but does not TRANSLATE; config intact; REPAIR not rebuild (SEC-031)
...
Read-only agent assessment, docs/audit/dc1-edge-assessment-20260802.txt (643 lines).
Decisive measurement: simultaneous tcpdump on BOTH edge taps while the rack pinged
1.1.1.1 -- the same packet appears on the LAN tap and 0.8 ms later on the WAN tap with
source STILL 10.12.64.2. vcloud's virbr1 masquerades only 172.30.3.0/24, so it is never
translated and no reply returns. Routing works, pf is not blocking, there is no outbound
NAT. My "config lost" hypothesis was half right and the half that mattered was wrong.
Config INTACT on positive evidence: fsck names every inode it deletes, all 50 enumerated,
/conf/config.xml not among them; the edge booted onto its ruled as-built addresses, not
the factory ones. Qualifier: 92 further casualties are inode-only, so strong not absolute.
What fsck destroyed is the FreeBSD base-system user DB -- /etc/master.passwd and
/etc/group, plus BOTH /var/backups copies. Hence "Configuring firewall.....failed." on
every boot since, no pf ruleset, and SEC-031: the edge is an open router serving its GUI
to the simulated ISP. Confirmed against a control (office1 edge: WAN ICMP 100% loss, GUI
000; dc1: 0% loss, GUI 200 in 0.014s). Bounded by lab topology; D-125 unaffected.
Repair not rebuild -- rebuilding discards a good config and re-incurs D-112(c)+D-113.
Blocked on there being no working credential path (sshd down, authed API hangs, console
login likely broken by the same passwd DB); repair needs boot -s, a mutation, so the
read-only agent stopped.
Two findings that outlive this: dc-egress-check's A2 is structurally blind to
forwards-but-does-not-translate and needs a positive "the edge TRANSLATES" assertion --
a defect in a gate built earlier the same day. And NEITHER edge had ever been
reboot-tested; 2026-08-01 was the first time dc1's as-built config ever booted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Fold: open the register, close both Class-A rows (F1 phase4, F12 SKILL.md)
...
Operator ruled fold-before-exercise, start at Phase 2, all DCs come up in their own
region. Register: docs/runbook-fold-register.md, 12 rows classed A/B/C.
Measured gap: D-138 and D-139 appear in NO runbook. Nothing in the chain builds a per-DC
MAAS region, a snap proxy, or the GUA carve. Running it as written would rebuild the
pre-D-132/D-138/D-139 shape and fail where we already fixed things by hand.
F1 (Class A): phase4 asserted "Both DCs deploy from the Office1 headend by the SAME
procedure" and "Every juju and maas command runs there". The 2026-07-27 ruling it cited
was about DC ordering, not run location; D-138 reversed the location half. Replaced with
a tool-split table; superseded text struck in place, not deleted.
F12 (Class A, worse): SKILL.md -- loaded by every session before any runbook -- had ZERO
mentions of D-138 while stating Plane 2 executes on voffice1. Now carries the split, the
structural reason, and the measured test: juju controllers on voffice1 -> "No controllers
registered"; on the dc0 rack -> the live vr1-dc0-controller.
Both record the racks-have-no-repo-clone trap and the sha256-verify-staged-copies rule.
A row closes on the edit AND on being exercised by the dc1 Phase-2 rebuild; an edit alone
is a ruled-but-not-built claim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R5 x2: D-135 amended (dc0 converges on the proxy at rebuild); D-140 PINNED (tofu/juju)
...
D-135 amendment. Operator: "if we have to rebuild in dc0 for any reason we will be using
a proxy rather than a full mirror rebuild." The 953 GB mirror is not reconstructed; a
rebuild comes back as an apt caching proxy on the dc1 pattern. This qualifies D-135's
standing "no fallback and no later convergence" -- none while the mirror stands, but a
rebuild converges. Nothing changes today: the running mirror is verified intact and stays.
Owed at the rebuild: capture the mirror arm's outcome BEFORE dismantling it, or the
experiment D-135 existed to run produces nothing. Consequence for the fold: the phase
runbooks can carry ONE artifact strategy instead of branching per DC.
D-140 PINNED. Operator: "I accept the opentofu plan. Hardened dc0 deployment then
opentofu management with a successful tested deployment method from dc0. Something to pin
for the end of deployment, review and add opentofu management to the remaining tasks."
Order: dc0 deploys and hardens -> method tested -> proven method is the translation input
-> reviewed at end-of-deployment. Not converted now because the bundle deploy has never
completed end to end and changing the mechanism first means two unknowns at once.
Value is measurably in the model/config layer, not topology. Cost to price at review: the
provider is resource-shaped not bundle-shaped, against a measured 56 apps / 108 relations.
Provider capabilities must be read, never recalled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CORRECTION: the edge break was UFS damage from an unclean power cut, not an update
...
My earlier "partially-applied OPNsense update" diagnosis was an inference from the shape
of the symptom and is refuted by the log: ZERO pkg-static / opnsense-update / firmware
lines in the entire 477 KB serial history.
What precedes the first missing-library error is an fsck salvage of a badly damaged UFS
root -- UNREF FILE x2533, UNEXPECTED SOFT UPDATE INCONSISTENCY x785, SALVAGE? yes x257,
then immediately ld-elf.so.1: libcrypto.so.17 not found. The libraries were fsck
casualties.
Causal chain: the 08-01 memory resize was an IN-PLACE tofu update, which this document
already records BOUNCES the guest (confirmed on dc1 as canary that day). A bounce of a
containment VM is a hard power cut to every inner VM. Under D-127 the DC edges are the
only inner VMs with autostart=true, so they were the only ones hard-cut and auto-
restarted onto damaged filesystems.
Confirmed by the dc1 control -- same event, different severity: dc0 2533/785 with ld-elf
errors, dc1 76/181 with zero. dc1's "healthy edge that does not forward", logged earlier
as a separate thread, is NOT separate -- same incident, lesser damage.
Durable finding is a procedure gap: the resize plan was asserted on content and passed a
capacity gate, and none of it covered the inner guests. "In-place" describes the tofu
resource, not the blast radius. Any future in-place change to a containment VM must shut
down its inner guests first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Close both egress gaps: dc-egress-check.sh + preflight P9, wired into restart + phase-4
...
Gap 1: nothing tested DC egress. dc-rack-net.sh check PASSED through a 19-hour outage
because it asserts legs and units -- a leg is not a path -- and the mirror answered 200
from its own nginx, which says nothing about upstream.
Gap 2: the post-reboot check set was my judgement, and it omitted egress.
New scripts/dc-egress-check.sh: site-keyed, runs on the rack, LAYERED (route -> edge
answers -> traffic leaves, ICMP and TCP -> the three upstreams), reporting the FIRST
failure as the cause while the rest SKIP. A4 asserts the UPSTREAM the local artifact
services sync FROM, which is independent of whether they serve.
Proven live on two different failure modes: dc0 fails at A2 naming the dead edge; dc1
passes A2 and fails A3/A4 -- traffic not leaving a healthy edge.
preflight P9 WARNs off-rack with the exact command, FAILs on the rack when broken, WARNs
if the checker is absent. Gap 2 is closed by the WIRING: restart procedure Stage 0
(before anything fetches) and phase-4 Step 3.9 (before the deploy), both stating P9 is
not a substitute for running it on the rack.
Mutation pass caught a decorative test of mine: T11 never reached A4's unrecognised-code
branch, so flipping it to ok() left the suite green. T14 added to exercise A4 alone.
tests/dc-egress-check 14/14; tests/preflight 39 -> 43/43; 5 mutations across both, each
killing a named case, scripts restored sha256-identical. Gauntlet ALL GREEN (97),
manifest 96 -> 97; repo-lint 0 fail / 1 warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 edge root cause: partially-applied OPNsense update, exposed (not caused) by the reboot
...
Console read: the edge sits at a single-user prompt with libcrypto.so.17 and
libpython3.13.so.1.0 missing, 10-configd / 15-templates / 90-carp all failing. The
running system had survived on already-mapped libraries; the 05:49 boot is where it fell
over. Symptom appears nowhere in appendix-A -- a new class.
Scope measured, not assumed: dc0 only. dc1's edge log has zero matches and booted clean
to a login prompt with its WAN addressed. Upstream healthy -- vcloud and voffice1 both
reach the internet, both uplink nets are active NAT, both racks hold default routes.
Recovery material: opnsense-26.7-nano.qcow2 is in the same pool, but the broken disk has
an empty backingStore -- standalone, no CoW base to roll back to.
Logged not conflated: dc1's rack also lacks egress despite a healthy edge; a dc1
forwarding/NAT question, off the deploy path, not diagnosed here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Step 3.5 done; BLOCKER: the dc0 edge has been down since the 08-01 05:48 reboot
...
Step 3.5 complete: model vr1-dc0 created (credential vr1-dc0-cred), spaces gate PASS
0 fatal, apt-mirror verified at http://10.12.8.4/ubuntu. Spaces script run from the
07-30 staged copy on the rack, all three files sha256-verified against the repo first.
Blocker found by dc-mirror.sh check dc0 -- the content-assertion fix from 2026-07-27
earning its keep. The rack cannot reach archive.ubuntu.com, streams.canonical.com,
api.snapcraft.io via the snap proxy, 1.1.1.1, or its own default gateway 10.12.4.1.
ip neigh shows 10.12.4.1 FAILED while .5 and .6 on the same segment answer. The edge
VM is running with both NICs attached, qga not connected, qemu log last written at the
reboot. Last successful mirror sync predates it. Step 4 cannot proceed: the deploy
needs the juju agent stream and snap upstream, both dead.
Two gaps exposed: no gate anywhere tests DC egress (dc-rack-net check passes with it
dead), and this session's own post-reboot verification cited five true facts none of
which tested egress -- a listener is not a path, and a 200 from a local service says
nothing about upstream.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 steps 2 and 3 EXECUTED for vr1-dc0: the nodes are on GUA
...
Step 2: dc-plane-ipam.sh carve-v6 --commit, applied=5, each GUA /64 on the same vlan as
its v4 twin and read back. 2 errors are correct refusals (lbaas-mgmt and oob have no v4
plane to pair with). Independent read: 11 v6 subnets, 6 GUA + 5 ULA, every GUA paired.
Step 3: dc-node-v6-carve.py replace --v6-family gua --commit, applied=45 skipped=9
errors=0, READ-BACK 45/45. Independent query: PRE 54 v6 links (GUA 9 / ULA 45) -> POST 54
(GUA 54 / ULA 0), and ZERO NICs carry more than one global v6, so G19's sole-global
predicate holds. v4 was not touched -- the ordering step 3 exists to enforce.
Gates after: dc-node-v6-carve check --v6-family gua PASS (54 correct, 0 missing, 0
errors); dc-plane-ipam check 29 pass / 2 fail, both expected absences and neither new
(lbaas-mgmt is step 5, oob has no MAAS plane). Was 7 fail before step 2.
Reversible: replace --v6-family ula --commit. Nothing was deleted; both families remain
in MAAS until step 6 retires the ULA rows. vr1-dc1 untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-node-v6-carve: v6 FAMILY guard + a `replace` action for the D-139 step-3 migration
...
Two defects and one gap, all on the critical path.
FAMILY GUARD. v6_by_vlan[vid] = s was last-writer-wins. Once step 2 put a GUA /64
alongside every ULA /64 on the same vlan, that decided the family of all 54 node statics
by MAAS's JSON array order. Measured live: it picked GUA -- luck, not a contract, and the
FOURTH copy of the defect the 2026-07-29 chain audit found in the apex readers. Now keyed
by (vlan, family): --v6-family selects, ambiguity without it REFUSES naming the flag, and
two subnets of the SAME family on one vlan still refuse.
REPLACE ACTION. The nodes hold pre-D-139 ULA statics and `apply` correctly refuses to add
a second address -- two globals on a NIC break G19's sole-global predicate, which has no
exemption. `replace` unlinks the old-family link then links the new one, per interface,
with read-back. It refuses a same-family change (that is an address edit, not a
migration), refuses when a NIC carries more than one v6, and refuses when the old link
carries no link id -- without an id there is no unlink, and falling through would ADD a
second global. Unlink failure does NOT attempt the link: a half-done interface is worse
than an untouched one. Real MAAS links were measured to carry ids (326, 410).
GAP FOUND BY WRITING THE TEST: `want in have` short-circuited to skipped++, so a NIC
carrying the wanted address AND a stray second global counted as "already correct" --
silently masking exactly the half-done migration G19 forbids. Now an error. F8's
assertion was REPLACED with that new invariant, not deleted, and F9 added to cover the
refusal path F8 was originally written for.
Harness 14 -> 23. Three mutations each kill named cases; restored sha256-identical.
Gauntlet ALL GREEN (96); repo-lint 0 fail / 1 warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-08-01 |
GA-R5: VPN (:e0) deferred to Roosevelt design time, recorded not left silent
...
Operator utterance: "Defer to Roosevelt design time".
Recorded as a DEFERRAL rather than left open, because an unrecorded hole is exactly how
the OOB omission survived until the operator caught it. Measured basis: VPN is a
RESERVATION at both precedents, never a carved plane -- VR0 DC0 and Willamette hold the
:e0::/60 parent ONLY, no /64 and no IPv4, unlike OOB which holds /60+/64 at both. VR1's
VPN is Tailscale under D-129(iii), addressed from fd7a:115c:a1e0::/48, which this
deployment does not carve, so a VR1 :e0::/60 would reserve space with no occupant.
Consequence stated plainly: VR1's octet map diverges from VR0 and Willamette on exactly
one hextet, by decision, and ruling B's "conforms to an existing org standard" claim
should be read with that documented exception. Nothing in D-139's execution list, the
Stage-5 deploy, or any gate depends on :e0. Trigger: Roosevelt VPN design, alongside
D-132 and D-131 sub-4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
