| 2026-07-27 |
Second gap sweep: U15-U17 closed; DC data path has no IPv6 at ANY layer
...
Operator: "Close the remaining gaps". All three measurable items are now
measured; what remains open is blocked by a known missing credential or is a
deliberate deferral, and both are stated with cause rather than left implicit.
U15 -- voffice1 transit addressing is REBOOT-DURABLE. A positive result, which
is worth recording as clearly as a defect: live enp2s0 172.31.0.1/30 (dc0 leg)
and enp3s0 172.31.0.5/30 (dc1 leg), both netplan-persistent via
/etc/netplan/60-transit.yaml and 61-transit-dc1.yaml. The D-128 Plane-2 path
survives a headend reboot and no site-baseleg row is owed for it.
U16 -- RETROFIT_WAIT=30m has NO recorded provenance. git log -S traces it to a
single bulk commit ("New Phase Scripts") with no rationale, and a repo-wide grep
finds no constant anywhere in scripts/ documented as nested-virt or depth-4
calibrated. It is an inherited default that has never been validated against the
nested I/O it will actually run on at Stage 5 Step 9. Log-only, but it should
not be mistaken for a tuned value. Bonus finding from reading the file: that
script's own preconditions require BOTH the openstack and juju clients, and no
host currently has both -- vcloud has neither, voffice1 has juju only. It cannot
run anywhere today, which reinforces S-1.
U17 -- the DC data path carries NO IPv6 at any layer measured. On BOTH racks:
zero global v6 addresses on any plane bridge, no v6 default route, and
accept_ra=1 (so they would accept a router advertisement; none is arriving).
Combined with the existing measurements -- no v6 subnet on any of the 12 MAAS
plane fabrics, no v6 link on any of the 18 nodes -- this is a THREE-LAYER
confirmation of L2-3. It also WIDENS the R2 propagation task: it is not only
"carve into MAAS", the rack bridges have no v6 either.
STILL OPEN, with cause rather than silence: the two DC edges' own interface-level
v6 configuration. dc0 is blocked by a known missing credential, not by a failure
to look -- ~/vr1-dc0-creds/ contains no opnsense-api.txt while ~/vr1-dc1-creds/
does, which is SEC-021(a) directly visible on disk, the residual item the
2026-07-27 consolidation batch deliberately excluded because the re-mint is a
live edge mutation. dc1's API is credentialed but measured unreachable from
vcloud (timeout; the path runs from the rack, where the edge creds are correctly
not staged per SEC-015). U17 answers the substantive question from the rack side.
repo-lint/gauntlet ON voffice1 remain deliberately deferred until precondition
0.1 advances that clone -- running them against a 105-commit-stale tree would
produce a number that means nothing. That is a judgement, not an omission, and
it is recorded as such.
Revert: git revert this commit; register and CURRENT-STATE changes are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
UNMEASURED-gap sweep: 14 deferred items closed; juju restore-backup CONFIRMED absent
...
Operator challenge: did the committee actually measure live state, or take
shortcuts? Answer: some of both. This commit is the accounting, plus the sweep
that closes what was closeable.
WHO DID WHAT, precisely, because the accounting matters:
- The apex was never polled by ANY lens. Lens 2 declared it out of scope and did
so HONESTLY -- its UNMEASURED note explicitly warned "D-101's literals may
exist in NetBox and simply not be carved into MAAS". No overclaim. But it is a
key system that lens had reach and time to measure. Under-scoped, not dishonest.
- The two-day-old dump was MINE, in the correction, not an agent's.
- Sharpest instance of the general problem: lens 6 declared juju restore-backup
unverifiable because "no Juju client exists on this host", while juju 3.6.27
was installed on voffice1 and lens 5 was running juju help against it in the
same session. One lens called impossible what another was measuring.
CLOSED BY THE SWEEP (14 items). The consequential ones:
- LIVE apex polled via netbox/office1-record-dump.py: 139 prefixes, 103 IPv6,
ZERO added, ZERO removed vs the dump. The conclusion was right; the method was
not, and that distinction is the whole point.
- juju restore-backup DOES NOT EXIST on 3.6.27 -- "not a juju command ... Did you
mean: create-backup". L6-14 goes from flagged RISK to CONFIRMED DEFECT: Stage 6
Step 9's D-104 restore drill and its DoD bullet are unsatisfiable as written.
- juju bind DOES exist as documented -- lens 6 was RIGHT to exclude it from L6-7
rather than assert it wrong. A correctly-handled uncertainty.
- dc0 compute provider MACs measured (52:54:00:1b:19:e6 / 52:54:00:18:ab:b4),
matching lib-hosts and CONFIRMING L3-8.
- curl present on both racks, so L4-13's hole is theoretical, not live.
- The pinned charm channels DO resolve (2024.1, 2.4, squid all present via juju
info on voffice1) -- proving L4-2's root cause and its fix: P3's 33 warns are
purely the missing juju binary on vcloud, not a charmhub problem.
NEW FINDING: preflight's "MAAS unreachable: 'maas admin subnets read' failed" is
a MISDIAGNOSIS -- the maas binary is simply ABSENT on vcloud. Together with the
absent juju (L4-2) and absent openstack (S-1), THREE separate preflight/deploy
failures on this jumphost are all "the client is not installed", and each is
reported as something else. A tool-absence must never be reported as a
target-unreachable. Its own DOCFIX.
STILL-OPEN gaps that are measurable and were NOT done are listed in the
register's section 3 explicitly, not buried: DC edge v6 state, repo-lint/gauntlet
on voffice1 (deferred on purpose -- that clone is 105 commits stale, so running
them today would measure a stale tree), voffice1 transit reboot-durability, and
the provenance of RETROFIT_WAIT=30m.
STANDING LESSON added for the next committee: before declaring an item
unmeasurable, VERIFY THE CONSTRAINT YOU ARE ASSERTING. The UNMEASURED escape
hatch is load-bearing for honesty and must not become a way to avoid looking.
Revert: git revert this commit; the register is new and the CURRENT-STATE
paragraph is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CORRECTION: the v6 literals are ASSIGNED (D-111), not pending -- R2a withdrawn
...
The operator asked whether the Office1 NetBox apex had actually been polled. It
had not. It has now been read, and the answer inverts a claim I made one commit
ago and a question I put in front of them.
WHAT IS TRUE: the v6 literals are assigned, ratified and recorded, and have been
since 2026-07-11 under D-111 (ADOPTED). Measured from
netbox/draft/vr1-office1-current-20260725.json -- 139 prefixes, 103 IPv6, every
relevant row tagged D-101/D-111: ULA fd50:840e:74e2::/48 with DC0 planes at
:220/:221/:230/:240/:250::/64 and DC1 at :320/:321/:330/:340/:350::/64; GUA
provider-public DC0 2602:f3e2:f02:10::/64 + VIP f02:11::/64, DC1
2602:f3e2:f03:10::/64 + VIP f03:11::/64.
WHAT I GOT WRONG: the R2 ruling note claimed D-101's "Remaining open item" (the
org ULA /48 and per-DC GUA carve) had become a Stage-5 precondition because
"dual-stack cannot deploy against literals that do not exist". They exist. R2a
("which literals") is WITHDRAWN as never having been open; no operator utterance
is owed on it.
THE REAL PRECONDITION IS NARROWER AND BETTER: propagation, not assignment, and it
needs no ruling. The ratified values are absent from the two places Stage 5
actually reads -- scripts/lib-net.sh carries no v6 arm at all, and MAAS carries
no v6 on any of the 12 DC plane fabrics. Both are mechanical copies from an
authoritative source, so both move from Part A (needs a decision) to the Phase-3
mechanical batch.
Note CURRENT-STATE's "DRIFT-FREE across netbox/lib-net/artifacts" is NOT
contradicted: drift-free means values appearing in more than one place agree, not
that coverage is complete. lib-net.sh simply never gained a v6 arm.
PROCESS FAILURE, OWNED AND RECORDED IN THE DECISION ITSELF: the audit's own lens 2
listed the apex as UNMEASURED and warned, in terms, that "D-101's literals may
exist in NetBox and simply not be carved into MAAS". That warning was correct and
available, and I put a question to the operator anyway on the strength of D-101's
stale prose. Trusting stale decision prose over an available measurement is
exactly what this audit was convened to catch.
NEW DOCFIX-CLASS FINDING, logged not fixed (hard rule 1): D-101's "Remaining open
item" paragraph still reads "pending NetBox assignment (gap #3)" for literals
D-111 adopted on 2026-07-11 -- a decision's own prose contradicted by a later
ruling, DOCFIX-200/204 class, and the direct cause of this error. Queued in the
Phase-3 batch.
The R2 ruling itself STANDS -- dual-stack as ruled. Only its stated effect changed.
Revert: git revert this commit to restore the prior (incorrect) framing; the R2
ruling in the preceding commit is independent and should not be reverted with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R2 RULED: dual-stack as originally decided; the v6 literals move onto the critical path
...
GA-R5: question and exact utterance quoted in the Status block, dated, pushed
before dependent work.
Operator utterance: "Carve v6 and deploy dual-stack as ruled (Recommended)".
Recorded as a D-101 RULING NOTE -- a re-confirmation, amending nothing, in the
same shape as the 2026-07-25 note it follows.
WHY IT WAS A LIVE QUESTION AT ALL: the 07-25 ruling was measured against the
substrate for the first time by this audit and found unimplemented. Exactly ONE
IPv6 subnet exists cloud-wide and it is on the Office1 base fabric; none of the
12 DC plane fabrics carries one; zero v6 links across all 18 nodes; every node
reports default_gateways.ipv6 = NONE. D-101's matrix requires ULA on
data-tenant/storage/replication plus legs on metal-admin/metal-internal, with
nothing to bind against.
THE CONSEQUENTIAL EFFECT, and the reason this commit matters beyond recording a
choice: D-101's own "Remaining open item" -- the org ULA /48 and per-DC GUA carve
-- has been carried since authoring as "pending NetBox assignment ... not a
ratification question", i.e. non-blocking. It is now a STAGE-5 PRECONDITION.
Dual-stack cannot be deployed against literals that do not exist.
Downstream, all now inheriting dual-family: R9 (where dc1's literals live), R11
(vault/designate VIPs, which must be dual-family from the outset or their cert
SANs get re-issued on a live cloud), and the L3-9 overlay collision, which must
be reconciled BEFORE either authority location is populated. Worth restating: of
the two merge orders the audit measured, the DANGEROUS one is the one that
PASSES -- vips-overlay-last silently replaces every v6 leg and reports green.
R8 (octavia's family) is explicitly NOT resolved by this ruling.
NEW SUB-QUESTION OPENED, queued not ruled: R2a, which literals. The ruling
directs that they be assigned; it does not assign them. I have deliberately not
proposed prefixes -- that is the operator's address space, and hard rule 2
forbids inventing a literal. Only the SHAPE is presented.
Revert: git revert this commit; the ruling note and R2a are additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R1 RULED: ceph-osd gets a real second disk on the storage nodes (D-121 amendment)
...
GA-R5: question as presented and the operator's exact utterance are both quoted
in the Status block, dated, and pushed BEFORE any dependent work.
Operator utterance: "Add an OSD volume to node-vm (Recommended)".
Recorded as a D-121 AMENDMENT rather than a new D-number. D-121 is the decision
that ratifies modules/node-vm sizing/count, and its own capacity re-validation
already records "Ceph disk re-run for 4 storage/DC = PASS 5.31 TiB" -- the second
disk was BUDGETED and never BUILT. GA-R3: architectural consequence, but it
amends a ruled node layout rather than establishing new ground.
TWO SCOPE CORRECTIONS landed with the ruling, both of which narrow it:
- It is EIGHT volumes, not eighteen. ceph-osd is placed on the four storage
nodes per DC (bundle.yaml:556-557, to: ["5","6","7","8"]). The other five
nodes per DC neither run ceph-osd nor need a second device. My earlier framing
said "all 18 nodes" -- true of the measured defect, wrong about the fix.
- Rack headroom measured before asserting feasibility: dc0 2.0T available, dc1
2.9T available under /var/lib/libvirt, thin-provisioned qcow2.
Why (a) over (b): MINIMIZE DELTA TO ROOSEVELT -- Roosevelt storage nodes have
real dedicated disks. Option (b) was also not safe to pick as presented: whether
ceph-osd at the pinned squid/stable supports a directory- or partition-backed
OSD was explicitly UNRESEARCHED, and the audit refused to assert it either way.
THE APPLY IS NOT AUTHORISED BY THIS RULING. Four preconditions recorded, the
sharpest being that re-commissioning is required for MAAS to see the new device,
and whether that preserves the D-134 statics and pinned MACs is UNVERIFIED --
the 2026-07-20 MAC-regeneration incident is the precedent for exactly that class.
modules/node-vm is shared by both DCs, so the change must be additive/opt-in or
it plans against all 18 domains.
Revert: git revert this commit; the amendment is additive and no artifact changed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CURRENT-STATE: land the dc1 egress measurement (repairs the prior commit's L10)
...
The previous commit (c1f5dc8) touched docs/audit/ without the paired
CURRENT-STATE update that GA-R1 amendment C1 requires, so repo-lint L10 failed
on it. Owned rather than silently amended: the commit is pushed, so this is a
fix-forward.
The measurement genuinely belongs in CURRENT-STATE under amendment C2. The
Stage-5 bootstrap entry has carried the hard ordering caveat -- bootstrap fetches
the juju agent stream and juju/juju-db snaps pre-apt, outside the apt proxy, so
it works ONLY while the dc1 edge egress is open -- but it carried no measurement
of whether egress actually IS open. It now does, probed from the dc1 rack with
--noproxy so an apt-cacher hit could not fake it: streams.canonical.com/juju/tools/
returns 200, api.snapcraft.io answers, archive.ubuntu.com 200, ping 1.1.1.1 0%
loss, default route via 10.12.64.1.
Stated with its shelf life: re-probe immediately before bootstrap, because the
D-135/D-107 narrowing is the closing mutation and nothing prevents it landing
first.
Revert: git revert this commit; the edit is additive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: lens 5 + the three deliverables
...
VERDICT: Stage 5 would NOT run error-free today, and would fail early. The
substrate underneath is excellent -- all three OpenTofu roots ZERO DIFF, 18
nodes Ready exact to D-121 Option C, MACs and power addresses matching
lib-hosts, D-134 statics perfect, 17 fabrics, zero orphaned interfaces, both
artifact paths serving, gauntlet ALL GREEN (81). What is not ready is the
layer between the substrate and the deploy.
LENS 5 (relaunched after an API error) returned the ordered precondition list
and the largest runbook defect found: Step 4 says "follow phase-01 verbatim",
and phase-01 ACTIVELY REFUSES dc1 -- its VIP guard greps bundle.yaml for
eleven 10.12.4.x VIPs, dc1's live in an overlay on 10.12.64/68/72, so it
takes the "ABORT: VIP guard failed" branch. After the ruled VIP extraction it
aborts for dc0 too. It also carries hardcoded VR0 system_ids, a jumphost-local
libvirt loop over disks that do not exist, and a 4-machine plan gate against a
9-machine bundle.
Three Stage-5 gate commands cannot execute at all, measured against the juju
actually installed (3.6.27): juju run used for a shell command when it is the
action runner; download-backup given a backup-id when it takes a controller
path; and the geneve gate grepping ovn-central for a config key the same
runbook says does not exist. Four of the five VERIFY-LIVE gates the record
says Stage 5 owes have NO step in the runbook -- including the keystone
policyd-override check, which is RULED.
Two convergent confirmations raise confidence in the whole set: the Ceph OSD
blocker was found independently by two lenses using different methods, and the
stale-clone blocker independently by this session and lens 5.
DELIVERABLES:
- docs/audit/stage5-readiness-20260727.md -- ordered precondition checklist,
READ FIRST. 5 phases, each row with status/evidence/what-breaks.
- docs/audit/stage5-committee-raw-20260727.md -- all 7 lenses verbatim.
- docs/audit/queued-rulings-20260727.md -- 11 Stage-5-blocking + 4 standing
questions, GA-R5 shape, one exchange each, blank utterance lines. NONE
adopted; a batch answer rules NOTHING.
MECHANICAL FIX TAKEN (exactly one, deliberately): the G3 gate row read OPEN
with a standing FREEZE while its own cited evidence file records G3 CLOSED and
the freeze lifted. Left standing, that clause would have blocked the very
DOCFIX batch this audit queues. No ruling was required -- two surfaces already
declared it closed.
The 21-item DOCFIX remediation batch is LOGGED NOT EXECUTED. Nearly every
runbook fix interlocks with an unanswered ruling, so landing them now would
encode assumptions about questions the operator has not answered.
Next-free numbers unchanged (D 138 / DOCFIX 205 / BUNDLEFIX 053) -- no number
was assigned, correctly, since nothing was remediated.
Revert: git revert this commit; the deliverables are new files and the
CURRENT-STATE edits are additive plus the one G3 correction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: lens 4 (gate integrity) + privileged creds re-run
...
LENS 4 broke seven gates with CONSTRUCTED, MEASURED fixtures. Independently
re-verified by this session before landing:
- repo-lint reports "PASS (0 fail, 0 warn)" over ZERO files when given a
one-character typo of either its documented flag or the repo path. This is
the gate whose 0-fail every GA-R6 stage-close commit cites; the verdict is
not evidence the repo was read. Reproduced here exactly.
- provider-bundle-check PASSES on decorative HA: `grep -rn cluster_count
scripts/ tests/` returns NOTHING, while the overlay carries 20 such lines.
A 3->1 rewrite of every cluster_count yields a byte-identical PASS.
- preflight's aggregator ignores every sub-gate exit code that is not 1 or 2,
so a gate that crashes (127) leaves "PREFLIGHT: PASS -- clear to deploy".
- P3 verified ZERO of 33 charm-channel pins TODAY because juju is not on the
host's PATH, and a missing BINARY is folded into the same WARN as an
unreachable charmhub. This CORRECTS this session's own earlier capture,
which waved the 33-warn line through as "expected" -- the count was
expected, the cause was not.
- G17's named check cannot fail: curl -sI exits 0 on 404/500 and the dc0 URL
is an autoindex root that answers 200 with nothing behind it.
Lens 4 also returned an 11-item COULD-NOT-BREAK list; notably the 2026-07-27
dc-mirror last-sync fix HOLDS under all five status arms, and creds-matrix's
tier-2 probe states could not be made to turn "could not look" into green.
CREDS: privileged sweep re-run (operator-authorized). Exit 1, still exactly
the same 7 findings -- no drift in a day. All three root-owned locations read
via sudo -n escalation, metadata only; ZERO E3 findings. Recorded as a
RE-CONFIRMATION of the 2026-07-26 privileged run, not a newly-closed item.
Six of seven lenses are now in docs/audit/stage5-committee-raw-20260727.md.
Lens 5 (ordered preconditions) is still running after an API-error relaunch.
Revert: git revert this commit; all files are audit artifacts plus additive
CURRENT-STATE paragraphs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage-5 grounding audit: live measurement sweep + first five lens returns
...
MEASUREMENTS (capture docs/audit/stage5-live-measurement-20260727.txt):
- ALL THREE OpenTofu roots plan ZERO DIFF -- outer on vcloud, and for the first
time both inner roots on voffice1 in the same session. Validity of the inner
pair was established FIRST by proving those roots are byte-identical to main
despite the voffice1 clone being 105 commits behind; a non-empty diff would
have made them UNMEASURED, not green.
- Gauntlet ALL GREEN (81). repo-lint 0 fail / 1 warn (the L1 legacy carve-out).
- preflight exits 1 for EXACTLY the known reasons -- nothing new has joined.
- dc0 full mirror and dc1 caching proxy both genuinely serving; dc0 sync
attested OK 2026-07-27T08:43:46Z.
CURRENT-STATE corrections under GA-R1 amendment C2 (measurement wins, corrected
in the capture's own commit -- this is also what repo-lint L10 requires):
- Juju on the headend measured 3.6.27, not the recorded 3.6.25. In-channel
patch refresh, which D-071 permits, so policy-compliant drift, not an
incident -- but it is the client that will bootstrap the controller.
- NEW pin row: the openstack client is ABSENT on BOTH hosts while ten
Stage-5/6/7 scripts invoke it. No prior surface carried this.
- Section 5 convergence history extended with the three-root re-confirmation.
OWNED MEASUREMENT ERROR, recorded rather than quietly corrected: the artifact
checkers were first run LOCALLY on vcloud and printed a comprehensive FAIL with
every probe MISS -- output indistinguishable from "never installed". Both
scripts are designed to be PIPED TO THE RACK. Corrected by direct functional
probe; lens 2 independently ran the canonical piped form and got PASS on all
three. The scripts are not at fault.
FIVE LENS RETURNS landed verbatim in docs/audit/stage5-committee-raw-20260727.md
(lenses 1, 2, 3, 6, 7). Findings are NOT yet verified -- verification is the next
step and a refuted finding stays in that file with its refutation, never deleted.
Lens 5 died on an API error and was relaunched; lens 4 still running.
Revert: git revert this commit. The two CURRENT-STATE edits are additive; the
audit files are new.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
close-out durability sweep: 4 transcript-only items landed; dc1-is-not-a-mirror warning
...
Operator-requested sweep for content that existed only in the session transcript, before
clearing. Precedent: docs/audit/queued-findings-20260726.txt. Capture:
docs/audit/queued-findings-20260727.txt.
THE CONSEQUENTIAL ONE. scripts/dc-mirror.sh's dc1 site row carried NO warning that dc1 is no
longer a mirror site, so `dc-mirror.sh install dc1` would have SILENTLY REBUILT everything the
same-day teardown removed -- units, nginx vhost, sync helper, and an ENABLED daily debmirror
timer that would start a fresh ~950G pull on a rack whose ruled artifact path is the proxy. That
is precisely the command someone would reach for after treating a failing `check dc1` as a
regression. The row now states dc1 is proxy-only per the D-135 amendment, that `check dc1` FAILS
BY DESIGN, and that `install dc1` is a deliberate strategy change and never a repair. The row is
RETAINED so its measured values survive a legitimate rebuild.
QUEUED, not built (hard rule 1): a RUNTIME guard in do_install for a site whose ruled path is
not the mirror. A comment is strictly weaker than a guard, and prose-only prevention has already
failed twice on this project (DoD item 8 missed dc1; D-137's wiring sat as prose and missed BOTH
DC standups). Recording that the interim is an interim.
A WRONG CLAIM CORRECTED. docs/audit/stage4-mirror-gate-20260727.txt said `systemctl reset-failed`
would re-arm the paused dc1 debmirror. It cannot: reset-failed on the SERVICE does not start an
INACTIVE TIMER. The real vector was a REBOOT -- `enabled` starts the timer at boot and
`Persistent=yes` fires it IMMEDIATELY because its window had passed. Correction APPENDED rather
than edited in place: the file is dated evidence, so a wrong analytical claim is superseded
openly, not quietly rewritten.
THE GENERALISABLE LESSON, previously unrecorded, is now platform-traps section 5:
5a "stopped is not dormant across a reboot" -- enabled + inactive + EMPTY NextElapse looks
harmless and is not; plus reset-failed does not re-arm, and Result=signal/status=15 on a
oneshot means it was SIGTERM'd and never wrote a terminal state (so a stale RUNNING is a
corpse, not progress).
5b a `oneshot` with RemainAfterExit=yes does NOT undo its work on stop, so stopping it proves
NOTHING about dependents -- which is exactly why the dc1 teardown test deleted the live
address and route instead of stopping the mirror net unit.
Three rows added to the verbatim-error index. Note dc0-mirror-sync.timer is still enabled with
Persistent=yes, so 5a describes dc0 today -- benign while its syncs succeed, same shape.
HARD RULE 2. dc-cache-proxy.sh stated that the two net units coexist safely on one host as
FACT. It is REASONED (idempotent replace verbs, identical measured values), never MEASURED --
the proxy has only run on dc1, where the mirror unit was then removed, so nobody has run both.
Marked accordingly, with an instruction to verify rather than trust the sentence.
VERIFICATION: dangling-reference sweep over docs/, runbooks/, scripts/ and the skill -- every
path this session introduced or cited RESOLVES; all pre-existing dangles are legitimate
(deleted-as-history, not-yet-built, or the deliberately-absent octavia PKI overlay preflight P4
fails on). ledger-scan reconciled against the machine-derived block. Gauntlet ALL GREEN (81),
repo-lint 0-fail, dc-mirror 24/24, dc-cache-proxy 20/20.
Revert: text + comments only; the capture correction is an append.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CURRENT-STATE: Stage 4 CLOSED 2026-07-27 -- merged 6f5701d, branch retired
...
GA-R1 C1: the merge is a status change, so the status authority records it in the same change
set. Stage 4 / Phase 3 flips OPEN -> CLOSED with its evidence: operator-gated merge to `main`
as merge commit 6f5701d (2 parents, NOT squashed; 77 commits), branch
`dc-dc-stage4-phase3-maas-deploy` retired local + remote after confirming containment with
`git branch --merged main`. Post-merge verification ON main: gauntlet ALL GREEN (81),
repo-lint 0-fail.
Recorded as travelling forward BY DESIGN rather than silently dropped: gate G17 (node-side
artifact reachability, triggers at Stage 5 first boot), 21 open SEC rows, and 7 residual
credential-register findings including the dc0 edge-API re-mint that was deliberately excluded
from the consolidation batch.
The next stage branches off post-merge `main`.
Revert: text-only. The merge itself is 6f5701d.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 4 close-out set: GA-R2 consolidation, skill sweep, snapshot, GA-R7 review
...
Every substantive Stage 4 item is closed or split to gate row G17. This is the GA-R6 close-out
set; only the operator-gated merge to `main` remains.
GA-R2 CONSOLIDATION: the 7 stage changelogs archived to docs/archive/changelogs/ (top-level
docs/ 25 -> 18, 113 archived total) and the stage record written at
docs/archive/stage-records/vr1-stage4-record.md. It records what the stage actually delivered
(READY handoff, not deployed nodes), the three things a future session should read first, and
the false-green pattern this stage kept surfacing. 12 stale `docs/changelog-*` paths in live
surfaces rewritten to the archive -- including several already dangling from the G12 close, so
every changelog reference in CURRENT-STATE / session-ledger / security-ledger now resolves.
SKILL SWEEP: three new INVARIANTS folded in --
- per-DC ARTIFACT DELIVERY is a per-DC STRATEGY (dc0 full mirror / dc1 caching proxy, D-135
amendments); checking dc1 as a mirror fails BY DESIGN. Corollary, learned the hard way: a
per-DC utility service OWNS ITS OWN network prerequisites -- never let one strategy's
teardown be able to break the other's.
- Stage 4 hands off READY nodes, NOT deployed ones (DOCFIX-200). Any surface saying "nodes
Deployed" as a Stage-4 exit is a defect; four were found and fixed. Consequence: nothing
node-side is probeable in Stage 4, which is why G17 exists.
- "A CHECKER THAT CANNOT FAIL IS NOT A GATE", with the working rules that came out of two
measured false greens: assert on CONTENT not existence; an unrecognised state REFUSES;
"could not look" is never "nothing there"; ENUMERATE what exists rather than only auditing
what is declared, because a register cannot report the absence of a row it does not have;
and when a test asserts a literal finding string, REPLACE the assertion, never delete it.
Plus two routing rows (per-DC artifact checks; D-137 as the credential-lifecycle authority).
SNAPSHOT REGENERATED: .claude/skills/openstack-cloud-ops-consolidated-20260727.md (1589 lines,
ASCII + LF byte-verified, all 7 sections). The superseded 20260725 snapshot is REMOVED rather
than kept alongside -- a stale snapshot being uploaded is the exact failure
docs/audit/skill-divergence-20260725.md records (a failed clone + a mis-shaped handoff), and it
is a derived artifact regenerable from any commit.
GA-R7 MEMORY REVIEW: NO new memory written -- everything durable from this stage graduated to
the skill and the repo, which is the correct GA-R7 outcome, not an omission. Both existing
entries verified against the repo (the finding-class baseline they cite does exist) and extended
with the two reasoning traps this stage produced: the register cannot see a credential it has no
ROW for, so enumerate what exists; and diagnose a denied call by checking whether an existing
rule failed to MATCH before blaming the classifier (the measured 2026-07-26 root cause). Also
marked the ask-vs-allow precedence claim UNVERIFIED so no future session relies on it.
Noted, not actioned (hard rule 1): T24's finding-class baseline covers TIER 1 only, so tier-2/3
classes have no baselined red state and a future false green there would not be caught.
Gauntlet ALL GREEN (81), repo-lint 0-fail.
Revert: git revert this commit; the changelog moves are renames and the snapshot is derived.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
set-interface-v4: reload the pf filter on --commit (RULED); appendix-A rewritten
...
RULING (GA-R5) 2026-07-27. Question as presented: (a) add `configctl filter reload` to the
script's --commit path; (b) leave the script and rely on DC standup DoD item 8; (c) put it behind
an opt-in flag. Operator answer, exact utterance: "a". OPS under GA-R3 -- a script fix completing
an already-documented operation, governed by D-113; no new D-number.
DEFECT (appendix-A, measured on the fresh dc1 edge 2026-07-23): --commit ran `configctl interface
reconfigure`, which applies the address to the kernel but does NOT regenerate the pf ruleset, so
OPNsense never creates its automatic outbound-NAT rules for the (re-)addressed LAN. Forwarded LAN
packets left the WAN un-NAT'd and replies never returned -- the edge passed NO LAN traffic while
looking perfectly configured. Self-masking: any later plugin install or config apply reloads the
filter as a side effect, so dc0's 07-20 qga work hid it there until dc1 exposed it.
Option (b) rejected on evidence, not taste: DoD item 8 already existed and did NOT fire at dc1,
and prose-only controls have a measured failure record here (D-137's own enforcement sat as prose
at phase-3:498 and missed BOTH DC standups).
PLACEMENT IS THE SUBSTANCE OF THE FIX, and my first attempt got it wrong -- caught before
shipping. Putting the reload on the next line of the `interface reconfigure` heredoc would have
been a fix that fixes nothing exactly when it matters: re-addressing the interface you arrived on
drops the session DURING the reconfigure, so a following command in that stream never executes --
and that drop case IS this incident's own scenario (dc1's LAN moved 192.168.1.1 -> 10.12.64.1
over the LAN itself). As shipped the reload runs over a FRESH connection to PROBE_HOST.
Also added a `pfctl -s nat` read-back with the `proto carp` boilerplate filtered out, since
boilerplate-only is precisely appendix-A's broken-state signature. REPORTED, not gated: an edge's
first addressing may legitimately have no gateway and so no NAT to generate, and hard-failing
would block a correct run. The hard gate stays address-on-the-kernel.
Harness 59/59 (was 53), cases 15-15f. Case 15f caught a real inconsistency -- appendix-A still
said "not yet ruled". Case 15e caught a bug in my own first draft, which scanned to EOF and so
failed the script for having the very hard gate the case exists to preserve.
appendix-A rewritten: the FIX stays for an edge already in this state, plus a PREVENTED-IN-THE-TOOL
block recording the placement detail and why prose prevention was rejected.
Gauntlet ALL GREEN (81), repo-lint 0-fail. No live edge touched -- both DC edges are already
addressed, so this affects only the NEXT edge addressing.
With this, every substantive Stage 4 in-stage item is closed or split to gate row G17; only the
GA-R6 close-out set remains.
Revert: git revert this commit's script + harness + appendix-A changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
SEC-024 retention RULED: keep both state-surgery snapshots (GA-R5)
...
Question as presented 2026-07-27: delete both `pre-*` state-surgery snapshots (cleanest, closes
SEC-024 fully), keep both and let preflight P5 keep watching them, or keep pre-G6 and delete
pre-G16 -- noting both gates are CLOSED, both carry the Office1 MAAS API key in plaintext per
DOCFIX-175, and deletion is irreversible with no other copy of that pre-surgery state.
Operator answer, exact utterance: "Keep both".
opentofu/terraform.tfstate.pre-G6-20260719 and .pre-G16-20260721 are RETAINED at 0600, neither
git-tracked, as the only record of state prior to the two direct tfstate edits (G6 autostart
injection, G16 channels reconcile).
SEC-024 stays OPEN as a standing WATCH rather than an open remediation: the mode defect is
fixed, but the umask CAUSE was out of ruled scope so a future apply may rewrite 0664, and P5 is
the detection. Recorded in the SEC-024 disposition cell and in CURRENT-STATE. Open SEC count
unchanged at 21.
repo-lint 0-fail.
Revert: text-only; the ruling record reverts with git revert. No file was deleted (that is the
point of the ruling).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 4 cleanup: carve residue 108 fabrics -> 17; dc1 leftover nginx purged
...
Both items were queued to the stage close at the 2026-07-23 carve.
CARVE RESIDUE. Deleted subnet id=8 192.168.1.0/24 (the superseded OPNsense FACTORY LAN -- both
edges were long since re-addressed to 10.12.4.1 / 10.12.64.1), then fabric-5, then the 90
auto-created empties (ids 6..95). 108 fabrics -> 17.
The audit caught an ORDERING DEPENDENCY the original flag did not state: 192.168.1.0/24 was the
ONLY occupant of fabric-5, so deleting the fabric first would have cascaded the subnet away
instead of removing it cleanly. Subnet first, then the fabric.
METHOD, chosen because of this repo's own history: a MAAS fabric delete cascades to its VLANs,
and the 2026-07-21 pod-delete incident cost 9 machine records precisely because the association
check ran AFTER the delete. So emptiness was PROVEN per fabric -- zero subnets, zero ipranges,
zero node interfaces across all its VLANs -- against a fresh occupancy snapshot (fabrics +
subnets + ipranges + every machine's interface_set) taken immediately before the batch, and the
cascade check was re-run afterwards too.
KEPT, all verified occupied: office1 base + GUA (fabric-0), office1 compose with its 2 LXD
guests (fabric-1), both transits (fabric-2 / fabric-141), both metal-admin/boot fabrics with 9
nodes each (fabric-4 / fabric-142), libvirt default (fabric-3), and the 10 named plane fabrics.
POST-STATE: 18 Ready + 2 Deployed office1 guests, all 18 still power_type=virsh, 7 interfaces
each (6 flat planes per D-133 + br-ex), ZERO orphaned interfaces, placement tags 9+9 intact,
both DCs' artifact paths re-verified PASS.
NGINX ON dc1. Purged nginx + nginx-common (dry-run first: nothing else depended on them, no
nginx-core was installed), plus autoremove of liblockfile-simple-perl and liblog-agent-perl --
debmirror's orphaned perl deps. /etc/nginx gone, nothing on :80, proxy still PASS on :3142. dc0
keeps its nginx and was re-verified unaffected (it serves the mirror).
Capture docs/audit/stage4-carve-residue-cleanup-20260727.txt. repo-lint 0-fail.
Revert: fabrics/subnets are MAAS records, not config -- no undo, and nothing needs them (90
were auto-created empties; 192.168.1.0/24 was a superseded factory address). nginx reverts with
apt-get install nginx, which dc1 has no use for while it is the proxy DC.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc1 full mirror REMOVED (D-135 amendment); dc-cache-proxy.sh now owns its net layer
...
Operator clarified intent, exact utterance: "DC0 is the test of a full mirror. DC1 is the test
of the mirror proxy. DC1 needed the partially created full mirror and associated
script/files/settings removed." Recorded as a D-135 AMENDMENT 2026-07-27 -- the per-DC split
is the DELIBERATE EXPERIMENT, not an interim state. That supersedes both the "PAUSED ... kept
dormant as the fallback" record and the "DC1's proxy is REPLACED by a full mirror afterward"
claim, which sat in THREE places (D-135, CURRENT-STATE:109, dc-cache-proxy.sh:6); all corrected.
dc0 FIRST (gated): sync re-run -> OK 2026-07-27T08:43:46Z ubuntu=0 uca=0 in 20s, confirming
the transient-500 diagnosis. The fixed check now reports PASS -- trustworthy precisely because
the same check FAILED on the same rack twenty minutes earlier. Stage 4 DoD bullet 5 is now MET
for both DCs (dc0 mirror PASS, dc1 proxy PASS), node-side half at G17.
WHY THE ORDER MATTERED. Asked to weigh rip-down-and-rebuild against surgical removal, the
rebuild path turned out to be BLOCKED, not merely slower: dc-cache-proxy.sh did not own its
utility net layer -- it REUSED dc-mirror.sh's <site>-mirror-net.service and refused to install
without it. So on dc1, whose ruled path is the PROXY, three files named dc1-mirror-* (net unit,
apply helper, resolved DNS drop-in) were load-bearing FOR THE PROXY. Consequences: the recorded
removal procedure would have silently killed the proxy, the bullet-5 PASS and the Stage-5 dc1
apt path; and the proxy could not be REBUILT afterward, since nothing but `dc-mirror.sh
install` created the net layer -- rebuilding a PROXY meant reinstalling the whole MIRROR
apparatus, enabled daily debmirror timer included. Operator direction: net layer first, then
rip it all down.
1. NET LAYER MOVED: <site>-cache-proxy-net.service + -net-apply + its own resolved drop-in,
reusing dc-mirror's proven mechanism (bridge from the libvirt net NAME at runtime, hard
rule 3). Proxy-named so a proxy-only DC has NO mirror-named dependency. Idempotent
replace verbs, so both net units coexist safely where a site runs both -- dc0 unaffected.
check asserts the ARTIFACTS, not just the live address/route: asserting only the live
effect would PASS while the layer was really the mirror's, which is the exact failure.
Harness 20/20 (was 15). T12 REPLACED not deleted -- it asserted the superseded design, so
it failed by design; now inverted, plus T12b-T12f. T12b is scoped to non-comment lines on
purpose: the header's account of the removed trap cites dc-mirror.sh install, and deleting
that history to satisfy a grep would lose why the trap existed.
2. INDEPENDENCE PROVEN BEFORE DELETING: a oneshot with RemainAfterExit=yes does not undo its
work on stop, so stopping the mirror unit would have proven nothing. Removed the live state
outright (addr count -> 0, default route count -> 0), applied the proxy-owned unit ALONE,
both restored; check dc1 PASS with dc1-mirror-net.service disabled.
3. WHOLESALE REMOVAL: all mirror units, timer, sync helper, nginx vhost, resolved drop-in, the
debmirror package, and 330G of partial mirror (33,037 files, rm -rf 47.6s; disk 339G ->
9.0G). Zero mirror-named residue, every proxy artifact present, proxy PASS throughout and
never down. Capture docs/audit/dc1-mirror-teardown-20260727.txt.
nginx left installed deliberately: a mirror prereq, but with the vhost gone it is a generic
server on its stock default vhost; removing the metapackage while nginx-core runs is a
half-measure and a full purge is scope creep on a rack host.
Gauntlet ALL GREEN (81), repo-lint 0-fail, dc-cache-proxy 20/20, dc-mirror 24/24.
Revert: `dc-mirror.sh install dc1` (needs debmirror reinstalled) then `sync dc1` to re-pull
~950G; git revert dc-cache-proxy.sh + harness to restore the mirror-owned layer. The 330G
partial is NOT recoverable -- deliberately, per the amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
G17 opened by operator ruling; Stage 4 DoD bullets 5-6 repaired (DOCFIX-204)
...
RULING (GA-R5), 2026-07-27. Question as presented: "The node-side half of bullet 5. Nodes are
powered off by the READY-handoff ruling, so no node-side probe can run as things stand. Either a
gated rescue-boot check on one node per DC now (closes it inside Stage 4), or split it into its
own gate row targeted at Stage 5 first boot (GA-R6 E3 explicitly permits this; a conditional
close is not permitted)." Operator answer, exact utterance: "split it into its own gate row".
Recorded in the G17 row of docs/CURRENT-STATE.md (the authority); pushed before dependent work.
G17 "Per-DC artifact source reachable FROM A NODE", [V], OPEN. dc0 -> curl 200 from a booted
node against the D-135 item-1 mirror; dc1 -> apt fetches through the proxy at 10.12.68.4:3142,
since dc1's ruled path is the CACHING PROXY with no node-facing mirror (D-135 amendment) and
checking it as one fails by design. Trigger: Stage 5 first boot. The row states that G17 is NOT
a Stage-5 precondition -- Stage 5's bootstrap needs open edge egress for the juju agent stream
and snaps (D-135 items 2-3 unbuilt), a different path from the apt artifact source.
Stage 4 retains bullet 5's rack-side half: the source answers on its own address with an
attested-current sync. dc1's proxy PASSES; dc0 pends the sync re-run.
DOCFIX-204 -- bullet 6 was UNSATISFIABLE and its stale text had spread to four surfaces. "NTP
from the DC's own OPNsense edge working" is superseded by D-129(iv) (RULED 2026-07-21, "Keep
MAAS hierarchy", no NTP role on the edge): the DoD asked for confirmation the edge is the time
source, which the ruling had already refused, so a correctly-built DC could never meet it.
Repaired in the phase-3 DoD (bullet 6 struck, bullet 5 rewritten per-DC), the phase-3 Step 7
(REWRITTEN -- it branched on gap #5 that D-135 resolved, applied one mirror check to both DCs
which fails on dc1 by design, and asked for checks "from a deployed node" that the READY handoff
makes impossible), the workflow-doc Gate cell (which also still said "Nodes deployed",
contradicting DOCFIX-200), the buildout design, and the phase-4 prerequisites (same "nodes
Deployed" error -- exactly the expectation DOCFIX-200 removed to stop Stage 5 breaking).
phase-4 also gains a block making it the OWNER of G17's captures, warning that first boot is the
observation window and missing it forces a deliberate rescue-boot.
Text and gate-record only; no live state touched. repo-lint 0-fail. DOCFIX next-free -> 205.
Revert: git revert this commit; all changes are documentation and the G17 gate row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 4 close-out: fix dc-mirror.sh check false-greening the mirror gate
...
Operator directed closing Stage 4 before resuming the DC1 Stage-5 chain. Verifying the DoD's
fifth bullet ("per-DC mirror reachable") found that the check the gate would close on COULD
NOT FAIL.
scripts/dc-mirror.sh:271-273 tested `[ -f last-sync.status ]` and printed the file's contents
behind an unconditional "OK" -- the status word was displayed but never asserted on, so the
check exited PASS regardless. MEASURED on both racks:
- dc0 read "FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0" -> printed OK, PASS
- dc1 read "RUNNING 2026-07-23T21:49:35Z", four days stale,
left by the debmirror the D-135 amendment killed by signal -> printed OK, PASS
GA-R6 requires a stage to close on a NAMED executable check whose captured output the closing
commit cites. A check that cannot distinguish "synced" from "failed" or "died mid-run" is not
that check.
FIXED: the status word is case-analysed. OK* passes. FAIL* misses. An ABSENT file misses (it
was a passing `note`, but nothing has attested the content). An unrecognised word REFUSES
instead of defaulting to success. RUNNING* is deliberately neither pass nor plain fail --
in-flight and died-mid-run are indistinguishable from the file alone, so it reports an explicit
UNKNOWN and cross-checks the unit, naming a RUNNING marker with no active unit as a corpse.
Harness 19 -> 24; T20-T24 lock each status word and T24 locks out the return of the exact
defective shape.
PROOF, re-run against both live racks -- the gate is now honestly NOT MET for the first time:
dc-mirror check (dc0): FAIL / dc-mirror check (dc1): FAIL. Capture
docs/audit/stage4-mirror-gate-20260727.txt.
SUBSTANCE (measured, not inferred):
- dc0's mirror CONTENT is complete: 949G ubuntu + 342M cloud-archive, all three dists plus
pool. Only the overnight INCREMENTAL failed -- "Download of dists/jammy/Release failed: 500
read timeout" aborted the ubuntu leg with 255; the UCA leg succeeded. Transient upstream
fetch failure, not mirror damage. Timer re-arms 2026-07-28 00:14:48.
- dc1's PROXY -- its RULED artifact path per the D-135 amendment -- checks PASS genuinely
(apt-cacher-ng on .4:3142 serving archive + UCA Release 200). Its dormant fallback debmirror
is dormant only ACCIDENTALLY: the timer is `enabled` with an EMPTY next-elapse because the
unit sits in failed/Result=signal, so the ruling is enforced by nothing and a reset-failed
would re-arm a 330G -> ~949G pull.
ALSO RECORDED, not yet actioned: DoD bullet 6 ("NTP from the DC's own OPNsense edge") is STALE
and unsatisfiable -- superseded by D-129(iv) 2026-07-21 ("Keep MAAS hierarchy", no NTP role on
the edge) -- and survives in four surfaces needing a DOCFIX before it can be checked at all.
The node-side half of bullet 5 needs an operator decision (gated rescue-boot check vs its own
gate row per GA-R6 E3); nodes are powered off by the READY-handoff ruling, so no node-side
probe can run as things stand. There is no conditional close.
No live state was altered: both racks were read only. repo-lint 0-fail, dc-mirror 24/24.
Revert: git revert the scripts/dc-mirror.sh + tests/dc-mirror/run-tests.sh changes; the check
returns to passing unconditionally on last-sync.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
creds consolidation batch: SEC-021(b) dc0 power key, SEC-025 NetBox GUI, V2 ruled deferral
...
Operator asked whether vcloud holds a consolidated set of login creds for every account that
exists, and what is missing / needs creating / was never moved in. Audited, then cleared the
consolidation batch on operator direction ("clear the whole consolidation batch first").
The audit's own headline: creds-audit read CLEAN on all three sites while the matrix returned
13 findings -- and the two most consequential gaps were in NEITHER surface. They were found by
enumerating the accounts that ACTUALLY EXIST. Absence of a ROW is invisible to the register,
which generalises D-137's founding argument one level up.
VERIFIED POSITIVE (both previously only asserted):
- MAAS account set COMPLETE: all 6 live accounts accounted for; admin + operator passwords on
vcloud, juju-vr1-dc0/dc1 random+unstored BY RULING with API keys present, MAAS +
maas-init-node MAAS-internal.
- tier-3 V1 now MEASURES maas-admin-password byte-identical to the headend source-of-record,
so SEC-020's stale-trap risk is clear as of this date.
DONE:
- dc0 SEC-012 power key consolidated to vcloud + .pub DERIVED (SEC-021(b) as written).
MEASURED FIRST, because the naming looked like a defect and was not: the headend
maas-virsh_ed25519 and the snap's id_ed25519 are the SAME key; it IS dedicated (distinct
from the dc0 service key, which is what SEC-012 requires); and dc0 using the snap's default
identity is SEC-016's RULED design. So no re-mint, no live power path touched.
- dc1 svc .pub backfilled to the headend store (public-key material, sha256-verified).
- NetBox web-GUI admin password consolidated -- a HUMAN login that had never left the VM it
was minted on, the third instance of the SEC-009/SEC-020 miss class. SEC-025 OPENED for the
at-rest exposure the copy CREATES, not the gap it closed. Open rows 20 -> 21.
- V2 taught the ruled-deferral state: SEC-006's standing "revoke at completion of this
deployment" ruling is now ACKNOWLEDGED (still naming the credential live and exposed)
instead of failing every run. Reissuing the token would have CONTRAVENED that ruling, so the
register was what needed to change. Match demands the cased state word AND explicit operator
ruling attribution, and cannot cross a ledger table cell -- T58/T59 lock the evasion out.
- Added --ledger (default unchanged): V2 read the register via --repo, which the harness must
point at the real repo for S4, so V2 had shipped with ZERO harness cases. T56-T59 added.
Findings 13 -> 7. Matrix 82 rows, harness 60/60 (was 56), creds-audit CLEAN x3, gauntlet ALL
GREEN (81), repo-lint 0-fail. The register is NOT green and is not expected to be.
RESIDUAL, all recorded: dc0-edge-api x2 (the opnsense-api.txt re-mint is a live edge mutation,
deliberately EXCLUDED from this batch), S5 x3 (RULED by SEC-016 -- needs a ruled-exception
mechanism, operator decision), S6 conflation x1 (the SEC-020 defect), E4 x2 (Stage-5/6 rows).
LOGGED NOT ACTIONED (hard rule 1): no registered root/console credential at EITHER DC edge
(measured absence of row/manifest/SEC row; what those passwords ARE is UNKNOWN and
deliberately unprobed per hard rule 2 -- vector is the LAN-reachable GUI and serial console,
not SSH, which is key-only and proven); plus the two structural blind spots that let it hide
-- S5 compares only cardinality=per-DC while all six per-site rows are office1-only, and
vm-secret-locations declares no rack/edge/cloud/unit/client location though the checker
accepts them.
Revert: per-item reverts in docs/changelog-20260727-creds-consolidation.md. The headend, snap
and NetBox VM source copies were never modified, so every file move is reversible by deletion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
close-out sweep: capture what existed only in the session transcript
...
Operator-requested final sweep before clearing. Two classes of content lived
only in the transcript and are now durable in
docs/audit/queued-findings-20260726.txt.
PART A -- the secrets-storage advice. Checked FIRST what the repo already
carries, and most of it was already there: OpenBao/BUSL/the fork question and
auto-unseal are in docs/D-068-vault-1.8-vs-1.16-analysis.md, and creds-mint.sh
is specified in detail in D-137 item 2(a). Only three items were genuinely
unrecorded: Tang/Clevis as the no-HSM unseal mechanism; MAAS 3.7's Vault
integration measured `status: disabled` together with the MAAS/Vault circular
dependency that must be designed around on bare metal; and a Vault SSH CA to
retire the static keypairs. The capture is framed so a future session does not
re-propose what is already ruled -- re-proposing settled decisions is this
project's measured failure mode.
PART B -- ten committee findings acknowledged but deliberately not acted on.
Most consequential: mint-ref line pins rot SILENTLY (S4 checks existence and
EOF, never content, so every pin becomes wrong-but-passing when the runbooks
are rewritten for Roosevelt), and ruling-5 remediation is indistinguishable
from ruling-5 evasion to S6. cardinality (B6) needs an operator ruling.
PART C -- items deferred by ruling, recorded so a later reader does not mistake
them for oversights.
Also repaired a dangling sentence fragment in CURRENT-STATE left by an earlier
edit this session, and noted the repair rather than making it silently.
Gauntlet ALL GREEN (81), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-26 |
D-137 tier 3 behavioural half: V2 declared-state + creds-probe.sh
...
Boundary and cadence operator-ruled: "Full -- including OpenStack service
credentials", "Stage close + DC standup close". I had recommended
passive-only and flagged the costs; the operator ruled otherwise.
V2 (passive, zero authentication) fires TODAY on a real defect: SEC-006 has
recorded the NetBox upstream token as BURNED since 2026-07-12 while the
matrix carried it as a healthy consolidated row. Nothing reconciled the two
surfaces. Keyword matching on ledger prose is fragile, so it is conservative
and reports the matched text as its basis.
creds-probe.sh is SEPARATE by design: a probe must USE a credential, and
creds-matrix.py is metadata-only by an enforced invariant (T22/T51). Folding
probes in would have relaxed that guard, and a guard relaxed once stays
relaxed.
Safety: never retry a failed auth (retry loops lock accounts; the MAAS admin
identity backs 19 call sites), never argv, never echo, stage-aware skips, and
per-tenant credentials REFUSED BY CONSTRUCTION -- a declaration cannot opt
them back in (D-069, ruling 3). Not a deploy gate.
FALSE GREEN in my own first run: ssh inside a while-read loop ate the loop's
stdin, so 1 of 13 probes ran and the run exited 0. A partial run reporting
clean is the exact failure class this tool exists to catch. Fixed with ssh -n
and locked by a total-accounting check that makes a partial run VOID.
13 probes declared: 3 active, 4 stage-pending, 6 declared-only with reasons
recorded. OpenStack credentials declared now, skipping until Stage 5.
tests/creds-probe 9/9; gauntlet 80 -> 81; phase-3 DoD wired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: local half blocking in P5; --pending-stage -> coupled projection
...
Two operator decisions, both taking the recommendation.
GATE SHAPE: local-only blocking, remote operator-run. preflight P5 now runs
creds-matrix.py --tier2 -- tier 1 STATIC plus tier 2 LOCAL, hermetic (no ssh,
no sudo). It already catches SEC-021's three dc0 on-disk absences without
leaving the jumphost. The remote half stays an operator-run check at
DC-standup close and stage close.
Wiring --remote --privileged into the deploy gate would let a host outage, a
NOPASSWD change, or a MAAS snap refresh turn the gate red for reasons that
are not about credentials, and a gate that fails environmentally is one
people learn to bypass. The repo already made this call once
(run-tests-all.sh:7-8, "different cadence"), and preflight already shows the
failure mode: P4 reads "MAAS unreachable" because it runs on vcloud while
MAAS lives on the headend.
STAGE INPUT: --pending-stage replaced by creds-manifests/stages-reached, a
machine-readable projection of CURRENT-STATE, coupled by new repo-lint L12
(the L10 pattern). A hand-typed gate parameter can be widened silently, and
GA-R1 puts status in CURRENT-STATE alone. A drifted projection would make the
gate defer REAL absences -- a false green.
Exhaustive both ways: an unclassified mint-stage is a hard FAIL, a
classified-but-unused one is a stale-declaration FAIL. That caught `stage4`
immediately -- Stage 4 is open and executing but mints no credential.
Self-inflicted break caught by the existing harness: I first renamed L10's
message to "L10/L12", breaking five cases that grep the established L10 id.
Reverted -- new rules get their own message id, they do not rebrand old ones.
Harnesses: creds-matrix 52 -> 56, repo-lint 45 -> 47, preflight 10.
Gauntlet ALL GREEN (80), repo-lint 0-fail, baseline unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
SEC-024 REMEDIATED (mode): chmod 600, operator-approved; retention deferred
...
Operator-approved mutation, executed and read-back verified: chmod 600 on
terraform.tfstate.backup and terraform.tfstate.pre-G16-20260721. All four
state files now 0600, matching the live state file, and the checker's
E2 WORLD-READABLE finding cleared.
Severity was corrected in the preceding commit BEFORE any remediation: the
row as opened said "group- and world-readable", but opentofu/ is 0700 and ~
is 0750, so no other account could reach it, and neither file is git-tracked.
Defence-in-depth defect, not live exposure. A ledger row that overstates an
exposure is its own defect.
Retention of the two pre-* state-surgery snapshots is operator-DEFERRED to
the stage close-out: both gates (G6, G16) are CLOSED, but deletion is
irreversible with no other copy of that state.
Deliberately NOT done: the umask cause (out of ruled scope -- a future apply
may write 0664 again and the P5 gate will catch it), and the underlying
DOCFIX-175 issue that the key reaches state files in plaintext at all, whose
rotation is owed under SEC-018/-019.
Caveat recorded in creds-matrix-notes.md: E2 reports file MODE, not
reachability. That is the right default for a gate, but whoever writes up an
E2 finding owes a reachability check before using the word "exposed" -- this
defect is exactly why.
Gauntlet ALL GREEN (80), repo-lint 0-fail, finding-class baseline unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 3 (VALIDITY) BUILT: cross-copy sha256 provenance
...
Closes the gap the plan names: creds-audit PARSES declared provenance and
never verifies it, so SEC-020's stale-trap warning has been unchecked prose
and its sha256 equality was a ONE-TIME manual act on 2026-07-25.
sha256sum runs as an EXTERNAL command on both sides so only a 64-hex digest
enters the checker -- hashing bytes in Python would meet the letter of "never
transfer content" and not the spirit.
RESULT: 4 groups compared, all byte-identical, INCLUDING the SEC-020 pair
(maas-region-admin, region source-of-record vs jumphost copy). Now
machine-verified instead of remembered. The [ok] line names the groups.
TWO FALSE POSITIVES IN MY OWN DRAFT, caught by verifying before reporting.
Keying digests by (role, site) let an identity's DISTINCT artifacts at one
scope overwrite each other, so it compared unrelated files and reported drift
on maas-region-admin and dc1-svc-key. Manual sha256sum showed both IDENTICAL.
Reporting that would have raised a false credential-rotation alarm.
Root cause was a schema gap: nothing expressed which rows are byte-copies.
custody gains `verbatim-copy` [OPS]. V1 now compares only rows sharing
(id, filename) across scopes, or a source-of-record row against declared
verbatim-copy rows -- SEC-020's exact shape. Everything else is skipped with
a reason: a pair we cannot justify comparing is not a pair we may report
drift on. verbatim-copy is manifest-bearing; it asserts byte-identity, it
does not move the file.
Harness 44 -> 52: T47/T48 regression-lock both false positives; T49 keeps
SEC-020's case from falling into T48's skip; T51 asserts the digest path uses
sha256sum and that no content-transfer verb exists anywhere in the checker.
NOT built: tier 3's behavioural probes. Liveness, expiry, revocation state
and grant/scope drift remain unverified.
Gauntlet ALL GREEN (80), repo-lint 0-fail, finding-class baseline unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 ruling 4 EXECUTED: SEC-009 demoted to a pointer, policy moves to D-137
...
Ruling 4 was already RULED and its stated trigger (tier 2 built) had passed,
so this is execution, not a decision.
The standing consolidation rule moved from docs/security-ledger.md into
D-137, restated as six numbered points and updated to D-137's own mechanisms
(matrix is the register; manifests are DERIVED; discovery bounded by
vm-secret-locations; one identity one principal; enforcement is blocking P5).
The ledger keeps a pointer and returns to being purely an exposure/rotation
register -- which is what lets a gate cite a D-number instead of a SEC row.
Kept as history: SEC-009's founding addendum (the NetBox token that lived
only on the VM and went un-consolidated until needed). That miss established
the convention, and history is what the register is for.
Carried forward WITH CORRECTIONS: the moved text's "11 entries", "7/7", and
its description of manifests as hand-declared were all stale and are not
reproduced.
Callers repointed in the same commit, because a demotion that leaves callers
citing the moved text is not done:
- dc-dc-phase3-maas-enlist-deploy.md:491 is the ONLY credential DoD in any
runbook and it told operators to hand-edit creds-manifests/$DC.manifest --
a GENERATED artifact under sub-ruling 2. The live runbook was teaching the
anti-pattern. It now directs a matrix row plus a locations row, and its
close-out requires both creds-audit CLEAN and no new creds-matrix finding
class for that DC.
- docs/vr1-office1-as-built.md restated the convention; now a pointer.
Gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 ruling 1 EXECUTED: tier 1 wired as the blocking preflight P5
...
Operator answer, exact utterance: "wire tier 1 as the blocking preflight Pn"
(2026-07-26), resolving the open sequencing question. Recorded on D-137.
preflight.sh gains P5: credential matrix (D-137 tier 1), blocking, with
worst-exit aggregation, placed AHEAD of the stage-2 reminders block (now P6)
so its verdict participates in the deploy decision. Tier 1 only -- it is
offline and its findings ARE the three ruled failure classes. Tier 2 needs
--remote/--privileged and a caller-supplied --pending-stage, which do not
belong in an unattended gate; that stays an open decision.
FAIL-OPEN in my own wiring, caught by the new test: a missing creds-matrix.py
makes python3 exit 2, which `note` maps to WARN -- so deleting the gate would
have downgraded it to a warning rather than stopping the deploy. P4 already
guarded this class; P5 did not until T9 failed. Now guarded explicitly.
tests/preflight 7 -> 10: T8 a red register FAILS the deploy (blocking, not
advisory); T9 fails closed when the checker is absent; T10 the gate runs
before the reminders block.
CONSEQUENCE, not overclaimed: preflight exits 1 with P5 among the reasons,
but preflight was ALREADY exiting 1 before this change (P4: octavia-pki.yaml
absent, MAAS unreachable from the jumphost). P5 adds a fifth reason to an
already-red gate; it flipped nothing and closes no open deploy path.
Verified by stashing the change and re-running.
Gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 phase 3: re-measure the register; SEC-024 opened
...
~21 inferred filenames corrected against their own mint-refs. The session had
owned six; those were only the rows the first sweep physically touched.
Octavia's 8 real basenames plus its THREE SUBDIRECTORIES (the bare
~/octavia-pki/* pattern matched ZERO of the 8 artifacts, two of them CA
private keys); vault's init.txt; the tenant rows' <client>- instance prefix,
with a placeholder matcher so they can match a real basename at all;
kube-config -> config; .maas.cli -> .maascli.db (measured: ~/.maas.cli does
not exist).
sec-ref mis-attribution corrected: both juju-maas-user rows cited SEC-020;
the on-subject rows are SEC-018/-019. This is the SEC-015 class recurring --
one instance was fixed by hand earlier without checking whether it was
systemic. It was.
mint-ref corrected: admin.pass cited a CONSUMER (:452), not the mint (:450);
the RGW key cites a line that takes it as INPUT, so it is operator-terminal.
Rows added: the libvirt SSH power password (standing rotation obligation from
a 2026-06-26 plaintext exposure, in neither the matrix nor the exclusions),
vault-ca-root, and the tfstate siblings. ~/as-executed/ deliberately NOT
declared, reason recorded: it holds session logs, so declaring it made every
log a false UNDECLARED.
Four further defects in my own phase-2 changes, found by RUNNING the sweep:
shlex.quote broke ~ expansion; a non-existent parent read as unreadable
rather than absent; site-key `-` needed shared-location semantics; the
tfstate siblings needed rows.
SEC-024 OPENED -- new exposure found by the fixed checker on its first clean
run: opentofu/terraform.tfstate.backup is mode 0664 and carries the MAAS API
key in plaintext (DOCFIX-175); the live state file is 0600. Invisible to
every prior control because the world-readable check was custody-gated and
the siblings were undeclared. LOGGED, NOT ACTIONED (hard rule 1).
Findings 35 -> 13, all true. Matrix 81 rows. Harness 44/44, creds-audit
15/15, gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 committee audit + phase 1: correct the record
...
Six-lens read-only review found the delivered tier-1/tier-2 build has eight
reproduced false-greens and that the records overstated what was verified.
Four lenses converged independently on tier 2 being SITE-BLIND.
Record corrections only in this commit (no code, no data):
- NEW docs/audit/d137-location-listing-20260726.txt: the actual stat listing.
CURRENT-STATE asserted remote filenames/modes citing the tier-2 capture,
which is a VERDICT file containing none of them -- a GA-R1 rule 2 defect.
- .maas.cli was NEVER MEASURED and is wrong; the MAAS snap CLI profile is at
~/snap/maas/current/.maascli.db (measured). It sat inside the fix for the
inferred-filename miss, so the corrected count is ~21, not six.
- WITHDRAWN: "ZERO undeclared files ... fully accounted for" (rested on the
site-blind check). CORRECTED: "SEC-021 on-disk half REPRODUCED" now reads
power-key artifacts only -- opnsense-api.txt's absence was masked.
- CURRENT-STATE's "virsh gap NOT fixed" corrected; it was fixed in 6d43619,
whose commit did not touch the status authority because L10's trigger set
excludes .claude/settings.json.
- Superseding figures recorded; stale "14 NOT JUDGED" -> 2; glob is *.pass.
- 12-column amendment recorded on D-137 in design-decisions.md; it previously
lived only in a changelog (non-citable per GA-R2 rule 1) and the build spec.
- Repaired citations my own edits broke (guard moved 71-73 -> 90-92), the
plan's 12-vs-27 operator-terminal count, the "two the ruling did not name"
claim (it is one), and an unsupported ruling date in the notes.
Code and data remediation follows in later commits.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: privileged sweep completed; permission gap fixed; 6 inferred filenames corrected
...
The block was a PATTERN GAP, not a policy conflict. Bash(ssh * sudo *) was
already in the project ask list, but it needs a literal space before `sudo`
and the command was ssh <host> 'sudo ...' -- the quote meant no rule matched,
so it fell through to the classifier. Added the quoted variants plus a
targeted rule for the privileged invocation; all `ask`, never `allow`.
FLAGGED NOT FIXED (hard rule 1): Bash(ssh * virsh *) has the identical gap.
MEASURED (docs/audit/d137-tier2-privileged-20260726.txt): the region secrets
dir holds exactly admin.apikey/admin.pass/db.pass/lxd-trust.pass at 0600 --
precisely the SEC-020(i) carve-out list; netbox holds admin.pass/api.token/
secret_key at 0600; the SEC-022 shadow stores are confirmed and now audited.
ZERO undeclared files at any declared location: SEC-020 and SEC-022 are
accounted for.
OWNED: the first matrix draft carried SIX filenames I authored plausibly
instead of measuring -- a hard-rule-2 miss. The checker's undeclared-detection
caught every one on first contact with real hosts. Corrected to measured
values, real locations declared, and five colocated non-secret artifacts
declared so the check stays meaningful. Matrix 72 -> 77 rows.
9 findings remain, all true: SEC-021 (3 declaration + 2 on-disk), the ruling-5
conflation, and 3 S5 asymmetries -- dc0's snap power key is the generic
id_ed25519 while dc1's is the dedicated id_dc1_power, the SEC-012/SEC-016
per-DC-isolation divergence made visible.
Harness 35/35, gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: wire --privileged escalation (run BLOCKED by the auto-mode classifier)
...
--privileged retries a remote probe with `sudo -n` ONLY where the
unprivileged probe returned `unreadable`, keeping the privileged surface as
small as ruling 3 keeps the search surface. Metadata only; the T22
metadata-only guard still passes; verified inert (zero escalations) when
nothing is unreadable.
The operator approved the privileged read, but the auto-mode classifier
denied the remote-sudo shape -- the same wall the 2026-07-23 close recorded,
whose noted fix is manual permission mode. NOT worked around: reading a
root-owned directory is exactly the privilege being denied, so any
alternative would fail identically or misrepresent the result.
SEC-022 shadow-store contents and the SEC-020 region secrets stay
UNCONFIRMED; the sweep reports them as E0 UNREADABLE FAILs, which is the
correct outcome for an audit that could not look.
Also de-staled the docstring tier table (tier 2 is built).
Harness 35/35, gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
