| 2026-07-29 |
Chain-audit deploy path: DC-aware preflight, dc1's real overlay set, Fork-3 role tags
...
Findings 16-20 from the D-136 chain audit, actioned.
preflight P2 is DC-AWARE and validates the deploy's ACTUAL overlay set. It
hardcoded --dc vr1-dc0 with no dc1 branch while dc-dc-phase4 invokes it BARE
as the dc1 gate and tells the operator to expect PASS -- so a dc1 operator
got a green gate that validated dc0's numbers against dc0's bands and never
parsed a dc1 artifact. It also passed one overlay where the deploy passes
two. DC=vr1-dc1 now merges vips + machines for that site.
dc1's deploy command names ALL its overlays. phase-4 Step 4 named only the
vips overlay, so a dc1 deploy run as written merged a machines block still
saying tags=openstack-vr1-dc0 -- grep for vr1-dc1-machines across runbooks
and scripts returned ZERO, i.e. that overlay was in no executable path. The
step now carries the full command, states dc-ha-scaleup is deliberately
excluded per R6, and adds the --dry-run machines-merge confirmation the
overlay's own VERIFY-LIVE note required and no step performed.
phase-6 deployed with NO VIP overlay and named a file that never existed.
Both blocks -- including the real apply against a live model -- passed
overlays/${DC}-hostnames.yaml, never authored, and no VIP overlay;
post-ruling-3 that is a merged input with zero VIPs including the designate
those commands exist to add.
scripts/maas-role-tags.sh ships Fork 3 (harness 9/9). Every machine
constrains on two tags; only the site tag is ever created, so the three role
tags exist nowhere and juju's allocation constraint matches no machine.
Role is DERIVED from two signals that must agree -- the ruled D-134 octet
band and the host's own name token -- and disagreement REFUSES rather than
picking one. Nodes match by pinned boot MAC, never hostname. Dry by default,
every write read back, absent CLI distinguished from unreachable service.
apply --commit is a live MAAS mutation and is NOT run here.
OWNED, third appearance of the IFS trap: ROLES="control compute storage"
under IFS=$'\n\t' does not word-split, so the script reported one absurd tag
named 'control compute storage'. Caught by its own harness pre-ship. Same
class as commit 1's CHECK 1 death and phase-05's aborting captures -- these
strict-bash gates should use arrays, never space-separated string lists.
Also fixed: the role-derivation refusal swallowed its own diagnostic.
Gauntlet ALL GREEN (87) on vcloud; repo-lint 0 fail / 614 files; preflight
verdict unchanged in kind.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-136 step 6: the chain audit -- R9's drift gate built, three refusals, two dead cases
...
Three read-only lenses over data source -> renderer -> deployment mechanism,
the ruled final step of D-136 option (D). Lens 3 was adversarial and made
the chain produce wrong-but-green THREE times, all executed.
All three converged on one root cause: dc1's overlay was byte-gated and
dc0's was gated by nothing. render/values/ was referenced by zero scripts
and zero harnesses, while preflight P2 feeds the deploy gate dc0's overlay.
R9's render-drift check had been RULED and never built.
Executed wrong-but-green:
- transposing two apps' octets in the dc0 overlay (consistent, in-band,
correct prefixes) -> PASS
- changing ONE WORD, family: dual -> v4, silently reverting dc0 to IPv4-only
against R2's ruling -> PASS, because dropping prefer-ipv6 and the v6 legs
together satisfies invariant 9, and vip_dual is printed but never asserted
tests/render-drift/ is R9's gate, built. It iterates render/values/*.yaml,
reads each file's own output: field (inert until now), renders and
byte-compares. A new per-DC values file is gated the moment it is committed.
Non-zero floor plus a proof-of-teeth case that mutates a copy and requires
red.
Three refusals where the chain took the last writer silently:
- duplicate (role, kind) in the apex, in BOTH readers. Appending one spare
prefix -- routine IPAM work -- re-homed every dc0 VIP into a different /64
and went green, because the gate checks overlay-vs-apex agreement and both
sides read the same wrong value. The winner depended on JSON array order.
- duplicate app name / duplicate octet / out-of-band octet in APP_OCTET. A
duplicate name emitted two YAML blocks, safe_load kept the last, and one
app's ruled octet vanished with nothing red.
Two harness cases that could not fail:
- T3b's byte-compare was CIRCULAR -- header derived --from-overlay $LIVE
then compared --against $LIVE, so a third of the bytes were self-copied.
Proven by rewriting a header line to a false claim and still matching.
- T29's own fixture was inert; its assertion held from the overlay alone.
OWNED: my first T29 re-point asserted a --show-legs flag THAT DOES NOT
EXIST. I invented an interface instead of reading one. The harness caught it
immediately; replaced with real behaviour and recorded rather than quietly
fixed.
Gauntlet ALL GREEN (86) on vcloud; repo-lint 0 fail / 612 files. The larger
deploy-path half is logged not actioned -- see the changelog, items 16-25.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Octavia: R8a BUILT (record was false), two stale surfaces, R7 generator de-frozen
...
Follows an upstream/vendor research pass answering "can Octavia run full
IPv6" -- yes; nothing inside Octavia requires IPv4. charm-octavia creates
the lb-mgmt subnet with ip_version: 6 and has no IPv4 code path at all; the
health manager is v6-capable both directions; the amphora agent binds '::'
and verifies certs by amphora UUID, not IP; and Nova metadata is not a
dependency (config_drive=True). Surviving v4 pressure is soft or external.
R8a BUILT. phase-05-octavia-verify.sh now compares o-hm0's MTU against
lb-mgmt-net's (resolved BY TAG, never by name), fails in either direction,
and REFUSES when a value cannot be read. This closes the LP#2018998
obligation, live on our exact pin after its 2025-12-31 recurrence.
THE DECISION RECORD HAD CLAIMED THIS SHIPPED. design-decisions.md carried a
present-tense "Delivery: the assertion ships in ..." written on the day of
the ruling, while the file held zero MTU references and its last commit
predated the ruling by a month. Corrected in place. Sharper than the usual
ruled-but-not-built class: the decision doc was the false witness.
TWO OWNED ERRORS:
- I concluded "no octavia harness exists" from a NAME grep. The script's
harness has always been tests/phase-05. Duplicate deleted, R8a cases
folded into the real one.
- My first MTU draft ABORTED the script: a $( ) under set -euo pipefail with
inherit_errexit exits the run rather than reaching the refusal branch. The
pre-existing harness I had just declared nonexistent caught it. Same class
as commit 1's IFS word-splitting trap; both now locked by cases.
Two live surfaces still contradicted R8 -- dc-dc-phase4 and the workflow doc
both called lb-mgmt IPv6 "a real, open risk" and recommended keeping it
v4-only, citing the two refuted bugs. A reader would have re-litigated a
closed ruling in the wrong direction. Both superseded in place.
R7 executed: phase-01 Step 1.0-GEN gains a $DC selector; the baked
/CN=VR0 DC0 CA subjects derive from it, and the VIP gate derives this DC's
provider prefix from the same overlay it read the VIP from -- R7's explicit
"read the MERGED input" caveat. dc1's 10.12.64.57 used to hard-ABORT, so no
dc1 artifact could be produced. Verified both DCs, negative control still
aborts.
Logged not built: the controller cert's CN/DNS SANs still carry dc0.vr0
(outside R7's ruled scope, inert while os-public-hostname is unset), and it
gains no IPv6 IP SAN though the VIP is dual-family -- a gap, not a break,
and adding v6 SANs needs its own ruling.
tests/phase-05 14 cases (was 9); gauntlet ALL GREEN (85) on vcloud;
repo-lint 0 fail / 611 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Ruling 3 commit 2: dual-stack ADD -- R2 and R11 are now BUILT, L3-9 resolved
...
Both per-DC VIP overlays re-rendered dual-family: every app carries
prefer-ipv6: true and a six-address vip, and vault .61 + designate .62 are
built. Measured both DCs: 13 clustered VIPs (13 dual-family) + 12 hacluster
principals all carrying a VIP -> PASS. With dc-ha-scaleup stacked: 13
principals, PASS -- R6's ruled ordering (VIPs before the HA overlay) is now
satisfied and executable. Two ruled-but-never-built decisions became
artifacts in one pass.
L3-9 RESOLVED. dc-dc-ipv6-family-matrix.yaml is narrowed to ceph-mon only;
its ten duplicate vip + prefer-ipv6 pairs (every value an unrendered token)
are gone, removed TOGETHER -- removing the vips alone would have recreated
the defect deliberately. Both merge orders now PASS, i.e. order-independent,
which is the actual fix. ceph-mon's ULA-only networks survive because they
exist in no other file. The two Launchpad citations arguing octavia lb-mgmt
IPv6 was an open risk are removed: both were read in full and both failed,
and R8 closed that question on 2026-07-27.
Also corrected at source: CURRENT-STATE described L3-9 as "the dangerous
merge order is the one that PASSES". Measured -- that stopped being true
when invariant 9 shipped on 2026-07-28. The merge semantics it describes
remain accurate; only the green-over-loss conclusion is retired.
Gate repairs the agents caught BEFORE the edit:
- phase-01's deploy guard would have ABORTED THE DEPLOY AGAIN -- its HI regex
(5[0-9]|60) excludes .61/.62, so it would have read 13/11/0 against a
required 11/11/0. Widened, counts moved to 13, verified 13/13/0 -> DEPLOY.
Second near-miss on this one guard across two commits.
- pre-flight-checks CHECK 1 hardcoded if(n!=3) and would have failed all 13.
Now takes a triple OR a sextet and asserts a sextet's last three legs are
really v6. Proven able to fail on both new paths.
Harnesses re-pointed, never deleted. T16/T17 asserted R11's gaps were OPEN;
inverted to the surviving invariant and PAIRED with new T16b/T17b that remove
a VIP and demand the check still fires. T20/T21 had become no-ops that would
have gone GREEN testing nothing -- inverted. A v4-only twin fixture keeps the
pre-dual-stack cases testing their own invariants. The renderer harness now
proves reproduction on both shapes, including new T3b against the live
dual-family overlay.
provider-bundle-check 33/33; render-dc-overlays 18/18; render-baseline 10/10;
preflight 16/16; gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 611.
PREFLIGHT: BOTH STANDING REDS CLEARED, 3 fatal -> 2 fatal. The remaining two
are the deliberately-absent octavia-pki overlay and MAAS unreachable from
vcloud, which R10 measured clears on voffice1.
FLAGGED, not hidden: octavia's dual-family API VIP is an inference from R2
applied evenly, not a quoted ruling. Worth one operator confirmation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Values-file provenance symmetry; repair the mangled ledger rotation note
...
Two small corrections found on review.
(a) The values file recorded `apex_record` only on the dual-family path, so
the dc0 v4 file carried no provenance at all -- and the asymmetry would have
read as a bug once commit 2 gives dc1 the apex field. Added `source_v4`,
recorded on BOTH families, because the v4 prefixes come from lib-net.sh and
NEVER from the apex: a values file carrying only `apex_record` misattributes
half its own inputs. Re-derived dc0; the rendered overlay is byte-identical
(a3b30e40...) and dc1 still reproduces exactly.
(b) docs/session-ledger.md's third rotation note was left MANGLED by an
earlier edit -- a sentence broke off mid-number and ran into a stray fragment
of the 2026-07-25 MAAS-admin-recovery summary. Flagged at this session's open;
repaired here rather than only reported, with a note saying so and a pointer
to that summary's archived body.
Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Ruling 3 commit 1: bundle.yaml is VIP-free; dc0's VIPs are now a RENDERED overlay
...
The 11 inline vip: lines are gone from bundle.yaml and live in
overlays/vr1-dc0-vips.yaml, GENERATED by scripts/render-dc-overlays.py from
render/values/vr1-dc0-vips.yaml -- the renderer's first real output.
placement's then-empty options: block was dropped rather than left parsing
as null. Ruling 3's dual-stack ADD is commit 2 and is NOT in here.
PROVEN NEUTRAL by the ruling's own named check: provider-bundle-check on the
pre-extraction bundle and on the merged pair produce line-for-line identical
output, same known designate failure included. A structural diff separately
confirmed only vip moved -- applications 56 -> 56, relations 108 -> 108,
every other key byte-identical.
The 11 inline comments were carried, not dropped: extracted verbatim by
parser rather than retyped, via new optional per-app comment support in the
renderer. An app with no comment emits none, so dc1 still reproduces
byte-for-byte at 1634 bytes. The B1/B5 tokens stay DEFINED in bundle.yaml's
header and the overlay header names that, so the references do not dangle.
The blast-radius repair is the bulk of this, and every item was found BEFORE
the edit by the read-only agent sweep:
- preflight.sh P2 validated the bare base (11 phantom failures post-move) ->
merged input.
- pre-flight-checks.sh CHECK 1 read the base as raw text and would have seen
ZERO VIPs. This is BLOCKER-1, fixed the ruled way and not by bumping the
count. Measured trap: the file sets IFS=$'\n\t', so a space-joined string
reached grep as ONE filename, grep exited 2, pipefail propagated and the
gate died silently mid-check. Fixed with a bash array.
- phase-01's RUN block gates juju deploy on 11/11/0 and would have read
0/0/0 and ABORTED THE DEPLOY. Both guards now span base + overlay, and a
second latent bug was fixed in the same edit: grep -c over two files prints
one count PER FILE, so the bare $( ) captured a two-line string and every
numeric test would have failed anyway. Now awk-summed; verified 11/11/0.
- phase-01's Octavia SAN derivation raised KeyError, swallowed by || true
into an empty VIP. Now reads the overlay, deriving 10.12.4.57.
- phase-00 teardown, dc-dc-phase4's "NO VIP edits" guidance, appendix-A L3,
and render-baseline's now-unverifiable bundle hash all repaired.
Harness 30 -> 31. The fixture base is now bundle + dc0 overlay, since that
pair is the deploy input; T16 renamed to match. NEW T31 asserts the bare base
fails for every clustered principal, so "deployed without its overlay" is a
tested state rather than an assumption.
Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 610 files scanned.
Still red and expected: CHECK 1 at OK=11 want 13, and designate -- R11's
.61/.62 remain ruled-but-not-built, which is commit 2's job.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Render-pipeline step 4: the renderer, reproducing a reviewed artifact byte-for-byte
...
scripts/render-dc-overlays.py. Two stages, split exactly where forward item
F2 froze the interface: `derive` (apex + lib-net.sh -> a per-DC values file)
and `render` (values -> overlay text). `render` is PURE -- no apex, no
network, no clock -- so it is offline-testable and a future CI job re-runs
only `derive`.
ORDER INVERTED BY MEASUREMENT. The plan was ruling-3's VIP extraction first.
Three read-only agents measured two facts that reversed it: the byte-for-byte
reproduction window is still open (the live dc1 overlay is still identical to
the frozen fixture) and closes on its own when the R2/R11 reconciliation
lands; and the extraction's blast radius is far larger than ruling 3's "2
commits" -- it breaks the phase-01 deploy gate (juju deploy runs only on
11/11/0), the Octavia SAN derivation, preflight in two places, and 8 of 30
harness cases. Better done BY the renderer, so dc0's overlay is a validated
output rather than a hand edit.
A TEXT emitter, not yaml.dump: a round-tripper destroys the comment header,
normalises the quoted vip scalar, and imposes its own key order -- all three
are load-bearing bytes. Two things are INPUT rather than invented: the 12-line
comment header (lifted verbatim via --from-overlay, so a reproduction run is
honest about which bytes the tool actually generates) and the ascending-octet
emission order (sorting by name reproduces nothing). The ruled app->octet map
is READ with ast out of dc-plane-apex-import.py rather than restated as a
second drifting copy.
RESULT: output hashes to 3f93ecb3..., identical to both the committed overlay
(1634 bytes) and the frozen fixture. PROVEN ABLE TO FAIL, not merely observed
passing -- three seeded faults all fail the compare (wrong octet, dropped
header, dc0 prefixes against the dc1 artifact).
THE ARITY WORK PAID FOR ITSELF WITHIN THE HOUR. The first dual-family output
was malformed: the values file stores each v6 /64 base with trailing colons
stripped, and the emitter joined with a single colon -- 2602:f3e2:f03:11:50,
not a valid address, and it still looks like one. provider-bundle-check,
fixed yesterday for exactly this class, rejected all 13 applications. Locked
by T13/T13b.
End to end: apex -> values -> renderer -> overlay -> gate. The dual-family
dc1 render with R11's ruled-but-unbuilt apps produces 13 VIPs, all
dual-family, PASSING provider-bundle-check --dc vr1-dc1 including the new
hacluster invariant. Nothing committed has been regenerated by it yet.
tests/render-baseline gains T6, the reproduction case its README invited --
ADDED, with T1-T5 deliberately not repointed at live, and comparing against
the FIXTURE because live is supposed to diverge. T6 proven able to fail by
making the renderer sort alphabetically.
New harness 17/17; render-baseline 10/10 (was 9); gauntlet ALL GREEN (85) on
vcloud, manifest re-recorded after the drift gate correctly caught the
addition; repo-lint 0 fail / 608 files scanned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-28 |
VIP arity gap CLOSED + R11's three ruled gate changes executed
...
provider-bundle-check.py could not express what R2 and R11 already ruled:
:137 required exactly 3 addresses per vip (a dual-family VIP is 6, so it
would have failed EVERY application) and :149's octet extraction returned
the whole string on a v6 literal, leaving every v6 leg trivially unique
and unchecked. The apex now holds the dual-family set a renderer will
emit, so the only gate that validates overlays could not validate what is
about to be produced.
Arity: a vip is now a v4 triple OR a dual-family sextet. The v6 legs are
validated against the per-DC v6 /64s READ FROM THE APEX RECORD (D-136
option (D)), reusing dc-plane-ipam.sh's (role, kind) keying -- the
provider leg takes the dedicated GUA VIP /64, admin/internal their plane
/64s. The v6 host part must MIRROR the v4 octet textually. prefer-ipv6
and the v6 legs are COUPLED in both directions: the measured L3-9 finding
is that the merge order which keeps prefer-ipv6 while dropping the v6
legs is the one that exits 0. An unreadable apex REFUSES at exit 2; a
v4-only bundle needs no apex.
R11 (D-020 amendment, ruled 2026-07-27), same pass: band 50-60 -> 50-99
in both separately-named sites; VIP_COUNT_EXPECT 11 -> 13;
EXPECT_PUBLIC_VIP deliberately stays 11 (measured -- neither vault nor
designate has a public binding); and an hacluster principal with no vip
now FAILS, since cluster_count is asserted nowhere and a 3->1 rewrite of
all 20 values produces a byte-identical PASS.
Measured consequence, not glossed: two sub-checks flip PASS -> FAIL
(designate's missing VIP; CHECK 1 at OK=11 want 13). preflight was
ALREADY exit 1 before this change and is still exit 1 after -- no deploy
path that was open is closed. Both reds are the ruled gate reporting real
work owed; per R6 the .61/.62 VIPs land before the HA overlay.
Harness 15 -> 28. Three cases RE-POINTED rather than deleted: the fixture
base splits into a pristine repo bundle and one carrying designate's
ruled .62, with new case T16 asserting the pristine bundle DOES trip the
new invariant. When .62 lands, T16 gets re-pointed, not deleted. Proven
able to both fail and pass on the same check (T16 red / T18 green).
Gauntlet ALL GREEN (84) on vcloud; repo-lint 0 fail / 604 files scanned.
Scope was operator-gated: "Fix the arity gap first, then start the
renderer", with the arity-alone vs arity-plus-R11 fork put separately and
answered "Arity gap + R11's three ruled changes (Recommended)". OPS under
GA-R3, no D-number.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-node-v6-carve.py: mirror each node's v4 octet into v6, dry by default
...
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc1 FIP pool RULED 10.12.65.0-10.12.67.254 (GA-R5); lib-net dc1 arm + assertions re-pointed
...
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-plane-ipam: v6 bands are VERIFIED against MAAS's default reservation, not created
...
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-plane-ipam: carve-v6 and reserve actions, dry by default, harness 25/25
...
Both are DRY BY DEFAULT, IDEMPOTENT, and READ BACK every write -- this repo has
been bitten by a script whose `apply` always silently dry-ran, so a create that
reports success is re-read before being called done.
carve-v6 places each v6 plane on the SAME MAAS vlan as its v4 twin, so the plane
is genuinely dual-stack on one L2 rather than a parallel fabric, and REFUSES to
create anything if that vlan cannot be resolved. The provider GUA VIP /64 is
skipped by design.
reserve does the D-134 bands plus the FIP pool. Two deliberate refusals:
- dc1's FIP pool is UNSET in lib-net BY DESIGN ("UNSET so any use fails loud").
Mirroring dc0's shape would be an inferred value, so it refuses and reports
that R4 cannot be fully executed for dc1 until the pool is ruled. Real finding.
- a same-bounds range of the wrong type is REFUSED, not overwritten.
v6 bands are skipped until carve-v6 has run -- R4's forced sequencing, surfaced
rather than silently passing.
TWO BUGS CAUGHT PRE-SHIP, neither by review:
- printf '%x' 50 yields 32, so the v6 bands would have been created at ::32-::63
-- a plausible-looking band that is NOT the one ruled. The ruling mirrors the
DIGITS (v4 .50 -> ::50). T19 locks it.
- `shift 2` with a single argument fails, leaving "$@" holding the action, which
the option loop rejected as an unknown option instead of printing usage. Found
by the harness.
The header claiming the mutating actions were absent is also corrected -- it went
stale the moment they landed.
Gauntlet ALL GREEN (83), repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-plane-ipam: add carve-v6 and reserve actions, dry by default
...
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-plane-ipam.sh: read-only gate for D-134 bands + R2 v6 carve (WIP, check only)
...
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R15(2): the gauntlet pins a harness MANIFEST, completing the R15 gate work
...
tests/HARNESS-MANIFEST (82 names) plus a drift gate in run-tests-all.sh, with
--record-manifest for deliberate re-recording. The zero-floor already existed;
what was missing was any pin on WHICH harnesses ran, so a renamed or deleted
harness was neither run nor failed while the gauntlet still printed ALL GREEN.
The figure 81 existed only as prose.
Proven able to fail on both cases, including the one a bare count cannot catch:
renaming a harness is reported, and renaming one WHILE adding a decoy -- so the
count holds at exactly 82 -- is still caught, naming both the missing and the
unpinned entry. That case is why the ruling chose a manifest over a count.
Sequenced LAST of R15's three deliberately: the ruling warns a manifest must be
seeded from a verified tree, not from whatever is on disk. Seeded here from a
tree with the gauntlet ALL GREEN and repo-lint 0 fail in the same session.
The gate runs only on a full gauntlet -- a filtered run legitimately executes a
subset. A missing manifest is itself a FAIL, since ALL GREEN would otherwise be
unfalsifiable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R15(3): preflight no longer fails open on an unexpected sub-gate exit
...
Reproduced first, in isolation with the exact note() from preflight.sh:27 --
sub-gate exits 127 (command not found), 126 (not executable), 130 (interrupted)
and 3 (undefined) ALL left "PREFLIGHT: PASS -- clear to add-model / deploy".
This is the gate that authorises the deploy.
rc=2 handled PER GATE rather than blanket-remapped, because R15's stated
rationale is only partly right on measurement: repo-lint and pre-flight-checks
both DOCUMENT rc2 as a legitimate warning, while provider-bundle-check and
creds-matrix use it only for could-not-evaluate. A blanket 2->FAIL would have
turned the standing L1 legacy-ASCII warn into a deploy blocker. channel_assert
uses 2 for BOTH meanings, so it is ambiguous in the checker and kept WARN; its
missing-PyYAML root cause is caught by P2 anyway, which shares it and is strict.
Flagged rather than guessed -- fixing that belongs in channel_assert.py.
Harness 16/16 (was 10): T11-T13 lock the fail-open cases, T14/T15 lock
could-not-evaluate -> FAIL, and T16 locks the non-over-correction (P1 rc=2 stays
WARN). Gauntlet ALL GREEN (82). Real-repo preflight still exits 1 for the same
known reasons with no new failure classes fired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
R15(1): repo-lint can no longer report PASS over zero files
...
Reproduced before fixing, both cases printing "PASS: repo lint (0 fail, 0 warn)"
with exit 0: `repo_lint.py --recordd` and `repo_lint.py /nonexistent-path`. Only
the two known flags were stripped, so any other --flag became argv[0] i.e. the
ROOT; it resolved nowhere, rglob yielded nothing, and zero findings read as
clean. This is the gate whose "0-fail" every GA-R6 stage close in this project's
history cites, and it could not tell a clean repo from an unexamined one.
Fix: reject unrecognised options (the actual root cause) and >1 positional root;
require the root to be a directory carrying repo markers, refusing rather than
defaulting to success; add a files-scanned floor; and PRINT the files-scanned
count -- Phase 0 found repo-lint emitted no such figure at all, so a reader could
not distinguish a thorough pass from a vacuous one.
The floor SCALES, and a flat one was wrong: the first implementation used a flat
100 and failed 20 of the 47 existing harness cases, because tests/repo-lint
legitimately builds ~7-file fixture repos. The strong floor now applies only to a
full checkout of THIS repo, detected by two files no fixture creates; any other
tree must still scan at least one file. Caught by running the harness.
Harness 54/54 (was 47) -- T48-T54 lock each reproduced defect, including that the
floor does NOT punish a legitimate minimal tree. Gauntlet ALL GREEN (82).
repo-lint on the real repo: 0 fail, 1 standing warn, 596 files scanned.
No consumer breaks -- preflight P1 reads the exit code, not the summary text;
harness regexes match the unchanged "PASS: repo lint" prefix. Both verified.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-27 |
close-out durability sweep: 4 transcript-only items landed; dc1-is-not-a-mirror warning
...
Operator-requested sweep for content that existed only in the session transcript, before
clearing. Precedent: docs/audit/queued-findings-20260726.txt. Capture:
docs/audit/queued-findings-20260727.txt.
THE CONSEQUENTIAL ONE. scripts/dc-mirror.sh's dc1 site row carried NO warning that dc1 is no
longer a mirror site, so `dc-mirror.sh install dc1` would have SILENTLY REBUILT everything the
same-day teardown removed -- units, nginx vhost, sync helper, and an ENABLED daily debmirror
timer that would start a fresh ~950G pull on a rack whose ruled artifact path is the proxy. That
is precisely the command someone would reach for after treating a failing `check dc1` as a
regression. The row now states dc1 is proxy-only per the D-135 amendment, that `check dc1` FAILS
BY DESIGN, and that `install dc1` is a deliberate strategy change and never a repair. The row is
RETAINED so its measured values survive a legitimate rebuild.
QUEUED, not built (hard rule 1): a RUNTIME guard in do_install for a site whose ruled path is
not the mirror. A comment is strictly weaker than a guard, and prose-only prevention has already
failed twice on this project (DoD item 8 missed dc1; D-137's wiring sat as prose and missed BOTH
DC standups). Recording that the interim is an interim.
A WRONG CLAIM CORRECTED. docs/audit/stage4-mirror-gate-20260727.txt said `systemctl reset-failed`
would re-arm the paused dc1 debmirror. It cannot: reset-failed on the SERVICE does not start an
INACTIVE TIMER. The real vector was a REBOOT -- `enabled` starts the timer at boot and
`Persistent=yes` fires it IMMEDIATELY because its window had passed. Correction APPENDED rather
than edited in place: the file is dated evidence, so a wrong analytical claim is superseded
openly, not quietly rewritten.
THE GENERALISABLE LESSON, previously unrecorded, is now platform-traps section 5:
5a "stopped is not dormant across a reboot" -- enabled + inactive + EMPTY NextElapse looks
harmless and is not; plus reset-failed does not re-arm, and Result=signal/status=15 on a
oneshot means it was SIGTERM'd and never wrote a terminal state (so a stale RUNNING is a
corpse, not progress).
5b a `oneshot` with RemainAfterExit=yes does NOT undo its work on stop, so stopping it proves
NOTHING about dependents -- which is exactly why the dc1 teardown test deleted the live
address and route instead of stopping the mirror net unit.
Three rows added to the verbatim-error index. Note dc0-mirror-sync.timer is still enabled with
Persistent=yes, so 5a describes dc0 today -- benign while its syncs succeed, same shape.
HARD RULE 2. dc-cache-proxy.sh stated that the two net units coexist safely on one host as
FACT. It is REASONED (idempotent replace verbs, identical measured values), never MEASURED --
the proxy has only run on dc1, where the mirror unit was then removed, so nobody has run both.
Marked accordingly, with an instruction to verify rather than trust the sentence.
VERIFICATION: dangling-reference sweep over docs/, runbooks/, scripts/ and the skill -- every
path this session introduced or cited RESOLVES; all pre-existing dangles are legitimate
(deleted-as-history, not-yet-built, or the deliberately-absent octavia PKI overlay preflight P4
fails on). ledger-scan reconciled against the machine-derived block. Gauntlet ALL GREEN (81),
repo-lint 0-fail, dc-mirror 24/24, dc-cache-proxy 20/20.
Revert: text + comments only; the capture correction is an append.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
set-interface-v4: reload the pf filter on --commit (RULED); appendix-A rewritten
...
RULING (GA-R5) 2026-07-27. Question as presented: (a) add `configctl filter reload` to the
script's --commit path; (b) leave the script and rely on DC standup DoD item 8; (c) put it behind
an opt-in flag. Operator answer, exact utterance: "a". OPS under GA-R3 -- a script fix completing
an already-documented operation, governed by D-113; no new D-number.
DEFECT (appendix-A, measured on the fresh dc1 edge 2026-07-23): --commit ran `configctl interface
reconfigure`, which applies the address to the kernel but does NOT regenerate the pf ruleset, so
OPNsense never creates its automatic outbound-NAT rules for the (re-)addressed LAN. Forwarded LAN
packets left the WAN un-NAT'd and replies never returned -- the edge passed NO LAN traffic while
looking perfectly configured. Self-masking: any later plugin install or config apply reloads the
filter as a side effect, so dc0's 07-20 qga work hid it there until dc1 exposed it.
Option (b) rejected on evidence, not taste: DoD item 8 already existed and did NOT fire at dc1,
and prose-only controls have a measured failure record here (D-137's own enforcement sat as prose
at phase-3:498 and missed BOTH DC standups).
PLACEMENT IS THE SUBSTANCE OF THE FIX, and my first attempt got it wrong -- caught before
shipping. Putting the reload on the next line of the `interface reconfigure` heredoc would have
been a fix that fixes nothing exactly when it matters: re-addressing the interface you arrived on
drops the session DURING the reconfigure, so a following command in that stream never executes --
and that drop case IS this incident's own scenario (dc1's LAN moved 192.168.1.1 -> 10.12.64.1
over the LAN itself). As shipped the reload runs over a FRESH connection to PROBE_HOST.
Also added a `pfctl -s nat` read-back with the `proto carp` boilerplate filtered out, since
boilerplate-only is precisely appendix-A's broken-state signature. REPORTED, not gated: an edge's
first addressing may legitimately have no gateway and so no NAT to generate, and hard-failing
would block a correct run. The hard gate stays address-on-the-kernel.
Harness 59/59 (was 53), cases 15-15f. Case 15f caught a real inconsistency -- appendix-A still
said "not yet ruled". Case 15e caught a bug in my own first draft, which scanned to EOF and so
failed the script for having the very hard gate the case exists to preserve.
appendix-A rewritten: the FIX stays for an edge already in this state, plus a PREVENTED-IN-THE-TOOL
block recording the placement detail and why prose prevention was rejected.
Gauntlet ALL GREEN (81), repo-lint 0-fail. No live edge touched -- both DC edges are already
addressed, so this affects only the NEXT edge addressing.
With this, every substantive Stage 4 in-stage item is closed or split to gate row G17; only the
GA-R6 close-out set remains.
Revert: git revert this commit's script + harness + appendix-A changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc1 full mirror REMOVED (D-135 amendment); dc-cache-proxy.sh now owns its net layer
...
Operator clarified intent, exact utterance: "DC0 is the test of a full mirror. DC1 is the test
of the mirror proxy. DC1 needed the partially created full mirror and associated
script/files/settings removed." Recorded as a D-135 AMENDMENT 2026-07-27 -- the per-DC split
is the DELIBERATE EXPERIMENT, not an interim state. That supersedes both the "PAUSED ... kept
dormant as the fallback" record and the "DC1's proxy is REPLACED by a full mirror afterward"
claim, which sat in THREE places (D-135, CURRENT-STATE:109, dc-cache-proxy.sh:6); all corrected.
dc0 FIRST (gated): sync re-run -> OK 2026-07-27T08:43:46Z ubuntu=0 uca=0 in 20s, confirming
the transient-500 diagnosis. The fixed check now reports PASS -- trustworthy precisely because
the same check FAILED on the same rack twenty minutes earlier. Stage 4 DoD bullet 5 is now MET
for both DCs (dc0 mirror PASS, dc1 proxy PASS), node-side half at G17.
WHY THE ORDER MATTERED. Asked to weigh rip-down-and-rebuild against surgical removal, the
rebuild path turned out to be BLOCKED, not merely slower: dc-cache-proxy.sh did not own its
utility net layer -- it REUSED dc-mirror.sh's <site>-mirror-net.service and refused to install
without it. So on dc1, whose ruled path is the PROXY, three files named dc1-mirror-* (net unit,
apply helper, resolved DNS drop-in) were load-bearing FOR THE PROXY. Consequences: the recorded
removal procedure would have silently killed the proxy, the bullet-5 PASS and the Stage-5 dc1
apt path; and the proxy could not be REBUILT afterward, since nothing but `dc-mirror.sh
install` created the net layer -- rebuilding a PROXY meant reinstalling the whole MIRROR
apparatus, enabled daily debmirror timer included. Operator direction: net layer first, then
rip it all down.
1. NET LAYER MOVED: <site>-cache-proxy-net.service + -net-apply + its own resolved drop-in,
reusing dc-mirror's proven mechanism (bridge from the libvirt net NAME at runtime, hard
rule 3). Proxy-named so a proxy-only DC has NO mirror-named dependency. Idempotent
replace verbs, so both net units coexist safely where a site runs both -- dc0 unaffected.
check asserts the ARTIFACTS, not just the live address/route: asserting only the live
effect would PASS while the layer was really the mirror's, which is the exact failure.
Harness 20/20 (was 15). T12 REPLACED not deleted -- it asserted the superseded design, so
it failed by design; now inverted, plus T12b-T12f. T12b is scoped to non-comment lines on
purpose: the header's account of the removed trap cites dc-mirror.sh install, and deleting
that history to satisfy a grep would lose why the trap existed.
2. INDEPENDENCE PROVEN BEFORE DELETING: a oneshot with RemainAfterExit=yes does not undo its
work on stop, so stopping the mirror unit would have proven nothing. Removed the live state
outright (addr count -> 0, default route count -> 0), applied the proxy-owned unit ALONE,
both restored; check dc1 PASS with dc1-mirror-net.service disabled.
3. WHOLESALE REMOVAL: all mirror units, timer, sync helper, nginx vhost, resolved drop-in, the
debmirror package, and 330G of partial mirror (33,037 files, rm -rf 47.6s; disk 339G ->
9.0G). Zero mirror-named residue, every proxy artifact present, proxy PASS throughout and
never down. Capture docs/audit/dc1-mirror-teardown-20260727.txt.
nginx left installed deliberately: a mirror prereq, but with the vhost gone it is a generic
server on its stock default vhost; removing the metapackage while nginx-core runs is a
half-measure and a full purge is scope creep on a rack host.
Gauntlet ALL GREEN (81), repo-lint 0-fail, dc-cache-proxy 20/20, dc-mirror 24/24.
Revert: `dc-mirror.sh install dc1` (needs debmirror reinstalled) then `sync dc1` to re-pull
~950G; git revert dc-cache-proxy.sh + harness to restore the mirror-owned layer. The 330G
partial is NOT recoverable -- deliberately, per the amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 4 close-out: fix dc-mirror.sh check false-greening the mirror gate
...
Operator directed closing Stage 4 before resuming the DC1 Stage-5 chain. Verifying the DoD's
fifth bullet ("per-DC mirror reachable") found that the check the gate would close on COULD
NOT FAIL.
scripts/dc-mirror.sh:271-273 tested `[ -f last-sync.status ]` and printed the file's contents
behind an unconditional "OK" -- the status word was displayed but never asserted on, so the
check exited PASS regardless. MEASURED on both racks:
- dc0 read "FAIL 2026-07-27T00:54:54Z ubuntu=255 uca=0" -> printed OK, PASS
- dc1 read "RUNNING 2026-07-23T21:49:35Z", four days stale,
left by the debmirror the D-135 amendment killed by signal -> printed OK, PASS
GA-R6 requires a stage to close on a NAMED executable check whose captured output the closing
commit cites. A check that cannot distinguish "synced" from "failed" or "died mid-run" is not
that check.
FIXED: the status word is case-analysed. OK* passes. FAIL* misses. An ABSENT file misses (it
was a passing `note`, but nothing has attested the content). An unrecognised word REFUSES
instead of defaulting to success. RUNNING* is deliberately neither pass nor plain fail --
in-flight and died-mid-run are indistinguishable from the file alone, so it reports an explicit
UNKNOWN and cross-checks the unit, naming a RUNNING marker with no active unit as a corpse.
Harness 19 -> 24; T20-T24 lock each status word and T24 locks out the return of the exact
defective shape.
PROOF, re-run against both live racks -- the gate is now honestly NOT MET for the first time:
dc-mirror check (dc0): FAIL / dc-mirror check (dc1): FAIL. Capture
docs/audit/stage4-mirror-gate-20260727.txt.
SUBSTANCE (measured, not inferred):
- dc0's mirror CONTENT is complete: 949G ubuntu + 342M cloud-archive, all three dists plus
pool. Only the overnight INCREMENTAL failed -- "Download of dists/jammy/Release failed: 500
read timeout" aborted the ubuntu leg with 255; the UCA leg succeeded. Transient upstream
fetch failure, not mirror damage. Timer re-arms 2026-07-28 00:14:48.
- dc1's PROXY -- its RULED artifact path per the D-135 amendment -- checks PASS genuinely
(apt-cacher-ng on .4:3142 serving archive + UCA Release 200). Its dormant fallback debmirror
is dormant only ACCIDENTALLY: the timer is `enabled` with an EMPTY next-elapse because the
unit sits in failed/Result=signal, so the ruling is enforced by nothing and a reset-failed
would re-arm a 330G -> ~949G pull.
ALSO RECORDED, not yet actioned: DoD bullet 6 ("NTP from the DC's own OPNsense edge") is STALE
and unsatisfiable -- superseded by D-129(iv) 2026-07-21 ("Keep MAAS hierarchy", no NTP role on
the edge) -- and survives in four surfaces needing a DOCFIX before it can be checked at all.
The node-side half of bullet 5 needs an operator decision (gated rescue-boot check vs its own
gate row per GA-R6 E3); nodes are powered off by the READY-handoff ruling, so no node-side
probe can run as things stand. There is no conditional close.
No live state was altered: both racks were read only. repo-lint 0-fail, dc-mirror 24/24.
Revert: git revert the scripts/dc-mirror.sh + tests/dc-mirror/run-tests.sh changes; the check
returns to passing unconditionally on last-sync.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
creds consolidation batch: SEC-021(b) dc0 power key, SEC-025 NetBox GUI, V2 ruled deferral
...
Operator asked whether vcloud holds a consolidated set of login creds for every account that
exists, and what is missing / needs creating / was never moved in. Audited, then cleared the
consolidation batch on operator direction ("clear the whole consolidation batch first").
The audit's own headline: creds-audit read CLEAN on all three sites while the matrix returned
13 findings -- and the two most consequential gaps were in NEITHER surface. They were found by
enumerating the accounts that ACTUALLY EXIST. Absence of a ROW is invisible to the register,
which generalises D-137's founding argument one level up.
VERIFIED POSITIVE (both previously only asserted):
- MAAS account set COMPLETE: all 6 live accounts accounted for; admin + operator passwords on
vcloud, juju-vr1-dc0/dc1 random+unstored BY RULING with API keys present, MAAS +
maas-init-node MAAS-internal.
- tier-3 V1 now MEASURES maas-admin-password byte-identical to the headend source-of-record,
so SEC-020's stale-trap risk is clear as of this date.
DONE:
- dc0 SEC-012 power key consolidated to vcloud + .pub DERIVED (SEC-021(b) as written).
MEASURED FIRST, because the naming looked like a defect and was not: the headend
maas-virsh_ed25519 and the snap's id_ed25519 are the SAME key; it IS dedicated (distinct
from the dc0 service key, which is what SEC-012 requires); and dc0 using the snap's default
identity is SEC-016's RULED design. So no re-mint, no live power path touched.
- dc1 svc .pub backfilled to the headend store (public-key material, sha256-verified).
- NetBox web-GUI admin password consolidated -- a HUMAN login that had never left the VM it
was minted on, the third instance of the SEC-009/SEC-020 miss class. SEC-025 OPENED for the
at-rest exposure the copy CREATES, not the gap it closed. Open rows 20 -> 21.
- V2 taught the ruled-deferral state: SEC-006's standing "revoke at completion of this
deployment" ruling is now ACKNOWLEDGED (still naming the credential live and exposed)
instead of failing every run. Reissuing the token would have CONTRAVENED that ruling, so the
register was what needed to change. Match demands the cased state word AND explicit operator
ruling attribution, and cannot cross a ledger table cell -- T58/T59 lock the evasion out.
- Added --ledger (default unchanged): V2 read the register via --repo, which the harness must
point at the real repo for S4, so V2 had shipped with ZERO harness cases. T56-T59 added.
Findings 13 -> 7. Matrix 82 rows, harness 60/60 (was 56), creds-audit CLEAN x3, gauntlet ALL
GREEN (81), repo-lint 0-fail. The register is NOT green and is not expected to be.
RESIDUAL, all recorded: dc0-edge-api x2 (the opnsense-api.txt re-mint is a live edge mutation,
deliberately EXCLUDED from this batch), S5 x3 (RULED by SEC-016 -- needs a ruled-exception
mechanism, operator decision), S6 conflation x1 (the SEC-020 defect), E4 x2 (Stage-5/6 rows).
LOGGED NOT ACTIONED (hard rule 1): no registered root/console credential at EITHER DC edge
(measured absence of row/manifest/SEC row; what those passwords ARE is UNKNOWN and
deliberately unprobed per hard rule 2 -- vector is the LAN-reachable GUI and serial console,
not SSH, which is key-only and proven); plus the two structural blind spots that let it hide
-- S5 compares only cardinality=per-DC while all six per-site rows are office1-only, and
vm-secret-locations declares no rack/edge/cloud/unit/client location though the checker
accepts them.
Revert: per-item reverts in docs/changelog-20260727-creds-consolidation.md. The headend, snap
and NetBox VM source copies were never modified, so every file move is reversible by deletion.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-26 |
D-137 tier 3 behavioural half: V2 declared-state + creds-probe.sh
...
Boundary and cadence operator-ruled: "Full -- including OpenStack service
credentials", "Stage close + DC standup close". I had recommended
passive-only and flagged the costs; the operator ruled otherwise.
V2 (passive, zero authentication) fires TODAY on a real defect: SEC-006 has
recorded the NetBox upstream token as BURNED since 2026-07-12 while the
matrix carried it as a healthy consolidated row. Nothing reconciled the two
surfaces. Keyword matching on ledger prose is fragile, so it is conservative
and reports the matched text as its basis.
creds-probe.sh is SEPARATE by design: a probe must USE a credential, and
creds-matrix.py is metadata-only by an enforced invariant (T22/T51). Folding
probes in would have relaxed that guard, and a guard relaxed once stays
relaxed.
Safety: never retry a failed auth (retry loops lock accounts; the MAAS admin
identity backs 19 call sites), never argv, never echo, stage-aware skips, and
per-tenant credentials REFUSED BY CONSTRUCTION -- a declaration cannot opt
them back in (D-069, ruling 3). Not a deploy gate.
FALSE GREEN in my own first run: ssh inside a while-read loop ate the loop's
stdin, so 1 of 13 probes ran and the run exited 0. A partial run reporting
clean is the exact failure class this tool exists to catch. Fixed with ssh -n
and locked by a total-accounting check that makes a partial run VOID.
13 probes declared: 3 active, 4 stage-pending, 6 declared-only with reasons
recorded. OpenStack credentials declared now, skipping until Stage 5.
tests/creds-probe 9/9; gauntlet 80 -> 81; phase-3 DoD wired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: local half blocking in P5; --pending-stage -> coupled projection
...
Two operator decisions, both taking the recommendation.
GATE SHAPE: local-only blocking, remote operator-run. preflight P5 now runs
creds-matrix.py --tier2 -- tier 1 STATIC plus tier 2 LOCAL, hermetic (no ssh,
no sudo). It already catches SEC-021's three dc0 on-disk absences without
leaving the jumphost. The remote half stays an operator-run check at
DC-standup close and stage close.
Wiring --remote --privileged into the deploy gate would let a host outage, a
NOPASSWD change, or a MAAS snap refresh turn the gate red for reasons that
are not about credentials, and a gate that fails environmentally is one
people learn to bypass. The repo already made this call once
(run-tests-all.sh:7-8, "different cadence"), and preflight already shows the
failure mode: P4 reads "MAAS unreachable" because it runs on vcloud while
MAAS lives on the headend.
STAGE INPUT: --pending-stage replaced by creds-manifests/stages-reached, a
machine-readable projection of CURRENT-STATE, coupled by new repo-lint L12
(the L10 pattern). A hand-typed gate parameter can be widened silently, and
GA-R1 puts status in CURRENT-STATE alone. A drifted projection would make the
gate defer REAL absences -- a false green.
Exhaustive both ways: an unclassified mint-stage is a hard FAIL, a
classified-but-unused one is a stale-declaration FAIL. That caught `stage4`
immediately -- Stage 4 is open and executing but mints no credential.
Self-inflicted break caught by the existing harness: I first renamed L10's
message to "L10/L12", breaking five cases that grep the established L10 id.
Reverted -- new rules get their own message id, they do not rebrand old ones.
Harnesses: creds-matrix 52 -> 56, repo-lint 45 -> 47, preflight 10.
Gauntlet ALL GREEN (80), repo-lint 0-fail, baseline unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 3 (VALIDITY) BUILT: cross-copy sha256 provenance
...
Closes the gap the plan names: creds-audit PARSES declared provenance and
never verifies it, so SEC-020's stale-trap warning has been unchecked prose
and its sha256 equality was a ONE-TIME manual act on 2026-07-25.
sha256sum runs as an EXTERNAL command on both sides so only a 64-hex digest
enters the checker -- hashing bytes in Python would meet the letter of "never
transfer content" and not the spirit.
RESULT: 4 groups compared, all byte-identical, INCLUDING the SEC-020 pair
(maas-region-admin, region source-of-record vs jumphost copy). Now
machine-verified instead of remembered. The [ok] line names the groups.
TWO FALSE POSITIVES IN MY OWN DRAFT, caught by verifying before reporting.
Keying digests by (role, site) let an identity's DISTINCT artifacts at one
scope overwrite each other, so it compared unrelated files and reported drift
on maas-region-admin and dc1-svc-key. Manual sha256sum showed both IDENTICAL.
Reporting that would have raised a false credential-rotation alarm.
Root cause was a schema gap: nothing expressed which rows are byte-copies.
custody gains `verbatim-copy` [OPS]. V1 now compares only rows sharing
(id, filename) across scopes, or a source-of-record row against declared
verbatim-copy rows -- SEC-020's exact shape. Everything else is skipped with
a reason: a pair we cannot justify comparing is not a pair we may report
drift on. verbatim-copy is manifest-bearing; it asserts byte-identity, it
does not move the file.
Harness 44 -> 52: T47/T48 regression-lock both false positives; T49 keeps
SEC-020's case from falling into T48's skip; T51 asserts the digest path uses
sha256sum and that no content-transfer verb exists anywhere in the checker.
NOT built: tier 3's behavioural probes. Liveness, expiry, revocation state
and grant/scope drift remain unverified.
Gauntlet ALL GREEN (80), repo-lint 0-fail, finding-class baseline unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 ruling 1 EXECUTED: tier 1 wired as the blocking preflight P5
...
Operator answer, exact utterance: "wire tier 1 as the blocking preflight Pn"
(2026-07-26), resolving the open sequencing question. Recorded on D-137.
preflight.sh gains P5: credential matrix (D-137 tier 1), blocking, with
worst-exit aggregation, placed AHEAD of the stage-2 reminders block (now P6)
so its verdict participates in the deploy decision. Tier 1 only -- it is
offline and its findings ARE the three ruled failure classes. Tier 2 needs
--remote/--privileged and a caller-supplied --pending-stage, which do not
belong in an unattended gate; that stays an open decision.
FAIL-OPEN in my own wiring, caught by the new test: a missing creds-matrix.py
makes python3 exit 2, which `note` maps to WARN -- so deleting the gate would
have downgraded it to a warning rather than stopping the deploy. P4 already
guarded this class; P5 did not until T9 failed. Now guarded explicitly.
tests/preflight 7 -> 10: T8 a red register FAILS the deploy (blocking, not
advisory); T9 fails closed when the checker is absent; T10 the gate runs
before the reminders block.
CONSEQUENCE, not overclaimed: preflight exits 1 with P5 among the reasons,
but preflight was ALREADY exiting 1 before this change (P4: octavia-pki.yaml
absent, MAAS unreachable from the jumphost). P5 adds a fifth reason to an
already-red gate; it flipped nothing and closes no open deploy path.
Verified by stashing the change and re-running.
Gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 phase 3: re-measure the register; SEC-024 opened
...
~21 inferred filenames corrected against their own mint-refs. The session had
owned six; those were only the rows the first sweep physically touched.
Octavia's 8 real basenames plus its THREE SUBDIRECTORIES (the bare
~/octavia-pki/* pattern matched ZERO of the 8 artifacts, two of them CA
private keys); vault's init.txt; the tenant rows' <client>- instance prefix,
with a placeholder matcher so they can match a real basename at all;
kube-config -> config; .maas.cli -> .maascli.db (measured: ~/.maas.cli does
not exist).
sec-ref mis-attribution corrected: both juju-maas-user rows cited SEC-020;
the on-subject rows are SEC-018/-019. This is the SEC-015 class recurring --
one instance was fixed by hand earlier without checking whether it was
systemic. It was.
mint-ref corrected: admin.pass cited a CONSUMER (:452), not the mint (:450);
the RGW key cites a line that takes it as INPUT, so it is operator-terminal.
Rows added: the libvirt SSH power password (standing rotation obligation from
a 2026-06-26 plaintext exposure, in neither the matrix nor the exclusions),
vault-ca-root, and the tfstate siblings. ~/as-executed/ deliberately NOT
declared, reason recorded: it holds session logs, so declaring it made every
log a false UNDECLARED.
Four further defects in my own phase-2 changes, found by RUNNING the sweep:
shlex.quote broke ~ expansion; a non-existent parent read as unreadable
rather than absent; site-key `-` needed shared-location semantics; the
tfstate siblings needed rows.
SEC-024 OPENED -- new exposure found by the fixed checker on its first clean
run: opentofu/terraform.tfstate.backup is mode 0664 and carries the MAAS API
key in plaintext (DOCFIX-175); the live state file is 0600. Invisible to
every prior control because the world-readable check was custody-gated and
the siblings were undeclared. LOGGED, NOT ACTIONED (hard rule 1).
Findings 35 -> 13, all true. Matrix 81 rows. Harness 44/44, creds-audit
15/15, gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 phase 2: fix the eight confirmed false-greens, regression-lock each
...
Tier 2 is now scoped by (host-role, SITE-KEY). Keying on role alone merged
all three jumphost creds folders into one basename namespace, so a credential
present at ANY site satisfied the row for EVERY site -- how SEC-021's dc0
opnsense-api.txt absence passed a green sweep while tier 1 flagged it. Four
committee lenses converged on this. vm-secret-locations gains a site column.
Also fixed, each reproduced by the committee and each now regression-locked:
- probe_remote called a successfully-read location UNREACHABLE when its glob
matched nothing, gating the role and turning every absence FAIL there into
an [ok]; a __DONE__ liveness marker now separates "ran, found nothing" from
"could not reach".
- literal-file locations skipped absent/unreadable detection entirely (the
T33 false green surviving in the other path form).
- an EMPTY locations list bypassed ruling 3's refusal.
- a zero-row matrix reported clean over the empty set.
- a mint-ref pointing at a directory crashed with exit 1 -- indistinguishable
from findings -- and skipped S5/S6/S7, silently dropping the conflation and
asymmetry findings.
- mode enforcement was custody-gated, leaving 43 of 77 rows unchecked
including the SEC-020 and SEC-022 stores; a world-readable check now
applies at any custody.
- S5's dc1->dc0 direction was untested; deleting it left the harness green.
- rows with no declared location for their scope are disclosed, not skipped.
- declared paths/targets are charset-validated and shell-quoted: an
unvalidated `;` reached a remote shell, under sudo when --privileged.
- creds-audit sprawl globs now cover dotfiles; the prior widening could not
see .admin.pass, the dotfile sibling of its own motivating secret.
T24 replaced with a checked-in finding-CLASS baseline. The old form grepped
for the literal "IDENTITY CONFLATION", so REMEDIATING that defect would have
turned the gauntlet red for whoever did the right thing -- making "edit the
test" the cheapest response exactly when the guard mattered.
Harness 35 -> 43, creds-audit 13 -> 15, gauntlet ALL GREEN (80), lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: wire --privileged escalation (run BLOCKED by the auto-mode classifier)
...
--privileged retries a remote probe with `sudo -n` ONLY where the
unprivileged probe returned `unreadable`, keeping the privileged surface as
small as ruling 3 keeps the search surface. Metadata only; the T22
metadata-only guard still passes; verified inert (zero escalations) when
nothing is unreadable.
The operator approved the privileged read, but the auto-mode classifier
denied the remote-sudo shape -- the same wall the 2026-07-23 close recorded,
whose noted fix is manual permission mode. NOT worked around: reading a
root-owned directory is exactly the privilege being denied, so any
alternative would fail identically or misrepresent the result.
SEC-022 shadow-store contents and the SEC-020 region secrets stay
UNCONFIRMED; the sweep reports them as E0 UNREADABLE FAILs, which is the
correct outcome for an audit that could not look.
Also de-staled the docstring tier table (tier 2 is built).
Harness 35/35, gauntlet ALL GREEN (80), repo-lint 0-fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-137 tier 2: live read-only sweep + two probe false-greens it exposed
...
SEC-021's ON-DISK half REPRODUCED: vr1-dc0-maas-power_ed25519{,.pub} are
genuinely absent from the dc0 jumphost creds folder, not merely undeclared.
Capture: docs/audit/d137-tier2-sweep-20260726.txt (read-only, no sudo,
stat-over-ssh metadata only; jumphost + voffice1 + office1-netbox).
Running it against real hosts exposed two false-greens in the probe:
1. "absent" vs "could not look". /root/maas-secrets and /root/netbox-secrets
reported "does not exist" -- they exist; SEC-020 is precisely about
admin.pass living in the first. An unprivileged [ -d ] on a root-owned
parent is indistinguishable from a missing directory. The probe now tests
parent traversability and reports UNREADABLE as a FAIL, not a skip.
2. Role-level aggregation manufactured false absences: skipping one location
let a role's other locations cover it, so credentials in the unprobed
location reported EXPECTED-BUT-ABSENT. Absence is now asserted only over
fully-probed roles (14 rows report NOT JUDGED). Refinement: "absent" is a
conclusive observation and does not gate a role; only unreadable and
unreachable do.
Harness 33 -> 35 (T33 unreadable location, T34 absent location).
Gauntlet ALL GREEN (80), repo-lint 0-fail.
OUTSTANDING: the two root-owned dirs need a privileged read, so SEC-022's
shadow-store contents and the SEC-020 region secrets stay UNCONFIRMED. That
is a remote-sudo shape and was not run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
