| 2026-07-30 |
Region images synced (jammy+noble); pre-migration carve captured
...
Both releases now Synced on the dc0 region, 6 rows each, matching Office1.
jammy is NOT seeded by `maas init` (noble only) and the OpenStack nodes
require it. The first import raced the selection's creation and returned
noble-only while reporting is-importing=false -- assert on Synced ROWS, never
on the import call returning.
Measured and recorded so the two are not conflated: MAAS boot images do NOT
come from the DC mirror. The region pulls images.maas.io; the mirror 404s on
every simplestreams index path because dc-mirror.sh mirrors debmirror APT
packages only. Three artifact classes, one local. Consequence: the
D-135/D-107 egress narrowing would BREAK image sync and juju bootstrap, not
just slow them, and nothing sequences it after them.
Pre-migration carve captured to docs/audit (179 lines): fabrics, VLANs,
spaces, subnets, D-134 bands, tags, per-node identity/power and the full
per-interface carve. MAAS cannot move machines between regions, so the
migration must recreate all of it.
The destructive half has NOT started -- the Office1 region still owns all 9
dc0 nodes, correctly carved and Ready. Nothing is half-applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 MAAS region LIVE; three creds minted, consolidated and REGISTERED (SEC-027)
...
Region: noble + MAAS 3.7.2 + PostgreSQL 16.14 snaps, byte-identical versions
to the Office1 region (checked, not assumed). regiond/rackd/apiserver ACTIVE,
dhcpd INACTIVE so there is no DHCP conflict with the Office1 rack still
serving that VLAN. API answers 200 on 10.12.8.6:5240 -- the address jujud
could not reach at Office1, which is the whole point of D-132's amendment.
Carve verified BOTH legs (the lesson of today): metal-admin 10.12.8.6 + v6
::6, provider-public 10.12.4.6 with gw 10.12.4.1.
Credentials, per operator direction: DB password, region admin password and
admin API key minted ON the VM, never printed, consolidated to
~/vr1-dc0-creds/ at 0600 with sha256 digests compared to prove the copies are
identical, API key format-verified parts=3. SEC-027 opened (open SEC 22->23),
recording the residual argv exposure honestly -- `maas init` has no stdin or
file option for --database-uri.
The creds MATRIX now expects them at BOTH DCs: 5 rows per DC, a new `region`
host-role token in the enum, 4 vm-secret-locations rows (definition-of-done
for a new mint site), dc0's manifest declares its three. dc1's three now FAIL
S2 BY DESIGN -- the forward register making dc1's absence detectable, which is
what D-137 exists for. Harness 65/65 PASS, repo-lint 0 fail.
Also: durable ssh aliases for DC nodes. Deployed nodes carry the
MAAS-registered vr1-office1-svc key, not the rack service key, so each hop
needs its own identity -- a bare `ssh -i <key> -J voffice1,<rack>` fails
because -i forces one key on every hop. No credential was copied anywhere.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc0 MAAS region VM APPLIED, MACs pinned, converged; commissioning trap recorded
...
Capacity gate re-measured PASS (RAM 870/1024 = 85%, FIT, 154 GiB headroom).
Plan asserted on content -- exactly two resources, both the new VM. Applied
2 added / 0 changed / 0 destroyed; MACs measured and pinned; converged ZERO
DIFF with live virsh domiflist byte-identical to the pin; virsh power set.
Enlisted as hot-kid / tw7ptw.
Corrected against measurement: the "1 add" figure in both new comments and in
D-104's amendment prose is wrong at resource level -- node-vm creates a
domain AND a volume, so it is 2 to add. The half that mattered, 0 change /
0 destroy, held.
Ordering trap measured and recorded as a standing rule: the first apply sets
running=true, so the VM PXE-enlists and auto-commissions within ~90s, and
pinning MACs is an in-place libvirt_domain update that BOUNCES the guest --
stalling that commission at "Loading ephemeral". After pinning MACs on a
freshly-applied VM, assume commissioning was interrupted and re-commission
deliberately. Recovery is abort -> New -> commission.
dc1's region VM is authored but NOT applied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pin vr1-dc0-maas-01 MACs from measurement, pre-enlistment
...
Measured with virsh domiflist on vvr1-dc0 in the same session as the first
apply, before the VM enlists -- the boundary modules/node-vm warns about and
the one the 2026-07-20 MAC regeneration made expensive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Author the per-DC MAAS region VM into both substrate roots (D-132 addendum)
...
vr1-dc0-maas-01 and vr1-dc1-maas-01: 4 vCPU / 8 GiB / 150 GiB, macs = []
for first apply. Additive on a for_each-keyed caller, so each plans
1 add / 0 change / 0 destroy.
Sizing is the shape the capacity gate actually modelled
(dc-dc-whole-host-budget.py --containment-overhead-mem-gib 32
--containment-overhead-vcpu 12 -> RAM 870/1024 = 85 pct, FIT, 154 GiB
headroom). Disk is 150 not 100 because this VM stores the region's
PostgreSQL AND its boot-image set.
Both entries carry the carve requirement in the comment: metal-admin AND
provider-public at enlistment. Carving metal-admin alone is exactly the
under-carve that cost three bootstrap attempts on the juju controller VM
today -- no default route, no image sync.
MACs deliberately empty, to be pinned from measurement immediately after
first apply, per the same rule that the 2026-07-20 MAC regeneration
established.
opentofu-validate: root + 12 modules + both extra roots all PASS.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-132 placement RULED: per-DC MAAS region VM at utility .6; D-134 map extended
...
GA-R5. Operator utterance: "Dedicated region VM per DC at utility .6
(Recommended)".
The region DB must not share fate with the hypervisor running every node it
manages -- also a precondition for D-132 q3 cross-site backup custody, which
stays pinned. The D-134 standing octet map now reads .4 artifact service /
.5 juju controller / .6 MAAS region, binding at every future DC standup.
Build notes carried, not ruled: author BOTH metal-admin and provider-public
legs at creation (the identical under-carve on the juju controller VM cost
three bootstrap attempts today); re-measure capacity before applying.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-132 q1 RULED for VR1: a MAAS region controller per DC; Stage 5 blocked
...
GA-R5, before dependent work. Operator utterance: "Put a MAAS region
controller in each DC".
Third bootstrap got PAST the agent fetch (downloaded attempt 1 -- the
controller carve closed that failure class) and failed at the next layer:
jujud, running ON the node, cannot reach the MAAS region API at
10.10.0.20:5240. Measured: rack reaches it (200, rack-originated, permitted
by SEC-010); the node cannot (forwarded, blocked); and the rack does NOT
proxy it -- nginx listens on 5248 only. All six required flows enumerated;
exactly one crosses the boundary.
The finding underneath: SEC-010 as written is incompatible with the
deployment's own control-plane topology (D-104 controller in-DC + MAAS
region at Office1). D-138 was necessary but NOT sufficient -- it moved the
cloud-facing client into the DC, and jujud is itself a provider client whose
MAAS dependency I did not enumerate when scoping it.
Per-DC regions remove the requirement rather than excepting it, so SEC-010
is PRESERVED UNAMENDED. Single, not HA (utterance is singular); D-132 q2/q3
stay pinned to the next deployment.
Consequence stated plainly: this reopens the MAAS layer of Stages 3 and 4,
both closed and merged. A region owns its own DB; nothing migrates between
regions. Open sub-question blocking the build: where each DC's region lives.
Also landed: both controllers carved symmetric with gw-bearing
provider-public legs. Trap recorded -- MAAS `interface update vlan=`
returned a full JSON object while changing nothing on a Deployed machine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
G17 dc0 CAPTURED (both assertions PASS); bootstrap root cause = under-carved controller
...
G17's one-shot first-boot window opened during the second bootstrap attempt and
was taken rather than lost. Capture docs/audit/g17-dc0-firstboot-20260730.txt,
exit 0: mirror asserted on FOUR body fields of a real package path (not the
autoindex root), and the node time source is the MAAS rack 10.12.8.2 chaining
to the region, not the DC edge. Scope stated: the node is the dc0 CONTROLLER
VM, not a role node; G17's dc1 half stays OPEN.
Two defects in G17's own text, both measured: it names `chronyc sources` and
chrony is NOT installed on the MAAS jammy image (node runs systemd-timesyncd),
so the gate as written can only REFUSE; and a first pass false-FAILED on
`printf | grep -q` under pipefail -- the same trap recorded earlier the same
day. Capture uses here-strings.
Root cause of both bootstrap failures is measured and is NOT architectural:
the controller VMs are UNDER-CARVED. D-138's path works -- the second attempt
reached "Connected to 10.12.8.5" -- then failed fetching the juju agent
because the node has NO DEFAULT ROUTE. Role nodes carry provider-public with
gw 10.12.4.1; the controllers carry metal-admin only, because they were added
after both carves and Stage 4's 90 NIC re-homes covered the 18 role nodes.
Owned: this session's egress re-probe was run FROM THE RACK and reported the
window open. The rack holds a default route nodes lack, so it did not
discharge the caveat for a node-side fetch. Same wrong-host class as the
dc-mirror instrument note, made after writing that note.
Blocked on a harness permission wall for four MAAS interface calls, not on a
decision. Not retried in altered shapes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-138 ADOPTED: the cloud-facing client lives IN the DC; SEC-026 opened
...
GA-R5, before dependent work. Operator utterance: "Move the cloud-facing
client into the DC (Recommended)".
Resolves the contradiction the Stage-5 bootstrap failure exposed: SEC-010 +
D-052 make metal-admin DC-local and forbid the route juju needs, while D-100
names Juju as fiber traffic and D-128 puts the client on voffice1. SEC-010's
"pinning is free" was priced against tools that proxy at the app layer; juju
dials the machine at L3 and was not in scope.
The client moves; the boundary does not. SEC-010, D-052 and D-125 are
UNCHANGED -- nothing punctured, no plane opened. D-128 is amended to exclude
cloud-facing tools. Scope was set by enumeration first: keystone's VIP is on
provider-public, so routing would have opened two planes per DC across a
dozen ports.
SEC-026 opened for the consequence: a MAAS admin-scoped key becomes resident
on a DC-local host, over a region shared by both DCs. Isolation is the
control -- each DC's client host gets ONLY its own credential.
Counters: 22 open SEC, next-free D 139.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 5 BLOCKED: no juju-client->node-plane path; D-134 .5 now BUILT both DCs
...
Bootstrap attempted and failed. Machine selection was correct (7n87bt, the
tagged VM) and MAAS deployed jammy end to end -- confirming the
--bootstrap-base ubuntu@22.04 pin against a live deploy. juju then could not
SSH the machine and released it.
Root cause: voffice1 has NO route to any DC node plane and two deliberate
controls forbid one -- SEC-010's transit FORWARD-drop on the rack, and
libvirt's blanket reject into the isolated plane bridges. The DC edge has no
metal-admin leg. Nothing regressed; this path never existed.
This is a contradiction between RULED surfaces: SEC-010/D-052 make
metal-admin DC-local and forbid the region routing to 10.12.8.0/22, while
D-100 says the fiber carries Juju traffic and D-128 puts the juju client on
voffice1. SEC-010's "pinning is free" justification was priced against MAAS,
which proxies at the app layer; juju dials the machine at L3 and was not in
scope. Needs a ruling, not a firewall edit. No reachability change made.
BUILT, both DCs (operator: "Fix now: static .5 + v6, re-bootstrap"): the
controller VMs held AUTO v4-only addresses; now static 10.12.8.5 +
fd50:840e:74e2:220::5 and 10.12.68.5 + fd50:840e:74e2:320::5. v6 prefixes
confirmed by VLAN pairing, not inferred. D-134's amendment was
ruled-but-not-built until now.
Capture: docs/audit/stage5-bootstrap-reachability-20260730.txt
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Stage 5: Step 1 + 2.0 gates passed; bootstrap constraint-flag ruled
...
GA-R5, before dependent work. Operator utterance: "Use both flags".
juju 3.6.27's own help assigns machine-targeting to --bootstrap-constraints
and model-defaulting to --constraints; the runbook and D-104's mechanism
sentence both name only --constraints. D-104 is NOT amended -- only the flag
implementing it is clarified. DOCFIX owed, recorded in the changelog.
Read-only gates, all on voffice1: selectors exit 0 (6 planes, 10 hosts);
credential gate outcome 1 of 3 so the mint is skipped (SEC-018 sprawl);
exactly one machine carries juju-controller-vr1-dc0, Ready.
Artifact sources PASS both DCs; egress re-probed from both racks with
--noproxy '*' -- bootstrap window OPEN. Instrument note recorded: the DC
checkers run ON the rack host and read all-MISS when run from voffice1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
STAGE 5 OPEN: P5 accepted by ruling, entry gate captured on voffice1
...
Opens Stage 5 / Phase 4 (Juju controller + OpenStack bundle, per DC) on the
standing operator directive. GA-R1/C1: the status change and CURRENT-STATE
land in one commit. GA-R5: the P5 ruling is committed and pushed BEFORE any
dependent work.
Entry gate, measured ON voffice1 (the only host whose reading counts):
P1/P2/P3/P4 PASS, P7 PASS 37/0 with the literal zone line, P5 FAIL on 6
pre-existing credential-register findings. Capture
docs/audit/stage5-preflight-dc0-20260730.txt.
Operator ruling, exact utterance: "Accept and proceed to deploy (Recommended)".
The acceptance covers those six findings, enumerated, and nothing else.
Logged not executed (hard rule 1): bundle.yaml:592 gives ceph-osd the stale
VR0 constraint tags=openstack, a tag measured absent from the VR1 region.
Decided at Step 4.2's dry-run, where its impact is first observable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 session close: F9 closed live, the reissue tool, P7, lib-identity
...
BOOKEND (bounded, 15 lines) appended to docs/session-ledger.md. Machine-derived
block RE-SEEDED from ledger-scan (DOCFIX next-free 205 -> 206). Ledger stood at
exactly 300 lines after the append, so the two oldest live summaries (the
2026-07-27 grounding-audit close and its 14-rulings addendum) rotated VERBATIM to
docs/archive/session-ledger-rotated-20260730.md: 300 -> 235. Rotated HERE rather
than left owed, because the next session is directed straight at the deploy.
STANDING OPERATOR DIRECTIVE recorded at the TOP of CURRENT-STATE section 1, where
the next session's bookend hits it first: 'we have to continue to juju deployment
next session no matter what'. Precondition work is DONE and further hardening is
OUT OF SCOPE unless it blocks the deploy. This is the GA-F06 circuit-breaker made
explicit -- the directive removes the judgement call.
Deliberately NOT put in auto-memory: GA-R7 rule 1 forbids operator policy,
priorities and postures there. It belongs on a repo surface, so it is on the
status authority.
Entry facts recorded so the next session does not re-derive them: run FROM
voffice1; preflight FAILs on P5 ONLY (pre-existing credential register -- decide
accept-or-remediate AT the gate, do not re-audit); every other gate green and P7
[ok] both DCs; the deploy input is bundle + that DC's three overlays; G17 arms at
first boot and is a Stage-5 artefact, not a precondition.
GA-R7 memory review done: two memories, both observations/conventions, both still
accurate, neither claiming operator intent. No changes needed.
Session totals: 19 commits. F9 CLOSED live on both DCs; octavia-pki 21/21 -> 51/51;
preflight 26/26 -> 33/33; gauntlet ALL GREEN (89) both hosts; repo-lint 0 fail;
scan unchanged at 3 decisions / SEC 21.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
lib-identity.sh: the estate's name and DNS base in ONE place
...
Operator: 'The specific server and region names do not matter as they change
during every deployment and hardening of the deployment workflow.' Then, after
scoping the two options: 'Yes, take option 2.'
WITHDREW MY OWN FIRST RECOMMENDATION. I proposed retiring the vr0-dc0 selector
as removing 'the ambiguity class outright'. Measured, wrong on both halves:
D-119 already closed it by region-qualifying (the hazard was the BARE dc0, which
is rejected loudly today); lib-net.sh:110 documents that arm as a NO-OP over the
file's flat defaults; and the footprint is 43 files, mostly the separate VR0
phase-NN track the VR1 runbooks cite as precedent. Medium cost, near-zero
benefit. Logged, not executed.
NEW scripts/lib-identity.sh -- CLOUD_NAME + CLOUD_DOMAIN, no side effects,
env-overridable. With <dc> and <region> already derived from the site token,
these were the LAST typed identity in the shell surface. A rebuild that renames
the estate now edits one file and the certs, zones, SANs and P7 all follow.
DELIBERATELY NOT lib-net.sh: sourcing that bare populates a full flat plane/VIP
namespace (its own header says so). A certificate checker needs two strings, not
a network namespace, and octavia-pki.sh sources NOTHING today -- pulling in
VR0-shaped defaults it never asked for is the R9 hazard in miniature. Sourced
from SCRIPT_DIR (sibling, survives the harness REPO override) and FAILS CLOSED
with REFUSE 3 if absent: a cert gate that invents the estate's identity is worse
than none.
Consumers: octavia-pki.sh derive_zone(), phase-01 1.0-GEN.c, dc-dc-phase6 Step 0.
The tofu side was ALREADY parameterised (opentofu/variables.tf domain_suffix
passed explicitly into modules/dc-planes), so nothing there changed.
HARNESS 48/48 -> 51/51:
- T45 the centralisation must be REAL, not decorative -- overrides both values
and requires the zone to follow BOTH (acme.dc0.vr1.example.test). A constant
nothing can vary is indistinguishable from a literal.
- T46 a missing lib-identity.sh REFUSES rather than guessing.
- T47 SHELL vs OPENTOFU drift. HCL cannot source a shell file, so two copies of
one fact exist; if they part, the libvirt plane domains and the cert zones
describe different estates and nothing else would notice. PROVEN able to fail
by injecting a drifted value, then restored.
Also: repo-lint L6 flagged '. $REPO/scripts/...' as a bare invocation. Its own
docstring blesses a bash|source|python3 prefix and sourcing needs no exec bit,
so the RUNBOOK moved to the documented 'source' form -- the gate was not changed
for style. And ${REPO:?} is now guarded at its FIRST use in GEN.c, not only at
:538 below it.
STILL typed, deliberately: the site allowlists. One that accepts anything is how
a typo'd DC name becomes a wrong-target write.
gauntlet ALL GREEN (89); repo-lint 0 fail; GEN.c bash -n clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
P7: the Octavia amphora PKI becomes an actual pre-deploy gate
...
Operator: 'Yes, I accept the recommendation. Convert to a gate.'
THE GAP: octavia-pki.sh was invoked by NOTHING executable. P4's CHECK 0 asserts
only that the overlay FILE EXISTS, and the harness runs against throwaway
fixtures -- so 'gauntlet ALL GREEN' said nothing about the live PKI and the only
real check ran when a human remembered to type it. That is how F9 survived.
Three problems solved, each from a measured precedent:
1. rc 3 must not reach note()'s *) arm. octavia-pki.sh uses exit 3 for REFUSE,
but note() calls any rc>=3 'UNEXPECTED exit code'. P7 maps rc explicitly, so
a REFUSE on the headend FAILS and is NAMED as an unevaluated trust domain.
2. HOST AUTHORITY. The PKI lives on the headend (D-109 (b)); elsewhere P7 says
NOT EVALUATED (WARN) -- never a silent pass, never a hard FAIL for being on
the jumphost. The binding is read from the SAME host-identity file
octavia-pki.sh reads, so they cannot drift. P5's F6 defect designed out.
3. A PASS VERDICT IS NOT SUFFICIENT. While os-public-hostname is unset, A12/A13
RECORD a wrong-zone cert rather than failing it, so verify exits 0 on exactly
the defect this gate exists to catch. P7 also requires the literal line
'DNS SANs are all in this DC.s expected zone'. Satisfiable by construction
now that GEN.c derives the zone and both DCs were reissued.
Also skips cleanly for vr0-dc0 (a gate that fires on a target it does not apply
to is how gates get bypassed) and FAILS CLOSED if the checker is absent.
HARNESS 26/26 -> 33/33. Ten existing cases went RED when P7 landed -- correctly,
the fixtures had no octavia-pki.sh. Fixed by making mkfix model a healthy headend,
NOT by patching cases one by one: that is the moment a gate gets demoted to a
warning to go green. T27 is the case that matters -- a wrong-zone cert on which
verify exits 0 must still FAIL P7.
Two harness bugs of my own, caught by running not reading: a stub emitted DC.s
where the real line has an apostrophe, then \x27\x27\x27 produced three of them
because \x27 in a printf FORMAT already yields one. Fixed the stub, not the gate.
Preflight's overall verdict is still FAIL -- that is P5 working as designed on
the pre-existing red register. P7 itself passes on the headend.
gauntlet ALL GREEN (89); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F8 + F9 fixed AT SOURCE in 1.0-GEN.c -- the next DC standup will not recreate them
...
Everything before this repaired the two EXISTING DCs. The GENERATION path still
carried both defects, so the next DC standup would have minted them again.
F9 at source: GEN.c baked omega.dc0.vr0 as a literal in THREE places (CN + both
DNS SANs). It now DERIVES DC_ZONE from $DC with the same two expansions the tool
uses (${DC%%-*} / ${DC#*-}), so runbook and gate cannot disagree, and echoes the
zone for confirmation before the sign.
F8 at source: the cat > controller.cnf heredoc is replaced by one printf per
line, values passed as %s ARGUMENTS. The 2026-07-29 note deferred this as 'an
untested rewrite ... reasonable when someone can run a real generation'. That
condition is now MET: the identical shape was exercised end to end by two real
mints (both DCs) plus 48 harness cases.
Plus structural assertions BEFORE signing -- four sections present, subjectAltName
wired, CN equals the derived zone, exactly 2 DNS entries -- because printf removes
the paste hazard but not the failure CLASS: a typo'd section name still yields a
SAN-less cert behind a wall of OK output.
The stale 'NOT changed -- outside R7's ruled scope' note is superseded in place,
keeping the reasoning worth carrying: R7 recorded this cert's SAN as 'already
DERIVED per-DC by design', true of the IP SAN and NOT the DNS names -- a claim
accurate about one half of a field, read as covering both. That is how F9 survived.
F10 HANDLED DELIBERATELY. Editing GEN.c shifts every mint-ref anchored below it.
All 13 octavia anchors re-resolved BY MARKER in a SINGLE PASS keyed by row id --
never sequential seds, because a line number can be simultaneously an old value
for one row and a new value for another -- then each verified to point at its
correct command. 12 rows rewritten; creds-matrix S4 CLEAN; block bash -n checked.
octavia-pki 48/48; creds-matrix 65/65; creds-matrix 101 rows / 5 findings, all
pre-existing; gauntlet ALL GREEN (89); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Q2 withdrawn: D-137 fork 1 was ruled 2026-07-25, not open
...
Third stale-premise item in queued-findings-20260730.txt (after Q1). Q2 opens
'D-137 OPEN FORK 1 (enforcement strength) -- still unruled'. Sub-ruling 1 ruled
it 2026-07-25 at option (b), utterance 'Blocking in preflight', and its
CONSEQUENCE block explicitly declines option (c): 'NOT adopted: the PreToolUse
guard (so an agent is not constrained at write time, only at the deploy gate)'.
So guard-destructive.py is NOT D-137's enforcement point -- it is the older
CLAUDE.md/DOCFIX-006 mechanism, and hardening it needs no ruling to proceed.
This was verified BEFORE the reissue minter was built, which is why 'Full script
minter now' was consistent with the ruling rather than a side-effect decision.
The misfire evidence stays valuable and the tally is now SEVEN. The two newest
were collected today: the guard blocked a command whose only purpose was to TEST
it (it read nothing), and then blocked the heredoc writing this very correction
into the audit file -- documentation prose, no file read. Both are the same class
as the earlier five: the matcher cannot distinguish reading secret material from
merely naming it.
Correction APPENDED, not rewritten (the stage4-mirror-gate precedent).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Reissue backup custody: transferred to vcloud, and REGISTERED
...
Operator: 'Update the matrix as needed. Transfer the artifacts to the vcloud.
keep the pin that during the secrets workflow planning that the creds and certs
from this step need to be included.'
CUSTODY (Step 1.0-REISSUE.4, executed): both pre-reissue archives pulled from the
headend to the jumphost SEC-009 creds folders, sha256 compared and IDENTICAL at
both ends (e30cab7f...3487 dc0, 17c831da...6fe3 dc1), 0600, each listing 15
entries including the deploy overlay and the CA serial. Headend staging copies
removed; ~/octavia-pki/backups/ gone; no .reissue-* residue. The sha256 compare
happens BEFORE the delete because a truncated scp would otherwise leave a
verified-looking backup of nothing.
REGISTER: two octavia-reissue-backup rows (per-DC, jumphost, consolidated,
templated filename <site>-octavia-pki-<stamp>.tar.gz), the DERIVED manifests
updated to match (D-137 ruling 2 -- manifests are generated, not hand-authored),
and a new n-reissue-backup note. It is deliberately a SEPARATE id from
n-pki-backup: that one is the generation-time workspace archive, this one is
per-ROTATION and additionally carries the deploy overlay, which the generation
archive does not -- restoring only the workspace would leave the half that
actually reaches the charm unrecoverable.
Measured after: tier1 101 rows / SAME 5 pre-existing findings; tier2 the same 7,
with NO reissue-attributable finding -- and tier2 FOUND both archives at their
declared location, which is the point of registering them. Harness 65/65.
HAZARD RECORDED: these archives contain controller-ca.cert.srl, the CA's issuance
state. Restoring one over a workspace that has issued since rolls the serial
BACKWARDS and the next mint reuses a serial the estate already holds. Serials
burned 2026-07-30: ...250E (dc0), ...674DE (dc1).
PINNED: the note carries the operator's standing requirement that the creds and
certs from this step be included in the secrets-storage workflow planning, with
the full scope enumerated. The jumphost creds folder is the INTERIM home only.
gauntlet ALL GREEN (89); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
F9 CLOSED: controller certs reissued into their own zones, both DCs
...
Executed on voffice1 via Step 1.0-REISSUE at HEAD 7b6a2e4. Operator-gated PER DC
('Go ahead' each), never batched; dc0 verified clean before dc1 was touched.
Capture: docs/audit/octavia-reissue-executed-20260730.txt
vr1-dc0 -> omega.dc0.vr1.cloud.neumatrix.local, IP 10.12.4.57 + f02:11::57,
serial ...250D -> ...250E
vr1-dc1 -> omega.dc1.vr1.cloud.neumatrix.local, IP 10.12.64.57 + f03:11::57,
serial ...674DD -> ...674DE
dc1's OLD cert carried dc0's LABEL as well as the wrong region -- the cross-DC
case A11 exists for. A11 confirms per-DC independence on both DCs after the mint.
Both: fresh EC P-256 key with the rotation PROVEN (public key differs from the
outgoing cert's), neither CA regenerated so the amphora trust domain is intact,
overlay surgery measured other_values_identical=4 changed_lines=1 line_count=8.
ACCEPTANCE was the literal positive line, not the verdict: while os-public-hostname
is unset, A12/A13 RECORD rather than fail, so PASS alone is compatible with a wrong
certificate. Both DCs now carry 'DNS SANs are all in this DC's expected zone'.
verify PASS 37/0 both DCs (was 29/0; +8 from the new assertions). creds-matrix:
same 5 pre-existing findings, no new finding, no S4 drift. No staging residue,
neither run reached exit 5.
The hardened re-run guard proved itself live: re-running dc0 immediately after
returned exit 4 'already correct ... and verify returns clean' -- the fix for the
defect that had it refusing to repair certs verify was failing.
STILL OWED, operator-gated: backup custody. Two archives on the headend hold BOTH
CA private keys, BOTH passphrases and the overlay; 1.0-REISSUE.4 pulls them to the
jumphost creds folder and removes the headend copy. They are also an unregistered
credential class (no creds-matrix row, no retention rule). Logged, not actioned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
octavia-pki: a hardened, tool-driven controller-cert REISSUE path (F9's remedy)
...
Operator: 'Let's reissue now. Engineer the artifact, variable, based workflow to
reduce the chance of errors.' Five GA-R5 rulings, one exchange each, quoted in
the changelog; ruling 5 was 'Full script minter now'.
THE TOOL IS BUILT; THE MINT IS NOT YET RUN (gated operator action, 1.0-REISSUE).
scripts/octavia-pki.sh gains 'reissue <site> [--force] [--dry-run]': controller
LEAF only, signed by the EXISTING controller CA (neither CA regenerated, amphora
trust domain untouched), fresh P-256 key, zone DERIVED from the site token and
never typed, printf-built config (the F8 payoff), stage-assert-promote, and a
single-value overlay surgery that never puts the bundle in argv.
SIX GATE DEFECTS FOUND BY VALIDATION AGENTS, ALL CLOSED. Mutation testing is
what earned this: deleting the three assertions I had just written left the
harness fully GREEN every time, so they were decoration until each got a
failing-direction fixture.
- A15 keyUsage/EKU on the ISSUED cert. A cert with NEITHER minted, promoted and
passed everything. keyUsage appeared once in the file: the printf writing it.
- A16 validity. openssl x509 -req defaults to -days 30, and checkend/notAfter/
enddate appeared ZERO times in verify -- a 30-day cert passed forever.
- A17 the overlay must decode to what the workspace holds. A10 graded shape,
A1-A16 graded the workspace, nothing joined them; a desynced overlay read
PASS 0-failed and the overlay is what reaches the charm.
- The re-run guard REFUSED to fix broken certs: names-correct + stale IP SAN
gave verify FAIL and reissue exit 4 'nothing to fix'. Now gated on verify.
- A8's negative proves its instrument loads first (openssl verify exits non-zero
for 'could not load the CA' too, so a corrupt CA passed it vacuously).
- Two pre-existing overlay defects moved pre-mint, so they stop producing a
false exit-5 with the workspace already promoted.
Harness 23/23 -> 48/48. Runbook Step 1.0-REISSUE appended AFTER GEN.e
deliberately (F10 line-anchored mint-refs; creds-matrix reports no S4 findings
after the append). Skill gains two standing invariants for future DC standups:
every per-DC secret needs a ROTATION tool not just a generation recipe -- a
REFUSE-IF-PRESENT generation gate makes the dangerous path the only path -- and
a hand-pasted openssl chain is a defect class, with F8 and F9 as the receipts.
Also corrected: D-137 fork 1 is NOT open (sub-ruling 1, 2026-07-25, ruled
'Blocking in preflight' and explicitly declined the PreToolUse guard), so the
queued Q2 premise is stale in the same way Q1 was.
gauntlet ALL GREEN (89); repo-lint 0 fail; creds-matrix 5 findings, all
pre-existing and named; ledger-scan unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 sweep: correct the artifact that filed Q1, and one stale residue
...
Advisor review caught that correcting CURRENT-STATE and the ledger was not
enough: docs/audit/queued-findings-20260730.txt -- the sweep capture the 07-30
bookend points at -- still presented Q1 as an open ruling that 'blocks any FQDN
cert', enforced by an A12 refusal that no longer exists. A next session reading
it would re-derive Q1 exactly as this one did, reopening the loop just closed.
- queued-findings-20260730.txt: APPENDED correction (not a rewrite; the
stage4-mirror-gate-20260727.txt precedent), naming the two claims that are now
false and pointing at the evidence. Q2 explicitly untouched.
- dc-dc-deployment-workflow.md:257: described phase-6 as proposing the retired
dc1/dc2-hostnames.yaml spelling. MEASURED: the runbook itself was already
corrected to the region-qualified form by an earlier session -- only the
workflow doc's description of it lagged. Under D-119 IS vr1-dc0, so the
-hostnames.yaml interpolation used elsewhere is correct as written.
- The DOCFIX-205 capture's section 1 quoted a 3-line span starting mid-sentence
and left a clause orphaned. Repaired to two verbatim quotes, both re-verified
present in design-decisions.md. CURRENT-STATE cites this file as evidence and
this repo's discipline is exact captures.
octavia-pki 23/23; repo-lint 0 fail (L10 gated this commit correctly -- the
docs/audit/ change set required CURRENT-STATE to move with it).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
L10 repair: cite the DOCFIX-205 capture in CURRENT-STATE
...
The preceding commit added docs/audit/docfix205-d117-annotation-20260730.txt
without touching CURRENT-STATE.md and FAILED repo-lint L10 (GA-R1 rule 8 / C1).
It was pushed anyway because the lint was chained with ';' rather than '&&', so
a red gate did not stop the commit -- the same silenced-error class logged in
the 2026-07-29 close. Recorded here rather than force-pushed over.
NOTE for the next session: repo-lint exits 2 on WARN, so a bare '&&' chain also
fails on the EXPECTED legacy L1 warn. Gate on '(0 fail,' in the output, not on
the exit code -- that is why ';' was reached for in the first place.
CURRENT-STATE's superseding block now cites the capture; tree is lint-clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205 capture: the D-117 supersession, annotation coverage 0/4 -> 4/4, live A12
...
Evidence file cited by CURRENT-STATE's superseding block. Carries the quoted
D-117 Status line, D-119's selector-only discharge, measured annotation coverage,
the phase-6 expansion old-vs-new in a real shell, live A12 on the headend
(PASS 29/0 both DCs, both INERT), and both hosts' gate results.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205: Q1 withdrawn -- D-117 ruled it 2026-07-13; annotate the four it owed
...
The operator asked to rule on queued Q1 (is substrate vr1-dc1 'dc1' by token or
'dc2' by position, for cert/DNS identity). Measured before presenting options.
The question was WITHDRAWN, not ruled: D-117 (ADOPTED 2026-07-13, four days after
D-106) names the supersession in its own Status line -- "Supersedes ... the D-106
dc1/dc2 zone labels" -- and rules the replacement: "The repo's dc1/dc2 labels are
retired in favour of dc0/dc1". vr1-dc0 -> dc0, vr1-dc1 -> dc1.
WHY IT RESURFACED, and this is the reusable half: D-117 ruled that the ADOPTED
decision texts D-101/D-106/D-111/D-115 be ANNOTATED in place. Measured: ZERO of
the four carried any D-117 annotation. It stayed invisible because D-117's OWN
Status line claimed "FULLY EXECUTED BY D-119", while D-119 scopes its discharge
to the SELECTOR half only. A reader checking whether the annotation was owed was
told it was already done. Ruled is not built -- check the artifact.
Landed (operator-approved batch, "All four + code fixes"):
- All four decisions annotated in place, per D-117's own ruled treatment. D-101
gets the hardest one: it carries BOTH namespaces, so it is annotated BY DATE
(dc1 means different datacenters in its two halves -- D-117 TRAP 2).
- D-117's Status line corrected by measurement (GA-R1/C2) to "EXECUTED IN TWO
HALVES", with the standing lesson that a Status line is a CLAIM about
execution, not evidence of it.
- phase-6 EXECUTABLE DEFECT fixed: os-public-hostname was built as
"keystone.omega.${DC}.vr1..." with $DC the D-119 selector, expanding to
omega.vr1-dc0.vr1... -- the region TWICE -- and would have been baked into
Vault-issued SANs at Stage 7. Both labels are now DERIVED from the site token
(${DC%%-*} / ${DC#*-}); verified by running it, not by reading it.
- octavia-pki.sh A12: refusal -> assertion on the full derived zone; a SAN-less
cert now also fails. Harness 21/21 -> 23/23, with T21 REPLACED (not deleted)
and new T21b (the other DC's label in the right region FAILS -- the cross-DC
mix-up region-only checking passed) and T21c (per-DC derivation proven).
- CURRENT-STATE + ledger corrected in the same commit (L10). "BLOCKING" was
wrong as stated: A12 measures INERT and PASSES live on both DCs. F9's reissue
obligation stands.
gauntlet ALL GREEN (89); repo-lint 0 fail; ledger-scan unchanged (3 decisions,
21 SEC, D 138 / BUNDLEFIX 053; DOCFIX 205 -> next-free 206).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
GA-R4 session close: bounded bookend + the headend reconcile
...
Bounded ledger bookend, 14 bullets against the GA-R4 rule-1 cap of 15; ledger 280 lines against
the F1 cap of 300, so no rotation is owed. Changelog body extended with items 13-17, each carrying
a revert. Status remains in CURRENT-STATE only.
The headend reconcile the operator asked for found a real defect rather than confirming a clean
state: the gauntlet had never actually been run on voffice1, and it FAILED there while passing on
vcloud. Cause was opentofu-validate's fmt check walking gitignored local tfvars that exist only on
the host which runs the applies -- so the gate was red where it mattered and green where it did
not. Fixed, scoped, pinned both ways. Both hosts now ALL GREEN (89), repo-lint 0 fail on both,
headend working tree clean, tfstate byte-identical across every pull.
Also reconciled: dc1's provider lock is now tracked, matching dc0's precedent, after ruling out
content divergence by measurement.
Owned in the bookend: the operator's own working commands were not in the repo until the
pre-bookend sweep asked for them; 664 CA certs were waved through as harmless when the exposure is
integrity rather than secrecy; and two errors were silenced by shell chaining -- `| grep -q` under
pipefail, and `git pull -q &&` hiding a real pull failure that made me re-run a gauntlet against a
stale tree and misread the result.
No stage opened or closed. Scan unchanged: 3 open decisions, SEC 21, D 138 / DOCFIX 205 /
BUNDLEFIX 053.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
voffice1 reconcile: a gate that was RED on the only host that deploys
...
Operator asked whether the headend clone was fully reconciled. It was not, and running the
gauntlet there -- which had never actually been done -- found a real host-dependent gate defect.
RECONCILE. The headend was 1 commit behind with 0 local commits, now at HEAD, tfstate sha256
byte-identical before and after. No stale remote-tracking refs. One asymmetry found and fixed:
opentofu/vr1-dc1-substrate/.terraform.lock.hcl was UNTRACKED and not gitignored, while dc0's
equivalent IS tracked. Same provider version (0.9.8), so a content divergence was ruled out by
measurement, but a fresh clone could not reproduce dc1's provider resolution the way it can
dc0's. Landed, byte-identical to the headend copy.
THE GATE DEFECT. The gauntlet read ALL GREEN (89) on the jumphost and 1/89 FAILED on the headend,
on opentofu-validate. Cause: `tofu fmt -recursive` walks EVERY .tf/.tfvars under the tree,
including GITIGNORED local files. The sole flagged file was
opentofu/vr1-dc0-substrate/d124-inner.auto.tfvars -- gitignored operator input that exists only
on the host which runs the applies. So the gate was RED ON THE ONLY HOST THAT DEPLOYS and green
where it did not matter, the same host-dependent-verdict class as the LC_ALL manifest sort.
Ruled out first: both hosts run tofu 1.12.4, so this was not version drift.
The fmt check is now scoped to repo content, and a gitignored file is SKIPPED with the reason
printed rather than silently. Proven both ways: a badly-formatted TRACKED file still FAILS (T16),
a gitignored local tfvars is skipped and does not fail (T17). T17 also fails loudly if the ignore
rule ever stops covering per-DC tfvars, so the case cannot silently stop testing what it claims.
Owned: my first T16 read as a failure because `bash "$SCRIPT" | grep -q` inherits the gate's
correct exit 1 under `set -o pipefail`, even though grep matched -- a passing assertion reading as
a failure. Same strict-bash family as the IFS and inherit_errexit traps this repo has hit before.
Captured first, then grepped.
opentofu-validate harness 16/16; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Pre-bookend sweep: the operator's working commands were NOT in the repo
...
Operator: "I'm worried that the fixes we made to the defective commands will be lost when we
close this session ... Complete a full sweep before the bookend."
THE CONCERN WAS WELL FOUNDED. The corrected Step 5/6 commands -- the ones that actually produced
both DCs' live PKI -- had been reviewed in conversation, judged equivalent, and never folded in.
Four items now landed:
1. Portable base64. GEN.d shipped the GNU-only `-w0` flag; what ran was
`base64 < f | tr -d '\r\n'`. The runbook described a command nobody executed.
2. The operator's VIP_OVERLAY:? guard in GEN.c.
3. The heredoc column-1 requirement, as a CAUTION heading rather than a footnote, because that
failure is silent: no alt_names section yields a certificate with NO SANs while every
openssl command still prints OK.
4. ${DC_LABEL:?} at both CA call sites -- F8's sibling, where an unset label bakes a DC-less
subject into a 10-year CA with no error anywhere.
Not done, with the reason recorded: the heredoc was NOT rewritten as printf lines. More
paste-proof, but an untested rewrite of a step that mints 10-year CA material and cannot be
exercised end-to-end from an agent session. Risk bounded instead by A9 plus harness T2.
F9 is now structural rather than remembered: A12 arms itself from os-public-hostname appearing as
a real option key. Live PASS 29/0 both DCs, harness 21/21.
A new ruling-shaped gap, now blocking rather than filed: D-008's shape plus D-106:2563's VR1
instantiation do not say whether substrate vr1-dc1 is dc1 by token or dc2 by position -- the
DC1/DC2 ambiguity item 3.1 retired elsewhere, here deciding certificate identity. Both live certs
carry dc0.vr0, the VR0 region, wrong under either reading. A12 refuses rather than picking.
A precedence bug the harness caught: REFUSE was checked before FAIL, so once A12 armed a
CONFIRMED wrong-region SAN reported as "could not evaluate". A known defect outranks an
unevaluated one.
Sweep capture docs/audit/queued-findings-20260730.txt carries F10-F13 and two ruling-shaped
questions. F10 is the one worth acting on: mint-ref line numbers drift silently and S4 cannot see
it, since it asserts only within-EOF. That bit three times in two sessions and was caught by hand
every time, never by a gate. All 24 octavia refs re-anchored in a single-pass mapping keyed by row
id -- 391 was simultaneously an old and a new value, so sequential seds would have corrupted it.
repo-lint 0 fail; octavia-pki 21/21; creds-matrix 65/65.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
A12: the DNS SAN check ARMS ITSELF, so F9 cannot be missed
...
Operator direction: make the DNS SANs part of the process so it is not a step that gets missed
in future.
The risk in F9 is not the defect, it is that a future session forgets. So A12 arms from the
change that makes the SANs load-bearing -- os-public-hostname appearing as a real option key in
the merged deploy input -- rather than from a note somebody has to read. Unarmed, it states the
names are inert and why; armed, it asserts them.
WHAT IT CAN ASSERT, AND A GAP IN A RULED DECISION THAT IT CANNOT. D-008 fixes the shape
<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local and D-106:2563 instantiates VR1 as
"omega.dc1.vr1" / "dc2.vr1" -- while the substrate's DCs are vr1-dc0 and vr1-dc1. So vr1-dc1 is
"dc1" by token or "dc2" by position: the DC1/DC2 ambiguity item 3.1 retired elsewhere, surviving
inside a ruling, where it decides certificate identity. The REGION is unambiguous and is
asserted. The DC LABEL REFUSES pending a ruling rather than blessing a name that cannot be
validated. Both live certs currently say dc0.vr0 -- the VR0 region, wrong for a VR1 DC under
either reading.
A PRECEDENCE BUG THE HARNESS CAUGHT. Because the armed branch always refuses on the label, a
definite wrong-region SAN was being reported as "could not evaluate" -- REFUSE was checked before
FAIL and masked a confirmed defect. A known defect outranks an unevaluated item, so the verdict
now reports FAIL first and still names the refusal, which is never lost.
Harness 21/21 (was 18). T19 proves the unarmed live posture still PASSES -- without it, arming
A12 would have broken every current verify, the same trap T17 guards for the deleted
intermediates. T20 is F9 armed. T21 proves the right region still refuses on the unruled label.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Both backup paths taken and proven restorable; E2 and E3 fully closed
...
Operator executed; the session verified read-only.
TFSTATE RESTORE PROVEN, not asserted. Both archives decompress to valid JSON, report serial 9
(dc0) and 4 (dc1) with 29 resources each, and are byte-identical to the live state. Those are the
same hashes measured at session open, before the headend pull -- so the states have not drifted
and the backups match what tofu actually uses. Archives are 600 inside 700 folders, differ
between DCs, and the staging copies came off the headend only after the checksum-and-parse gate.
P5 went 9 -> 7 on the jumphost, and the two that cleared were exactly the EXPECTED-BUT-ABSENT
rows the register had been demanding: the gate asked for the backup, then confirmed it. Zero E2
and zero E3 findings remain on any octavia or tfstate artifact. The residual 7 are the
pre-existing set carried into this session, none introduced by this work.
The serial chmod closed the last E2: P5 on the headend fell 8 -> 6 and octavia-pki.sh verify
reads PASS 26/0 on both DCs.
Net effect of the day's credential work: the Octavia PKI went from a generator that would have
destroyed dc0's CA when dc1 was generated, an overlay one rename away from being committable in
a public repo, a register structurally blind to a missing second CA set, and a gate suite that
asserted neither the SAN set nor any file mode -- to two independent trust domains, both verified
26/0, both backed up with a proven restore, and both fully declared. Every one of those defects
was found by measurement rather than inherited from a document.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
tfstate backup path built; E2/E3 closed live; a gate disagreement found and fixed
...
BACKUP PROVEN BY RESTORE, not by existence. Both PKI archives were extracted and compared
byte-for-byte against the live tree: 12/12 sha256 MATCH across both DCs, covering both encrypted
CA keys, both passphrases, both CA serials and both controller bundles. Archives are 600 inside
700 folders and differ between DCs, so per-DC independence survives into the backup.
E2/E3 CLOSED LIVE. Operator ran the chmod and the intermediate removal; octavia-pki.sh verify
now reads PASS 26/0 on BOTH DCs, workspaces 12 -> 10 files, and P5 on the headend fell 18 -> 8.
A GATE DISAGREEMENT, recorded as the pattern rather than the incident. verify read 26/0 while
creds-matrix E2 still reported the CA serial mode 664 on both DCs. The chmod list predated the
serial being declared, and verify's mode assertions covered six private files and three certs --
the serial was in neither. Two gates that can both see a file must not disagree about it. Fixed
in three places: verify now mode-checks the serial (integrity, not secrecy -- rewriting it forces
the next issuance to reuse a serial), the generator's chmod includes it, and T18 pins it. The
harness then caught its own fixture creating the serial at the inherited umask; baseline went red
until the fixture matched the corrected generator. 18/18.
TFSTATE BACKUP as dc-dc-phase2 step 13, 2 rows, notes key n-tfstate-backup. Register 99 rows;
schema, render-drift, mint-ref and notes checks clean.
Three deliberate differences from the PKI backup: tfstate is DYNAMIC so the serial is recorded
both sides and the step must be re-run after every apply (restoring a stale state is actively
dangerous -- tofu then believes everything created since does not exist); the pull block proves
RESTORE by decompressing to check the archive parses as JSON and reports a serial, because a
truncated gzip passes a size check; and mint-stage is stage3, which is REACHED, so P5 correctly
reports these EXPECTED-BUT-ABSENT until the step runs rather than deferring them.
Classification recorded honestly: the outer state is credential-bearing (DOCFIX-175, MAAS API key
in plaintext); the inner states show no credential-shaped key names, consistent with the inner
root using only libvirt over qemu+ssh. Strong evidence, NOT proof -- a name-based absence cannot
exclude material inside a value -- so they are registered and stored as if sensitive. Uncertainty
resolves toward more auditing, and an unaudited backup directory beside audited ones is how the
SEC-022 shadow stores happened.
Two of my own defects caught by the gates: repo-lint L9 rejected a hardcoded clone name in the
new step (the repo has been renamed once, D-110) -- now $REPO; and two mint-refs had drifted when
a later edit added lines above their anchors, which S4 cannot catch because it only checks
line <= EOF. Both re-anchored.
Also corrected: phase-01:605 said the overlay must be in the backup set. It is derivable from the
archived workspace by 1.0-GEN.d, so the line implied a gap that does not exist.
Known, not fixed: both live inner states are 664, group-writable, and they are the authority tofu
trusts. Tightening needs its own gated change plus proof the provider preserves the mode.
creds-matrix 65/65, octavia-pki 18/18, repo-lint 0 fail / 621 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
