| 2026-08-01 |
d139-gua-carve: add the OOB plane (:f0::/60 + :f0::/64) per the 2026-08-01 ruling
...
CARVE gains (0xf0, "oob", "oob", True). It was absent because design-decisions.md:3066
declared lbaas-mgmt(:80), vpn(:e0) and oob(:f0) out of scope of the six-plane tool;
D-139 brought :80 back and left the other two, so the carve was incomplete against the
org standard ruling B cites as its reason. VR0 DC0 and Willamette both hold :f0::/60 +
:f0::/64 with role oob, so the v6 side CONFORMS. VPN (:e0) is the same omission and is
deliberately NOT added -- it was never ruled.
Harness 71 -> 75. Count literals REPLACED, not deleted, per the standing rule: T07 and
T12 11 -> 13, T17's three commit-path tallies 11 -> 13, and T08/T12 gained the two OOB
CIDR+role rows. stub-apex ROLES gains "oob" so the role-existence check resolves.
Mutation: removing the OOB row from CARVE turns T07 and both T08 oob cases red; tool
restored sha256-identical. Live dry run now reports CREATE 13 | EXISTS 3 |
RETIRE-REPORT 9 with both oob rows present; nothing written.
Gauntlet ALL GREEN (96); repo-lint 0 fail / 1 warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
P8 host guard: && -> || (live false FAIL on voffice1); dc0 MAAS path restored
...
P8's guard warned only when BOTH terraform.tfstate and opentofu/.terraform were
absent. voffice1 has no state file but does have .terraform -- D-128 has it run the
INNER tofu roots, and `tofu init` leaves a provider cache -- so neither warn branch
fired, P8 ran a real `tofu plan`, and it died on a gitignored tfvar -> exit 1 -> hard
FAIL on the very host the Stage-5 runbook designates for preflight. terraform.tfstate
is the OWNERSHIP marker; .terraform is only a cache. Both must be present to evaluate.
No test caught it because T38 removes the whole opentofu/ directory, so both markers
vanish together; the fixture itself sat in the defective state and passed only because
the && was wrong. Fixture now creates both markers; new T39 covers voffice1's real
shape (cache present, state absent -> WARN). Mutation: reverting || to && turns T39
and only T39 red; script restored sha256-identical.
Also: the dc0 MAAS path was down -- the vr1-dc0-region profile points at an SSH
forward (127.0.0.1:5241) that died with the 05:48 rack reboot. Restored and proven
(plan vr1-dc0 PASS exit 0, was REFUSE exit 3). Not an SEC-010 puncture: an SSH forward
originates on the rack. voffice1 pulled to HEAD. Preflight re-run: 12 [FAIL] lines,
11 = the ruled-accepted P5 set compared by identity, 1 = P8.
tests/preflight 38 -> 39/39; gauntlet ALL GREEN (96); repo-lint 0 fail / 1 warn.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Both DC VMs 416 -> 480 GiB; 128 GiB swap; new preflight gate P8 (substrate drift)
...
POST-BOOKEND: the GA-R4 bookend (4b8ba3c) was committed BEFORE this work, so the
ledger close summary and the 08-01 sweep do not cover it. The ledger line is
AMENDED in this commit rather than left stale.
Operator: "Option 2 look sgood me. I approve the sequencing, process as
autonomously as possible" -- +64 GiB to EACH DC with swap first, over my
recommendation of +48. The deciding arithmetic, put up before the ruling: at +64
each, 994 GiB allocated against a 1007.4 GiB host = 13.4 GiB residual while the
host's OWN measured footprint is 18.0 GiB, with 1 GiB swap free. Swap is what
makes +64 safe, hence the sequencing.
PROBLEM FIXED, MEASURED: dc0's rack ran 402 GiB of inner guests in a 409 GiB host
(98.3%), leaving 7 GiB for the rack OS, mirror, snap proxy and page cache. Each
rack now has ~71 GiB.
SWAP: operator-run -- sudo -n is NOT available on vcloud so that half could not be
automated. /swap2.img 128 GiB, 600 root:root, live + fstab. 135 GiB total.
CHANGED THROUGH TOFU, the only correct place: a virsh setmaxmem would have been
reverted by the next apply. The derivation now lives IN the variable comment.
PLAN ASSERTED ON CONTENT (2026-07-20: an in-place apply silently regenerated 9
node MACs): per DC 1 in-place, 0 create/destroy/replace, 0 MAC changes, memory the
only changed attribute. A THIRD resource appeared and was resolved BEFORE
applying: module.office1_opnsense ~ id = 2 -> 11 sits under "has changed outside
of OpenTofu" (drift OBSERVED), not "will be updated in-place" (action PLANNED) --
libvirt's domain id is a runtime value. It was not touched.
dc1 FIRST AS CANARY, and it answered the open question: the in-place update
BOUNCES the guest (domain id 9 -> 12, uptime 0 min). Guest sees 472.2 GiB.
>>> FINDING: vr1-dc0-maas-01 (MAAS region) and vr1-dc0-juju-01 (juju controller)
have autostart=disable. <<< Only the edge auto-recovers. Checking first is the
only reason the dc0 bounce did not come back looking catastrophic. Both started by
hand and verified. LOGGED NOT FIXED: a host reboot leaves dc0 with no region and
no controller -- a standing exposure independent of this change.
RESULT: host used 809 -> 63 GiB, available 197 -> 944 GiB; the restarts also
released the stranded RSS (vvr1-dc1 354 -> 12 GiB), discharging 08-01 sweep F1 as
a side effect. Recovery verified end to end: MAAS region API 000 -> 502 -> 200 (a
real boot progression), snap proxy listening, mirror 200, juju controller model
"Last connection: just now".
GATE P8 -- substrate drift. Built because answering "does tofu need updates
logged" produced a measurement: preflight.sh and pre-flight-checks.sh contained
ZERO tofu plan checks and the opnsense drift had sat unseen. DESIGN POINT:
-detailed-exitcode returns 2 for BOTH a pending change and a harmless observation,
so P8 asserts on CONTENT -- pending ACTION FAILS, observed-only drift WARNS,
unrecognised shape REFUSES. A gate permanently red on benign drift gets ignored.
It WARNS rather than fails when it cannot look, since the outer root lives on
vcloud and "not the substrate host" is a legitimate state.
Harness 33 -> 38 (T34-T38). Baseline proven by stashing: my first draft's 7
failures were MINE, not pre-existing. Three mutations each killed a NAMED test,
script restored sha256-identical. T38 exercises the absence by removing the
fixture STATE, never the fake tofu from fakebin -- the real tofu is on the system
PATH and a "closed" PATH would fall through to the LIVE substrate. PROVEN LIVE:
tofu exit=0, 0 pending, 0 drift -> [ok] zero diff.
SCOPE: tofu owns the SUBSTRATE only. MAAS carves, rack services and the v6 node
carve are script-driven by design (Model B / D-123); P8 does not check them.
gauntlet ALL GREEN (96); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
G19 BUILT and PASSED LIVE; finds a latent defect in D-139's execution list
...
scripts/dc-node-v6-verify.sh (270 lines, 24-line header) + tests/dc-node-v6-verify
(55 cases); manifest 95 -> 96. All three subcommands exercised LIVE this session:
plan vr1-dc0 PASS exit 0 (9 machines, v6 plane set IDENTICAL across all 9, 6
planes, prefixes DERIVED); node vr1-dc0 <spec> on storage-01 against live peer
storage-02 PASS exit 0 with 12 assertions -- six address-presence, six
peer-reachability, every plane "up, global, DAD complete, sole global on the NIC"
and "replies, neighbour DELAY"; bridges vr1-dc0 reporting 7 bridges. Harness
55/55, gauntlet ALL GREEN (96), three mutations each turning a NAMED test red.
RUN AGAINST A REAL NODE while the boot window was open specifically so it would
not ship fixture-green -- the snap proxy's harness was green for a week before
anything was proven end to end.
NO v6 PREFIX LITERAL EXISTS IN THE FILE, and that was a constraint not a nicety:
D-139 retires the ULA /48 for GUA, so a baked table would be right today and wrong
on carve day. plan derives from live MAAS and emits a SPEC; node consumes it and
knows no prefixes. A harness case drives the same green path with a ULA fixture
AND a GUA fixture, so the property is EXECUTED, not asserted.
>>> LATENT DEFECT IN D-139's OWN EXECUTION LIST <<<
D-139 names dc-node-v6-carve.py to "re-carve 54 node v6 statics". VERIFIED against
that script's source: it assigns v6 "on every plane WHERE IT ALREADY CARRIES
IPv4", picks the v6 subnet "on the SAME MAAS vlan as that v4 link", derives the
host part from "the last octet of the node's own v4 address", and its loop body is
a bare `if not v4: continue`. Under D-139 five planes lose v4 and lb-mgmt never
had a v4 twin, so after v4 removal it would SILENTLY CARVE FOUR FEWER PLANES PER
NODE rather than fail. The v6 carve MUST run BEFORE v4 removal, or the tool must
be rewritten -- compounding the already-recorded fact that D-139's "remove v4
LAST" clause is unsatisfiable. LOGGED, NOT FIXED. My own agent mandate stated this
tool's input source WRONGLY (apex/D-136; it is live MAAS); the agent checked
rather than inheriting the error, which is why the defect surfaced.
THREE MORE FINDINGS: (i) the multicast reading is ESTATE-WIDE -- all six plane
bridges plus the WAN bridge at snooping=1/querier=0 on dc0 AND dc1, and the
jumphost's uplinks too, i.e. the default here, not a chosen setting; the
subcommand tags it SUSPECT and asserts no Linux behaviour, correctly, since
cold-start multicast ND was already measured working across it. (ii) plan's PEER
SELECTION IS DEPLOYMENT-BLIND -- it picked control-01, which is not deployed, so
the derived SPEC cannot pass in a PARTIAL deployment; worked around by
substituting the live peer, logged not fixed. (iii) a harness bug caught by the
instrument-currency rule, NINTH of its kind: absent-ip/absent-virsh REFUSE cases
were driven with PATH=/usr/bin:/bin, which still contains both, so they were red
for the WRONG REASON.
TWO OPEN CHECKS SETTLED on the live nodes: every NIC carries exactly ONE global
(sole-global predicate holds today), and an OVS-internal br-ex does report
LOWER_UP (predicate safe).
GA-R2: the build agent wrote its own changelog; folded VERBATIM into this
session's single changelog and removed. Crossing midnight does not start a new
session. repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
RULED "B plus C": D-139 ruling A narrowed to an experiment + upstream fix
...
Operator, exact utterance: "B plus C". Ruling B's GUA carve is unaffected and
proceeds; ruling A's five-plane v6-only scope narrows to a bounded experiment;
metal-internal, data-tenant and lb-mgmt stay DUAL-STACK pending upstream fixes.
That last sentence is recorded as an INTERPRETATION, not as the utterance.
>>> I SCOPED THE EXPERIMENT WRONG AND THIS CORRECTS IT. <<<
MEASURED against shipped ceph-osd rev 953, ceph_hooks.py:544-553: prefer-ipv6 is
ONE switch that sets ms_bind_ipv4 = False GLOBALLY, so it governs Ceph's PUBLIC
network (storage) and CLUSTER network (replication) TOGETHER. There is no state
in which Ceph runs cluster on v6 and public on v4. I recommended "replication
alone, smallest blast radius" -- not achievable. The experiment is storage +
replication as a PAIR, with a larger blast radius than I represented.
A PATH THE EARLIER ANALYSIS MISSED: the buggy bare get_ipv6_addr() (LP #2061836,
dynamic/SLAAC-only, never fixed on ceph-osd, while our nodes hold MAAS statics) is
reached ONLY under `if not public_network:` / `if not cluster_network:`. Setting
ceph-public-network and ceph-cluster-network to the v6 CIDRs means it is never
used for address selection -- the defect becomes AVOIDABLE BY CONFIG. This is a
CODE READING, NOT A MEASUREMENT, and it does not address the get_mon_hosts()/
get_host_ip race (LP #2109798).
REVIEW PASS on netbox/d139-gua-carve.py (independent second agent, per operator
standing instruction). 4 real defects fixed, 4 reported, 1 charter miss upheld.
D1 is the mandate's own named class "a refusal that does not refuse": an
unreachable apex gave a raw traceback and EXIT 1 -- the tool's own header defines
exit 1 as a write error -- on a dry run that wrote nothing. _req caught only
HTTPError so URLError walked past the REFUSE handler. Fixed, REFUSE(2).
D2: a comment asserted ruling B was "FLAGGED" -- already false at commit time.
Replaced with a citation; prose in a script rots against governance state, a
citation cannot. D3/D4: two greps over the tool's OWN SOURCE and two vacuous
negatives replaced with behavioural coverage.
main() decomposition DECLINED on a reusable argument: the `if not a.commit:
return 0` guard is ADJACENT to the write block, and extracting commit_writes()
would split that safety proof in two. Line count is not the criterion.
R1 IS A REAL SECURITY FINDING, MEASURED: urllib carries the Authorization header
across a CROSS-HOST redirect (loopback probe -- the target received the token
verbatim). SANDBOX_HOSTS gates the supplied url, not a redirect from it. LOW
severity, logged not executed.
Harness 65/65 -> 71/71; all 5 original mutations RE-RUN and still standing, plus
3 new; gauntlet ALL GREEN (95); dry-run body diffed BYTE-FOR-BYTE identical to the
committed capture; apex 139 prefixes before and after. QUALIFIED: a dry run
executes zero POSTs, so D1's change to the WRITE path's failure semantics is
UNTESTED and is declared, not implied.
PROCESS FAILURE REPEATED AND OWNED: the review mandate required a docs/audit/
capture and forbade touching CURRENT-STATE.md -- jointly unsatisfiable against
L10. The agent correctly left the tree RED and said so rather than working around
it. Standing fix, now twice learned: an agent mandate must name a repo path for
findings AND leave the committer able to satisfy L10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 ruling A NOT achievable at these charm revisions; apex carve tool built
...
TWO agent results, both recorded; nothing actioned, nothing pushed to NetBox.
>>> RULING A (v6-only on five planes) IS BLOCKED BY UPSTREAM CHARM DEFECTS <<<
RULING B (full GUA) IS ORTHOGONAL AND STANDS -- GUA-vs-ULA is independent of
whether a plane keeps v4, so the carve tool is unaffected and a GUA DUAL-STACK
carve is achievable today. This needs an operator ruling.
RE-VERIFIED INDEPENDENTLY against the shipped ceph-osd rev 953 artifact, because
it decides the ruling. hooks/utils.py:203-217 get_host_ip() returns
get_ipv6_addr()[0] when prefer-ipv6 is set, else socket.inet_aton(hostname) and
on failure dns.resolver.query(hostname, 'A') -- an IPv4-ONLY lookup -- under the
charm's OWN comment "...just let it kill the hook". So ceph-osd fails on a
v6-only plane in BOTH directions: false -> inet_aton raises on a v6 literal then
NXDOMAINs an IN A lookup (LP #2109798, New); true -> bare get_ipv6_addr() (also
ceph_hooks.py:547, WITHOUT dynamic_only=False) returns DYNAMIC/SLAAC only while
every VR1 node holds a MAAS STATIC v6 (LP #2061836 -- Fix Committed on ceph-mon,
New and never fixed on ceph-osd, confirmed absent from rev 953).
Reported from the capture and NOT re-verified here, recorded as such: mysql /
mysql-router bare user:pw@addr URIs invalid for an unbracketed v6 literal;
hacluster 2.4/stable rev 166 (released 2026-06-29) still hard-writes
ip_version: ipv4 though LP #2111852 is Fix Committed -- FIX COMMITTED IS NOT FIX
RELEASED; OVN documenting encap as "The IPv4 address of the encapsulation tunnel
endpoint" with zero v6 encap values in OVN 24.03's test suite.
Counted locally: metal-internal appears 273 times in bundle.yaml across ~44 apps;
rabbitmq-server is DEPLOYED by the bundle but ABSENT from PREFER_IPV6_CHARMS --
the only mismatch across all 33 bundle pairs, where invariant 9a would emit a
factually false message.
APEX CARVE TOOL (netbox/d139-gua-carve.py, 232 lines) + tests/d139-gua-carve
(65 cases, offline stub apex); manifest 94->95. Live dry run: CREATE 22 |
EXISTS 6 untouched | RETIRE-REPORT 18, and it surfaced 52 DEPENDENT ip-addresses
inside the retiring ULA /64s that a later delete would orphan. Proof no write
occurred: apex prefix count 139 before and after. 5 mutations, each turning a
NAMED test red, tool restored sha256-identical.
OWED: (a) the foundational measurement is STILL not taken -- does network-get
return v6 on a v6-only bound space? (b) D-139's "remove v4 LAST, after each is
proven" is UNSATISFIABLE as written; per-plane conversion is atomic.
(c) the rabbitmq-server mismatch.
PROCESS FAILURE, OWNED: the research was written to /tmp and would have been LOST
-- I scoped the agent read-only and gave it no repo path. Rescued, sha256-verified.
STANDING CONSEQUENCE: every agent mandate must NAME A REPO PATH for its findings;
"read-only" constrains what it may CHANGE, not where it may RECORD.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
D-139 ADOPTED: VR1 goes IPv6-only east-west, whole carve moves to GUA
...
TWO GA-R5 rulings, taken in SEPARATE exchanges, recorded separately.
A -- family matrix. Operator: "IPv6 on all planes except for metal-admin and
provider-public which will remain dual stack". metal-internal, data-tenant,
storage, replication and lb-mgmt become IPv6-ONLY. Amends D-101's matrix twice:
metal-internal's "datastore east-west stays v4-bound" is superseded, and lb-mgmt
(ruled v6-only by D-101 but NEVER carved anywhere) becomes a first-class plane.
B -- addressing model. Operator: "Full GUA on every plane (Recommended)". Every
plane carves from its DC's GUA /48 on the :10/:11/:20/:21/:30/:40/:50/:80 octet
map VR0 DC0 and the Willamette site ALREADY use -- conforming to an existing org
standard, not inventing one. The ULA /48 fd50:840e:74e2::/48 is RETIRED for VR1.
Amends D-101 and D-111. Deciding reason is measurable: RFC 6724 ranks IPv4-mapped
at precedence 35 and ULA at 3 while GUA falls under ::/0 at 40, so on a dual-stack
plane a ULA leg LOSES address selection to IPv4 and is decorative. SEC-010, D-052,
D-125 and D-107 unchanged -- containment lives at the forwarding layer.
THE RECORDED ROOT CAUSE OF THE v4-ONLY CONTAINERS WAS WRONG AND THIS REPO CARRIED
IT (GA-R1 C2 -- measurement corrects the document). CURRENT-STATE said MAAS "has
nothing to give for v6". Measured, and re-verified independently: subnet
statistics on fd50:840e:74e2:220::/64 returns available_string "100%",
num_available 18446744069414584320. Zero ipranges rows means zero RESTRICTIONS,
not zero availability. dc-plane-ipam.sh:368-372 has carried the correct behaviour
since 2026-07-27, in the tree, contradicting the authoritative doc the whole time.
The real mechanism is juju-side and there is NO knob: EthernetDeviceForBridge
(tag v3.6.27) takes addrs[0] from an UNSORTED query and derives one CIDR -> one
LinkSubnetArgs. gomaasapi documents "Any number of STATIC links can exist on an
interface", so MAAS would accept both families; the limit is juju's. LP #1723240
is this exact symptom, Triaged/Low since 2017, and the theory this repo recorded
is the one its reporter rebutted in-thread. Consequence: on a container plane
DUAL-STACK IS NOT EXPRESSIBLE while v6-only IS -- ruling A is the achievable
configuration, not merely the desired one.
SNAP PROXY: dc0 INSTALLED, RUNNING, check dc0 PASS exit 0 (16 assertions,
re-verified independently). A REAL SNAP PAYLOAD WAS FETCHED THROUGH IT -- HTTP 206,
first bytes hsqs. No longer fixture-green. BUG-3 confirmed by measurement, no
assertion changed. BUG-4 fixed at BOTH sites (the review named one); T20b/T41b NEW.
OWNED: my BUG-1 fix was WRONG. ip route get 10.12.8.4 -> src 10.12.8.2; .4 is a
SECONDARY and Linux never auto-selects a secondary as a source, so the rule is
still dead. Not fixed now -- editing the generator turns the LIVE gate red until
install re-runs. T23b annotated so nobody "fixes" the test to a wrong value.
Generalisable: any ACL keyed to a service ALIAS (the whole D-134 utility .4 band)
misses that host's own traffic -- same exposure in dc-mirror.sh, dc-cache-proxy.sh.
OWED, unverified: the deployed jammy /etc/gai.conf, which can override the RFC 6724
default table that ruling B rests on. Nothing of D-139 is executed.
repo-lint 0 fail; tests/dc-snap-proxy 55/55; 4 mutations killed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-31 |
Snap proxy tool BUILT and REVIEWED (not yet applied); 4 real bugs logged
...
Ruling 2's repo side. squid 6.14 as a dedicated instance on each DC rack at the
D-134 utility .4:3129, CONNECT-only, restricted to Canonical's documented store
and CDN hosts, consumed via juju snap-http-proxy/snap-https-proxy. Design cited
to vendor sources, not preference: every snap endpoint is :443 so a CONNECT
proxy is mandatory and CACHING IS IMPOSSIBLE (squid cannot read a CONNECT
tunnel), which removes the apt-style caching argument entirely; squid is in
main, tinyproxy is universe with six unevaluated CVEs and defaults to allowing
all CONNECT ports.
MEASUREMENTS THAT CHANGED THE DESIGN:
- the failing apps are LXD containers sourcing from METAL-ADMIN (10.12.8.122),
NOT the 10.12.12.116 juju status displays. An ACL on the displayed address
would have denied every client the proxy exists for.
- ports 3128 AND 8000 are already held wildcard-bound by MAAS's own squid on
both racks -- hence 3129 and a dedicated instance.
- no new D-134 octet needed: .4 is already aliased on both racks.
- LOGGED NOT ADOPTED: MAAS's squid ALREADY CONNECT-proxies api.snapcraft.io,
measured working from the failing container. snap-https-proxy pointed at
10.12.8.6:8000 may unblock the deploy today with nothing built. Rejected as
the shipped design (per-revision path, NO destination restriction, the
hidden-coupling shape D-135 already has a scar from) but it is a real
operator option, not a dismissal.
QUALITY: 53 mutations, 53 killed, 0 survived -- including the dc-mirror.sh
existence-vs-content false-green class reproduced and caught. One survivor was
a MALFORMED mutation that broke bash -n and therefore tested nothing; recorded
as malformed and redone rather than counted. Three of the harness's own cases
were found too weak and were STRENGTHENED, not left green.
INDEPENDENT REVIEW: ruled design HONOURED on all four counts -- utility band, no
direct node egress (in fact stronger: destination-allowlisted, and that scope is
flagged as a build-time choice NOT a ruling), site-keyed both DCs with the dc1
limit stated honestly, and it OWNS its own network prerequisites rather than
borrowing another service's -- with the coupling anti-pattern now under test
(M09/M10) rather than only under comment.
FOUR REAL BUGS LOGGED, NOT FIXED (hard rule 1 -- the file was frozen for review):
BUG-1 dead snap_probe ACL; the probes are actually admitted by snap_clients,
and the comment misdescribes its own mechanism
BUG-2 | grep -q under pipefail -- INHERITED from dc-mirror.sh and
dc-cache-proxy.sh alike; a three-script sweep, not a local fix
BUG-3 the deny probe's 403 shape was measured from a PORT deny, never a
DSTDOMAIN deny. If squid answers differently the gate REFUSES
PERMANENTLY -- and a gate that cannot PASS blocks a GA-R6 close as hard
as one that cannot fail. Requires a one-time apply-time confirmation.
BUG-4 dpkg -s reports a deinstall-ok-config-files package as present
SIZE: 636 lines / 320 code vs dc-mirror 385/227 and dc-cache-proxy 396/200. The
reviewer's landing zone is ~535/~250 and it RECOMMENDS AGAINST forcing 385 --
two extra generators (75 lines) are measured-forced because the packaged squid
cannot be repointed, and further cuts would delete mutation-proven assertions.
~50 lines of the excess is header prose, which carries zero bug risk.
TWO BLOCKERS BEFORE THE PLAN IS APPLIED: the S8 consolidation invalidates
M32-M39/M42-M47/M49-M52 as evidence, so the mutation pass MUST be re-run; and
BUG-3 must be confirmed once against the built proxy before check is cited to
close anything.
NOTHING IS PROVEN END TO END -- no proxy installed, no snap fetched through one.
The harness green is FIXTURE green and says so.
repo-lint 0 fail; gauntlet ALL GREEN (94).
OWNED: my earlier teardown commit e57ad09 used 'git add -A' while the agent was
writing and swept its audit capture and CURRENT-STATE paragraph into a commit
about the model teardown. This commit is staged EXPLICITLY by path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Build the (b) ruling: prefer-ipv6 emitted for NO application; 9b re-pointed
...
Renderer: the emission condition is DISABLED, not deleted -- when the v6
completion work lands (allocatable ranges on the six v6 plane subnets, rack-side
v6 legs, a v6 default route, v6-reachable services) that one line turns it back
on, and PREFER_IPV6_CHARMS is still the correct set to gate it by.
Both overlays re-rendered: exactly 7 lines removed per DC and nothing else --
all 13 apps keep 6-leg dual-family vips at both DCs.
INVARIANT 9b RE-POINTED, not deleted: it required the option and the v6 legs to
travel together on a declaring charm (the L3-9 defect), which would now fail on
the ruled artifact. The rule is the option must be ABSENT everywhere. L3-9 cannot
recur while it is absent -- there is nothing to keep while legs are dropped. It
returns to the coupling form when v6 is operational.
HARNESS re-pointed rather than deleted, and three stale fixtures fixed:
- T20 was 'option without v6 legs FAILS'; now 'option on a DECLARING charm FAILS'
- T21 was the inverse; now the POSITIVE CONTROL -- v6 legs WITHOUT the option
must PASS, so 9b cannot be satisfied by rejecting everything
- T22/T23/T30 each set prefer-ipv6 because the OLD coupling required it
alongside v6 legs. Under the absence rule 9b fires FIRST and short-circuits,
so their v6-band assertions would never have been reached and they would have
passed for the wrong reason. The option is removed from those fixtures.
Mutation-proven: disabling the 9b absence check kills T20; re-enabling renderer
emission kills render-drift. Source restored byte-identical after each.
Gauntlet ALL GREEN (93); repo-lint 0 fail; harness 55/55.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Build the D-135 amendment: jammy-backports at source, T10 re-pointed
...
scripts/dc-mirror.sh's generated sync script gains jammy-backports in --dist=,
with a comment at the generator recording why it is not optional: the MAAS
jammy image ships backports in its sources.list and juju's apt-mirror
model-config rewrites every suite to this mirror, so a mirror without it 404s
and every charm install hook fails. That cost 22 units today.
FIXED AT SOURCE, not only on the live rack -- a live-only edit would leave the
generator regenerating the old scope at the next install or at any future DC
standup.
tests/dc-mirror T10 asserted the old dist string verbatim. RE-POINTED to the
new invariant with the reason recorded in-file, never deleted to go green.
Mutation-proven: reverting the generator's dist line turns T10 red (23/1);
restored byte-identical, 24/24.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Option D half 1: explicit base on all 56 apps, gated by new invariant 12
...
PROVEN AGAINST THE REAL MODEL, NOT INFERRED. The earlier one-app test only
showed the error MOVING to the next app. The fully-based bundle was staged as a
throwaway on the rack and --dry-run against the LIVE 23-application model --
the same command that errored on barbican 265-vs-261 -- exits 0.
Subordinates are in scope, measured: mysql-router is 1154 at 22.04 and 1178 at
24.04. hacluster sharing rev 166 across both bases is not a property to depend
on.
TWO SILENT UNDER-MATCHES, both caught by cross-checking the edit against the
parsed file rather than by review:
(1) a first pass keyed on the block form inserted 44 of 56 -- the twelve
-hacluster apps are single-line FLOW MAPPINGS. The inserter asserts its
app list equals what yaml.safe_load sees, and refuses otherwise.
(2) overlays/dc-ha-scaleup.yaml DEFINES vault-hacluster, an application that
exists nowhere in bundle.yaml, and it had no base. That overlay is a LATER
deploy step, so this would have re-opened the trap AFTER the deploy
succeeded. Found by the new gate on its FIRST run, in a file the fix was
not looking at -- the same later-step-overlay class as the ceph-mon miss
earlier today.
INVARIANT 12 (NEW): every application carries an explicit base equal to the
bundle's OWN default-base -- keyed on that value, not a literal. A bundle with
no default-base REFUSES rather than passing vacuously. Without this, deleting
the 56 lines is a byte-identical PASS and the trap returns at the next partial
failure, at any DC.
Five cases, each mutation-proven individually (harness 50 -> 55/55): neutering
nobase kills T49/T50/T52, wrongbase kills T51, the refuse branch kills T53.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Fix a defect in my own build: invariant 9a was VIP-scoped and the charm list was short
...
Found by a repo-wide grep, not by any gate. Two faults, one cause -- I
measured and gated the population the QUESTION named (the 13 VIP charms)
rather than the population the INVARIANT covers (every application juju
validates).
(1) PREFER_IPV6_CHARMS was WRONG. Re-measured across ALL 33 charms in
bundle.yaml: TWELVE declare the option, not seven. The five missing are
ceph-mon 491, ceph-osd 953 (squid/stable), nova-compute 894, hacluster 166
(2.4/stable) and mysql-innodb-cluster 164 (8.0/stable).
(2) 9a was written INSIDE the VIP loop, so an application with no vip never
reached it -- and the repo ALREADY has that case:
overlays/dc-dc-ipv6-family-matrix.yaml sets prefer-ipv6 on ceph-mon, which
carries no VIP. That overlay is a LATER deploy step, so the miss would have
passed attempt 2 cleanly and surfaced at the step after, looking like a new
fault.
9a now runs over EVERY application, before the VIP loop; the VIP loop skips an
app 9a already flagged so the diagnosis is not doubled. ceph-mon's own in-file
claim of the option is now INDEPENDENTLY VERIFIED from the artifact rather
than taken from the comment.
Harness 48 -> 50/50. T47 (non-VIP app whose charm lacks the option FAILS) and
T48 (ceph-mon PASSES) both mutation-proven: re-scoping 9a back to the VIP loop
kills T47; dropping ceph-mon from the list kills T48.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Build the 2026-07-31 ruling: renderer + invariant 9 replaced, overlays re-rendered
...
Three parts, all mutation-proven.
(i) provider-bundle-check.py gains PREFER_IPV6_CHARMS -- the measured authority,
keyed by CHARM name, carrying the revisions and a re-measure warning -- and
invariant 9 is REPLACED, never deleted:
9a prefer-ipv6 on a charm that does NOT declare it FAILS. Asserted on
PRESENCE, not truthiness: 'prefer-ipv6: false' is the same fatal
'unknown option' to juju as true.
9b on a DECLARING charm it still travels with the v6 legs -- the L3-9
protection is retained unchanged for those seven.
9c arity.
The old text's mechanism claim ("prefer-ipv6 makes HAProxy bind :::port") is
refuted by this session's measurement and is recorded as such in-file.
(ii) render-dc-overlays.py emits the option only for a declaring charm, READING
the set from provider-bundle-check with ast rather than restating it (the same
rule APP_OCTET already uses). render() keeps its purity property: the set is a
PARAMETER and the default REFUSES rather than defaulting to empty, because an
empty set renders a plausible artifact with the option nowhere. Both overlays
re-rendered: exactly SIX lines removed per DC and nothing else -- every vip
string byte-identical, so all v6 legs are retained.
(iii) tests/provider-bundle-check 44 -> 48/48. T43 reproduces the attempt-1
defect; T44 proves presence-not-truthiness; T45 is the positive control; T46
asserts the DIAGNOSIS.
MUTATION PASS -- six mutations, every one killed tests. And one NEW assertion
was found to be DECORATION and was REPLACED rather than kept: T46's first form
could not fail under any mutation, because a non-declaring charm with no option
reaches neither branch. Re-written to catch the fold-9a-into-9b simplification
and re-proven.
Gauntlet ALL GREEN (93); repo-lint 0 fail. The fixed dc0 deploy input now
sweeps CLEAN on option names: 75 assignments, 20 schemas, 0 unknown.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-node-carve: convert the commissioning auto link, found by a live dry-run
...
The first live plan refused enp1s0 on every machine: it holds an `auto` link on
metal-admin out of enlistment, and the "wrong address on the right subnet is a
FAILURE" rule was too broad. A commissioning link is the expected starting
state, not a conflict -- so the tool could not carve the metal-admin leg at all.
Narrowed BY MODE: auto/dhcp/link_up gets unlinked and re-created as STATIC
(MAAS will not convert a link's mode in place); a `static` link with the wrong
address is still refused, because something deliberate put it there.
The `raw` fixture modelled the auto link correctly -- what was missing was any
assertion that `apply` over the raw state EXITS 0. T21-T24 only grepped for
planned strings, so a refusal on the very first NIC read as green. T26/T27 now
cover both directions and both are mutation-proven.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-30 |
dc-node-carve.sh: the v4 node interface carve, mutation-proven 40/40
...
The last no-tool gap between the dc0 region migration and the Juju deploy:
60 NIC re-homes, 9 OVS br-ex bridges, 54 statics. The Stage-4 equivalent was
done ad-hoc and logged outside the repo, so there was nothing to re-run when
D-132 q1 forced a re-enrolment.
Proved the four-call MAAS cycle on ONE live interface (canary 6q4syf enp3s0)
before writing the tool -- nothing had exercised `interface update vlan=` on a
link_up-only interface on an auto fabric, which is the state all 60 NICs are in.
Measured in passing: link-subnet REPLACES a link_up link, so only the br-ex
member needs an explicit unlink.
No default profile; maas-profile-assert.sh before any read or write; every id
resolved live by stable key (boot MAC / CIDR / interface name); every mutation
read back and compared. Role asymmetry derived from the host token, and the
controller's deliberately-uncarved NICs are asserted ABSENT.
Mutation pass found the br-ex static compare could not fail -- Pattern B was
being caught by the type/parent assertions instead. Added the brexip fixture
and re-proved the kill. Also found that neutering an assertion's MESSAGE only
proves it exists, not that its predicate works.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
lib-hosts: the node carve is ROLE-DEPENDENT, not uniform
...
Measured from the pre-migration capture. Role nodes carve all six planes
plus an OVS br-ex parented on enp2s0 carrying the provider-public static,
with enp2s0 itself holding no L3 link. The juju controller carves only TWO
planes -- enp1s0 metal-admin and enp2s0 provider-public as a raw NIC WITH
the gateway -- has no br-ex, and leaves enp3s0..enp6s0 on their
auto-created VLANs.
Evidence: 7n87bt (controller) reads enp3-6 on vlans 5200-5203, which are
auto fabrics; 8mtpxq (storage-02) reads enp3-6 on 5190-5193, the named
planes, plus br-ex.
This matters because assuming uniformity fails in both directions: it would
either strand the controller without a default route -- the under-carve that
cost three bootstrap attempts on 2026-07-30 -- or give it four legs it must
not have. The node carve has no repo tool yet, so this was about to be
re-derived by hand.
dc-selector 95 checks; the depths-must-differ guard proven able to fail, and
the controller-presence assertion checked inside a per-DC subshell because
the main shell still holds the flat VR0 fleet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
lib-hosts: record the MEASURED VR1 node NIC -> plane order
...
The substrate roots pin six MACs per node and the module attaches them in
list order, so macs[i] is enp<i+1>s0. Verified end to end on
vr1-dc0-control-01 against the live carve: enp1s0 metal-admin (PXE),
enp2s0 provider-public (br-ex parent), enp3s0 metal-internal, enp4s0
data-tenant, enp5s0 storage, enp6s0 replication.
This is NOT lib-net.sh's PLANE_CIDRS order, which starts with
provider-public. Anything walking PLANE_CIDRS positionally to place NICs
swaps enp1s0 and enp2s0 and puts the PXE leg on provider-public, stranding
commissioning on a plane with no DHCP. The node carve has no repo tool yet,
so this ordering was about to be re-derived by hand; recording it measured,
with a guard that the two orders differ, is the cheap half of that tool.
dc-selector 88 checks; the trap guard proven able to fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-node-v6-carve.py: honour MAAS_PROFILE; refuse instead of tracebacking
...
The script was env-blind -- default 'admin', --profile only, no environment
read anywhere. Every bash tool here honours MAAS_PROFILE, so
`export MAAS_PROFILE=vr1-dc0-region` left THIS tool silently targeting the
Office1 region. During the per-DC region migration that is the
unrecoverable direction: against the wrong region a carve is an idempotent
no-op that prints PASS while the region you meant stays untouched.
It now reads MAAS_PROFILE, keeps --profile as an override, and prints which
region it targets and where that came from -- a capture that does not name
the profile cannot be audited for wrong-region work.
An absent maas CLI raised FileNotFoundError and exited 1 with a traceback,
indistinguishable from a real failure and contrary to this script's own
REFUSE contract. It now refuses with exit 2.
Harness 14/14 (was 9); both fixes proven able to fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-region-topology: RENAME the auto fabric instead of moving the subnet
...
Moving a subnet between fabrics does NOT bring interface links along. The
first apply created vr1-dc0-provider-public and moved 10.12.4.0/22 onto it,
leaving the region VM's enp2s0 holding a static 10.12.4.6 on a VLAN whose
subnet had left -- the region's provider-public leg, which it needs for
image sync, and the same under-carve class that cost three bootstrap
attempts on the Juju controller. Live connectivity was unaffected because
the OS netplan is independent of MAAS's model, so every other assertion
passed 39/39 with the model inconsistent.
Renaming the auto fabric that already carries the subnet moves nothing. A
MOVE is kept only for the case where the fabric carries other planes, and
now prints a warning that links do not follow.
Adds the assertion that would have caught it: every interface link must be
on its subnet's VLAN. Harness 47/47; the rename path and the stranded-link
check each proven able to fail.
Live remediation already applied to the dc0 region: subnet returned to
vlan 5002, the wrongly-created fabric deleted, fabric-1 renamed, space
re-bound. enp2s0 now reads fabric=vr1-dc0-provider-public with its static
intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-region-topology: explicit dry-run placeholders instead of empty ids
...
An empty `vlan=` in a dry run reads like a defect and would hide a real
one. On a from-scratch build the target fabric does not exist yet when the
MOVE and space-bind lines are planned, so their ids cannot resolve; they
now print <vlan-of-NAME> / <fabric-for-PLANE>. Four assertions added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
dc-region-topology.sh: the plane-topology builder the repo never had
...
A read-only survey found THREE dc0 carve components with no repo tool at
all -- the named plane fabrics, the six v4 plane subnets, and the site
placement tag were created ad-hoc in the Stage-4 window, with the mutations
landing only in ~/as-executed/2026-07-23-stage4-carve.log, which is not in
the repo. When D-132 q1 required rebuilding that carve on a new region
there was nothing to re-run. This closes the gap so dc1 and every Roosevelt
DC standup rebuild from a tool rather than a transcript.
check/apply over: five named plane fabrics, six spaces, six v4 plane
subnets on the right VLANs with the ruled gateways, VLAN->space bindings,
and the openstack-<site> tag. Expected values derive from lib-net.sh;
fabric names derive from the site token. Numeric MAAS ids are resolved
per-run by name or CIDR and never persisted -- the pre-migration capture's
id= values are database row ids, not 802.1Q tags, and nothing reads them.
metal-admin deliberately keeps MAAS's own auto-created fabric: it is a
discovery artifact created when the rack registers, Juju binds spaces
rather than fabrics, and renaming it would fight MAAS for ownership of its
own object for no functional gain. The check asserts only that it does not
share a fabric with a named plane.
Every mutating path is gated by maas-profile-assert.sh first -- against
the wrong profile this apply is an idempotent no-op that prints PASS.
--profile has NO default, because the repo-wide default 'admin' resolves
to the Office1 region.
Harness 35/35; 4 mutations each killed tests (region gate removed, profile
defaulted to admin, space-binding assertion, fabric-placement assertion).
Fixtures corrected against a live read: MAAS reports an unbound VLAN as
the string "undefined", not null, so the first fixtures graded a payload
MAAS never emits.
Live read-only check against the new dc0 region: 5 passed, 22 failed,
exit 1 -- the expected shape of a region that has only auto-discovered its
own two NICs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Per-region power address, region power-key tool, and two red gates fixed
...
lib-hosts.sh: MAAS power ops originate from the REGION, so the reachable
rack address depends on which region dials. Measured from vr1-dc0-maas-01:
the rack's transit leg 172.31.0.2:22 is CLOSED, its metal-admin leg
10.12.8.2:22 is OPEN. Adds VIRSH_POWER_ADDRESS_FROM_{OFFICE1,DCREGION} for
both DCs (dc1's 10.12.68.2 measured on the dc1 rack, not inferred from
dc0's .2). VIRSH_POWER_ADDRESS still aliases the Office1 form so existing
callers are byte-for-byte unchanged. dc-selector 81 checks; 4 mutations
each killed tests.
New scripts/maas-region-power-key.sh (harness 44/44) installs and verifies
the per-DC MAAS->libvirt key inside a region's snap, ending on a real virsh
call rather than on config. Runbook prerequisite 5 carried this as prose,
and prose is not a gate -- it must also be re-asserted after every snap
refresh. LIVE at dc0: 9/9 exit 0, virsh enumerates 12 domains. The key is
the SEC-012 dc0 key already authorized on that rack, so per-DC isolation
holds.
Three defects its own testing found: ssh-keygen -lf reads an adjacent .pub
and reports ITS fingerprint; a snap's data root (/var/snap) and exec root
(/snap) are not interchangeable; the snap's virsh must run inside the snap
runtime or it fails on missing libraries.
tests/node-vm T8/T9 were RED at HEAD before any of this session's work
(proven by stashing and re-running): commits 086c827 + 447315f added the
region VMs, moving the dc0 root to 11 macs lists / 66 MAC literals against
assertions reading 10 / 60, and the gauntlet was not re-run. Re-pointed to
the new invariant, reason recorded in-file, re-proven able to fail.
HARNESS-MANIFEST re-recorded (91) for the two new harnesses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Region-resolution gate: maas-profile-assert.sh + named dc0 region profile
...
The wrong-REGION hazard is the migration's unrecoverable failure mode: 13
repo scripts default to MAAS_PROFILE=admin, which on voffice1 resolved to
the OFFICE1 region. Against the wrong profile a carve recreate is a silent
no-op that prints PASS, and machine delete destroys the real record.
New scripts/maas-profile-assert.sh asserts which region a profile resolves
to by rack-controller IDENTITY (a machine count is not proof -- two regions
can hold the same number). Harness 20/20; mutation pass found one of its own
assertions could not fail (nameless rack entries were caught by a different
branch) and it was made branch-specific.
Live both directions: admin -> 23 machines / office1 trio (exit 0);
vr1-dc0-region -> 0 machines / hot-kid (exit 0); admin asserted as hot-kid
(exit 1). Capture docs/audit/dc0-region-profile-assert-20260730.txt.
Tunnel voffice1 -> rack -> 10.12.8.6:5240 because no single host has repo +
CLI + new-region reach + NetBox apex reach. Application-layer and
rack-originated, so SEC-010 is not punctured; nothing was rerouted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
lib-identity.sh: the estate's name and DNS base in ONE place
...
Operator: 'The specific server and region names do not matter as they change
during every deployment and hardening of the deployment workflow.' Then, after
scoping the two options: 'Yes, take option 2.'
WITHDREW MY OWN FIRST RECOMMENDATION. I proposed retiring the vr0-dc0 selector
as removing 'the ambiguity class outright'. Measured, wrong on both halves:
D-119 already closed it by region-qualifying (the hazard was the BARE dc0, which
is rejected loudly today); lib-net.sh:110 documents that arm as a NO-OP over the
file's flat defaults; and the footprint is 43 files, mostly the separate VR0
phase-NN track the VR1 runbooks cite as precedent. Medium cost, near-zero
benefit. Logged, not executed.
NEW scripts/lib-identity.sh -- CLOUD_NAME + CLOUD_DOMAIN, no side effects,
env-overridable. With <dc> and <region> already derived from the site token,
these were the LAST typed identity in the shell surface. A rebuild that renames
the estate now edits one file and the certs, zones, SANs and P7 all follow.
DELIBERATELY NOT lib-net.sh: sourcing that bare populates a full flat plane/VIP
namespace (its own header says so). A certificate checker needs two strings, not
a network namespace, and octavia-pki.sh sources NOTHING today -- pulling in
VR0-shaped defaults it never asked for is the R9 hazard in miniature. Sourced
from SCRIPT_DIR (sibling, survives the harness REPO override) and FAILS CLOSED
with REFUSE 3 if absent: a cert gate that invents the estate's identity is worse
than none.
Consumers: octavia-pki.sh derive_zone(), phase-01 1.0-GEN.c, dc-dc-phase6 Step 0.
The tofu side was ALREADY parameterised (opentofu/variables.tf domain_suffix
passed explicitly into modules/dc-planes), so nothing there changed.
HARNESS 48/48 -> 51/51:
- T45 the centralisation must be REAL, not decorative -- overrides both values
and requires the zone to follow BOTH (acme.dc0.vr1.example.test). A constant
nothing can vary is indistinguishable from a literal.
- T46 a missing lib-identity.sh REFUSES rather than guessing.
- T47 SHELL vs OPENTOFU drift. HCL cannot source a shell file, so two copies of
one fact exist; if they part, the libvirt plane domains and the cert zones
describe different estates and nothing else would notice. PROVEN able to fail
by injecting a drifted value, then restored.
Also: repo-lint L6 flagged '. $REPO/scripts/...' as a bare invocation. Its own
docstring blesses a bash|source|python3 prefix and sourcing needs no exec bit,
so the RUNBOOK moved to the documented 'source' form -- the gate was not changed
for style. And ${REPO:?} is now guarded at its FIRST use in GEN.c, not only at
:538 below it.
STILL typed, deliberately: the site allowlists. One that accepts anything is how
a typo'd DC name becomes a wrong-target write.
gauntlet ALL GREEN (89); repo-lint 0 fail; GEN.c bash -n clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
P7: the Octavia amphora PKI becomes an actual pre-deploy gate
...
Operator: 'Yes, I accept the recommendation. Convert to a gate.'
THE GAP: octavia-pki.sh was invoked by NOTHING executable. P4's CHECK 0 asserts
only that the overlay FILE EXISTS, and the harness runs against throwaway
fixtures -- so 'gauntlet ALL GREEN' said nothing about the live PKI and the only
real check ran when a human remembered to type it. That is how F9 survived.
Three problems solved, each from a measured precedent:
1. rc 3 must not reach note()'s *) arm. octavia-pki.sh uses exit 3 for REFUSE,
but note() calls any rc>=3 'UNEXPECTED exit code'. P7 maps rc explicitly, so
a REFUSE on the headend FAILS and is NAMED as an unevaluated trust domain.
2. HOST AUTHORITY. The PKI lives on the headend (D-109 (b)); elsewhere P7 says
NOT EVALUATED (WARN) -- never a silent pass, never a hard FAIL for being on
the jumphost. The binding is read from the SAME host-identity file
octavia-pki.sh reads, so they cannot drift. P5's F6 defect designed out.
3. A PASS VERDICT IS NOT SUFFICIENT. While os-public-hostname is unset, A12/A13
RECORD a wrong-zone cert rather than failing it, so verify exits 0 on exactly
the defect this gate exists to catch. P7 also requires the literal line
'DNS SANs are all in this DC.s expected zone'. Satisfiable by construction
now that GEN.c derives the zone and both DCs were reissued.
Also skips cleanly for vr0-dc0 (a gate that fires on a target it does not apply
to is how gates get bypassed) and FAILS CLOSED if the checker is absent.
HARNESS 26/26 -> 33/33. Ten existing cases went RED when P7 landed -- correctly,
the fixtures had no octavia-pki.sh. Fixed by making mkfix model a healthy headend,
NOT by patching cases one by one: that is the moment a gate gets demoted to a
warning to go green. T27 is the case that matters -- a wrong-zone cert on which
verify exits 0 must still FAIL P7.
Two harness bugs of my own, caught by running not reading: a stub emitted DC.s
where the real line has an apostrophe, then \x27\x27\x27 produced three of them
because \x27 in a printf FORMAT already yields one. Fixed the stub, not the gate.
Preflight's overall verdict is still FAIL -- that is P5 working as designed on
the pre-existing red register. P7 itself passes on the headend.
gauntlet ALL GREEN (89); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
octavia-pki: a hardened, tool-driven controller-cert REISSUE path (F9's remedy)
...
Operator: 'Let's reissue now. Engineer the artifact, variable, based workflow to
reduce the chance of errors.' Five GA-R5 rulings, one exchange each, quoted in
the changelog; ruling 5 was 'Full script minter now'.
THE TOOL IS BUILT; THE MINT IS NOT YET RUN (gated operator action, 1.0-REISSUE).
scripts/octavia-pki.sh gains 'reissue <site> [--force] [--dry-run]': controller
LEAF only, signed by the EXISTING controller CA (neither CA regenerated, amphora
trust domain untouched), fresh P-256 key, zone DERIVED from the site token and
never typed, printf-built config (the F8 payoff), stage-assert-promote, and a
single-value overlay surgery that never puts the bundle in argv.
SIX GATE DEFECTS FOUND BY VALIDATION AGENTS, ALL CLOSED. Mutation testing is
what earned this: deleting the three assertions I had just written left the
harness fully GREEN every time, so they were decoration until each got a
failing-direction fixture.
- A15 keyUsage/EKU on the ISSUED cert. A cert with NEITHER minted, promoted and
passed everything. keyUsage appeared once in the file: the printf writing it.
- A16 validity. openssl x509 -req defaults to -days 30, and checkend/notAfter/
enddate appeared ZERO times in verify -- a 30-day cert passed forever.
- A17 the overlay must decode to what the workspace holds. A10 graded shape,
A1-A16 graded the workspace, nothing joined them; a desynced overlay read
PASS 0-failed and the overlay is what reaches the charm.
- The re-run guard REFUSED to fix broken certs: names-correct + stale IP SAN
gave verify FAIL and reissue exit 4 'nothing to fix'. Now gated on verify.
- A8's negative proves its instrument loads first (openssl verify exits non-zero
for 'could not load the CA' too, so a corrupt CA passed it vacuously).
- Two pre-existing overlay defects moved pre-mint, so they stop producing a
false exit-5 with the workspace already promoted.
Harness 23/23 -> 48/48. Runbook Step 1.0-REISSUE appended AFTER GEN.e
deliberately (F10 line-anchored mint-refs; creds-matrix reports no S4 findings
after the append). Skill gains two standing invariants for future DC standups:
every per-DC secret needs a ROTATION tool not just a generation recipe -- a
REFUSE-IF-PRESENT generation gate makes the dangerous path the only path -- and
a hand-pasted openssl chain is a defect class, with F8 and F9 as the receipts.
Also corrected: D-137 fork 1 is NOT open (sub-ruling 1, 2026-07-25, ruled
'Blocking in preflight' and explicitly declined the PreToolUse guard), so the
queued Q2 premise is stale in the same way Q1 was.
gauntlet ALL GREEN (89); repo-lint 0 fail; creds-matrix 5 findings, all
pre-existing and named; ledger-scan unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DOCFIX-205: Q1 withdrawn -- D-117 ruled it 2026-07-13; annotate the four it owed
...
The operator asked to rule on queued Q1 (is substrate vr1-dc1 'dc1' by token or
'dc2' by position, for cert/DNS identity). Measured before presenting options.
The question was WITHDRAWN, not ruled: D-117 (ADOPTED 2026-07-13, four days after
D-106) names the supersession in its own Status line -- "Supersedes ... the D-106
dc1/dc2 zone labels" -- and rules the replacement: "The repo's dc1/dc2 labels are
retired in favour of dc0/dc1". vr1-dc0 -> dc0, vr1-dc1 -> dc1.
WHY IT RESURFACED, and this is the reusable half: D-117 ruled that the ADOPTED
decision texts D-101/D-106/D-111/D-115 be ANNOTATED in place. Measured: ZERO of
the four carried any D-117 annotation. It stayed invisible because D-117's OWN
Status line claimed "FULLY EXECUTED BY D-119", while D-119 scopes its discharge
to the SELECTOR half only. A reader checking whether the annotation was owed was
told it was already done. Ruled is not built -- check the artifact.
Landed (operator-approved batch, "All four + code fixes"):
- All four decisions annotated in place, per D-117's own ruled treatment. D-101
gets the hardest one: it carries BOTH namespaces, so it is annotated BY DATE
(dc1 means different datacenters in its two halves -- D-117 TRAP 2).
- D-117's Status line corrected by measurement (GA-R1/C2) to "EXECUTED IN TWO
HALVES", with the standing lesson that a Status line is a CLAIM about
execution, not evidence of it.
- phase-6 EXECUTABLE DEFECT fixed: os-public-hostname was built as
"keystone.omega.${DC}.vr1..." with $DC the D-119 selector, expanding to
omega.vr1-dc0.vr1... -- the region TWICE -- and would have been baked into
Vault-issued SANs at Stage 7. Both labels are now DERIVED from the site token
(${DC%%-*} / ${DC#*-}); verified by running it, not by reading it.
- octavia-pki.sh A12: refusal -> assertion on the full derived zone; a SAN-less
cert now also fails. Harness 21/21 -> 23/23, with T21 REPLACED (not deleted)
and new T21b (the other DC's label in the right region FAILS -- the cross-DC
mix-up region-only checking passed) and T21c (per-DC derivation proven).
- CURRENT-STATE + ledger corrected in the same commit (L10). "BLOCKING" was
wrong as stated: A12 measures INERT and PASSES live on both DCs. F9's reissue
obligation stands.
gauntlet ALL GREEN (89); repo-lint 0 fail; ledger-scan unchanged (3 decisions,
21 SEC, D 138 / BUNDLEFIX 053; DOCFIX 205 -> next-free 206).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
voffice1 reconcile: a gate that was RED on the only host that deploys
...
Operator asked whether the headend clone was fully reconciled. It was not, and running the
gauntlet there -- which had never actually been done -- found a real host-dependent gate defect.
RECONCILE. The headend was 1 commit behind with 0 local commits, now at HEAD, tfstate sha256
byte-identical before and after. No stale remote-tracking refs. One asymmetry found and fixed:
opentofu/vr1-dc1-substrate/.terraform.lock.hcl was UNTRACKED and not gitignored, while dc0's
equivalent IS tracked. Same provider version (0.9.8), so a content divergence was ruled out by
measurement, but a fresh clone could not reproduce dc1's provider resolution the way it can
dc0's. Landed, byte-identical to the headend copy.
THE GATE DEFECT. The gauntlet read ALL GREEN (89) on the jumphost and 1/89 FAILED on the headend,
on opentofu-validate. Cause: `tofu fmt -recursive` walks EVERY .tf/.tfvars under the tree,
including GITIGNORED local files. The sole flagged file was
opentofu/vr1-dc0-substrate/d124-inner.auto.tfvars -- gitignored operator input that exists only
on the host which runs the applies. So the gate was RED ON THE ONLY HOST THAT DEPLOYS and green
where it did not matter, the same host-dependent-verdict class as the LC_ALL manifest sort.
Ruled out first: both hosts run tofu 1.12.4, so this was not version drift.
The fmt check is now scoped to repo content, and a gitignored file is SKIPPED with the reason
printed rather than silently. Proven both ways: a badly-formatted TRACKED file still FAILS (T16),
a gitignored local tfvars is skipped and does not fail (T17). T17 also fails loudly if the ignore
rule ever stops covering per-DC tfvars, so the case cannot silently stop testing what it claims.
Owned: my first T16 read as a failure because `bash "$SCRIPT" | grep -q` inherits the gate's
correct exit 1 under `set -o pipefail`, even though grep matched -- a passing assertion reading as
a failure. Same strict-bash family as the IFS and inherit_errexit traps this repo has hit before.
Captured first, then grepped.
opentofu-validate harness 16/16; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
A12: the DNS SAN check ARMS ITSELF, so F9 cannot be missed
...
Operator direction: make the DNS SANs part of the process so it is not a step that gets missed
in future.
The risk in F9 is not the defect, it is that a future session forgets. So A12 arms from the
change that makes the SANs load-bearing -- os-public-hostname appearing as a real option key in
the merged deploy input -- rather than from a note somebody has to read. Unarmed, it states the
names are inert and why; armed, it asserts them.
WHAT IT CAN ASSERT, AND A GAP IN A RULED DECISION THAT IT CANNOT. D-008 fixes the shape
<service>.<cloud>.<dc>.<region>.cloud.neumatrix.local and D-106:2563 instantiates VR1 as
"omega.dc1.vr1" / "dc2.vr1" -- while the substrate's DCs are vr1-dc0 and vr1-dc1. So vr1-dc1 is
"dc1" by token or "dc2" by position: the DC1/DC2 ambiguity item 3.1 retired elsewhere, surviving
inside a ruling, where it decides certificate identity. The REGION is unambiguous and is
asserted. The DC LABEL REFUSES pending a ruling rather than blessing a name that cannot be
validated. Both live certs currently say dc0.vr0 -- the VR0 region, wrong for a VR1 DC under
either reading.
A PRECEDENCE BUG THE HARNESS CAUGHT. Because the armed branch always refuses on the label, a
definite wrong-region SAN was being reported as "could not evaluate" -- REFUSE was checked before
FAIL and masked a confirmed defect. A known defect outranks an unevaluated item, so the verdict
now reports FAIL first and still names the refusal, which is never lost.
Harness 21/21 (was 18). T19 proves the unarmed live posture still PASSES -- without it, arming
A12 would have broken every current verify, the same trap T17 guards for the deleted
intermediates. T20 is F9 armed. T21 proves the right region still refuses on the unruled label.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
tfstate backup path built; E2/E3 closed live; a gate disagreement found and fixed
...
BACKUP PROVEN BY RESTORE, not by existence. Both PKI archives were extracted and compared
byte-for-byte against the live tree: 12/12 sha256 MATCH across both DCs, covering both encrypted
CA keys, both passphrases, both CA serials and both controller bundles. Archives are 600 inside
700 folders and differ between DCs, so per-DC independence survives into the backup.
E2/E3 CLOSED LIVE. Operator ran the chmod and the intermediate removal; octavia-pki.sh verify
now reads PASS 26/0 on BOTH DCs, workspaces 12 -> 10 files, and P5 on the headend fell 18 -> 8.
A GATE DISAGREEMENT, recorded as the pattern rather than the incident. verify read 26/0 while
creds-matrix E2 still reported the CA serial mode 664 on both DCs. The chmod list predated the
serial being declared, and verify's mode assertions covered six private files and three certs --
the serial was in neither. Two gates that can both see a file must not disagree about it. Fixed
in three places: verify now mode-checks the serial (integrity, not secrecy -- rewriting it forces
the next issuance to reuse a serial), the generator's chmod includes it, and T18 pins it. The
harness then caught its own fixture creating the serial at the inherited umask; baseline went red
until the fixture matched the corrected generator. 18/18.
TFSTATE BACKUP as dc-dc-phase2 step 13, 2 rows, notes key n-tfstate-backup. Register 99 rows;
schema, render-drift, mint-ref and notes checks clean.
Three deliberate differences from the PKI backup: tfstate is DYNAMIC so the serial is recorded
both sides and the step must be re-run after every apply (restoring a stale state is actively
dangerous -- tofu then believes everything created since does not exist); the pull block proves
RESTORE by decompressing to check the archive parses as JSON and reports a serial, because a
truncated gzip passes a size check; and mint-stage is stage3, which is REACHED, so P5 correctly
reports these EXPECTED-BUT-ABSENT until the step runs rather than deferring them.
Classification recorded honestly: the outer state is credential-bearing (DOCFIX-175, MAAS API key
in plaintext); the inner states show no credential-shaped key names, consistent with the inner
root using only libvirt over qemu+ssh. Strong evidence, NOT proof -- a name-based absence cannot
exclude material inside a value -- so they are registered and stored as if sensitive. Uncertainty
resolves toward more auditing, and an unaudited backup directory beside audited ones is how the
SEC-022 shadow stores happened.
Two of my own defects caught by the gates: repo-lint L9 rejected a hardcoded clone name in the
new step (the repo has been renamed once, D-110) -- now $REPO; and two mint-refs had drifted when
a later edit added lines above their anchors, which S4 cannot catch because it only checks
line <= EOF. Both re-anchored.
Also corrected: phase-01:605 said the overlay must be in the backup set. It is derivable from the
archived workspace by 1.0-GEN.d, so the line implied a gap that does not exist.
Known, not fixed: both live inner states are 664, group-writable, and they are the authority tofu
trusts. Tightening needs its own gated change plus proof the provider preserves the mode.
creds-matrix 65/65, octavia-pki 18/18, repo-lint 0 fail / 621 files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
