| 2026-07-13 |
Fold the AppArmor/libvirt rule into scripts/prereqs -- it gated every VR1 VM boot
...
`/var/lib/libvirt/vr1/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu had
been applied BY HAND on vcloud and existed ONLY as host state. Nothing in the repo
carried it, so a rebuild or a second host walked straight back into the same wall.
Why it matters: libvirt's stock abstractions/libvirt-qemu grants qemu only the DEFAULT
pool path (/var/lib/libvirt/images). VR1 uses a custom pool parent, which is not in that
abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The domain
DEFINES fine, then fails to start with a bare qemu "Permission denied" -- and NOTHING in
the libvirt error names AppArmor. It cost a session on 2026-07-12 (DOCFIX-186), was
logged as a PREREQ candidate, and was never actioned until the ledger collapse resurfaced it.
New: scripts/prereqs/install-apparmor-libvirt.sh, wired into install-all.sh (after
install-virtualization -- it edits libvirt's own profile dir) and reported by
check-prereqs.sh.
- Idempotent and deliberately QUIET on a good host: if the rule is present it exits 0
having touched nothing -- it does NOT reload apparmor or bounce libvirtd. Those happen
only on the run that actually adds the rule.
- Writes to local/, the vendor-sanctioned override include, so it survives package upgrades.
- Pool parent is a variable (VR1_POOL_PARENT, default /var/lib/libvirt/vr1), not a literal.
- Non-AppArmor hosts report N/A and SUCCEED -- no manufactured failure on a host it
does not apply to.
Verification: tests/prereqs 32/32 PASS (was 24/24) -- the new cases drive DETECTION off a
pool parent guaranteed absent from the profile, so the negative path is deterministic on
any host: a false OK from --check IS the bug, --dry-run must plan and mutate nothing, and
the planned rule must be exactly `<parent>/** rwk,`. Full gauntlet ALL GREEN (52 harnesses),
repo-lint 0 fail. Run live on vcloud: --check reports OK and leaves the profile byte-identical,
confirming the already-satisfied path is a true no-op.
No host state was mutated by this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2): API-key bootstrap -- no GUI click, no re-implemented crypto
...
The last unknown in D-113(a2) edge provisioning. The API path was proven (read AND write), but
every new edge still needed a MANUAL GUI CLICK to mint its key -- which would have put a human in
the middle of building DC1's and DC2's edges in Stage 3. Closed.
scripts/opnsense-mint-apikey.php runs ON the edge; calls OPNsense's OWN model
(Auth\User -> apikeys->add()) -- the exact code path
the GUI's ticket button invokes
scripts/opnsense-bootstrap-apikey.sh ships it over SSH, runs it, retrieves the key, wipes
the remote copies
tests/opnsense-bootstrap-apikey/ offline harness, 8 PASS (ssh/scp stubbed)
Gauntlet ALL GREEN (54 harnesses, was 53); repo-lint 0 fail.
VERIFIED END TO END on the live Office1 edge: minted a SECOND key via the vendor model ->
GET core/firmware/status returned HTTP 200 (the vendor-minted key AUTHENTICATES) ->
POST auth/user/del_api_key/<id> -> {"result":"deleted"} -> search_api_key shows 1 row remaining
(the operator's original key, untouched throughout) -> the retired key is now REJECTED (a real
negative test, not just an absence of errors). Temp creds shredded from the jumphost. The edge is
back to its prior state.
THE DESIGN DECISION, and why the alternative was rejected: OPNsense stores API secrets as
crypt(secret,'$6$') -- SHA-512 with an EMPTY salt (verified on the live 26.1 box: the stored hash
is "$6$$" + 86 chars). We COULD mint keys offline and seed them into a bootstrap config.xml. We
deliberately DO NOT. That would couple our provisioning to a vendor hash format we would have to
keep byte-compatible forever, and a mismatch FAILS SILENTLY -- keys that simply never
authenticate, with no error at generation time. Calling the vendor's own generator has no format
to keep in sync. Same principle as D-113(a2) itself: call the interface, do not re-implement the
internals -- which is exactly what DOCFIX-191/192/193 cost us. (An offline-hash attempt was
blocked by the permission classifier mid-session; the block was correct and produced this better
design.)
GUARD CLAUSE WORTH KEEPING (harness T5): the script REFUSES to overwrite an existing creds file.
An existing file means a LIVE key on the edge; overwriting the local copy would strand it there
PERMANENTLY, because the secret is hashed on save and can never be read back. That is
unrecoverable credential loss. T5 asserts both the refusal and that the existing file is left
byte-intact. The secret is never printed -- creation is the only moment it exists in cleartext.
WHAT THIS UNBLOCKS: edge provisioning is now automatable with no human in the loop --
boot factory nano -> D-112(c) console bootstrap (SSH + key) -> opnsense-bootstrap-apikey.sh
-> opnsense-api.sh for DHCP/firewall/interfaces.
There is NO config.xml anywhere in that chain.
STILL OPEN: the template reduction itself. And the chain above raises the real question -- the
template may not need REDUCING so much as DELETING, since the console bootstrap plus the key
minter cover first contact. NOT ruled; not done unilaterally. (Note the opnsense-edge module's
config_seed volume is INSTANTIATED -- it is in tfstate -- so removing the ISO wiring is a LIVE
change, not a docs change.)
Revert: git rm the two scripts + tests/opnsense-bootstrap-apikey/ (nothing live depends on them --
the live edge's key was minted via the GUI and is unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules
...
scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.
ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.
Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.
DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.
DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.
THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.
VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.
Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.
Nothing was instantiated; no live system is affected by either the bug or the fix.
Revert: git revert this commit (restores the two broken modules and the root-only gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SAFETY SWEEP: mark the config.xml push path as a LIVE HAZARD (no behaviour change)
...
After D-113(a2) was proven (Office1's config is now API-managed), every instruction in the repo
that still said "render a config.xml and push it to the edge" became ACTIVELY DANGEROUS: a full
config.xml push replaces /conf/config.xml wholesale and drops ~667 migration-populated elements
(measured), including the only 2 firewall pass rules on the box. Following the old runbook steps
against the live edge would clobber it.
Warnings and headers only. No behaviour changes, no live system touched.
Marked (5 files):
runbooks/dc-dc-phase1-office1-standup.md DANGER banner; the config.xml render + config-ISO
sub-steps removed as RUNNABLE instructions (they
would clobber the live edge). Image-prep retained;
history preserved in git + the build changelog.
runbooks/dc-dc-phase2-tofu-dc-substrate.md STOP banner on Step 4.
scripts/opnsense-render-config.sh DANGER header at point-of-use. NOT dead: under
D-113(a2) it is to be reduced to a MINIMAL bootstrap
render. Until then, safe only for a brand-new,
not-yet-booted edge.
scripts/opnsense-build-config-iso.sh RETIRED header quoting the upstream source proving
the importer can never fire on a nano image.
docs/dc-dc-deployment-workflow.md STALE-CONTENT warning atop the tooling gap register.
THE SWEEP ALSO CAUGHT A PRE-EXISTING LANDMINE, unrelated to today's work: dc-dc-phase2 Step 4
still instructs building a CONFIG ISO for DC1's edge -- but D-112 established that ISO can never
be read (opnsense-importer probes for a read-only root; on a pre-installed nano the root is
writable and a factory /conf/config.xml already exists, so it bootstrap_and_exit 0's without
enumerating a single device). That runbook has been telling anyone who follows it to build an
inert artifact and then wonder why the edge came up on factory defaults -- which is EXACTLY the
day lost on 2026-07-12. It was never corrected at the source. Now it is. Its WAN_IF/LAN_IF
"chicken-and-egg" discussion is moot for the same reason: that problem only exists if you try to
seed a full config before first boot, and D-112(c) measures vtnetN AFTER boot.
Changelogs deliberately NOT touched -- they are history, not instructions. Marking history is
noise; marking instructions is safety.
Verification: opnsense-render-config 24 PASS, opnsense-build-config-iso 2 PASS, opnsense-api
21 PASS; repo-lint 0 fail.
STILL OPEN (the actual D-113(a2) migration): reduce the template to a minimal bootstrap, retire
the config-ISO path and the opnsense-edge module's config_seed/cdrom wiring, and rewrite Stage 3's
edge steps around the API. This sweep makes the repo SAFE, not FINISHED.
Revert: git revert this commit (restores the previous, dangerous instructions verbatim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) step 1: thin OPNsense REST API client + harness
...
First delivery under the D-113 ruling (stay on OPNsense, move config off hand-authored
config.xml and onto the documented REST API).
scripts/opnsense-api.sh -- [--dry-run] <GET|POST> <api-path> [json-body]
tests/opnsense-api/run-tests.sh -- offline harness, 21 PASS, no network, no real key
Gauntlet ALL GREEN (53 harnesses, was 52). repo-lint 0 fail.
Design decisions that the harness now enforces:
- THE SECRET NEVER REACHES ARGV. Credentials are read from a file and handed to curl via
--config on STDIN. `curl -u key:secret` would put the secret in argv, where any user on the
box can read it out of `ps`. Harness T9 stubs curl with an argv recorder and asserts the
secret is absent from argv and present on stdin -- so a future "simplification" to `curl -u`
turns the harness red. That case is the point of the file.
- The API host is NEVER inferred (hard rule 2): $OPNSENSE_API_HOST unset is a loud failure, not
a silent default of 10.10.0.1 (T3).
- --insecure is deliberate and scoped: the edge's self-signed cert is REGENERATED ON EVERY BOOT
(measured -- "Created web GUI TLS certificate" appears in the config revision trail after each
reboot), so pinning it is pointless. Acceptable ONLY on the private lab LAN leg; never to be
copied to a tenant-facing surface.
- Dry-run reads no credentials (T11), which is what lets the harness prove URL construction
offline.
NOT RUN against the edge yet, and blocked on one thing: the API is alive (lighttpd on 443
answers 401) but root has 0 API keys. Minting one is a GUI action (System > Access > Users >
root > API keys), deliberately NOT automated -- writing apikeys into config.xml by hand is the
exact anti-pattern D-113 just retired. Key lands in ~/vr1-office1-creds/opnsense-api.txt,
operator-only, never read into agent context (SEC-007 covers that directory).
Proof-of-path once the key exists: drive yesterday's DHCP subnet through the API and confirm it
round-trips. If it does, the template retires to a minimal bootstrap config (sshd + root key +
console + apikey) and DHCP/firewall/interfaces move to the API. If it cannot express it, that is
the signal to revisit D-113.
Revert: git rm scripts/opnsense-api.sh && git rm -r tests/opnsense-api/ (nothing live touched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
| 2026-07-12 |
DOCFIX-193: LAN DHCP was never implemented -- add Kea to the edge config template
...
The operator's goal for the Office1 edge was "router + DHCP for office1-local". The
router half works; the DHCP half was NEVER BUILT. The template had no DHCP section at
all -- no <dhcpd>, no <Kea> -- and nothing served DHCP (measured on the live edge: no
kea/dhcpd/dnsmasq process, nothing on udp/67). The ledger's claim of "DHCP .100-.199"
was false; corrected in place.
OPNsense 26.1 uses KEA. ISC dhcpd is gone, so an old-style <dhcpd> block would have been
inert -- which is the trap this could easily have fallen into.
The Kea shape is NOT invented: it mirrors what OPNsense itself PERSISTS on the live 26.1
box (read back from its config.xml, which its own migrations had populated with an
enabled=0 skeleton), and the field names match the upstream model
src/opnsense/mvc/app/models/OPNsense/Kea/KeaDhcpv4.xml.
Render script now derives and VALIDATES, rather than trusting inputs:
- LAN_NETWORK_CIDR derived from LAN_IPADDR/LAN_SUBNET_BITS (the subnet must be the
NETWORK, not the host address -- 10.12.8.1/22 -> 10.12.8.0/22).
- subnet4 uuid is a DETERMINISTIC uuid5 of the LAN CIDR, so re-rendering is idempotent
and does not churn the config.
- DHCP_POOL_START/END are REQUIRED and validated: a pool outside the LAN subnet, a pool
that swallows the router's own LAN address, or reversed bounds all FAIL LOUD. Each of
those is XML-valid and would serve nobody (or fight the router) -- exactly the class of
silent misconfiguration this session has been finding all day.
Harness 15 -> 24 PASS (T10-T13, incl. three negative cases). Gauntlet ALL GREEN (52);
repo-lint 0 fail.
NOT YET APPLIED to the live edge -- that is a separate gated mutation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-191: the rendered edge config would have LOCKED MANAGEMENT OUT (no sshd, no key)
...
Found while building the D-112(c) bootstrap. The config template had:
<ssh><group>admins</group></ssh>
and no authorized key on root. That is OPNsense's DISABLED-sshd shape. So the config
we were about to push would have left the edge with sshd off and no key installed --
reachable only from the serial console. Under D-112(c), SSH *is* the management path,
so this is a defect in the config itself, independent of how it gets delivered. Had the
Importer worked back in July, we would have shipped an unmanageable edge and blamed the
network.
Element names are VERIFIED against upstream, not guessed:
- src/www/system_advanced_admin.php: `enabled` is the literal STRING "enabled";
`permitrootlogin` / `passwordauth` are PRESENCE-checked (isset), so omitting
passwordauth leaves password auth OFF.
- src/etc/inc/auth.inc: `$keys = base64_decode($user['authorizedkeys']);` -- the
<user><authorizedkeys> value is BASE64 and is written to ~/.ssh/authorized_keys.
Changes:
- template: <ssh> gains <enabled>enabled</enabled> + <permitrootlogin>; root <user>
gains <authorizedkeys>{{ROOT_AUTHORIZED_KEYS_B64}}</authorizedkeys>.
- render script: ROOT_AUTHORIZED_KEYS is now REQUIRED (no default -- an edge rendered
without a key is a lockout waiting to happen). PUBLIC key material only; the private
key is never read by this repo. Base64 is computed AFTER the req() loop so a missing
key fails with the friendly message, not a `set -u` crash.
- Key-only auth is deliberate; the serial console stays the break-glass fallback.
Harness 9 -> 12 PASS. T7b asserts sshd is actually enabled; T7c asserts <authorizedkeys>
base64-DECODES back to the supplied key (a round-trip, not a presence check -- a
wrongly-encoded key yields a well-formed config that silently grants no access); T8
asserts a missing key FAILS LOUD.
Note: the harness's existing T1c well-formed-XML check caught a bug in this very change
(an XML comment containing "--", which is illegal). Fixed. That is the harness earning
its keep.
Gauntlet ALL GREEN (52); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
Office1 OPNsense edge build: real-ISP-router config + first-boot fixes
...
Bringing the Office1 OPNsense edge online (router + DHCP for office1-local)
before the Office1 NetBox VM. First-ever OPNsense boot in this repo; several
first-execution findings fixed.
DOCFIX-184: opnsense-prep-image.sh decompresses the .img.bz2 via python3's
stdlib bz2 when bunzip2 is absent (no sudo install needed). Harness 3/3.
DOCFIX-185: the OPNsense edge is a normal simulated-ISP router, not an
egress-airgap. Operator clarified the transport model -- each site (DCs +
Office) has its own real ISP connection; dark fiber is East-West/replication
only; D-107 node-airgap is a separate DC concern. Stripped the WAN
egress-control rules (seq 20/21/99) + MIRROR_* tokens from the config template,
render script, and harness (8/8). Flagged a D-100/D-107 amendment note as
follow-up.
DOCFIX-186: instantiate module "office1_opnsense" in main.tf (first real
validation of modules/opnsense-edge). Two infra findings: (1) config_iso_path
must be OUTSIDE the pool dir (create.content.url collides with its own target
volume) -> stage the ISO; (2) apparmor blocks the custom /var/lib/libvirt/vr1/
pool path (foundational -- all VR1 VMs) -> needs a one-time local apparmor rule
(operator sudo). office1-wan NAT network created via virsh (D-103 debt).
repo-lint 0 fail; opnsense-prep-image 3/3; opnsense-render-config 8/8.
Boot blocked only on the apparmor fix (pending operator sudo).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-174: fix self-inflicted false positive in ledger-scan.sh next-free scan
...
DOCFIX-173's own changelog narrated its {3}->{3,} regex fix using a literal
illustrative example ("DOCFIX-1004"), which the widened regex then matched
as a real assignment -- inflating DOCFIX next-free from 174 to 1005.
Reworded the illustrative examples in prose, added a reproduc-line
exclusion to nextfree_mentions() as defense-in-depth, and added a
regression fixture to the test harness (37/37 PASS).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-173: full-project sweep -- script logic bugs + security hardening
...
Four confirmed logic bugs (unguarded pipelines/function calls under
set -e+pipefail), all independently reproduced in isolation before being
trusted -- including catching that a first proposed fix for one of them
was itself wrong: `|| true` after a failing pipeline under pipefail
silently discards PIPESTATUS, which would have made phase-06-capi-stack's
own gate check never fire again. The correct `if pipeline; then :; fi`
idiom was verified via reproduction before committing.
- carve-host-interfaces.sh: emit()'s unguarded `return 1` killed apply-mode
runs on the first MAAS error, before the FATAL-accumulator summary
could print.
- phase-06-capi-stack.sh: run_step's PIPESTATUS read was dead code.
- juju-spaces-check.sh, osd-blank-check.sh: one unguarded display-only
pipe each, safely `|| true`-able.
- ledger-scan.sh: DOCFIX/BUNDLEFIX next-free regex used a fixed {3} not
{3,} -- the same bug class already fixed once for the D-series, never
mirrored here. Dormant, not yet reachable, but a real landmine.
- dc-dc-rbd-mirror.sh: hardened the bootstrap token's default path
(random suffix + immediate chmod 600, was a predictable /tmp filename
with no permission hardening).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-166: $DC parameterization for reenroll-hosts/carve-host-interfaces/phase-00-maas-standup (gap #15, CLI half)
...
Each script gained an opt-in $DC env var, calling lib_net_select_dc/
lib_hosts_select_dc immediately after sourcing (unset $DC is byte-for-byte
unchanged VR0/DC0 behavior -- verified directly, no fixtures needed since
the selector calls sit before any MAAS call). DC=dc1 bash
scripts/phase-00-maas-standup.sh is now a real, correct, gated call
(D-101 makes dc1's target identical to dc0's); DC=dc1/dc2 for the other
two scripts still fails loud at the host selector until real per-DC host
data exists. New/extended test cases across three harnesses. Six of gap
#15's seven sub-findings remain open -- said plainly, not overclaimed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-165: Ceph replication tooling scripts, radosgw multisite + rbd-mirror (gap #5)
...
Extracts the Stage 6 runbook's manual command sequences into real,
parameterized scripts: scripts/dc-dc-radosgw-multisite.sh (master-init/
join-readonly/enable-two-way), scripts/dc-dc-rbd-mirror.sh
(bootstrap-primary/bootstrap-secondary --direction), scripts/dc-dc-dr-drill.sh
(failover/failback). The single most safety-critical property in the whole
drill -- demote the current primary before promoting the recovering side
during failback -- is hard-coded as fixed call order, not left to
documentation. Dry-run by default; --apply required for real execution,
plus a typed confirmation for the drill script. 19/19, 19/19, 23/23 tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-162: MTU/geneve + Ceph disk-budget calculator scripts (gap #7)
...
scripts/dc-dc-mtu-geneve-budget.sh reproduces D-101's own worked example
exactly (1500-56=1444) and adds an IPv6-minimum-link-MTU sanity floor.
scripts/dc-dc-ceph-disk-budget.sh computes size=3-vs-size=2 fit against a
measured total disk, requiring --backup-overhead-fraction explicitly (no
invented default) and naming size=2 as an operator-logged decision, never
silently applied. 19/19 and 16/16 tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-151: $DC selector convention for lib-net.sh/lib-hosts.sh (gap #1 closed)
...
Adds lib_net_select_dc() (dc0/dc1 no-op per D-101 inherited-layout ruling,
dc2 fails loud pending NetBox) and lib_hosts_select_dc() (dc0 no-op, dc1+dc2
both fail loud -- no per-DC host enrollment exists yet). Backward compatible;
21/21 new harness tests green; full gauntlet unaffected (A/B via git stash).
Unblocks the Stage 5 runbook's "high reuse" claim in the DC-DC deployment
workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-149: OPNsense config.xml template + tested renderer
...
Real design work, researched before drafting: fetched OPNsense's own actual
shipped config.xml.sample and its Route.xml static-routes model before
writing anything.
opentofu/templates/opnsense-config.xml.tmpl: {{TOKEN}}-parameterized (reuses
this repo's existing clientdocs convention), covering interfaces/gateway/
staticroutes/DNS/NTP and D-107-shaped firewall rules (default-deny WAN
egress except NTP + the per-DC mirror's upstream sync). The real sample
fetch directly confirmed last turn's audit finding with an actual example:
it ships literal placeholder device names inside each interface's own <if>
block, proving LAN/WAN role assignment really is that explicit per-block
mapping, not declaration order.
scripts/opnsense-render-config.sh: the renderer. Needs no external tool
(unlike every other opnsense-* script this session), so it's tested
END-TO-END, not just guard clauses -- 8/8, including well-formed-XML
validation. The harness caught a real bug before it shipped: the token
HOSTNAME collides with bash's own built-in $HOSTNAME variable (unset
doesn't actually clear it), silently passing a test that should have
failed. Renamed to OPNSENSE_HOSTNAME throughout.
opentofu/templates/README.md's token legend marks exactly which values are
real (NTP pool default, D-106 naming) vs. pending Stage 0 ratification
(D-100/D-101/D-107) vs. only measurable on a real boot (vtnetN assignment)
vs. a security requirement (root password hash must be freshly generated,
never the stock sample's own shipped default).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-148: OpenTofu module audit pass + provider-docs skill reference
...
Audited every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules
built this session rather than leaving them as-is. Found and fixed a real
documentation error: opnsense-edge's comments claimed devices.interfaces
list order directly sets OPNsense's LAN/WAN role. Researching OPNsense's own
interface-assignment model showed this conflates two things -- list order
plausibly controls which vtnetN device number a NIC gets (libvirt's own
docs: auto-assigned PCI addresses "usually match" XML order for a simple
topology), but the actual LAN/WAN role is a separate, explicit mapping set
inside config.xml itself ("vtnet0=WAN" is a convention some guides choose,
not an enforced rule). Corrected main.tf's comments and both
lan_network_name/wan_network_name variable descriptions -- no config.xml
exists in this repo yet, so this fixes documentation accuracy for whoever
writes it next, not current behavior.
Confidence upgraded (not just re-hedged) on three other items: node-vm's
boot={order=N} shape (confirmed the provider 1:1-mirrors libvirt's own XML,
and libvirt's own docs confirm the single-attribute shape), create.content.url
accepting local paths (second independent source), and the genisoimage
flags (confirmed standard, cleanly separated from the still-open question of
whether OPNsense's Importer actually reads the result). Confirmed safe:
maas_vm_host's zone/pool computed-if-unset behavior.
Added .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md:
indexes every provider/doc source used this session with exact URLs and
confirmed versions, plus the fetch methodology that actually worked
(Registry doc pages are JS-rendered, use GitHub instead; branch names vary,
check default_branch via the API; real example .tf files beat doc-summarized
prose for syntax questions). Wired into SKILL.md's routing table,
cross-referenced from opentofu/README.md in both directions without
duplicating content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-146: opnsense-edge module + nano-image/config-ISO prep scripts
...
Implements the mechanism identified in today's OPNsense deployment research
(cloud-init confirmed unreliable on FreeBSD; the real native mechanism is
OPNsense's Configuration Importer, which scans an attached volume for
/conf/config.xml and gained ISO9660 support specifically for VM/cloud
automation).
scripts/opnsense-prep-image.sh: downloads the nano image, decompresses,
converts raw->qcow2, resizes -- work create.content.url almost certainly
can't do itself (a plain fetch, no decompression). No hardcoded mirror URL;
OPNSENSE_MIRROR_BASE is required explicitly.
scripts/opnsense-build-config-iso.sh: builds the ISO9660 image containing
/conf/config.xml that the Configuration Importer picks up.
opentofu/modules/opnsense-edge: wires both together, mechanically identical
to modules/cloudinit-vm's cdrom-attach shape but with no libvirt_cloudinit_disk
resource (wrong format for this). Explicit lan_network_name/wan_network_name
variables rather than an ordered list, since OPNsense's interface-1-is-LAN
convention makes list-ordering a real place for a silent mistake to hide. No
PXE boot-order attribute used -- this VM boots from its own disk, not the
network, so node-vm's flagged-unverified boot field doesn't apply here.
Both new scripts' harnesses (4/4, 3/3) exercise REAL missing-tool guards --
qemu-img/genisoimage/xorriso are all genuinely absent from this environment.
Two things flagged, not presented as fact: whether the Importer's ISO9660
behavior holds on a real boot, and whether interface list-order reliably
maps to LAN/WAN NIC enumeration for this guest. config_iso_path has no
default -- real config.xml content per site is still undesigned. Not
instantiated in root main.tf.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-143: repo_lint.py no longer relies on git to clean up its own CRLF
...
The --record-clientdocs-sweep/--record-guide-skill-coupling writers used
write_text(), which translates \n to the platform newline (CRLF on Windows)
before writing. .gitattributes' eol=lf silently cleaned this up at every
`git add`, so no CRLF ever reached a commit, but the script was depending on
git to fix its own sloppy output -- flagged during the DOCFIX-141 commit.
Both writers now build the string, .encode("ascii") it, and write_bytes() it
directly. ascii (not utf-8) deliberately: this repo's own rule is ASCII-only,
so a future non-ASCII path/hash now raises loudly here instead of writing
content L1 would fail on anyway.
Verified live: re-ran --record-clientdocs-sweep, byte-inspected the receipt
(0 CR, was 41/41 CR/LF before). tests/repo-lint harness 34/34 unchanged.
repo-lint 0 fail / 1 documented warn.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-142: opentofu/ network + storage-pool scaffold (VR1 IaC, gap #2)
...
First OpenTofu content in the repo. Network/pool layer only: modules/dc-planes
(the six per-DC planes, D-052/D-100), modules/mesh-link (the D-100 dark-fiber
triangle), modules/dc-storage-pool, wired for DC1 (inherits DC0's CIDRs per
D-101) and Office1. DC2 deliberately not wired -- its CIDRs aren't assigned
yet (D-101 open sub-item).
Provider schema (dmacvicar/libvirt v0.9.8) verified against its actual current
docs this session, not memory -- it has diverged materially from older common
examples (network isolation is a nested forward.mode, not top-level mode;
libvirt_domain restructured to ~40 args mirroring raw libvirt XML). Node-VM/
volume resources, OPNsense config, and tc netem application are deliberately
deferred: that schema is too large/unfamiliar to author safely without a real
`tofu providers schema` pass on a connected machine -- see opentofu/README.md
for the full account and the concrete next step.
Added scripts/opentofu-validate.sh (fmt/init/validate wrapper) + a harness
testing what's actually exercisable without a tofu binary (none is available
anywhere this repo has been worked in this session -- logged, not hidden).
Extended repo_lint.py's L1 check to cover .tf files (uncovered before since
none existed). repo-lint 0 fail / 1 documented warn; both harnesses green.
Workflow doc + its companion visual tracker updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
| 2026-07-09 |
DOCFIX-139..141: repo-rename sweep (ipv4->dc-dc) + repo-lint L9 self-reference guard
...
New repo openstack-caracal-dc-dc still had stale operational references to the
source repo it was seeded from (openstack-caracal-ipv4), plus, after the
mechanical fix, hardcoded references to its own new name -- both break on any
future fork/rename. Fixes:
- DOCFIX-139/140: repaired 8 stale clone-path/URL references across the skill
and 4 runbook RUN blocks; fixed a repo_lint.py Windows path-separator bug
that silently turned a documented WARN into a FAIL (rel comparison used
str() instead of as_posix()).
- DOCFIX-141: new repo-lint check L9 fails any script/RUN-block/skill file
that hardcodes the current repo's own directory name. Narrowed the
.claude exclusion in repo_lint.py to .claude/worktrees/ only (it was
blanket-hiding checked-in .claude/skills and .claude/hooks from all
lint checks). Established a $REPO session-variable convention
(runbooks/README.md Conventions) and converted the 8 hardcoded paths to
require it explicitly, fail loud if unset.
repo-lint: 0 fail / 1 documented legacy WARN. tests/repo-lint harness 34/34.
Full details + per-item revert instructions in the two changelogs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
| 2026-07-08 |
DOCFIX-138: S8b -- centralize KEYSTONE_VIP default into lib-net.sh (single source of truth)
...
The keystone endpoint literal 10.12.4.50 was the default in 6 scripts (3 forms); now
all reference lib-net.sh KEYSTONE_VIP_DEFAULT (hard-rule #3). Sourced lib-net where
absent (verified source-safe). Each stays env-overridable. Resolve-checked; gauntlet
39/39, lint 0 fail (L7 re-recorded). S8 fully done (S8a+S8b); S8c won't-do.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-137: repo-lint L8 -- enforceable guide<->skill coupling guard
...
Make 'sweep the skill when the guide changes' a hard gate, not a note (skill-lag hit
twice: DOCFIX-135/136). L8 pins the Jenkins+K8s guide's sha256 in
clientdocs/guide-skill-coupling.txt; any guide change FAILS repo-lint (naming the
skill files to review) until re-recorded with --record-guide-skill-coupling. Mirrors
the L7 sweep-receipt pattern. Harness +5 (28/28), gauntlet 39/39, lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-134: S8a -- FIP pool literals to lib-net.sh (single source of truth)
...
D-003 FIP pool (10.12.5.0-10.12.7.254) was hardcoded in phase-04-network-create
(env-overridable) and -verify (pinned). Both already source lib-net.sh; added
FIP_POOL_START/END there and referenced them. Gauntlet 39/39. S8b (KEYSTONE_VIP,
needs sourcing into client-critical scripts) + S8c (python cross-source) deferred
for operator decision -- bigger than the S estimate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-132/133: S-class cleanups -- S7 dead code + S4 trust_filter.py consolidation
...
S7: remove unused CF (d011-04) + subsumed vr_is_ipv4 guard (d011-03), no behavior change.
S4: three inline python trust filters -> one fixture-tested scripts/trust_filter.py
(--fields/--trustor-in/--first/--error-token); SDIR added to offboard + acceptance;
new tests/trust_filter/ 9/9. Gauntlet 39 ALL GREEN, offboard 22/22, acceptance 16/16.
S8 re-scoped (bigger than estimate) -- operator decision logged in ledger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-131: tenant-offboard.sh Phase A cluster captures stdout-only
...
Fixes the last merged-stderr sites in the offboard (Phase C/D were DOCFIX-113).
A benign 'coe cluster list' stderr warning could word-split into the delete loop
and keep the wait loop from ever seeing empty -- a false 'not gone' exit 21 on an
already-clean tenant. New tenant-scoped tinv() (stdout-only, stderr surfaced as
NOTE); wait loop retries on list-error, final check fails closed. Harness un-excludes
coe from warnstderr + new coe-stderr-not-argv assertion (22/22). Gauntlet 38/38.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
addendum 45: gated script fixes -- DOCFIX-126 offboard E0 always-403 no longer FATALs before Phase F (harness 21/21) + DOCFIX-127 onboard cred files carry dashboard_url (env-overridable); live verify gated
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
