| 2026-07-13 |
D-114 follow-through: fix node-vm's nested-virt defect NOW; voffice1 pinned to a reserved address
...
Operator ruling: fixes land as we find them -- only DC1 DEPLOYMENT is gated, not DC1 fixes.
1. modules/node-vm: UNCONDITIONAL `cpu = { mode = "host-passthrough" }`. Same defect just found in
cloudinit-vm: with no cpu block libvirt hands the guest a generic emulated CPU with NO svm. A DC
node runs nova-compute, so it MUST run KVM guests -- there is no correct value other than this,
hence no knob that could only ever be set wrong (contrast cloudinit-vm, where it IS a real per-VM
decision, and opnsense-edge, which disables svm as correct hardening for a router). Not yet
instantiated, so no live call breaks. Without it the DC nodes would have come up looking healthy
and been unable to launch a single instance -- surfacing in Stage 5 as an inscrutable scheduler error.
2. voffice1 moved off the dynamic pool onto a RESERVED address. It first took 10.10.0.100 -- the
first address of Kea's pool (.100-.199). A MAAS region controller must not hold an address that
can move. Added a Kea host reservation via the D-113(a2) REST API (a second live exercise of that
write path): 52:54:00:6a:87:e5 -> 10.10.0.20, deliberately OUTSIDE the pool. add_reservation ->
"saved", service/reconfigure -> "ok". Rebooted; voffice1 came back on 10.10.0.20, .100 released,
/dev/kvm still present. Done now because nothing depends on the address yet.
3. runbooks/dc-dc-phase1-office1-standup.md rewritten to the D-114 model (was: three sibling service
VMs + an unruled Option A/B fork -- it would have actively misdirected whoever ran it). New
sequence: voffice1 via tofu -> first contact (lease/SSH/nested-KVM) -> MAAS on voffice1 -> LXD
(5.21 LTS) -> register LXD as a MAAS VM host -> MAAS-COMPOSE NetBox/GitBucket/Tailscale. The
MAAS/LXD command lines carry explicit PENDING VERIFICATION markers rather than fabricated flags.
docs/dc-dc-deployment-workflow.md Stage 2 + gap register updated to match.
repo-lint 0 fail. opentofu-validate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
...
APPLIED, operator-approved: 5 added, 0 changed, 0 destroyed. The running OPNsense edge was NOT
touched (confirmed by plan BEFORE apply -- no in-place update, so no repeat of the 2026-07-13
guest bounce).
THE MODULE WAS SILENTLY BROKEN FOR D-114'S ENTIRE MODEL. modules/cloudinit-vm's libvirt_domain
had NO `cpu` block. With none, libvirt renders a GENERIC EMULATED CPU exposing no `svm` -- and
LXD *virtual machines* are qemu/KVM guests, so NO service VM could ever have been composed into
voffice1. D-114 would have failed at its first step with a confusing "KVM not available" that
pointed nowhere near OpenTofu. Measured proof of the masking: host is an AMD EPYC 9965; a
default-CPU guest was handed an "Opteron_G3".
Fix: `cpu { mode = "host-passthrough" }` always, plus a new `expose_nested_virt` variable with NO
default -- a real per-VM decision. true = pass `svm` through (required for D-114's containment
VMs); false = disable it, matching modules/opnsense-edge. The operator asked whether the edge's
`svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a
triple-fault fix on 2026-07-12, did NOT resolve it (the real cause was the memory unit), and was
retained on hardening grounds. A router has no business seeing nested virt. It stays disabled there.
voffice1 as built (MEASURED): Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, single NIC on
office1-local, reports the real EPYC CPU with svm on all 16 cores and /dev/kvm PRESENT, egress to
1.1.1.1 at 4.6ms THROUGH the edge. Cloud-init deliberately MINIMAL (identity + key + guest agent);
MAAS and LXD go in as separate GATED steps rather than buried in a first-boot script that either
silently works or silently does not. The SSH pubkey is read via file(var...) at plan time -- key
material never enters a command line, the repo, or an agent's context; the tfvars holding the path
stays gitignored. network_config matches the NIC by GLOB, not a guessed kernel name (an inferred
value); it came up enp1s0.
TWO GATES CLEARED:
1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item: the daemon was proven, the
SERVICE never was (office1-local had no client). voffice1 -- the first client ever on that LAN
-- took 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state
active, read back through the D-113(a2) REST API. DHCP now works end to end.
2. D-114's NESTING PROBE PASSES AT L3. Stated honestly: this proves nested KVM is AVAILABLE. The
DEFINITIVE proof is a guest actually booting inside voffice1 -- that lands with the first
composed LXD VM. Not recording L3 as fully proven until then.
FINDING, logged NOT actioned (DC1 is gated behind Office1 per D-114): modules/node-vm has the SAME
missing-cpu-block defect. DC nodes run nova-compute, which needs working KVM -- without the fix
they will come up unable to run a single instance. MUST be fixed before Stage 3.
opentofu-validate PASS, repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules
...
scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.
ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.
Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.
DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.
DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.
THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.
VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.
Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.
Nothing was instantiated; no live system is affected by either the bug or the fix.
Revert: git revert this commit (restores the two broken modules and the root-only gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
| 2026-07-12 |
DOCFIX-193: LAN DHCP was never implemented -- add Kea to the edge config template
...
The operator's goal for the Office1 edge was "router + DHCP for office1-local". The
router half works; the DHCP half was NEVER BUILT. The template had no DHCP section at
all -- no <dhcpd>, no <Kea> -- and nothing served DHCP (measured on the live edge: no
kea/dhcpd/dnsmasq process, nothing on udp/67). The ledger's claim of "DHCP .100-.199"
was false; corrected in place.
OPNsense 26.1 uses KEA. ISC dhcpd is gone, so an old-style <dhcpd> block would have been
inert -- which is the trap this could easily have fallen into.
The Kea shape is NOT invented: it mirrors what OPNsense itself PERSISTS on the live 26.1
box (read back from its config.xml, which its own migrations had populated with an
enabled=0 skeleton), and the field names match the upstream model
src/opnsense/mvc/app/models/OPNsense/Kea/KeaDhcpv4.xml.
Render script now derives and VALIDATES, rather than trusting inputs:
- LAN_NETWORK_CIDR derived from LAN_IPADDR/LAN_SUBNET_BITS (the subnet must be the
NETWORK, not the host address -- 10.12.8.1/22 -> 10.12.8.0/22).
- subnet4 uuid is a DETERMINISTIC uuid5 of the LAN CIDR, so re-rendering is idempotent
and does not churn the config.
- DHCP_POOL_START/END are REQUIRED and validated: a pool outside the LAN subnet, a pool
that swallows the router's own LAN address, or reversed bounds all FAIL LOUD. Each of
those is XML-valid and would serve nobody (or fight the router) -- exactly the class of
silent misconfiguration this session has been finding all day.
Harness 15 -> 24 PASS (T10-T13, incl. three negative cases). Gauntlet ALL GREEN (52);
repo-lint 0 fail.
NOT YET APPLIED to the live edge -- that is a separate gated mutation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-192: the edge config silenced its own ONLY console (no serial, no video device)
...
Third lockout-class defect in the same config, found by executing D-112(c) for real.
After the config was applied and the box rebooted, the serial console went PERMANENTLY
SILENT -- no login prompt, no output, nothing (measured: raw console read returns empty).
The edge VM has NO VIDEO DEVICE, so the serial port is its only console and its only
break-glass path.
Cause: the config set neither <primaryconsole> nor <serialspeed>, so the applied config
did not select the serial console. Boot output still appears (the loader honours the nano
image's own /boot.config -S115200), which is why this looks like a hang rather than a
config choice -- the console dies exactly when OPNsense applies our config.
The box was NOT lost: DOCFIX-191 had just enabled sshd + installed the service key, so it
stayed reachable. That is the only reason this was a finding and not an outage. Two
lockout bugs in one config, and the first one saved us from the second.
Key names VERIFIED against upstream src/www/system_advanced_admin.php:
- `primaryconsole` and `serialspeed` are STRING values.
- There is NO `enableserial` key in OPNsense -- that is a pfSense-ism. Do not add it.
115200 matches the nano image's own /boot.config.
Harness 12 -> 15 PASS. T9b asserts the serial console is retained; T9c guards against
someone "fixing" this by re-adding the bogus pfSense <enableserial> key.
(T9c initially failed on my own template COMMENT, which names enableserial while warning
against it. Tightened to match an actual element. The harness caught it.)
Gauntlet ALL GREEN (52); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-191: the rendered edge config would have LOCKED MANAGEMENT OUT (no sshd, no key)
...
Found while building the D-112(c) bootstrap. The config template had:
<ssh><group>admins</group></ssh>
and no authorized key on root. That is OPNsense's DISABLED-sshd shape. So the config
we were about to push would have left the edge with sshd off and no key installed --
reachable only from the serial console. Under D-112(c), SSH *is* the management path,
so this is a defect in the config itself, independent of how it gets delivered. Had the
Importer worked back in July, we would have shipped an unmanageable edge and blamed the
network.
Element names are VERIFIED against upstream, not guessed:
- src/www/system_advanced_admin.php: `enabled` is the literal STRING "enabled";
`permitrootlogin` / `passwordauth` are PRESENCE-checked (isset), so omitting
passwordauth leaves password auth OFF.
- src/etc/inc/auth.inc: `$keys = base64_decode($user['authorizedkeys']);` -- the
<user><authorizedkeys> value is BASE64 and is written to ~/.ssh/authorized_keys.
Changes:
- template: <ssh> gains <enabled>enabled</enabled> + <permitrootlogin>; root <user>
gains <authorizedkeys>{{ROOT_AUTHORIZED_KEYS_B64}}</authorizedkeys>.
- render script: ROOT_AUTHORIZED_KEYS is now REQUIRED (no default -- an edge rendered
without a key is a lockout waiting to happen). PUBLIC key material only; the private
key is never read by this repo. Base64 is computed AFTER the req() loop so a missing
key fails with the friendly message, not a `set -u` crash.
- Key-only auth is deliberate; the serial console stays the break-glass fallback.
Harness 9 -> 12 PASS. T7b asserts sshd is actually enabled; T7c asserts <authorizedkeys>
base64-DECODES back to the supplied key (a round-trip, not a presence check -- a
wrongly-encoded key yields a well-formed config that silently grants no access); T8
asserts a missing key FAILS LOUD.
Note: the harness's existing T1c well-formed-XML check caught a bug in this very change
(an XML comment containing "--", which is illegal). Fixed. That is the harness earning
its keep.
Gauntlet ALL GREEN (52); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
D-112: record option (a) as MEASURED INFEASIBLE; rule the (c) bootstrap; make the edge console interactive
...
Operator asked to re-examine option (a) (bake /conf/config.xml into the nano) before
committing to (c). Done, and it is BLOCKED on this host -- measured, not assumed:
CONFIG_UFS_FS=m but "# CONFIG_UFS_FS_WRITE is not set" -> UFS driver is READ-ONLY
partition table: MBR, a single FreeBSD a5 partition -> UFS only, no FAT to write
fuse-ufs / ufs2-tools -> not present, not packaged
libguestfs (guestfish / virt-copy-in) -> not installed
Two ways (a) could be forced, both rejected and recorded so they are not re-litigated:
a FreeBSD helper VM (works, xorriso is present to remaster a live ISO -- but it means
maintaining a builder VM forever to deliver one 4.5 KB file), and a raw byte-patch of
the image (probably works; silently corrupts if the file is not block-contiguous, and
is NON-TRANSFERABLE -- you cannot byte-patch a real appliance on Roosevelt hardware,
which is exactly what this repo's governing constraint forbids).
Examining (a) properly STRENGTHENED (c): (a) is not merely less mainstream here, it is
unavailable without a detour costing more than (c)'s entire bootstrap.
Bootstrap RULED (operator): B1 = temporary host leg on virbr2 (reversible; no change to
the D-100 network definition). B2 = one interactive console login to enable SSH and
install the ed25519 pubkey already generated in ~/vr1-office1-creds/, after which
everything is plain SSH/SCP.
Module change enabling B2: the edge's serial becomes a unix socket + a log file, rather
than a write-only file source. Both roles are needed and they are different:
source.unix -> INTERACTIVE bidirectional console. A `file` source is write-only capture
and cannot be typed into, so it cannot bootstrap anything. A unix socket
is scriptable with `socat - UNIX-CONNECT:`, unlike a pty, which needs a
controlling TTY.
log -> preserves full boot capture from power-on (a socket loses anything
emitted before a client connects). That capture is what made both
2026-07-12 boot bugs legible; it is not being dropped.
S1+S2 guards PASS, harness 6/6, repo-lint 0 fail, tofu fmt/validate clean.
Live recreate + host leg remain separately gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-189: re-audit the max-context session's troubleshooting + docs
...
Operator-instructed re-evaluation. The 2026-07-12 session authored its docs and
module comments WHILE WORKING FROM A WRONG DIAGNOSIS, at a context limit, so its
reasoning was re-checked rather than just its one-line bug (DOCFIX-188).
Headline: the settings it changed are mostly fine and are RETAINED. The reasoning
it recorded was not -- it attributed the fault to whatever knob was being turned,
on no evidence, four separate times. It also left one silently-dead knob.
FALSE CAUSAL CLAIMS (corrected at the source, opnsense-edge/main.tf). The disk
shape, q35->i440fx, the serial console and svm-disable were EACH written up as a
contributing cause. None changed the symptom -- the fault sat at a deterministic
262 bytes through all of them. All four settings RETAINED (each defensible on its
own merits); only the false justifications are gone. The serial console earned its
keep, but it REVEALED the fault, it did not change it.
DEAD KNOB (real functional defect). `disk_size_bytes = 16 GiB` did nothing once
DOCFIX-187 made the disk a direct copy; unused variables are legal HCL so neither
validate nor plan complained. MEASURED: the live disk is 11.00 GiB, not 16.
Removed from main.tf + module variables; real sizing is opnsense-prep-image.sh's
GROW (+8G). Proof it was dead: tofu plan is byte-identical before/after removal.
STALE/UNSOURCED FACTS. "Opteron_G3" is a red herring -- the host is really an AMD
EPYC 9965 (Zen 5, family 26); libvirt's CPU-model DB doesn't know family 26 and
falls back to the oldest name, and the prior session partly built its nested-virt
theory on that artifact. "OPNsense's 3 GB min" is unsourced -- annotated, do not
propagate. DOCFIX-186's backing_store parenthetical is stale (same conclusion,
different mechanism). DOCFIX-187's changelog gains a SUPERSEDED-IN-PART header.
KEPT: DOCFIX-186's apparmor + config-iso-staging findings are real and measured.
DOCFIX-185 (operator transport ruling) untouched.
STANDING LESSON: when a change does not fix the symptom, do not write it up as
though it did. A fault that is deterministic and immune to every knob you turn
means you have not yet touched the cause.
Gauntlet ALL GREEN (52); repo-lint 0 fail; tofu fmt/validate/plan clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-187: opnsense-edge module boot fixes + boot-triple-fault incident report
...
Office1 OPNsense edge first-boot debugging. Several correct module fixes made;
the boot is NOT yet resolved (BTX-loader triple-fault under double-nested virt),
documented as an OPEN incident for the next session.
Module (opentofu/modules/opnsense-edge/main.tf), all kept (a working config
needs them, none resolved the fault):
- serial console (nano is serial-only; the module had none) + boot-log capture
- machine q35 -> i440fx (pc)
- disk COW-overlay -> direct per-VM copy (matches virt-install --import)
- CPU host-passthrough with AMD svm disabled (documented AMD nested-virt fix;
note the provider key is `features` plural, not `feature`)
Incident report docs/incident-20260712-opnsense-edge-boot-triplefault.md: full
symptom (boot2 echoes /boot.config then triple-faults loading /boot/loader, a
consistent 262-byte fault confirmed fresh each boot), environment (double-nested
AMD Opteron_G3), everything tried, and ranked next steps (full CPU flags
+kvm_pv_eoi/+kvm_pv_unhalt, add a video device for the -D dual console, mem 4G,
UEFI/OVMF, outer-hypervisor CPU). Ledger checkpoint added; session at context
limit.
repo-lint 0 fail; module tofu validate Success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 OPNsense edge build: real-ISP-router config + first-boot fixes
...
Bringing the Office1 OPNsense edge online (router + DHCP for office1-local)
before the Office1 NetBox VM. First-ever OPNsense boot in this repo; several
first-execution findings fixed.
DOCFIX-184: opnsense-prep-image.sh decompresses the .img.bz2 via python3's
stdlib bz2 when bunzip2 is absent (no sudo install needed). Harness 3/3.
DOCFIX-185: the OPNsense edge is a normal simulated-ISP router, not an
egress-airgap. Operator clarified the transport model -- each site (DCs +
Office) has its own real ISP connection; dark fiber is East-West/replication
only; D-107 node-airgap is a separate DC concern. Stripped the WAN
egress-control rules (seq 20/21/99) + MIRROR_* tokens from the config template,
render script, and harness (8/8). Flagged a D-100/D-107 amendment note as
follow-up.
DOCFIX-186: instantiate module "office1_opnsense" in main.tf (first real
validation of modules/opnsense-edge). Two infra findings: (1) config_iso_path
must be OUTSIDE the pool dir (create.content.url collides with its own target
volume) -> stage the ISO; (2) apparmor blocks the custom /var/lib/libvirt/vr1/
pool path (foundational -- all VR1 VMs) -> needs a one-time local apparmor rule
(operator sudo). office1-wan NAT network created via virsh (D-103 debt).
repo-lint 0 fail; opnsense-prep-image 3/3; opnsense-render-config 8/8.
Boot blocked only on the apparmor fix (pending operator sudo).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-175/176: OpenTofu state file security + VR1 teardown/rollback runbook
...
Found while reviewing what's missing from the VR1 buildout ahead of
tomorrow's execution.
DOCFIX-175: terraform.tfstate stores sensitive=true variables (like
maas_api_key) in plaintext regardless of the flag, which only redacts
CLI/plan output (verified against HashiCorp's own docs). Added .gitignore
coverage, a "State file handling" section in opentofu/README.md, an inline
caveat on maas_api_key's description, and a "SECURE THE STATE FILE" callout
at Stage 1 Step 10 -- the actual first real tofu apply (an early draft
wrongly assumed Stage 3; corrected after reading the runbooks directly).
DOCFIX-176: new runbooks/dc-dc-teardown-rollback.md for the OpenTofu/
libvirt/MAAS-vm_host layer, which D-061's existing teardown scripts don't
cover (different layer, juju/MAAS-machine). Two paths (scoped -target
teardown or full VR1 destroy), a rollback decision tree favoring
fix-forward over destroy, and explicit mesh-link shared-infrastructure
handling.
Also folded in: a fix to docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md
after re-running ledger-scan.sh caught that its own narration (and the
session-ledger mirror) had reintroduced the exact false-positive class
DOCFIX-174 had just fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-170: full-project sweep -- VR0 runbook + OpenTofu doc fixes
...
Part of an 8-dimension parallel sweep for errors/architectural problems
across the whole repo, requested after DOCFIX-169. phase-08-workload-
cluster-acceptance.md had the same cascading-staleness class as DOCFIX-169
(Designate framing never updated after DOCFIX-167's bundle.yaml edit) --
fixed with a nuanced correction distinguishing "charms now active/idle"
from "D-011.8's real acceptance criterion still needs Stage 7's overlay."
Also fixed a stale opentofu/modules/base-image/outputs.tf comment
referencing nonexistent cloudinit-vm variables. Session ledger updated
with the full 8-dimension sweep summary and two orphaned DOCFIX-number
closures (164, 167) caught as integration misses.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-163: Office1-local network module + OPNsense edge ownership (gaps #12+16)
...
New opentofu/modules/office1-network (dc-planes-shaped, one shared network
instead of six), instantiated for real in main.tf -- deliberately not a
reused host bridge, since that would live outside OpenTofu's D-103
inventory and break the virsh net-list --all completeness check used
throughout the runbooks. Decided gap #16: Office1 gets its own OPNsense
edge (a fourth site), owned by Stage 2's runbook (new Step 4b). Found a
deeper cross-site gap while wiring the edge's WAN side (no per-site
ISP-uplink network exists for ANY site yet, DC1/DC2 included) -- logged as
new gap #17 rather than defaulted to a mesh-link, which would contradict
D-100's management-only ruling for those legs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-149: OPNsense config.xml template + tested renderer
...
Real design work, researched before drafting: fetched OPNsense's own actual
shipped config.xml.sample and its Route.xml static-routes model before
writing anything.
opentofu/templates/opnsense-config.xml.tmpl: {{TOKEN}}-parameterized (reuses
this repo's existing clientdocs convention), covering interfaces/gateway/
staticroutes/DNS/NTP and D-107-shaped firewall rules (default-deny WAN
egress except NTP + the per-DC mirror's upstream sync). The real sample
fetch directly confirmed last turn's audit finding with an actual example:
it ships literal placeholder device names inside each interface's own <if>
block, proving LAN/WAN role assignment really is that explicit per-block
mapping, not declaration order.
scripts/opnsense-render-config.sh: the renderer. Needs no external tool
(unlike every other opnsense-* script this session), so it's tested
END-TO-END, not just guard clauses -- 8/8, including well-formed-XML
validation. The harness caught a real bug before it shipped: the token
HOSTNAME collides with bash's own built-in $HOSTNAME variable (unset
doesn't actually clear it), silently passing a test that should have
failed. Renamed to OPNSENSE_HOSTNAME throughout.
opentofu/templates/README.md's token legend marks exactly which values are
real (NTP pool default, D-106 naming) vs. pending Stage 0 ratification
(D-100/D-101/D-107) vs. only measurable on a real boot (vtnetN assignment)
vs. a security requirement (root password hash must be freshly generated,
never the stock sample's own shipped default).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-148: OpenTofu module audit pass + provider-docs skill reference
...
Audited every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules
built this session rather than leaving them as-is. Found and fixed a real
documentation error: opnsense-edge's comments claimed devices.interfaces
list order directly sets OPNsense's LAN/WAN role. Researching OPNsense's own
interface-assignment model showed this conflates two things -- list order
plausibly controls which vtnetN device number a NIC gets (libvirt's own
docs: auto-assigned PCI addresses "usually match" XML order for a simple
topology), but the actual LAN/WAN role is a separate, explicit mapping set
inside config.xml itself ("vtnet0=WAN" is a convention some guides choose,
not an enforced rule). Corrected main.tf's comments and both
lan_network_name/wan_network_name variable descriptions -- no config.xml
exists in this repo yet, so this fixes documentation accuracy for whoever
writes it next, not current behavior.
Confidence upgraded (not just re-hedged) on three other items: node-vm's
boot={order=N} shape (confirmed the provider 1:1-mirrors libvirt's own XML,
and libvirt's own docs confirm the single-attribute shape), create.content.url
accepting local paths (second independent source), and the genisoimage
flags (confirmed standard, cleanly separated from the still-open question of
whether OPNsense's Importer actually reads the result). Confirmed safe:
maas_vm_host's zone/pool computed-if-unset behavior.
Added .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md:
indexes every provider/doc source used this session with exact URLs and
confirmed versions, plus the fetch methodology that actually worked
(Registry doc pages are JS-rendered, use GitHub instead; branch names vary,
check default_branch via the API; real example .tf files beat doc-summarized
prose for syntax questions). Wired into SKILL.md's routing table,
cross-referenced from opentofu/README.md in both directions without
duplicating content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-147: opentofu/modules/maas-vm-host + modules/netem-link
...
Registers the vcloud host with MAAS as a virsh VM host, and applies tc netem
WAN-simulation parameters -- the two remaining unbuilt mechanisms from the
OpenTofu scope, researched before drafting per instruction.
maas-vm-host uses canonical/maas's maas_vm_host (register the chassis so
MAAS discovers already-existing VMs), deliberately not maas_vm_host_machine
(composes new VMs itself -- takes cores/memory/disks as inputs). Read both
schemas and cross-checked against D-103's explicit "MAAS does NOT compose
new ones" before picking one.
netem-link uses terraform_data (OpenTofu's own recommended replacement for
null_resource) with an SSH-wrapped local-exec provisioner -- confirmed
local-exec runs on the machine invoking tofu apply (Office1 per D-103), not
the vcloud host where the bridges live, so a bare local command would do
nothing useful.
Neither module instantiated: needs a real MAAS zone/pool, the vcloud host's
SSH target, and real netem parameters, none of which exist yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-146: opnsense-edge module + nano-image/config-ISO prep scripts
...
Implements the mechanism identified in today's OPNsense deployment research
(cloud-init confirmed unreliable on FreeBSD; the real native mechanism is
OPNsense's Configuration Importer, which scans an attached volume for
/conf/config.xml and gained ISO9660 support specifically for VM/cloud
automation).
scripts/opnsense-prep-image.sh: downloads the nano image, decompresses,
converts raw->qcow2, resizes -- work create.content.url almost certainly
can't do itself (a plain fetch, no decompression). No hardcoded mirror URL;
OPNSENSE_MIRROR_BASE is required explicitly.
scripts/opnsense-build-config-iso.sh: builds the ISO9660 image containing
/conf/config.xml that the Configuration Importer picks up.
opentofu/modules/opnsense-edge: wires both together, mechanically identical
to modules/cloudinit-vm's cdrom-attach shape but with no libvirt_cloudinit_disk
resource (wrong format for this). Explicit lan_network_name/wan_network_name
variables rather than an ordered list, since OPNsense's interface-1-is-LAN
convention makes list-ordering a real place for a silent mistake to hide. No
PXE boot-order attribute used -- this VM boots from its own disk, not the
network, so node-vm's flagged-unverified boot field doesn't apply here.
Both new scripts' harnesses (4/4, 3/3) exercise REAL missing-tool guards --
qemu-img/genisoimage/xorriso are all genuinely absent from this environment.
Two things flagged, not presented as fact: whether the Importer's ISO9660
behavior holds on a real boot, and whether interface list-order reliably
maps to LAN/WAN NIC enumeration for this guest. config_iso_path has no
default -- real config.xml content per site is still undesigned. Not
instantiated in root main.tf.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-145 + OPNsense deployment research
...
opentofu/modules/base-image + modules/cloudinit-vm: the cloud-init/pre-built-
image VM pattern for Office1's own service VMs (MAAS-region, NetBox,
GitBucket), verified against examples/alpine_cloudinit.tf. The base image's
.path is threaded across the module boundary as a real attribute reference
rather than reconstructed from pool+name strings. user_data/meta_data/
network_config are required inputs with no default -- the real Office1 VM
configs aren't designed yet, and a plausible DHCP default would silently
fail anyway since these planes carry no libvirt DHCP.
OPNsense research (no DOCFIX -- research delivery, not code): checked
OPNsense's own docs/forum/GitHub. Confirmed cloud-init is genuinely
unreliable on OPNsense (FreeBSD; OPNsense's own forum consensus), closing
the open question cloudinit-vm's build left. Found the real mechanism: the
Configuration Importer scans an attached volume for /conf/config.xml in a
2-3s boot window; ISO9660 support was added to it specifically for VM/cloud
automation -- mechanically identical to cloudinit-vm's cdrom-attach shape,
different payload. Use the nano image for KVM, not the installer images.
Fully sourced in opentofu/README.md. Not built yet, itemized: a decompress/
convert preprocessing script, an ISO9660 config.xml-seed builder, and the
real config.xml content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-144: opentofu/modules/node-vm + fix real syntax bug in DOCFIX-142
...
Built the D-103 node-VM pattern (blank disk, PXE-boot, MAAS-managed) for
Stage 3 -- this time from the provider's own real example .tf files
(examples/domain_with_network.tf, examples/alpine_cloudinit.tf), not
doc-summarization alone, since that confidence gap is exactly why the domain
module was deferred last time.
That surfaced a real bug in the already-pushed dc-planes/mesh-link/
dc-storage-pool modules (14d7382): they used classic HCL block syntax
(domain { ... }) where this provider's current schema needs attribute-style
objects (domain = { ... }) -- confirmed once real examples showed the actual
convention. All three fixed; opentofu/README.md's schema notes now state this
as a provider-wide assumption for any field not personally confirmed
otherwise.
One inference flagged, not presented as fact: the per-device boot-order
attribute's exact internal shape ({ order = N }) -- the field's existence is
confirmed, its shape is not. Not instantiated in root main.tf: node
count/sizing is a pending Phase-0 decision, not invented here. Still
deferred: the cloud-init/base-image VM pattern (Office1 services + OPNsense),
DC2 planes, netem application.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
Fold OPNsense's four separable sub-gaps into the tooling gap register
...
Gap #4 ("OPNsense automation + netem") was too coarse -- broke it into what
it actually is: (a) network substrate DONE (modules/mesh-link), (b) the edge
VM/image BLOCKED on the deferred domain module with no image source chosen,
(c) OPNsense's own routing/firewall config NOT STARTED with no mechanism
picked (config.xml is untouched by the libvirt provider), (d) tc netem WAN
simulation is a separate OS-level mechanism with no Terraform-native
equivalent, blocked on an unruled D-100 parameter decision before it can even
be scripted. Same breakdown mirrored into opentofu/README.md's deferred-items
list and the companion visual tracker.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-142: opentofu/ network + storage-pool scaffold (VR1 IaC, gap #2)
...
First OpenTofu content in the repo. Network/pool layer only: modules/dc-planes
(the six per-DC planes, D-052/D-100), modules/mesh-link (the D-100 dark-fiber
triangle), modules/dc-storage-pool, wired for DC1 (inherits DC0's CIDRs per
D-101) and Office1. DC2 deliberately not wired -- its CIDRs aren't assigned
yet (D-101 open sub-item).
Provider schema (dmacvicar/libvirt v0.9.8) verified against its actual current
docs this session, not memory -- it has diverged materially from older common
examples (network isolation is a nested forward.mode, not top-level mode;
libvirt_domain restructured to ~40 args mirroring raw libvirt XML). Node-VM/
volume resources, OPNsense config, and tc netem application are deliberately
deferred: that schema is too large/unfamiliar to author safely without a real
`tofu providers schema` pass on a connected machine -- see opentofu/README.md
for the full account and the concrete next step.
Added scripts/opentofu-validate.sh (fmt/init/validate wrapper) + a harness
testing what's actually exercisable without a tofu binary (none is available
anywhere this repo has been worked in this session -- logged, not hidden).
Extended repo_lint.py's L1 check to cover .tf files (uncovered before since
none existed). repo-lint 0 fail / 1 documented warn; both harnesses green.
Workflow doc + its companion visual tracker updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
