| 2026-07-12 |
DOCFIX-192: the edge config silenced its own ONLY console (no serial, no video device)
...
Third lockout-class defect in the same config, found by executing D-112(c) for real.
After the config was applied and the box rebooted, the serial console went PERMANENTLY
SILENT -- no login prompt, no output, nothing (measured: raw console read returns empty).
The edge VM has NO VIDEO DEVICE, so the serial port is its only console and its only
break-glass path.
Cause: the config set neither <primaryconsole> nor <serialspeed>, so the applied config
did not select the serial console. Boot output still appears (the loader honours the nano
image's own /boot.config -S115200), which is why this looks like a hang rather than a
config choice -- the console dies exactly when OPNsense applies our config.
The box was NOT lost: DOCFIX-191 had just enabled sshd + installed the service key, so it
stayed reachable. That is the only reason this was a finding and not an outage. Two
lockout bugs in one config, and the first one saved us from the second.
Key names VERIFIED against upstream src/www/system_advanced_admin.php:
- `primaryconsole` and `serialspeed` are STRING values.
- There is NO `enableserial` key in OPNsense -- that is a pfSense-ism. Do not add it.
115200 matches the nano image's own /boot.config.
Harness 12 -> 15 PASS. T9b asserts the serial console is retained; T9c guards against
someone "fixing" this by re-adding the bogus pfSense <enableserial> key.
(T9c initially failed on my own template COMMENT, which names enableserial while warning
against it. Tightened to match an actual element. The harness caught it.)
Gauntlet ALL GREEN (52); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
D-112: record option (a) as MEASURED INFEASIBLE; rule the (c) bootstrap; make the edge console interactive
...
Operator asked to re-examine option (a) (bake /conf/config.xml into the nano) before
committing to (c). Done, and it is BLOCKED on this host -- measured, not assumed:
CONFIG_UFS_FS=m but "# CONFIG_UFS_FS_WRITE is not set" -> UFS driver is READ-ONLY
partition table: MBR, a single FreeBSD a5 partition -> UFS only, no FAT to write
fuse-ufs / ufs2-tools -> not present, not packaged
libguestfs (guestfish / virt-copy-in) -> not installed
Two ways (a) could be forced, both rejected and recorded so they are not re-litigated:
a FreeBSD helper VM (works, xorriso is present to remaster a live ISO -- but it means
maintaining a builder VM forever to deliver one 4.5 KB file), and a raw byte-patch of
the image (probably works; silently corrupts if the file is not block-contiguous, and
is NON-TRANSFERABLE -- you cannot byte-patch a real appliance on Roosevelt hardware,
which is exactly what this repo's governing constraint forbids).
Examining (a) properly STRENGTHENED (c): (a) is not merely less mainstream here, it is
unavailable without a detour costing more than (c)'s entire bootstrap.
Bootstrap RULED (operator): B1 = temporary host leg on virbr2 (reversible; no change to
the D-100 network definition). B2 = one interactive console login to enable SSH and
install the ed25519 pubkey already generated in ~/vr1-office1-creds/, after which
everything is plain SSH/SCP.
Module change enabling B2: the edge's serial becomes a unix socket + a log file, rather
than a write-only file source. Both roles are needed and they are different:
source.unix -> INTERACTIVE bidirectional console. A `file` source is write-only capture
and cannot be typed into, so it cannot bootstrap anything. A unix socket
is scriptable with `socat - UNIX-CONNECT:`, unlike a pty, which needs a
controlling TTY.
log -> preserves full boot capture from power-on (a socket loses anything
emitted before a client connects). That capture is what made both
2026-07-12 boot bugs legible; it is not being dropped.
S1+S2 guards PASS, harness 6/6, repo-lint 0 fail, tofu fmt/validate clean.
Live recreate + host leg remain separately gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-189: re-audit the max-context session's troubleshooting + docs
...
Operator-instructed re-evaluation. The 2026-07-12 session authored its docs and
module comments WHILE WORKING FROM A WRONG DIAGNOSIS, at a context limit, so its
reasoning was re-checked rather than just its one-line bug (DOCFIX-188).
Headline: the settings it changed are mostly fine and are RETAINED. The reasoning
it recorded was not -- it attributed the fault to whatever knob was being turned,
on no evidence, four separate times. It also left one silently-dead knob.
FALSE CAUSAL CLAIMS (corrected at the source, opnsense-edge/main.tf). The disk
shape, q35->i440fx, the serial console and svm-disable were EACH written up as a
contributing cause. None changed the symptom -- the fault sat at a deterministic
262 bytes through all of them. All four settings RETAINED (each defensible on its
own merits); only the false justifications are gone. The serial console earned its
keep, but it REVEALED the fault, it did not change it.
DEAD KNOB (real functional defect). `disk_size_bytes = 16 GiB` did nothing once
DOCFIX-187 made the disk a direct copy; unused variables are legal HCL so neither
validate nor plan complained. MEASURED: the live disk is 11.00 GiB, not 16.
Removed from main.tf + module variables; real sizing is opnsense-prep-image.sh's
GROW (+8G). Proof it was dead: tofu plan is byte-identical before/after removal.
STALE/UNSOURCED FACTS. "Opteron_G3" is a red herring -- the host is really an AMD
EPYC 9965 (Zen 5, family 26); libvirt's CPU-model DB doesn't know family 26 and
falls back to the oldest name, and the prior session partly built its nested-virt
theory on that artifact. "OPNsense's 3 GB min" is unsourced -- annotated, do not
propagate. DOCFIX-186's backing_store parenthetical is stale (same conclusion,
different mechanism). DOCFIX-187's changelog gains a SUPERSEDED-IN-PART header.
KEPT: DOCFIX-186's apparmor + config-iso-staging findings are real and measured.
DOCFIX-185 (operator transport ruling) untouched.
STANDING LESSON: when a change does not fix the symptom, do not write it up as
though it did. A fault that is deterministic and immune to every knob you turn
means you have not yet touched the cause.
Gauntlet ALL GREEN (52); repo-lint 0 fail; tofu fmt/validate/plan clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-187: opnsense-edge module boot fixes + boot-triple-fault incident report
...
Office1 OPNsense edge first-boot debugging. Several correct module fixes made;
the boot is NOT yet resolved (BTX-loader triple-fault under double-nested virt),
documented as an OPEN incident for the next session.
Module (opentofu/modules/opnsense-edge/main.tf), all kept (a working config
needs them, none resolved the fault):
- serial console (nano is serial-only; the module had none) + boot-log capture
- machine q35 -> i440fx (pc)
- disk COW-overlay -> direct per-VM copy (matches virt-install --import)
- CPU host-passthrough with AMD svm disabled (documented AMD nested-virt fix;
note the provider key is `features` plural, not `feature`)
Incident report docs/incident-20260712-opnsense-edge-boot-triplefault.md: full
symptom (boot2 echoes /boot.config then triple-faults loading /boot/loader, a
consistent 262-byte fault confirmed fresh each boot), environment (double-nested
AMD Opteron_G3), everything tried, and ranked next steps (full CPU flags
+kvm_pv_eoi/+kvm_pv_unhalt, add a video device for the -D dual console, mem 4G,
UEFI/OVMF, outer-hypervisor CPU). Ledger checkpoint added; session at context
limit.
repo-lint 0 fail; module tofu validate Success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-148: OpenTofu module audit pass + provider-docs skill reference
...
Audited every "UNVERIFIED"/"inferred"/"assumed" marker across all 9 modules
built this session rather than leaving them as-is. Found and fixed a real
documentation error: opnsense-edge's comments claimed devices.interfaces
list order directly sets OPNsense's LAN/WAN role. Researching OPNsense's own
interface-assignment model showed this conflates two things -- list order
plausibly controls which vtnetN device number a NIC gets (libvirt's own
docs: auto-assigned PCI addresses "usually match" XML order for a simple
topology), but the actual LAN/WAN role is a separate, explicit mapping set
inside config.xml itself ("vtnet0=WAN" is a convention some guides choose,
not an enforced rule). Corrected main.tf's comments and both
lan_network_name/wan_network_name variable descriptions -- no config.xml
exists in this repo yet, so this fixes documentation accuracy for whoever
writes it next, not current behavior.
Confidence upgraded (not just re-hedged) on three other items: node-vm's
boot={order=N} shape (confirmed the provider 1:1-mirrors libvirt's own XML,
and libvirt's own docs confirm the single-attribute shape), create.content.url
accepting local paths (second independent source), and the genisoimage
flags (confirmed standard, cleanly separated from the still-open question of
whether OPNsense's Importer actually reads the result). Confirmed safe:
maas_vm_host's zone/pool computed-if-unset behavior.
Added .claude/skills/openstack-cloud-ops/references/opentofu-provider-docs.md:
indexes every provider/doc source used this session with exact URLs and
confirmed versions, plus the fetch methodology that actually worked
(Registry doc pages are JS-rendered, use GitHub instead; branch names vary,
check default_branch via the API; real example .tf files beat doc-summarized
prose for syntax questions). Wired into SKILL.md's routing table,
cross-referenced from opentofu/README.md in both directions without
duplicating content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-146: opnsense-edge module + nano-image/config-ISO prep scripts
...
Implements the mechanism identified in today's OPNsense deployment research
(cloud-init confirmed unreliable on FreeBSD; the real native mechanism is
OPNsense's Configuration Importer, which scans an attached volume for
/conf/config.xml and gained ISO9660 support specifically for VM/cloud
automation).
scripts/opnsense-prep-image.sh: downloads the nano image, decompresses,
converts raw->qcow2, resizes -- work create.content.url almost certainly
can't do itself (a plain fetch, no decompression). No hardcoded mirror URL;
OPNSENSE_MIRROR_BASE is required explicitly.
scripts/opnsense-build-config-iso.sh: builds the ISO9660 image containing
/conf/config.xml that the Configuration Importer picks up.
opentofu/modules/opnsense-edge: wires both together, mechanically identical
to modules/cloudinit-vm's cdrom-attach shape but with no libvirt_cloudinit_disk
resource (wrong format for this). Explicit lan_network_name/wan_network_name
variables rather than an ordered list, since OPNsense's interface-1-is-LAN
convention makes list-ordering a real place for a silent mistake to hide. No
PXE boot-order attribute used -- this VM boots from its own disk, not the
network, so node-vm's flagged-unverified boot field doesn't apply here.
Both new scripts' harnesses (4/4, 3/3) exercise REAL missing-tool guards --
qemu-img/genisoimage/xorriso are all genuinely absent from this environment.
Two things flagged, not presented as fact: whether the Importer's ISO9660
behavior holds on a real boot, and whether interface list-order reliably
maps to LAN/WAN NIC enumeration for this guest. config_iso_path has no
default -- real config.xml content per site is still undesigned. Not
instantiated in root main.tf.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
