| 2026-07-13 |
D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
...
APPLIED, operator-approved: 5 added, 0 changed, 0 destroyed. The running OPNsense edge was NOT
touched (confirmed by plan BEFORE apply -- no in-place update, so no repeat of the 2026-07-13
guest bounce).
THE MODULE WAS SILENTLY BROKEN FOR D-114'S ENTIRE MODEL. modules/cloudinit-vm's libvirt_domain
had NO `cpu` block. With none, libvirt renders a GENERIC EMULATED CPU exposing no `svm` -- and
LXD *virtual machines* are qemu/KVM guests, so NO service VM could ever have been composed into
voffice1. D-114 would have failed at its first step with a confusing "KVM not available" that
pointed nowhere near OpenTofu. Measured proof of the masking: host is an AMD EPYC 9965; a
default-CPU guest was handed an "Opteron_G3".
Fix: `cpu { mode = "host-passthrough" }` always, plus a new `expose_nested_virt` variable with NO
default -- a real per-VM decision. true = pass `svm` through (required for D-114's containment
VMs); false = disable it, matching modules/opnsense-edge. The operator asked whether the edge's
`svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a
triple-fault fix on 2026-07-12, did NOT resolve it (the real cause was the memory unit), and was
retained on hardening grounds. A router has no business seeing nested virt. It stays disabled there.
voffice1 as built (MEASURED): Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, single NIC on
office1-local, reports the real EPYC CPU with svm on all 16 cores and /dev/kvm PRESENT, egress to
1.1.1.1 at 4.6ms THROUGH the edge. Cloud-init deliberately MINIMAL (identity + key + guest agent);
MAAS and LXD go in as separate GATED steps rather than buried in a first-boot script that either
silently works or silently does not. The SSH pubkey is read via file(var...) at plan time -- key
material never enters a command line, the repo, or an agent's context; the tfvars holding the path
stays gitignored. network_config matches the NIC by GLOB, not a guessed kernel name (an inferred
value); it came up enp1s0.
TWO GATES CLEARED:
1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item: the daemon was proven, the
SERVICE never was (office1-local had no client). voffice1 -- the first client ever on that LAN
-- took 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state
active, read back through the D-113(a2) REST API. DHCP now works end to end.
2. D-114's NESTING PROBE PASSES AT L3. Stated honestly: this proves nested KVM is AVAILABLE. The
DEFINITIVE proof is a guest actually booting inside voffice1 -- that lands with the first
composed LXD VM. Not recording L3 as fully proven until then.
FINDING, logged NOT actioned (DC1 is gated behind Office1 per D-114): modules/node-vm has the SAME
missing-cpu-block defect. DC nodes run nova-compute, which needs working KVM -- without the fix
they will come up unable to run a single instance. MUST be fixed before Stage 3.
opentofu-validate PASS, repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-175/176: OpenTofu state file security + VR1 teardown/rollback runbook
...
Found while reviewing what's missing from the VR1 buildout ahead of
tomorrow's execution.
DOCFIX-175: terraform.tfstate stores sensitive=true variables (like
maas_api_key) in plaintext regardless of the flag, which only redacts
CLI/plan output (verified against HashiCorp's own docs). Added .gitignore
coverage, a "State file handling" section in opentofu/README.md, an inline
caveat on maas_api_key's description, and a "SECURE THE STATE FILE" callout
at Stage 1 Step 10 -- the actual first real tofu apply (an early draft
wrongly assumed Stage 3; corrected after reading the runbooks directly).
DOCFIX-176: new runbooks/dc-dc-teardown-rollback.md for the OpenTofu/
libvirt/MAAS-vm_host layer, which D-061's existing teardown scripts don't
cover (different layer, juju/MAAS-machine). Two paths (scoped -target
teardown or full VR1 destroy), a rollback decision tree favoring
fix-forward over destroy, and explicit mesh-link shared-infrastructure
handling.
Also folded in: a fix to docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md
after re-running ledger-scan.sh caught that its own narration (and the
session-ledger mirror) had reintroduced the exact false-positive class
DOCFIX-174 had just fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-147: opentofu/modules/maas-vm-host + modules/netem-link
...
Registers the vcloud host with MAAS as a virsh VM host, and applies tc netem
WAN-simulation parameters -- the two remaining unbuilt mechanisms from the
OpenTofu scope, researched before drafting per instruction.
maas-vm-host uses canonical/maas's maas_vm_host (register the chassis so
MAAS discovers already-existing VMs), deliberately not maas_vm_host_machine
(composes new VMs itself -- takes cores/memory/disks as inputs). Read both
schemas and cross-checked against D-103's explicit "MAAS does NOT compose
new ones" before picking one.
netem-link uses terraform_data (OpenTofu's own recommended replacement for
null_resource) with an SSH-wrapped local-exec provisioner -- confirmed
local-exec runs on the machine invoking tofu apply (Office1 per D-103), not
the vcloud host where the bridges live, so a bare local command would do
nothing useful.
Neither module instantiated: needs a real MAAS zone/pool, the vcloud host's
SSH target, and real netem parameters, none of which exist yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-142: opentofu/ network + storage-pool scaffold (VR1 IaC, gap #2)
...
First OpenTofu content in the repo. Network/pool layer only: modules/dc-planes
(the six per-DC planes, D-052/D-100), modules/mesh-link (the D-100 dark-fiber
triangle), modules/dc-storage-pool, wired for DC1 (inherits DC0's CIDRs per
D-101) and Office1. DC2 deliberately not wired -- its CIDRs aren't assigned
yet (D-101 open sub-item).
Provider schema (dmacvicar/libvirt v0.9.8) verified against its actual current
docs this session, not memory -- it has diverged materially from older common
examples (network isolation is a nested forward.mode, not top-level mode;
libvirt_domain restructured to ~40 args mirroring raw libvirt XML). Node-VM/
volume resources, OPNsense config, and tc netem application are deliberately
deferred: that schema is too large/unfamiliar to author safely without a real
`tofu providers schema` pass on a connected machine -- see opentofu/README.md
for the full account and the concrete next step.
Added scripts/opentofu-validate.sh (fmt/init/validate wrapper) + a harness
testing what's actually exercisable without a tofu binary (none is available
anywhere this repo has been worked in this session -- logged, not hidden).
Extended repo_lint.py's L1 check to cover .tf files (uncovered before since
none existed). repo-lint 0 fail / 1 documented warn; both harnesses green.
Workflow doc + its companion visual tracker updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
