| 2026-07-16 |
Review sweep A+B + R-3 storage-04: coherent record, whole-host calculator, R-3-compliant Model A fallback
...
Phase A (record amendments) + Phase B (scripts/dc-dc-whole-host-budget.py + harness) of the
Stage-3 adversarial-review sweep, plus the R-3 storage-04 addition (Model A layout now
3+2+4 = 9 nodes/DC). This commit is the R-3-COMPLIANT Model A state and becomes the re-cut
`model-a-fallback` anchor (the prior tag at 87a7a8a was R-3-stale -- 3 storage -- flagged by
the Model B design cross-check). NO cloud mutation; nothing applied.
- design-decisions.md: D-103/D-107/D-114/D-115/D-121/D-122/D-123/D-124 amendments (rulings R-1..R-4).
- scripts/dc-dc-whole-host-budget.py + tests/ (13/13): replaces the lost optc-calc.py (R3-F06).
- opentofu/main.tf: locals.vr1_dc0_nodes += storage-04 (R-3); tofu validate Success.
- security-ledger SEC-011 closed; session-ledger rulings + sweep progress; model-a-fallback plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
...
Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:
- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
(D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
the office1<->dc0 mesh leg; rack sizing 4/8192/80.
Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
(two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
/24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
D-121/122 folded in, stale gate corrected).
docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-14 |
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-13 |
D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
...
APPLIED, operator-approved: 5 added, 0 changed, 0 destroyed. The running OPNsense edge was NOT
touched (confirmed by plan BEFORE apply -- no in-place update, so no repeat of the 2026-07-13
guest bounce).
THE MODULE WAS SILENTLY BROKEN FOR D-114'S ENTIRE MODEL. modules/cloudinit-vm's libvirt_domain
had NO `cpu` block. With none, libvirt renders a GENERIC EMULATED CPU exposing no `svm` -- and
LXD *virtual machines* are qemu/KVM guests, so NO service VM could ever have been composed into
voffice1. D-114 would have failed at its first step with a confusing "KVM not available" that
pointed nowhere near OpenTofu. Measured proof of the masking: host is an AMD EPYC 9965; a
default-CPU guest was handed an "Opteron_G3".
Fix: `cpu { mode = "host-passthrough" }` always, plus a new `expose_nested_virt` variable with NO
default -- a real per-VM decision. true = pass `svm` through (required for D-114's containment
VMs); false = disable it, matching modules/opnsense-edge. The operator asked whether the edge's
`svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a
triple-fault fix on 2026-07-12, did NOT resolve it (the real cause was the memory unit), and was
retained on hardening grounds. A router has no business seeing nested virt. It stays disabled there.
voffice1 as built (MEASURED): Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, single NIC on
office1-local, reports the real EPYC CPU with svm on all 16 cores and /dev/kvm PRESENT, egress to
1.1.1.1 at 4.6ms THROUGH the edge. Cloud-init deliberately MINIMAL (identity + key + guest agent);
MAAS and LXD go in as separate GATED steps rather than buried in a first-boot script that either
silently works or silently does not. The SSH pubkey is read via file(var...) at plan time -- key
material never enters a command line, the repo, or an agent's context; the tfvars holding the path
stays gitignored. network_config matches the NIC by GLOB, not a guessed kernel name (an inferred
value); it came up enp1s0.
TWO GATES CLEARED:
1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item: the daemon was proven, the
SERVICE never was (office1-local had no client). voffice1 -- the first client ever on that LAN
-- took 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state
active, read back through the D-113(a2) REST API. DHCP now works end to end.
2. D-114's NESTING PROBE PASSES AT L3. Stated honestly: this proves nested KVM is AVAILABLE. The
DEFINITIVE proof is a guest actually booting inside voffice1 -- that lands with the first
composed LXD VM. Not recording L3 as fully proven until then.
FINDING, logged NOT actioned (DC1 is gated behind Office1 per D-114): modules/node-vm has the SAME
missing-cpu-block defect. DC nodes run nova-compute, which needs working KVM -- without the fix
they will come up unable to run a single instance. MUST be fixed before Stage 3.
opentofu-validate PASS, repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
| 2026-07-12 |
DOCFIX-189: re-audit the max-context session's troubleshooting + docs
...
Operator-instructed re-evaluation. The 2026-07-12 session authored its docs and
module comments WHILE WORKING FROM A WRONG DIAGNOSIS, at a context limit, so its
reasoning was re-checked rather than just its one-line bug (DOCFIX-188).
Headline: the settings it changed are mostly fine and are RETAINED. The reasoning
it recorded was not -- it attributed the fault to whatever knob was being turned,
on no evidence, four separate times. It also left one silently-dead knob.
FALSE CAUSAL CLAIMS (corrected at the source, opnsense-edge/main.tf). The disk
shape, q35->i440fx, the serial console and svm-disable were EACH written up as a
contributing cause. None changed the symptom -- the fault sat at a deterministic
262 bytes through all of them. All four settings RETAINED (each defensible on its
own merits); only the false justifications are gone. The serial console earned its
keep, but it REVEALED the fault, it did not change it.
DEAD KNOB (real functional defect). `disk_size_bytes = 16 GiB` did nothing once
DOCFIX-187 made the disk a direct copy; unused variables are legal HCL so neither
validate nor plan complained. MEASURED: the live disk is 11.00 GiB, not 16.
Removed from main.tf + module variables; real sizing is opnsense-prep-image.sh's
GROW (+8G). Proof it was dead: tofu plan is byte-identical before/after removal.
STALE/UNSOURCED FACTS. "Opteron_G3" is a red herring -- the host is really an AMD
EPYC 9965 (Zen 5, family 26); libvirt's CPU-model DB doesn't know family 26 and
falls back to the oldest name, and the prior session partly built its nested-virt
theory on that artifact. "OPNsense's 3 GB min" is unsourced -- annotated, do not
propagate. DOCFIX-186's backing_store parenthetical is stale (same conclusion,
different mechanism). DOCFIX-187's changelog gains a SUPERSEDED-IN-PART header.
KEPT: DOCFIX-186's apparmor + config-iso-staging findings are real and measured.
DOCFIX-185 (operator transport ruling) untouched.
STANDING LESSON: when a change does not fix the symptom, do not write it up as
though it did. A fault that is deterministic and immune to every knob you turn
means you have not yet touched the cause.
Gauntlet ALL GREEN (52); repo-lint 0 fail; tofu fmt/validate/plan clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
Office1 OPNsense edge build: real-ISP-router config + first-boot fixes
...
Bringing the Office1 OPNsense edge online (router + DHCP for office1-local)
before the Office1 NetBox VM. First-ever OPNsense boot in this repo; several
first-execution findings fixed.
DOCFIX-184: opnsense-prep-image.sh decompresses the .img.bz2 via python3's
stdlib bz2 when bunzip2 is absent (no sudo install needed). Harness 3/3.
DOCFIX-185: the OPNsense edge is a normal simulated-ISP router, not an
egress-airgap. Operator clarified the transport model -- each site (DCs +
Office) has its own real ISP connection; dark fiber is East-West/replication
only; D-107 node-airgap is a separate DC concern. Stripped the WAN
egress-control rules (seq 20/21/99) + MIRROR_* tokens from the config template,
render script, and harness (8/8). Flagged a D-100/D-107 amendment note as
follow-up.
DOCFIX-186: instantiate module "office1_opnsense" in main.tf (first real
validation of modules/opnsense-edge). Two infra findings: (1) config_iso_path
must be OUTSIDE the pool dir (create.content.url collides with its own target
volume) -> stage the ISO; (2) apparmor blocks the custom /var/lib/libvirt/vr1/
pool path (foundational -- all VR1 VMs) -> needs a one-time local apparmor rule
(operator sudo). office1-wan NAT network created via virsh (D-103 debt).
repo-lint 0 fail; opnsense-prep-image 3/3; opnsense-render-config 8/8.
Boot blocked only on the apparmor fix (pending operator sudo).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-163: Office1-local network module + OPNsense edge ownership (gaps #12+16)
...
New opentofu/modules/office1-network (dc-planes-shaped, one shared network
instead of six), instantiated for real in main.tf -- deliberately not a
reused host bridge, since that would live outside OpenTofu's D-103
inventory and break the virsh net-list --all completeness check used
throughout the runbooks. Decided gap #16: Office1 gets its own OPNsense
edge (a fourth site), owned by Stage 2's runbook (new Step 4b). Found a
deeper cross-site gap while wiring the edge's WAN side (no per-site
ISP-uplink network exists for ANY site yet, DC1/DC2 included) -- logged as
new gap #17 rather than defaulted to a mesh-link, which would contradict
D-100's management-only ruling for those legs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-147: opentofu/modules/maas-vm-host + modules/netem-link
...
Registers the vcloud host with MAAS as a virsh VM host, and applies tc netem
WAN-simulation parameters -- the two remaining unbuilt mechanisms from the
OpenTofu scope, researched before drafting per instruction.
maas-vm-host uses canonical/maas's maas_vm_host (register the chassis so
MAAS discovers already-existing VMs), deliberately not maas_vm_host_machine
(composes new VMs itself -- takes cores/memory/disks as inputs). Read both
schemas and cross-checked against D-103's explicit "MAAS does NOT compose
new ones" before picking one.
netem-link uses terraform_data (OpenTofu's own recommended replacement for
null_resource) with an SSH-wrapped local-exec provisioner -- confirmed
local-exec runs on the machine invoking tofu apply (Office1 per D-103), not
the vcloud host where the bridges live, so a bare local command would do
nothing useful.
Neither module instantiated: needs a real MAAS zone/pool, the vcloud host's
SSH target, and real netem parameters, none of which exist yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-142: opentofu/ network + storage-pool scaffold (VR1 IaC, gap #2)
...
First OpenTofu content in the repo. Network/pool layer only: modules/dc-planes
(the six per-DC planes, D-052/D-100), modules/mesh-link (the D-100 dark-fiber
triangle), modules/dc-storage-pool, wired for DC1 (inherits DC0's CIDRs per
D-101) and Office1. DC2 deliberately not wired -- its CIDRs aren't assigned
yet (D-101 open sub-item).
Provider schema (dmacvicar/libvirt v0.9.8) verified against its actual current
docs this session, not memory -- it has diverged materially from older common
examples (network isolation is a nested forward.mode, not top-level mode;
libvirt_domain restructured to ~40 args mirroring raw libvirt XML). Node-VM/
volume resources, OPNsense config, and tc netem application are deliberately
deferred: that schema is too large/unfamiliar to author safely without a real
`tofu providers schema` pass on a connected machine -- see opentofu/README.md
for the full account and the concrete next step.
Added scripts/opentofu-validate.sh (fmt/init/validate wrapper) + a harness
testing what's actually exercisable without a tofu binary (none is available
anywhere this repo has been worked in this session -- logged, not hidden).
Extended repo_lint.py's L1 check to cover .tf files (uncovered before since
none existed). repo-lint 0 fail / 1 documented warn; both harnesses green.
Workflow doc + its companion visual tracker updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
