| 2026-07-16 |
D-125 correction: uplink /24 is the ruled 172.30.2.0/24, not a HELD tfvar (no new NetBox object)
...
The initial D-125 records wrongly modeled the vcloud ISP /24 as a NEW,
NetBox-assigned HELD value (`var.vr1_dc0_uplink_cidr`) with an OPNsense WAN
re-address. That is factually wrong: bridge-in is single-NAT, so there is
exactly ONE simulated-ISP /24 for vr1-dc0, and D-115 already ruled it
(172.30.2.0/24, role edge, site vr1-dc0). NetBox models the prefix by role+site
and does not care which libvirt host runs the NAT, so the /24's IPAM identity is
UNCHANGED from Model A; only the libvirt realization moved (a vcloud NAT bridged
through vvr1-dc0). OPNsense WAN stays .2. (Caught reconciling the "create netbox
objects" request against dc-edge-wan-import.py, which already registers this /24.)
- opentofu/main.tf: module vr1_dc0_uplink cidr = "172.30.2.0/24" inlined (ruled
literal, matching the inner root's site-wan pattern) instead of the var.
- opentofu/variables.tf: REMOVED variable vr1_dc0_uplink_cidr (replaced with a
note explaining why it is a ruled literal, not a tfvar).
- Records corrected everywhere the false claim propagated: D-125 body + D-122
amendment, changelog-phaseC2-D125, model-a-fallback (B->A delta item 8 +
revert), session-ledger, dc-dc-deployment-workflow, phase-2 runbook.
Net effect: D-125 adds NO new NetBox object and NO new HELD gate. The only
NetBox work left for the DC substrate is D-124's transit /30 + rack IP.
Both roots validate; gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3 design gap
...
OBS-3: Model B nesting put vr1-dc0-wan (a libvirt NAT) inside vvr1-dc0, whose
only leg is the SEC-010 FORWARD-dropped transit -- so the per-DC simulated ISP
had no egress. Operator ruled bridge-in (single-NAT) over double-NAT.
- modules/wan-bridge (NEW): libvirt forward=bridge onto a pre-existing host
bridge (bridge={name}, schema measured from the provider, not guessed). No
subnet/NAT. Sibling to site-wan so every NAT user stays untouched. Covered by
opentofu-validate S3 (standalone PASS).
- OUTER root: vcloud ISP NAT vr1_dc0_uplink (the single NAT) + a 2nd IP-less
uplink NIC on vvr1-dc0 + the br-vr1-dc0-wan bridge declared in netplan (transit
stays NIC1 so SEC-010/--transit-if keep keying on it). New HELD var
vr1_dc0_uplink_cidr (no default; office1-netbox apex, hard rule 2).
- INNER root: vr1_dc0_wan site-wan(NAT) -> wan-bridge onto br-vr1-dc0-wan;
OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113, HELD).
- Bootstrap (site-headend-install.sh): --uplink-if/--wan-bridge; --check verifies
the bridge exists + uplink ENSLAVED (fail-open guard, same class as the transit
check); SEC-010 rationale corrected (bridge-in has no inner NAT) + br_netfilter
CONSTRAINT (keep the FORWARD-drop interface-scoped, NEVER global). Harness 49->54.
Records: D-125 (new); D-122 amendment (forward-pointer -- WAN realization
superseded, intent stands); SEC-010 (br_netfilter + hardened --check);
model-a-fallback B->A delta item 8 + revert; platform-traps 1g (bridge +
br_netfilter + nested-foreign-MAC trap).
NOT closed: the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in
equivalent of the depth-4 boot gate); fallback = double-NAT. HELD: vr1_dc0_uplink_cidr
+ OPNsense WAN re-address. Both roots validate; gauntlet ALL GREEN (63); repo-lint
0 fail. Present-only; NOT pushed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage-3 review sweep C+D: Model B two-root reshape + SEC-010 hardening
...
Phase C (Model B reshape, D-123 operator ruling):
- outer root (opentofu/main.tf): vvr1-dc0 resized to whole-DC containment VM
(108 vCPU / 425984 MiB / 3221225472000 bytes; expose_nested_virt=true;
single Office1 transit leg); node/plane/wan/opnsense modules removed from
vcloud level (MOVED markers), storage-04 added per R-3.
- inner root (opentofu/vr1-dc0-substrate/): NEW second root, provider
qemu+ssh to vvr1-dc0; reuses ../modules/* verbatim to build 9 nodes +
6 planes + wan + opnsense inside the containment VM. dmacvicar/libvirt
0.9.8 pinned (.terraform.lock.hcl committed). Both roots tofu-validate.
- variables.tf: vvr1_dc0_vcpu/memory_mib/disk_bytes with derivation comment.
Phase D (SEC-010, R3-F02): site-headend-install.sh --host-nodes node-host
bootstrap installs qemu-kvm+libvirt, persists+verifies nested=1, writes a
transit FORWARD-drop (nftables inet sec010) enforcing DC-LOCAL under Model B.
Gate hardened (advisor catch): --check now verifies the keyed transit
interface EXISTS, else nftables oifname loads clean but matches nothing
(fail-open). Harness 49/49.
Records: D-103/107/114/121/122/123/124 amendments (append-only);
model-a-fallback-plan synced; ledgers + Phase-C changelog.
KNOWN GAP (OBS-3, operator-confirmed 2026-07-16): under Model B the inner
vr1-dc0-wan NAT is nested inside vvr1-dc0 (single transit NIC), severing the
simulated-ISP egress that Model A had at vcloud level. DC0/DC1 ARE meant to
have ISP connections -- egress path resolution is follow-up design work, not
resolved by this commit.
Present-only; not pushed. Gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
...
Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:
- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
(D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
the office1<->dc0 mesh leg; rack sizing 4/8192/80.
Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
(two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
/24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
D-121/122 folded in, stale gate corrected).
docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-13 |
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SAFETY SWEEP: mark the config.xml push path as a LIVE HAZARD (no behaviour change)
...
After D-113(a2) was proven (Office1's config is now API-managed), every instruction in the repo
that still said "render a config.xml and push it to the edge" became ACTIVELY DANGEROUS: a full
config.xml push replaces /conf/config.xml wholesale and drops ~667 migration-populated elements
(measured), including the only 2 firewall pass rules on the box. Following the old runbook steps
against the live edge would clobber it.
Warnings and headers only. No behaviour changes, no live system touched.
Marked (5 files):
runbooks/dc-dc-phase1-office1-standup.md DANGER banner; the config.xml render + config-ISO
sub-steps removed as RUNNABLE instructions (they
would clobber the live edge). Image-prep retained;
history preserved in git + the build changelog.
runbooks/dc-dc-phase2-tofu-dc-substrate.md STOP banner on Step 4.
scripts/opnsense-render-config.sh DANGER header at point-of-use. NOT dead: under
D-113(a2) it is to be reduced to a MINIMAL bootstrap
render. Until then, safe only for a brand-new,
not-yet-booted edge.
scripts/opnsense-build-config-iso.sh RETIRED header quoting the upstream source proving
the importer can never fire on a nano image.
docs/dc-dc-deployment-workflow.md STALE-CONTENT warning atop the tooling gap register.
THE SWEEP ALSO CAUGHT A PRE-EXISTING LANDMINE, unrelated to today's work: dc-dc-phase2 Step 4
still instructs building a CONFIG ISO for DC1's edge -- but D-112 established that ISO can never
be read (opnsense-importer probes for a read-only root; on a pre-installed nano the root is
writable and a factory /conf/config.xml already exists, so it bootstrap_and_exit 0's without
enumerating a single device). That runbook has been telling anyone who follows it to build an
inert artifact and then wonder why the edge came up on factory defaults -- which is EXACTLY the
day lost on 2026-07-12. It was never corrected at the source. Now it is. Its WAN_IF/LAN_IF
"chicken-and-egg" discussion is moot for the same reason: that problem only exists if you try to
seed a full config before first boot, and D-112(c) measures vtnetN AFTER boot.
Changelogs deliberately NOT touched -- they are history, not instructions. Marking history is
noise; marking instructions is safety.
Verification: opnsense-render-config 24 PASS, opnsense-build-config-iso 2 PASS, opnsense-api
21 PASS; repo-lint 0 fail.
STILL OPEN (the actual D-113(a2) migration): reduce the template to a minimal bootstrap, retire
the config-ISO path and the opnsense-edge module's config_seed/cdrom wiring, and rewrite Stage 3's
edge steps around the API. This sweep makes the repo SAFE, not FINISHED.
Revert: git revert this commit (restores the previous, dangerous instructions verbatim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
| 2026-07-10 |
DOCFIX-175/176: OpenTofu state file security + VR1 teardown/rollback runbook
...
Found while reviewing what's missing from the VR1 buildout ahead of
tomorrow's execution.
DOCFIX-175: terraform.tfstate stores sensitive=true variables (like
maas_api_key) in plaintext regardless of the flag, which only redacts
CLI/plan output (verified against HashiCorp's own docs). Added .gitignore
coverage, a "State file handling" section in opentofu/README.md, an inline
caveat on maas_api_key's description, and a "SECURE THE STATE FILE" callout
at Stage 1 Step 10 -- the actual first real tofu apply (an early draft
wrongly assumed Stage 3; corrected after reading the runbooks directly).
DOCFIX-176: new runbooks/dc-dc-teardown-rollback.md for the OpenTofu/
libvirt/MAAS-vm_host layer, which D-061's existing teardown scripts don't
cover (different layer, juju/MAAS-machine). Two paths (scoped -target
teardown or full VR1 destroy), a rollback decision tree favoring
fix-forward over destroy, and explicit mesh-link shared-infrastructure
handling.
Also folded in: a fix to docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md
after re-running ledger-scan.sh caught that its own narration (and the
session-ledger mirror) had reintroduced the exact false-positive class
DOCFIX-174 had just fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-154/155/156: Stage 2/3/5 runbooks -- Office1 standup, DC substrate, per-DC bundle
...
Three DC-DC runbooks authored in parallel this evening:
- Stage 2 (Office1 headend standup): MAAS-region/NetBox/GitBucket/Tailscale,
each VM-creation step forking an explicit OpenTofu-blocked vs manual-debt
path rather than picking one silently. Flags a real gap: no Office1-local
network is modeled in opentofu/.
- Stage 3 (OpenTofu builds each DC substrate): DC1-first, DC2 hard-gated on
NetBox literals. Walks every open decision (node sizing, MAAS zone/pool,
config.xml tokens, netem params) and stops rather than inventing values.
- Stage 5 (Juju controller + bundle, per DC): describes adapting the
existing phase-01..08 loop to run twice, not duplicating it. Mandates the
$DC selector convention (DOCFIX-151) before every adapted command block.
Surfaces the undesigned IPv6 family-matrix overlay as blocking work.
All three: not yet executed (prep-only session, no live infrastructure
reachable tonight); repo-lint 0 fail/1 documented warn.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
