| 2026-07-22 |
G12 dc1: apex --commit executed + substrate step A applied (5/0/0 exact, converged)
...
Operator-gated session (as-executed: dc1-deploy, per-command wrap):
apex write CREATED 172.31.0.4/30 + 10.12.68.2/22 (preflight re-run +
idempotent read-back captured); dc1 svc keypair minted per manifest
(creds-audit CLEAN); tfvars authored (local, D-124 amendment verbatim);
outer saved-plan apply 5 added / 0 changed / 0 destroyed -- vvr1-dc1
RUNNING (108 vCPU / 416 GiB, measured host headroom ~140 GiB), prior
guests untouched; convergence ZERO DIFF
(docs/audit/outer-plan-20260722-postdc1-converged.txt).
Next gated step: dc1 step B bootstrap (CURRENT-STATE G12 row).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
G12 dc1: item-3 substrate landed + harness reconcile (gauntlet 76 ALL GREEN)
...
Lands the disconnected session's authored tree (vr1_dc1 vars, vvr1-dc1 +
uplink modules, inner root vr1-dc1-substrate/ with 54 pre-pinned MACs,
lib-net dc1 arm + one-selection guard) and delivers changelog item 4:
six harnesses reconciled to the ratified dc1 selector arm, phase-00
PLANES/lib-net parity guard (exit 2, refuses cross-DC planning),
rbd-mirror + radosgw parent-shell selection (dc1 reminder now shows
10.12.84.0/22, not dc0's). Gauntlet ALL GREEN 76
(docs/audit/gauntlet-20260722-g12-reconcile.txt); repo-lint 0-fail.
G12 next gated step unchanged: operator-gated apex --commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
| 2026-07-21 |
Ruling 1: vr1-dc0-maas RETIRED (repo removed); INCIDENT: pod delete cascaded to 9 machine records
...
- git rm opentofu/vr1-dc0-maas (operator-ruled 'Retire fully'); validate PASS
- maas vm-host delete 4 removed the stale pod AND the nine linked DC0
machine records (association check ran after the delete -- agent error,
owned; capture docs/audit/incident-20260721-pod-delete-cascade.txt)
- Substrate measured intact (10/10 domains, MAC pins config-carried);
CURRENT-STATE corrected same commit (GA-R1 C2); recovery flow gated,
pending operator decision; SEC-013 file cleanup HELD
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XRtm5sDQgUsZyTJk3k7dz
|
Stage-close batch: ledger-scan PARTIAL fix; DOCFIX-197; D-103/D-123 pod-refutation amendments; GA-R7 memory review
...
- scripts/ledger-scan.sh: PARTIALLY-RULED decisions now surface (D-131
blind spot); harness regression fixture, 46 checks ALL PASS
- DOCFIX-197: phase2 runbook Step 11 leg selection corrected to executed
reality (dc0<->dc1 leg; office1 leg carries live transit)
- D-103/D-123 amendments + modules/maas-vm-host header: 2026-07-20
measured pod refutation + per-machine-virsh ruling RECORDED (cited to
captures; no new ruling made)
- GA-R7 memory review DONE: no memory-only facts remain
- Read-only: stale MAAS pod object confirmed (id=4 vr1-dc0-inner, virsh)
for the pending retire-or-keep ruling
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XRtm5sDQgUsZyTJk3k7dz
|
Step E netem DONE -- G10 CLOSED; netem-link local mode; G16 opened (office1 channels residual)
...
- modules/netem-link: optional local execution (empty ssh target -> bare
sudo tc; D-128 outer root runs ON vcloud); NEW tests/netem-link harness
(12 cases); gauntlet 76 ALL GREEN
- outer root: module netem_vr1_dc0_vr1_dc1 wired -- virbr5 (measured at
wire time), placeholder profile 'delay 3ms 1ms loss 0.01%' (S6 lean,
PROVISIONAL; D-100 gap #11 unruled); runbook Step-11 leg divergence
flagged, DOCFIX queued
- 1/1/0 STOP honored; operator RULED 'Targeted netem apply (Recommended)';
saved -target plan exact 1/0/0, applied; qdisc live on virbr5,
virbr7/virbr3 untouched (docs/audit/stepE-netem-20260721.txt +
outer-{plan,apply}-20260721-netem*.txt)
- Convergence 0/1/0 = office1 channels=[] residual only
(outer-plan-20260721-postE-residual.txt) -> split per E3 into NEW gate
G16; CURRENT-STATE section 5 expected plan re-recorded 0/1/0 (GA-R1 C1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XRtm5sDQgUsZyTJk3k7dz
|
node-vm: MAAS owns node power -- ignore_changes on running (operator-ruled)
...
The MAC-pin verification plan (docs/audit/inner-plan-20260721-macpin.txt,
0/9/0, 54 mac adoptions, zero replaces) exposed 9 entangled out-of-band
power-ons: the module re-asserts running=true against nodes MAAS holds
OFF. Operator-ruled: guard first, then apply.
- lifecycle ignore_changes = [running]: create still boots (PXE
enlistment); afterwards MAAS owns power (virsh here, Roosevelt IPMI).
- tests/node-vm +3 cases (guard present, no scope creep, cites MAAS);
15/15 green, gauntlet 73 ALL GREEN, repo-lint 0 fail.
- CURRENT-STATE: MAC-pinning delivery status + pending gated apply.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STHXiHfoxHqq8fGRVvb66G
|
node-vm: pin NIC MACs (54 measured) -- in-place applies can no longer drift them
...
Queued delivery item from the 2026-07-21 commissioning diagnosis: an
"in-place" apply (0/9/0) regenerated all 9 unpinned boot MACs on
2026-07-20 and stranded the fleet in MAAS.
- modules/node-vm: optional interface_macs (all-or-nothing + format
validations); mac = { address = ... } shape verified against the
dmacvicar/libvirt 0.9.8 schema dump this session.
- vr1-dc0-substrate: all 9 nodes x 6 planes pinned to values MEASURED
this session (virsh domiflist on vvr1-dc0); metal-admin entries
cross-checked vs MAAS boot-interface records 9/9.
- NEW tests/node-vm harness (12 cases). Gauntlet 72 -> 73 ALL GREEN;
repo-lint 0 fail; opentofu-validate PASS.
NOT YET APPLIED on voffice1 (inner root state) -- the gated apply's plan
must show NO replace; any replace is a STOP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STHXiHfoxHqq8fGRVvb66G
|
| 2026-07-20 |
modules/node-vm: add serial console + boot log (parameterized dir)
...
Its absence blocked a live diagnosis: 6/9 dc0 nodes stall in MAAS 'Loading
ephemeral' and, with no console and no agent, a stuck node is unobservable.
Same pattern opnsense-edge already carries, for the same reason -- a
PXE-booting node emits everything interesting before anything sshable
exists. Unconditional (MAAS-managed nodes are rebuilt at will; no
live-router bounce risk like the D-129 qga channel). Log dir is a variable
rather than opnsense-edge's hardcoded vcloud literal, which is wrong on any
other libvirt host -- backport queued.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
NEW root opentofu/vr1-dc0-maas: MAAS VM-host registration, isolated from substrate roots
...
Third root by design, applying DOCFIX-179's own lesson rather than
repeating it: a provider 'maas' block forces EVERY plan in its root to
demand maas_api_url + the sensitive key, so coupling it to the substrate
roots would make routine substrate plans require MAAS creds. This root
touches MAAS only; the substrate roots stay credential-free. Registers
the pod via modules/maas-vm-host (discover, never compose -- D-103) with
power_address local to the rack per Model B. Validated explicitly (the
validator covers only the outer root + modules -- extending it to all
roots stays queued).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
modules/opnsense-edge: opt-in qga virtio channel (D-129); dc0 edge sets it true
...
Shape taken from the provider schema (devices.channels[].target.virt_io /
source.unix on dmacvicar 0.9.8), not inferred. Default OFF so existing
edges are never bounced by a plan; the new dc0 edge boots with it per the
runbook's no-retrofit intent for NEW DC edges. Without the channel the
os-qemu-guest-agent package is inert-but-present -- the false success the
D-129 review flagged -- so package and channel ship as one decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Deploy step C COMPLETE: inner apply 28/28 from voffice1, converged zero diff; 9 nodes + 26.7 edge RUNNING at depth-4; three first-real-apply fixes
...
voffice1 = Plane-2 executor (tofu 1.12.4, repo @29cf7bf, dc0 key,
keyscan'd known_hosts). Fixed forward, measured at first real use:
provider keyfile/sshauth URI params; wan-bridge illegal <mtu> in
bridge mode; base image uploads from the EXECUTING host (copied to
voffice1, description corrected); staging-dir literal created on
rack. Captures: inner-{plan,apply,converge}-20260720-stepC*.
Edge serial log at FreeBSD login prompt (26.7 D-112 first datapoint).
CURRENT-STATE position updated in this commit (GA-R1/C1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
wan-bridge: drop illegal <mtu> on bridge-mode network (measured at first real apply; MTU is the host bridge's)
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Inner root: provider keyfile+sshauth URI params (measured: provider ssh ignores ssh config/default identities); known_hosts multi-type note
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Deploy step B COMPLETE: rack bootstrapped + enrolled, SEC-010 CLOSED both ends (operator-ruled), OPNsense 26.7 base staged (operator-ruled); ledger-scan fail-open hardening
...
Bootstrap runs 1+2 (stale-lists 404 fixed by proxied apt update):
maas 3.7/stable, rack enrolled, libvirt+nested KVM, SEC-010 loaded
+ persistent on enp1s0; final --check EXIT 0 ALL items
(docs/audit/stepB-check-20260720-final.txt). voffice1-side SEC-010
pin applied per the row's follow-on clause. 26.7 nano prepped on the
rack (measured leaseweb mirror via region proxy); inner-root base
default -> 26.7. ledger-scan: FAIL-OPEN no longer counts as OPEN
(harness 45/45). CURRENT-STATE position/G10/G14 + section-5 updated
in this commit (GA-R1/C1). Open SEC count 8 -> 7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Step B transit reach ESTABLISHED: voffice1 .1/30 applied (Kea re-key incident fixed); vvr1-dc0 netplan set-name root cause fixed; zero-diff converged
...
voffice1 transit NIC applied 1/1/1 exact; MAC-regen broke the Kea
reservation (measured .108) -- re-keyed via edge API, .20 restored.
vvr1-dc0 both-NICs-silent root cause: netplan match.name+set-name
(one-boot rename lag, measured offline via libguestfs); fix drops
set-name, keeps measured enp1s0/enp2s0, instance-id bump, seed
-replace 2/0/2 exact. Verified: ssh -J voffice1 w/ dc0 key,
rack->region 0% loss. Outer plan ZERO DIFF converged.
CURRENT-STATE sections 3+5 updated in this commit (GA-R1/C1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
| 2026-07-19 |
Step B prereq: voffice1 transit NIC authored (172.31.0.1/30 region end, D-124); ruling recorded; expected plan 1/1/1
...
Measured: transit net isolated, only vvr1-dc0 attached; nothing held .1.
Operator-ruled mechanism 2026-07-19: voffice1 via IaC + restart.
Capture: docs/audit/outer-plan-20260719-voffice1-transit.txt (1/1/1).
CURRENT-STATE sections 3+5 updated in this commit (GA-R1/C1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
GA-F03: stale tofu 1.12.3 pins demoted to CURRENT-STATE pointers
...
opentofu/README.md:41 (historical prose keeps its date, drops the pin)
and docs/dc-dc-deployment-workflow.md:47/:360/:875 (the :360 live claim
"binary EXISTS: v1.12.3" was FALSE -- measured 1.12.4) now point at
docs/CURRENT-STATE.md section 7, the only pin authority (GA-R1 rule 3).
The :47 edit also drops a "DONE" status token (vocab overlap with 2.5).
Batch 2 item 2.2.
Revert: git revert this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
D-130 ADOPTED (a) + implemented: seed-volume ignore_changes guard (GA-F01, G4+G5 CLOSED)
...
Batch 1 items 1-4. G4 verifications captured: v8 = suppression CONFIRMED
(fresh baseline 7/2/7 identical to 2026-07-18, with-edit 6/2/6, zero
forces-replacement; DC0 creates unaffected); v7 = defect reproduced on
isolated throwaway-v7-* domain, replacement applied UNDER the running
domain with NO bounce (domid 3->3), teardown verified zero residue.
Operator ruled (a) per GA-R5 (utterance quoted in the D-130 entry).
Implemented: lifecycle { ignore_changes = [create] } on
libvirt_volume.seed in modules/cloudinit-vm + tests/cloudinit-vm
harness (6/6; guard placement, exact attribute, no leak onto data disk,
tofu validate). CURRENT-STATE: G4/G5 rows CLOSED + rotated, decision
queue item 1 rotated to the register, section 5 records post-fix 6/2/6
pending the G6 reconcile (C2: captures + doc same commit). 342/360 cap.
Revert: git revert this commit (restores the replace-pair behavior and
the pre-ruling CURRENT-STATE rows; captures remain in history).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
| 2026-07-17 |
D-127 IaC autostart matrix baked into deploy tooling + Office1 guest updates
...
Operator directive: bake the autoboot-vs-manual matrix into the IaC so DC0/DC1 get
correct autostart at creation, not by hand. Added an `autostart` bool var to modules
cloudinit-vm/opnsense-edge/node-vm (wired to libvirt_domain.autostart); set explicitly
per instance (shared modules => IaC is source of truth): voffice1=true,
office1-opnsense=true, vvr1-dc0=FALSE, vr1-dc0-opnsense=true, node-vm=FALSE.
HARD GATE verified: the two existing Office1 domains (voffice1, office1-opnsense) plan
as autostart false->true UPDATE-IN-PLACE, NOT replacement (replacement would destroy the
live host/edge). vvr1-dc0 is a new create with autostart=false baked in. Outer plan now
5 add / 2 in-place / 6 destroy; the 2 collapse to 0 once the bookend sets those two
autostart live pre-apply. tests/opentofu-validate T11-T15 pin the matrix against drift.
Standup runbook records the composed service VMs' LXD boot.autostart step. Confirmed
DC0/DC1 create NO LXD instances (inner root = node-vm + opnsense-edge + planes/pool/
wan-bridge) -- DC side is purely this VM matrix.
Office1 guest updates (agent sudo inside the guests): office1-netbox apt upgrade (no
reboot, netbox healthy); office1-tailscale full-upgrade -> kernel 6.8.0-136 + reboot
(back, re-advertising 10.10.0.0/22); voffice1 0-upgradable; LXD boot.autostart=true set
on both. vcloud + office1-opnsense stay in the operator's bookend.
Gauntlet ALL GREEN (66); opentofu-validate clean; repo-lint 0-fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-16 |
DC0 deploy-readiness handoff: per-env-key ruling implemented + netem ruled + prep checklist
...
Prep for a fresh session to run the Stage-3 Phase-2 DC0 build. Nothing applied;
the outer tofu apply remains the first (operator-gated) mutation.
Per-env-key (D-126, operator ruling 2026-07-16, option a): vvr1-dc0 gets its OWN
access key, not office1's. Minted ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 (creds-audit
CLEAN); added variable vr1_dc0_ssh_pubkey_path; vvr1-dc0 cloud-init now installs the
dc0 pubkey (voffice1 keeps office1's key -- only the vvr1_dc0 block changed). Re-
validated + re-planned: outer plan UNCHANGED at 5 add / 0 change / 6 destroy (the
cloud-init edit is create-only content, no new destroys/replaces of live infra).
netem (D-100 gap #11): RULED = placeholder for the rehearsal; final values deferred.
docs/dc0-deploy-readiness.md (NEW): the full readiness state (outer plan verified
safe -- vvr1-dc0 416GiB/108vcpu, 6 empty-plane relocation, 980GiB RAM headroom),
remaining prep (pre-apply / mid-deploy / non-blocking), fresh-session bootstrap, and
the gated A-E deploy sequence. Ledger points at it.
Gauntlet ALL GREEN (65 harnesses incl. the 2 new access harnesses); repo-lint 0-fail;
opentofu-validate clean (both roots + 12 modules). The dc0 private key + local tfvars
stay gitignored/off-repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125 correction: uplink /24 is the ruled 172.30.2.0/24, not a HELD tfvar (no new NetBox object)
...
The initial D-125 records wrongly modeled the vcloud ISP /24 as a NEW,
NetBox-assigned HELD value (`var.vr1_dc0_uplink_cidr`) with an OPNsense WAN
re-address. That is factually wrong: bridge-in is single-NAT, so there is
exactly ONE simulated-ISP /24 for vr1-dc0, and D-115 already ruled it
(172.30.2.0/24, role edge, site vr1-dc0). NetBox models the prefix by role+site
and does not care which libvirt host runs the NAT, so the /24's IPAM identity is
UNCHANGED from Model A; only the libvirt realization moved (a vcloud NAT bridged
through vvr1-dc0). OPNsense WAN stays .2. (Caught reconciling the "create netbox
objects" request against dc-edge-wan-import.py, which already registers this /24.)
- opentofu/main.tf: module vr1_dc0_uplink cidr = "172.30.2.0/24" inlined (ruled
literal, matching the inner root's site-wan pattern) instead of the var.
- opentofu/variables.tf: REMOVED variable vr1_dc0_uplink_cidr (replaced with a
note explaining why it is a ruled literal, not a tfvar).
- Records corrected everywhere the false claim propagated: D-125 body + D-122
amendment, changelog-phaseC2-D125, model-a-fallback (B->A delta item 8 +
revert), session-ledger, dc-dc-deployment-workflow, phase-2 runbook.
Net effect: D-125 adds NO new NetBox object and NO new HELD gate. The only
NetBox work left for the DC substrate is D-124's transit /30 + rack IP.
Both roots validate; gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
docs(opentofu/README): reconcile to Model B two-root + neutralize deleted config-ISO path
...
opentofu/README.md described a single-root pre-Model-B tree and, dangerously,
the DELETED OPNsense config.xml/config-ISO delivery as current. Added an
authoritative STATUS-RECONCILED banner (repo/ledger authoritative over the older
per-module prose) and inline SUPERSEDED/DELETED markers. Grep-verified:
- TWO ROOTS (D-123 Model B): outer opentofu/ + inner opentofu/vr1-dc0-substrate/,
bootstrap gate between; 9 node VMs (D-121/R-3); D-125 vcloud ISP uplink.
- Modules ADDED, missing from the Built list: site-wan (D-122), wan-bridge (D-125).
- INSTANTIATED (contradicts "not instantiated" prose): cloudinit-vm + base-image
(voffice1/vvr1-dc0/ubuntu_noble_base), node-vm (inner), dc-planes/site-wan/
wan-bridge/opnsense-edge (inner).
- DANGER neutralized: templates/opnsense-config.xml.tmpl,
scripts/opnsense-build-config-iso.sh, scripts/opnsense-render-config.sh are
DELETED (verified) -- a full config.xml push clobbers the live API-managed edge
(D-112/D-113(a2)). Banner + inline markers on the Built-list refs; deeper
2026-07-09 research/audit refs are framed historical by the banner.
- GitBucket out of scope (D-116) where listed.
Kept Stage-1 history (applied 2026-07-10, DOCFIX-179 provider mapping). Doc-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3 design gap
...
OBS-3: Model B nesting put vr1-dc0-wan (a libvirt NAT) inside vvr1-dc0, whose
only leg is the SEC-010 FORWARD-dropped transit -- so the per-DC simulated ISP
had no egress. Operator ruled bridge-in (single-NAT) over double-NAT.
- modules/wan-bridge (NEW): libvirt forward=bridge onto a pre-existing host
bridge (bridge={name}, schema measured from the provider, not guessed). No
subnet/NAT. Sibling to site-wan so every NAT user stays untouched. Covered by
opentofu-validate S3 (standalone PASS).
- OUTER root: vcloud ISP NAT vr1_dc0_uplink (the single NAT) + a 2nd IP-less
uplink NIC on vvr1-dc0 + the br-vr1-dc0-wan bridge declared in netplan (transit
stays NIC1 so SEC-010/--transit-if keep keying on it). New HELD var
vr1_dc0_uplink_cidr (no default; office1-netbox apex, hard rule 2).
- INNER root: vr1_dc0_wan site-wan(NAT) -> wan-bridge onto br-vr1-dc0-wan;
OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113, HELD).
- Bootstrap (site-headend-install.sh): --uplink-if/--wan-bridge; --check verifies
the bridge exists + uplink ENSLAVED (fail-open guard, same class as the transit
check); SEC-010 rationale corrected (bridge-in has no inner NAT) + br_netfilter
CONSTRAINT (keep the FORWARD-drop interface-scoped, NEVER global). Harness 49->54.
Records: D-125 (new); D-122 amendment (forward-pointer -- WAN realization
superseded, intent stands); SEC-010 (br_netfilter + hardened --check);
model-a-fallback B->A delta item 8 + revert; platform-traps 1g (bridge +
br_netfilter + nested-foreign-MAC trap).
NOT closed: the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in
equivalent of the depth-4 boot gate); fallback = double-NAT. HELD: vr1_dc0_uplink_cidr
+ OPNsense WAN re-address. Both roots validate; gauntlet ALL GREEN (63); repo-lint
0 fail. Present-only; NOT pushed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage-3 review sweep C+D: Model B two-root reshape + SEC-010 hardening
...
Phase C (Model B reshape, D-123 operator ruling):
- outer root (opentofu/main.tf): vvr1-dc0 resized to whole-DC containment VM
(108 vCPU / 425984 MiB / 3221225472000 bytes; expose_nested_virt=true;
single Office1 transit leg); node/plane/wan/opnsense modules removed from
vcloud level (MOVED markers), storage-04 added per R-3.
- inner root (opentofu/vr1-dc0-substrate/): NEW second root, provider
qemu+ssh to vvr1-dc0; reuses ../modules/* verbatim to build 9 nodes +
6 planes + wan + opnsense inside the containment VM. dmacvicar/libvirt
0.9.8 pinned (.terraform.lock.hcl committed). Both roots tofu-validate.
- variables.tf: vvr1_dc0_vcpu/memory_mib/disk_bytes with derivation comment.
Phase D (SEC-010, R3-F02): site-headend-install.sh --host-nodes node-host
bootstrap installs qemu-kvm+libvirt, persists+verifies nested=1, writes a
transit FORWARD-drop (nftables inet sec010) enforcing DC-LOCAL under Model B.
Gate hardened (advisor catch): --check now verifies the keyed transit
interface EXISTS, else nftables oifname loads clean but matches nothing
(fail-open). Harness 49/49.
Records: D-103/107/114/121/122/123/124 amendments (append-only);
model-a-fallback-plan synced; ledgers + Phase-C changelog.
KNOWN GAP (OBS-3, operator-confirmed 2026-07-16): under Model B the inner
vr1-dc0-wan NAT is nested inside vvr1-dc0 (single transit NIC), severing the
simulated-ISP egress that Model A had at vcloud level. DC0/DC1 ARE meant to
have ISP connections -- egress path resolution is follow-up design work, not
resolved by this commit.
Present-only; not pushed. Gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Review sweep A+B + R-3 storage-04: coherent record, whole-host calculator, R-3-compliant Model A fallback
...
Phase A (record amendments) + Phase B (scripts/dc-dc-whole-host-budget.py + harness) of the
Stage-3 adversarial-review sweep, plus the R-3 storage-04 addition (Model A layout now
3+2+4 = 9 nodes/DC). This commit is the R-3-COMPLIANT Model A state and becomes the re-cut
`model-a-fallback` anchor (the prior tag at 87a7a8a was R-3-stale -- 3 storage -- flagged by
the Model B design cross-check). NO cloud mutation; nothing applied.
- design-decisions.md: D-103/D-107/D-114/D-115/D-121/D-122/D-123/D-124 amendments (rulings R-1..R-4).
- scripts/dc-dc-whole-host-budget.py + tests/ (13/13): replaces the lost optc-calc.py (R3-F06).
- opentofu/main.tf: locals.vr1_dc0_nodes += storage-04 (R-3); tofu validate Success.
- security-ledger SEC-011 closed; session-ledger rulings + sweep progress; model-a-fallback plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
...
Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:
- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
(D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
the office1<->dc0 mesh leg; rack sizing 4/8192/80.
Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
(two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
/24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
D-121/122 folded in, stale gate corrected).
docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-14 |
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-13 |
D-114 follow-through: fix node-vm's nested-virt defect NOW; voffice1 pinned to a reserved address
...
Operator ruling: fixes land as we find them -- only DC1 DEPLOYMENT is gated, not DC1 fixes.
1. modules/node-vm: UNCONDITIONAL `cpu = { mode = "host-passthrough" }`. Same defect just found in
cloudinit-vm: with no cpu block libvirt hands the guest a generic emulated CPU with NO svm. A DC
node runs nova-compute, so it MUST run KVM guests -- there is no correct value other than this,
hence no knob that could only ever be set wrong (contrast cloudinit-vm, where it IS a real per-VM
decision, and opnsense-edge, which disables svm as correct hardening for a router). Not yet
instantiated, so no live call breaks. Without it the DC nodes would have come up looking healthy
and been unable to launch a single instance -- surfacing in Stage 5 as an inscrutable scheduler error.
2. voffice1 moved off the dynamic pool onto a RESERVED address. It first took 10.10.0.100 -- the
first address of Kea's pool (.100-.199). A MAAS region controller must not hold an address that
can move. Added a Kea host reservation via the D-113(a2) REST API (a second live exercise of that
write path): 52:54:00:6a:87:e5 -> 10.10.0.20, deliberately OUTSIDE the pool. add_reservation ->
"saved", service/reconfigure -> "ok". Rebooted; voffice1 came back on 10.10.0.20, .100 released,
/dev/kvm still present. Done now because nothing depends on the address yet.
3. runbooks/dc-dc-phase1-office1-standup.md rewritten to the D-114 model (was: three sibling service
VMs + an unruled Option A/B fork -- it would have actively misdirected whoever ran it). New
sequence: voffice1 via tofu -> first contact (lease/SSH/nested-KVM) -> MAAS on voffice1 -> LXD
(5.21 LTS) -> register LXD as a MAAS VM host -> MAAS-COMPOSE NetBox/GitBucket/Tailscale. The
MAAS/LXD command lines carry explicit PENDING VERIFICATION markers rather than fabricated flags.
docs/dc-dc-deployment-workflow.md Stage 2 + gap register updated to match.
repo-lint 0 fail. opentofu-validate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
...
APPLIED, operator-approved: 5 added, 0 changed, 0 destroyed. The running OPNsense edge was NOT
touched (confirmed by plan BEFORE apply -- no in-place update, so no repeat of the 2026-07-13
guest bounce).
THE MODULE WAS SILENTLY BROKEN FOR D-114'S ENTIRE MODEL. modules/cloudinit-vm's libvirt_domain
had NO `cpu` block. With none, libvirt renders a GENERIC EMULATED CPU exposing no `svm` -- and
LXD *virtual machines* are qemu/KVM guests, so NO service VM could ever have been composed into
voffice1. D-114 would have failed at its first step with a confusing "KVM not available" that
pointed nowhere near OpenTofu. Measured proof of the masking: host is an AMD EPYC 9965; a
default-CPU guest was handed an "Opteron_G3".
Fix: `cpu { mode = "host-passthrough" }` always, plus a new `expose_nested_virt` variable with NO
default -- a real per-VM decision. true = pass `svm` through (required for D-114's containment
VMs); false = disable it, matching modules/opnsense-edge. The operator asked whether the edge's
`svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a
triple-fault fix on 2026-07-12, did NOT resolve it (the real cause was the memory unit), and was
retained on hardening grounds. A router has no business seeing nested virt. It stays disabled there.
voffice1 as built (MEASURED): Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, single NIC on
office1-local, reports the real EPYC CPU with svm on all 16 cores and /dev/kvm PRESENT, egress to
1.1.1.1 at 4.6ms THROUGH the edge. Cloud-init deliberately MINIMAL (identity + key + guest agent);
MAAS and LXD go in as separate GATED steps rather than buried in a first-boot script that either
silently works or silently does not. The SSH pubkey is read via file(var...) at plan time -- key
material never enters a command line, the repo, or an agent's context; the tfvars holding the path
stays gitignored. network_config matches the NIC by GLOB, not a guessed kernel name (an inferred
value); it came up enp1s0.
TWO GATES CLEARED:
1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item: the daemon was proven, the
SERVICE never was (office1-local had no client). voffice1 -- the first client ever on that LAN
-- took 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state
active, read back through the D-113(a2) REST API. DHCP now works end to end.
2. D-114's NESTING PROBE PASSES AT L3. Stated honestly: this proves nested KVM is AVAILABLE. The
DEFINITIVE proof is a guest actually booting inside voffice1 -- that lands with the first
composed LXD VM. Not recording L3 as fully proven until then.
FINDING, logged NOT actioned (DC1 is gated behind Office1 per D-114): modules/node-vm has the SAME
missing-cpu-block defect. DC nodes run nova-compute, which needs working KVM -- without the fix
they will come up unable to run a single instance. MUST be fixed before Stage 3.
opentofu-validate PASS, repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
