| 2026-07-30 |
voffice1 reconcile: a gate that was RED on the only host that deploys
...
Operator asked whether the headend clone was fully reconciled. It was not, and running the
gauntlet there -- which had never actually been done -- found a real host-dependent gate defect.
RECONCILE. The headend was 1 commit behind with 0 local commits, now at HEAD, tfstate sha256
byte-identical before and after. No stale remote-tracking refs. One asymmetry found and fixed:
opentofu/vr1-dc1-substrate/.terraform.lock.hcl was UNTRACKED and not gitignored, while dc0's
equivalent IS tracked. Same provider version (0.9.8), so a content divergence was ruled out by
measurement, but a fresh clone could not reproduce dc1's provider resolution the way it can
dc0's. Landed, byte-identical to the headend copy.
THE GATE DEFECT. The gauntlet read ALL GREEN (89) on the jumphost and 1/89 FAILED on the headend,
on opentofu-validate. Cause: `tofu fmt -recursive` walks EVERY .tf/.tfvars under the tree,
including GITIGNORED local files. The sole flagged file was
opentofu/vr1-dc0-substrate/d124-inner.auto.tfvars -- gitignored operator input that exists only
on the host which runs the applies. So the gate was RED ON THE ONLY HOST THAT DEPLOYS and green
where it did not matter, the same host-dependent-verdict class as the LC_ALL manifest sort.
Ruled out first: both hosts run tofu 1.12.4, so this was not version drift.
The fmt check is now scoped to repo content, and a gitignored file is SKIPPED with the reason
printed rather than silently. Proven both ways: a badly-formatted TRACKED file still FAILS (T16),
a gitignored local tfvars is skipped and does not fail (T17). T17 also fails loudly if the ignore
rule ever stops covering per-DC tfvars, so the case cannot silently stop testing what it claims.
Owned: my first T16 read as a failure because `bash "$SCRIPT" | grep -q` inherits the gate's
correct exit 1 under `set -o pipefail`, even though grep matched -- a passing assertion reading as
a failure. Same strict-bash family as the IFS and inherit_errexit traps this repo has hit before.
Captured first, then grepped.
opentofu-validate harness 16/16; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |
Pin juju-01 MACs, add both controllers to lib-hosts at the ruled .5
...
Twelve measured MACs (six per DC, virsh domiflist) pinned into both
substrate roots; <dc>-juju-01 added to lib-hosts.sh with HOST_OCTET=5 per
the D-134 amendment and its measured boot MAC. Done now because both VMs are
ENLISTED -- the boundary modules/node-vm warns about, past which an in-place
apply can regenerate unpinned MACs and strand the node, as happened to the
nine role nodes on 2026-07-21.
No race with the running agents: they operate on the voffice1 clone; this
edit is on vcloud's, and nothing reaches their tree until someone pulls
there.
Recorded rather than smoothed over: dc1's juju-01 MACs are libvirt-generated
52:54:00: values, not dc1's schematic 52:54:01:d1: scheme -- a direct
consequence of the deliberate macs = [] at first apply.
Three exact-count assertions re-pointed to the new invariant and kept EXACT
rather than relaxed: dc-selector fleet 9 -> 10 (label corrected, since Option
C is a 9-node ruling and the 10th is D-104's), node-vm T8 9 -> 10 and T9
54 -> 60.
A real harness defect surfaced doing it: T8's grep -c 'macs = [' counts
COMMENTS, and my own comment explaining the deliberate macs = [] inflated the
count to 11 -- which reads exactly like a spurious extra node. Fixed to strip
comment lines. Same class as the ledger-scan next-free defect where a doc
merely quoting a token inflated the counter.
maas-role-tags.sh taught the utility band: adding juju-01 to HOSTS made it
refuse, because .5 is in no ROLE band -- correct behaviour meeting a new
fact. Per the amendment a utility-band host with no role token is now SKIPPED
and REPORTED, while the refusal keeps its teeth for genuine disagreement.
New cases T10 and T11; harness 9 -> 11.
Gauntlet ALL GREEN (87) on vcloud; opentofu-validate PASS; repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
Author both Stage-5 blockers (NOT applied); Octavia merge question resolved
...
Octavia RESOLVED as a clean negative. The chain audit flagged that
octavia-pki.yaml lands last and writes the same applications.octavia.options
map as the vips overlay, so under replace semantics octavia would lose its
vip and prefer-ipv6 with every offline gate still green. Researched against
source: juju 3.6 pins charm v12.1.1, and mergeStructs does
dstMap.SetMapIndex(srcKey, srcMapVal) per key for map-kind fields.
ApplicationSpec.Options is a Go map, so it deep-merges -- both overlays' keys
survive. The merge path is byte-identical between 2.9 and 3.6. The checker's
"MIRRORS juju's documented merge" comment is now sourced rather than assumed.
Logged: LP #2002371 (Triaged/High, unfixed) -- an EMPTY/null options: key
WIPES the base map rather than merging. Our overlays lack that shape and
nothing gates against acquiring it.
R1 AUTHORED. modules/node-vm gains osd_disk_size_bytes, default 0 = creates
nothing; the volume is count-gated and the disk entry appended via concat, so
an opted-out node has no extra resource and no plan diff -- precondition 4
verbatim. Four storage nodes per DC set osd_gib = 500: 8 volumes, not 18.
500 GiB is D-121's own budgeted figure (4x500Gi/DC = PASS, 5.31 TiB margin),
and D-121 records that footprint as an assumption pending an OSD-sizing
decision, so it is evidenced but not separately ruled. Headroom caveat
surfaced: dc0 has 2.0T available against 2.0 TiB nominal -- thin-provisioned,
so creation is cheap, but there is no room for those OSDs to fill.
D-104 AUTHORED. A 10th keyed entry per root, <dc>-juju-01, planning 1 add /
0 change / 0 destroy. The nine role nodes are untouched. 8 GiB / 4 vCPU are
the values the capacity gate actually modelled; disk 100 GiB is the authoring
value the amendment left open. MACs deliberately empty -- correct only before
enlistment, and they must be pinned from measurement after the first apply or
this reproduces the 2026-07-21 MAC-regeneration trap.
NEITHER IS APPLIED. opentofu-validate PASS on both roots; no apply run. Both
are live substrate mutations and individually operator-gated.
R1 precondition 3 is NOT satisfied and I am not treating it as such: the
amendment requires verifying BEFORE the apply that re-commissioning preserves
the D-134 statics and pinned MACs, and that needs live MAAS, absent here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-23 |
G13 CLOSED: office1 edge bundle -- qga channel live, 26.7.1, os-iperf in (operator-approved)
...
Saved-plan 0/1/0 channel apply (bounce = the ruled restart; guest-ping proven),
REST update 26.7->26.7.1 (no reboot), vr1-edge profile complete (both plugins
installed=1 by read-back), outer plan re-converged zero diff. Close capture
docs/audit/g13-close-20260723.txt; G13 row + section 5/7 + as-built updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
| 2026-07-22 |
G12 dc1 step B COMPLETE: transit live, rack enrolled, SEC-010 both ends, 26.7 staged
...
cloudinit-vm gains interface_macs (node-vm port; harness 10/10) -- voffice1
NIC3 + full MAC pins, vvr1-dc1 pin-adoption, 0/2/0 exact, post-bounce
battery ALL PASS, converged zero diff. Transit .5<->.6 proven with the dc1
key. Rack bootstrap dc1-parameterized via existing flags (dry-run caught the
dc0-flavored defaults); first run exit 4 on stale apt index, re-run exit 0;
region lists vvr1-dc1 (nmpcq4). SEC-010 region end extended to enp3s0.
OPNsense 26.7 base = hash-verified copy of dc0's proven artifact. Named
gate EXIT 0 (docs/audit/dc1-stepB-check-20260722-final.txt). Three queued
findings logged (changelog item 7).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
G12 dc1: remove spent binary saved plan (lint L1 fix)
...
tfplan-dc1-20260722 deviated from the *.tfplan gitignore naming and was
committed by mistake with a pipeline-masked lint failure; removed, lint
0-fail. Captures remain the applied-plan record. Changelog CORRECTION
appended.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
G12 dc1: apex --commit executed + substrate step A applied (5/0/0 exact, converged)
...
Operator-gated session (as-executed: dc1-deploy, per-command wrap):
apex write CREATED 172.31.0.4/30 + 10.12.68.2/22 (preflight re-run +
idempotent read-back captured); dc1 svc keypair minted per manifest
(creds-audit CLEAN); tfvars authored (local, D-124 amendment verbatim);
outer saved-plan apply 5 added / 0 changed / 0 destroyed -- vvr1-dc1
RUNNING (108 vCPU / 416 GiB, measured host headroom ~140 GiB), prior
guests untouched; convergence ZERO DIFF
(docs/audit/outer-plan-20260722-postdc1-converged.txt).
Next gated step: dc1 step B bootstrap (CURRENT-STATE G12 row).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
G12 dc1: item-3 substrate landed + harness reconcile (gauntlet 76 ALL GREEN)
...
Lands the disconnected session's authored tree (vr1_dc1 vars, vvr1-dc1 +
uplink modules, inner root vr1-dc1-substrate/ with 54 pre-pinned MACs,
lib-net dc1 arm + one-selection guard) and delivers changelog item 4:
six harnesses reconciled to the ratified dc1 selector arm, phase-00
PLANES/lib-net parity guard (exit 2, refuses cross-DC planning),
rbd-mirror + radosgw parent-shell selection (dc1 reminder now shows
10.12.84.0/22, not dc0's). Gauntlet ALL GREEN 76
(docs/audit/gauntlet-20260722-g12-reconcile.txt); repo-lint 0-fail.
G12 next gated step unchanged: operator-gated apex --commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
| 2026-07-21 |
Ruling 1: vr1-dc0-maas RETIRED (repo removed); INCIDENT: pod delete cascaded to 9 machine records
...
- git rm opentofu/vr1-dc0-maas (operator-ruled 'Retire fully'); validate PASS
- maas vm-host delete 4 removed the stale pod AND the nine linked DC0
machine records (association check ran after the delete -- agent error,
owned; capture docs/audit/incident-20260721-pod-delete-cascade.txt)
- Substrate measured intact (10/10 domains, MAC pins config-carried);
CURRENT-STATE corrected same commit (GA-R1 C2); recovery flow gated,
pending operator decision; SEC-013 file cleanup HELD
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XRtm5sDQgUsZyTJk3k7dz
|
Stage-close batch: ledger-scan PARTIAL fix; DOCFIX-197; D-103/D-123 pod-refutation amendments; GA-R7 memory review
...
- scripts/ledger-scan.sh: PARTIALLY-RULED decisions now surface (D-131
blind spot); harness regression fixture, 46 checks ALL PASS
- DOCFIX-197: phase2 runbook Step 11 leg selection corrected to executed
reality (dc0<->dc1 leg; office1 leg carries live transit)
- D-103/D-123 amendments + modules/maas-vm-host header: 2026-07-20
measured pod refutation + per-machine-virsh ruling RECORDED (cited to
captures; no new ruling made)
- GA-R7 memory review DONE: no memory-only facts remain
- Read-only: stale MAAS pod object confirmed (id=4 vr1-dc0-inner, virsh)
for the pending retire-or-keep ruling
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XRtm5sDQgUsZyTJk3k7dz
|
Step E netem DONE -- G10 CLOSED; netem-link local mode; G16 opened (office1 channels residual)
...
- modules/netem-link: optional local execution (empty ssh target -> bare
sudo tc; D-128 outer root runs ON vcloud); NEW tests/netem-link harness
(12 cases); gauntlet 76 ALL GREEN
- outer root: module netem_vr1_dc0_vr1_dc1 wired -- virbr5 (measured at
wire time), placeholder profile 'delay 3ms 1ms loss 0.01%' (S6 lean,
PROVISIONAL; D-100 gap #11 unruled); runbook Step-11 leg divergence
flagged, DOCFIX queued
- 1/1/0 STOP honored; operator RULED 'Targeted netem apply (Recommended)';
saved -target plan exact 1/0/0, applied; qdisc live on virbr5,
virbr7/virbr3 untouched (docs/audit/stepE-netem-20260721.txt +
outer-{plan,apply}-20260721-netem*.txt)
- Convergence 0/1/0 = office1 channels=[] residual only
(outer-plan-20260721-postE-residual.txt) -> split per E3 into NEW gate
G16; CURRENT-STATE section 5 expected plan re-recorded 0/1/0 (GA-R1 C1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XRtm5sDQgUsZyTJk3k7dz
|
node-vm: MAAS owns node power -- ignore_changes on running (operator-ruled)
...
The MAC-pin verification plan (docs/audit/inner-plan-20260721-macpin.txt,
0/9/0, 54 mac adoptions, zero replaces) exposed 9 entangled out-of-band
power-ons: the module re-asserts running=true against nodes MAAS holds
OFF. Operator-ruled: guard first, then apply.
- lifecycle ignore_changes = [running]: create still boots (PXE
enlistment); afterwards MAAS owns power (virsh here, Roosevelt IPMI).
- tests/node-vm +3 cases (guard present, no scope creep, cites MAAS);
15/15 green, gauntlet 73 ALL GREEN, repo-lint 0 fail.
- CURRENT-STATE: MAC-pinning delivery status + pending gated apply.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STHXiHfoxHqq8fGRVvb66G
|
node-vm: pin NIC MACs (54 measured) -- in-place applies can no longer drift them
...
Queued delivery item from the 2026-07-21 commissioning diagnosis: an
"in-place" apply (0/9/0) regenerated all 9 unpinned boot MACs on
2026-07-20 and stranded the fleet in MAAS.
- modules/node-vm: optional interface_macs (all-or-nothing + format
validations); mac = { address = ... } shape verified against the
dmacvicar/libvirt 0.9.8 schema dump this session.
- vr1-dc0-substrate: all 9 nodes x 6 planes pinned to values MEASURED
this session (virsh domiflist on vvr1-dc0); metal-admin entries
cross-checked vs MAAS boot-interface records 9/9.
- NEW tests/node-vm harness (12 cases). Gauntlet 72 -> 73 ALL GREEN;
repo-lint 0 fail; opentofu-validate PASS.
NOT YET APPLIED on voffice1 (inner root state) -- the gated apply's plan
must show NO replace; any replace is a STOP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STHXiHfoxHqq8fGRVvb66G
|
| 2026-07-20 |
modules/node-vm: add serial console + boot log (parameterized dir)
...
Its absence blocked a live diagnosis: 6/9 dc0 nodes stall in MAAS 'Loading
ephemeral' and, with no console and no agent, a stuck node is unobservable.
Same pattern opnsense-edge already carries, for the same reason -- a
PXE-booting node emits everything interesting before anything sshable
exists. Unconditional (MAAS-managed nodes are rebuilt at will; no
live-router bounce risk like the D-129 qga channel). Log dir is a variable
rather than opnsense-edge's hardcoded vcloud literal, which is wrong on any
other libvirt host -- backport queued.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
NEW root opentofu/vr1-dc0-maas: MAAS VM-host registration, isolated from substrate roots
...
Third root by design, applying DOCFIX-179's own lesson rather than
repeating it: a provider 'maas' block forces EVERY plan in its root to
demand maas_api_url + the sensitive key, so coupling it to the substrate
roots would make routine substrate plans require MAAS creds. This root
touches MAAS only; the substrate roots stay credential-free. Registers
the pod via modules/maas-vm-host (discover, never compose -- D-103) with
power_address local to the rack per Model B. Validated explicitly (the
validator covers only the outer root + modules -- extending it to all
roots stays queued).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
modules/opnsense-edge: opt-in qga virtio channel (D-129); dc0 edge sets it true
...
Shape taken from the provider schema (devices.channels[].target.virt_io /
source.unix on dmacvicar 0.9.8), not inferred. Default OFF so existing
edges are never bounced by a plan; the new dc0 edge boots with it per the
runbook's no-retrofit intent for NEW DC edges. Without the channel the
os-qemu-guest-agent package is inert-but-present -- the false success the
D-129 review flagged -- so package and channel ship as one decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Deploy step C COMPLETE: inner apply 28/28 from voffice1, converged zero diff; 9 nodes + 26.7 edge RUNNING at depth-4; three first-real-apply fixes
...
voffice1 = Plane-2 executor (tofu 1.12.4, repo @29cf7bf, dc0 key,
keyscan'd known_hosts). Fixed forward, measured at first real use:
provider keyfile/sshauth URI params; wan-bridge illegal <mtu> in
bridge mode; base image uploads from the EXECUTING host (copied to
voffice1, description corrected); staging-dir literal created on
rack. Captures: inner-{plan,apply,converge}-20260720-stepC*.
Edge serial log at FreeBSD login prompt (26.7 D-112 first datapoint).
CURRENT-STATE position updated in this commit (GA-R1/C1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
wan-bridge: drop illegal <mtu> on bridge-mode network (measured at first real apply; MTU is the host bridge's)
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Inner root: provider keyfile+sshauth URI params (measured: provider ssh ignores ssh config/default identities); known_hosts multi-type note
...
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Deploy step B COMPLETE: rack bootstrapped + enrolled, SEC-010 CLOSED both ends (operator-ruled), OPNsense 26.7 base staged (operator-ruled); ledger-scan fail-open hardening
...
Bootstrap runs 1+2 (stale-lists 404 fixed by proxied apt update):
maas 3.7/stable, rack enrolled, libvirt+nested KVM, SEC-010 loaded
+ persistent on enp1s0; final --check EXIT 0 ALL items
(docs/audit/stepB-check-20260720-final.txt). voffice1-side SEC-010
pin applied per the row's follow-on clause. 26.7 nano prepped on the
rack (measured leaseweb mirror via region proxy); inner-root base
default -> 26.7. ledger-scan: FAIL-OPEN no longer counts as OPEN
(harness 45/45). CURRENT-STATE position/G10/G14 + section-5 updated
in this commit (GA-R1/C1). Open SEC count 8 -> 7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Step B transit reach ESTABLISHED: voffice1 .1/30 applied (Kea re-key incident fixed); vvr1-dc0 netplan set-name root cause fixed; zero-diff converged
...
voffice1 transit NIC applied 1/1/1 exact; MAC-regen broke the Kea
reservation (measured .108) -- re-keyed via edge API, .20 restored.
vvr1-dc0 both-NICs-silent root cause: netplan match.name+set-name
(one-boot rename lag, measured offline via libguestfs); fix drops
set-name, keeps measured enp1s0/enp2s0, instance-id bump, seed
-replace 2/0/2 exact. Verified: ssh -J voffice1 w/ dc0 key,
rack->region 0% loss. Outer plan ZERO DIFF converged.
CURRENT-STATE sections 3+5 updated in this commit (GA-R1/C1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
| 2026-07-19 |
Step B prereq: voffice1 transit NIC authored (172.31.0.1/30 region end, D-124); ruling recorded; expected plan 1/1/1
...
Measured: transit net isolated, only vvr1-dc0 attached; nothing held .1.
Operator-ruled mechanism 2026-07-19: voffice1 via IaC + restart.
Capture: docs/audit/outer-plan-20260719-voffice1-transit.txt (1/1/1).
CURRENT-STATE sections 3+5 updated in this commit (GA-R1/C1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
GA-F03: stale tofu 1.12.3 pins demoted to CURRENT-STATE pointers
...
opentofu/README.md:41 (historical prose keeps its date, drops the pin)
and docs/dc-dc-deployment-workflow.md:47/:360/:875 (the :360 live claim
"binary EXISTS: v1.12.3" was FALSE -- measured 1.12.4) now point at
docs/CURRENT-STATE.md section 7, the only pin authority (GA-R1 rule 3).
The :47 edit also drops a "DONE" status token (vocab overlap with 2.5).
Batch 2 item 2.2.
Revert: git revert this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
D-130 ADOPTED (a) + implemented: seed-volume ignore_changes guard (GA-F01, G4+G5 CLOSED)
...
Batch 1 items 1-4. G4 verifications captured: v8 = suppression CONFIRMED
(fresh baseline 7/2/7 identical to 2026-07-18, with-edit 6/2/6, zero
forces-replacement; DC0 creates unaffected); v7 = defect reproduced on
isolated throwaway-v7-* domain, replacement applied UNDER the running
domain with NO bounce (domid 3->3), teardown verified zero residue.
Operator ruled (a) per GA-R5 (utterance quoted in the D-130 entry).
Implemented: lifecycle { ignore_changes = [create] } on
libvirt_volume.seed in modules/cloudinit-vm + tests/cloudinit-vm
harness (6/6; guard placement, exact attribute, no leak onto data disk,
tofu validate). CURRENT-STATE: G4/G5 rows CLOSED + rotated, decision
queue item 1 rotated to the register, section 5 records post-fix 6/2/6
pending the G6 reconcile (C2: captures + doc same commit). 342/360 cap.
Revert: git revert this commit (restores the replace-pair behavior and
the pre-ruling CURRENT-STATE rows; captures remain in history).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
| 2026-07-17 |
D-127 IaC autostart matrix baked into deploy tooling + Office1 guest updates
...
Operator directive: bake the autoboot-vs-manual matrix into the IaC so DC0/DC1 get
correct autostart at creation, not by hand. Added an `autostart` bool var to modules
cloudinit-vm/opnsense-edge/node-vm (wired to libvirt_domain.autostart); set explicitly
per instance (shared modules => IaC is source of truth): voffice1=true,
office1-opnsense=true, vvr1-dc0=FALSE, vr1-dc0-opnsense=true, node-vm=FALSE.
HARD GATE verified: the two existing Office1 domains (voffice1, office1-opnsense) plan
as autostart false->true UPDATE-IN-PLACE, NOT replacement (replacement would destroy the
live host/edge). vvr1-dc0 is a new create with autostart=false baked in. Outer plan now
5 add / 2 in-place / 6 destroy; the 2 collapse to 0 once the bookend sets those two
autostart live pre-apply. tests/opentofu-validate T11-T15 pin the matrix against drift.
Standup runbook records the composed service VMs' LXD boot.autostart step. Confirmed
DC0/DC1 create NO LXD instances (inner root = node-vm + opnsense-edge + planes/pool/
wan-bridge) -- DC side is purely this VM matrix.
Office1 guest updates (agent sudo inside the guests): office1-netbox apt upgrade (no
reboot, netbox healthy); office1-tailscale full-upgrade -> kernel 6.8.0-136 + reboot
(back, re-advertising 10.10.0.0/22); voffice1 0-upgradable; LXD boot.autostart=true set
on both. vcloud + office1-opnsense stay in the operator's bookend.
Gauntlet ALL GREEN (66); opentofu-validate clean; repo-lint 0-fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-16 |
DC0 deploy-readiness handoff: per-env-key ruling implemented + netem ruled + prep checklist
...
Prep for a fresh session to run the Stage-3 Phase-2 DC0 build. Nothing applied;
the outer tofu apply remains the first (operator-gated) mutation.
Per-env-key (D-126, operator ruling 2026-07-16, option a): vvr1-dc0 gets its OWN
access key, not office1's. Minted ~/vr1-dc0-creds/vr1-dc0_svc_ed25519 (creds-audit
CLEAN); added variable vr1_dc0_ssh_pubkey_path; vvr1-dc0 cloud-init now installs the
dc0 pubkey (voffice1 keeps office1's key -- only the vvr1_dc0 block changed). Re-
validated + re-planned: outer plan UNCHANGED at 5 add / 0 change / 6 destroy (the
cloud-init edit is create-only content, no new destroys/replaces of live infra).
netem (D-100 gap #11): RULED = placeholder for the rehearsal; final values deferred.
docs/dc0-deploy-readiness.md (NEW): the full readiness state (outer plan verified
safe -- vvr1-dc0 416GiB/108vcpu, 6 empty-plane relocation, 980GiB RAM headroom),
remaining prep (pre-apply / mid-deploy / non-blocking), fresh-session bootstrap, and
the gated A-E deploy sequence. Ledger points at it.
Gauntlet ALL GREEN (65 harnesses incl. the 2 new access harnesses); repo-lint 0-fail;
opentofu-validate clean (both roots + 12 modules). The dc0 private key + local tfvars
stay gitignored/off-repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125 correction: uplink /24 is the ruled 172.30.2.0/24, not a HELD tfvar (no new NetBox object)
...
The initial D-125 records wrongly modeled the vcloud ISP /24 as a NEW,
NetBox-assigned HELD value (`var.vr1_dc0_uplink_cidr`) with an OPNsense WAN
re-address. That is factually wrong: bridge-in is single-NAT, so there is
exactly ONE simulated-ISP /24 for vr1-dc0, and D-115 already ruled it
(172.30.2.0/24, role edge, site vr1-dc0). NetBox models the prefix by role+site
and does not care which libvirt host runs the NAT, so the /24's IPAM identity is
UNCHANGED from Model A; only the libvirt realization moved (a vcloud NAT bridged
through vvr1-dc0). OPNsense WAN stays .2. (Caught reconciling the "create netbox
objects" request against dc-edge-wan-import.py, which already registers this /24.)
- opentofu/main.tf: module vr1_dc0_uplink cidr = "172.30.2.0/24" inlined (ruled
literal, matching the inner root's site-wan pattern) instead of the var.
- opentofu/variables.tf: REMOVED variable vr1_dc0_uplink_cidr (replaced with a
note explaining why it is a ruled literal, not a tfvar).
- Records corrected everywhere the false claim propagated: D-125 body + D-122
amendment, changelog-phaseC2-D125, model-a-fallback (B->A delta item 8 +
revert), session-ledger, dc-dc-deployment-workflow, phase-2 runbook.
Net effect: D-125 adds NO new NetBox object and NO new HELD gate. The only
NetBox work left for the DC substrate is D-124's transit /30 + rack IP.
Both roots validate; gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
docs(opentofu/README): reconcile to Model B two-root + neutralize deleted config-ISO path
...
opentofu/README.md described a single-root pre-Model-B tree and, dangerously,
the DELETED OPNsense config.xml/config-ISO delivery as current. Added an
authoritative STATUS-RECONCILED banner (repo/ledger authoritative over the older
per-module prose) and inline SUPERSEDED/DELETED markers. Grep-verified:
- TWO ROOTS (D-123 Model B): outer opentofu/ + inner opentofu/vr1-dc0-substrate/,
bootstrap gate between; 9 node VMs (D-121/R-3); D-125 vcloud ISP uplink.
- Modules ADDED, missing from the Built list: site-wan (D-122), wan-bridge (D-125).
- INSTANTIATED (contradicts "not instantiated" prose): cloudinit-vm + base-image
(voffice1/vvr1-dc0/ubuntu_noble_base), node-vm (inner), dc-planes/site-wan/
wan-bridge/opnsense-edge (inner).
- DANGER neutralized: templates/opnsense-config.xml.tmpl,
scripts/opnsense-build-config-iso.sh, scripts/opnsense-render-config.sh are
DELETED (verified) -- a full config.xml push clobbers the live API-managed edge
(D-112/D-113(a2)). Banner + inline markers on the Built-list refs; deeper
2026-07-09 research/audit refs are framed historical by the banner.
- GitBucket out of scope (D-116) where listed.
Kept Stage-1 history (applied 2026-07-10, DOCFIX-179 provider mapping). Doc-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3 design gap
...
OBS-3: Model B nesting put vr1-dc0-wan (a libvirt NAT) inside vvr1-dc0, whose
only leg is the SEC-010 FORWARD-dropped transit -- so the per-DC simulated ISP
had no egress. Operator ruled bridge-in (single-NAT) over double-NAT.
- modules/wan-bridge (NEW): libvirt forward=bridge onto a pre-existing host
bridge (bridge={name}, schema measured from the provider, not guessed). No
subnet/NAT. Sibling to site-wan so every NAT user stays untouched. Covered by
opentofu-validate S3 (standalone PASS).
- OUTER root: vcloud ISP NAT vr1_dc0_uplink (the single NAT) + a 2nd IP-less
uplink NIC on vvr1-dc0 + the br-vr1-dc0-wan bridge declared in netplan (transit
stays NIC1 so SEC-010/--transit-if keep keying on it). New HELD var
vr1_dc0_uplink_cidr (no default; office1-netbox apex, hard rule 2).
- INNER root: vr1_dc0_wan site-wan(NAT) -> wan-bridge onto br-vr1-dc0-wan;
OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113, HELD).
- Bootstrap (site-headend-install.sh): --uplink-if/--wan-bridge; --check verifies
the bridge exists + uplink ENSLAVED (fail-open guard, same class as the transit
check); SEC-010 rationale corrected (bridge-in has no inner NAT) + br_netfilter
CONSTRAINT (keep the FORWARD-drop interface-scoped, NEVER global). Harness 49->54.
Records: D-125 (new); D-122 amendment (forward-pointer -- WAN realization
superseded, intent stands); SEC-010 (br_netfilter + hardened --check);
model-a-fallback B->A delta item 8 + revert; platform-traps 1g (bridge +
br_netfilter + nested-foreign-MAC trap).
NOT closed: the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in
equivalent of the depth-4 boot gate); fallback = double-NAT. HELD: vr1_dc0_uplink_cidr
+ OPNsense WAN re-address. Both roots validate; gauntlet ALL GREEN (63); repo-lint
0 fail. Present-only; NOT pushed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage-3 review sweep C+D: Model B two-root reshape + SEC-010 hardening
...
Phase C (Model B reshape, D-123 operator ruling):
- outer root (opentofu/main.tf): vvr1-dc0 resized to whole-DC containment VM
(108 vCPU / 425984 MiB / 3221225472000 bytes; expose_nested_virt=true;
single Office1 transit leg); node/plane/wan/opnsense modules removed from
vcloud level (MOVED markers), storage-04 added per R-3.
- inner root (opentofu/vr1-dc0-substrate/): NEW second root, provider
qemu+ssh to vvr1-dc0; reuses ../modules/* verbatim to build 9 nodes +
6 planes + wan + opnsense inside the containment VM. dmacvicar/libvirt
0.9.8 pinned (.terraform.lock.hcl committed). Both roots tofu-validate.
- variables.tf: vvr1_dc0_vcpu/memory_mib/disk_bytes with derivation comment.
Phase D (SEC-010, R3-F02): site-headend-install.sh --host-nodes node-host
bootstrap installs qemu-kvm+libvirt, persists+verifies nested=1, writes a
transit FORWARD-drop (nftables inet sec010) enforcing DC-LOCAL under Model B.
Gate hardened (advisor catch): --check now verifies the keyed transit
interface EXISTS, else nftables oifname loads clean but matches nothing
(fail-open). Harness 49/49.
Records: D-103/107/114/121/122/123/124 amendments (append-only);
model-a-fallback-plan synced; ledgers + Phase-C changelog.
KNOWN GAP (OBS-3, operator-confirmed 2026-07-16): under Model B the inner
vr1-dc0-wan NAT is nested inside vvr1-dc0 (single transit NIC), severing the
simulated-ISP egress that Model A had at vcloud level. DC0/DC1 ARE meant to
have ISP connections -- egress path resolution is follow-up design work, not
resolved by this commit.
Present-only; not pushed. Gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
