| 2026-07-15 |
C2 sub-task (i): sandbox re-verified faithful; fidelity-checker scope false-positive fixed
...
Re-ran the HARDENED fidelity check against office1-netbox (the VR1 IPAM apex,
DOCFIX-195) -- read-only dump via the consolidated sandbox token + an SSH tunnel.
Result: exit 0, "upstream draft + EXACTLY the planned delta (D-115/D-117/D-118)."
C2 sub-task (i) is CLOSED; the sandbox is proven faithful.
The hardened checker false-failed on 10.10.0.0/16 and 172.30.0.0/16 being
unscoped -- but those two role-container /16s are unscoped BY DESIGN
(d115-office-carve.py), matching the 27 unscoped role containers already in the
upstream apex. A full cross-check of all 44 delta prefixes against their intended
scope was 44/44 (every DC prefix on the correct DC site; no wrong-target binding).
Fix (netbox/sandbox-fidelity-check.py): assert scope == INTENDED per delta prefix
(EXPECTED_PREFIX_SCOPE; DC half generated by _dc_prefix_scopes so the CIDR set and
scope map cannot drift). Strictly stronger than the old "has a scope?" test -- still
catches a dropped scope, and now catches a WRONG-TARGET binding (a DC0 prefix bound
to vr1-dc1, the D-117 near-miss). NOT fixed by exempting the two CIDRs.
Harness moved to dump format (scope_site), T1 sets each prefix's scope from the
checker's own map, +T9 (wrong-target fails) +T10 (legit-unscoped passes). 10->12.
Gauntlet ALL GREEN (58); repo-lint 0 fail.
SEC-009: the sandbox NetBox token had been MISSED by the consolidation (lived only
at /root/netbox-secrets/api.token on office1-netbox); now consolidated to
~/vr1-office1-creds/vr1-netbox-sandbox.env (0600), copied off the VM without
printing. Ledger addendum recorded.
No NetBox state changed -- dump and check are read-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
DOCFIX-195: office1-netbox holds the VR1 IPAM apex role; apex write-back deferred; C2 redefined
...
Operator ruling 2026-07-15: office1-netbox (10.10.1.10) HOLDS THE IPAM APEX
ROLE during VR1 -- the authoritative NetBox every consuming system (OpenTofu,
MAAS, overlays, bundle) derives its values from. It is a working draft (edited
as the buildout proceeds), but it is the apex. netbox.baldurkeep.com takes NO
write during VR1; it is the eventual production apex, frozen as a read-only v1
reference draft. Merge-back is deferred to end-of-deployment (a maybe), NOT a
Stage 2 gate.
The repo encoded the opposite roles and defined close-out C2 (the last Stage 2
gate, and the branch->main merge trigger) as an operator-gated write UPSTREAM.
Under the ruling there is no apex write in Stage 2, so C2 is redefined:
office1-netbox holds the COMPLETE validated dataset -- carve + six-plane roles
+ both DCs + the D-120 compose-/24 child ranges + the two service IPs -- and it
is VERIFIED. C2 stays OPEN.
Finding surfaced while drafting (logged, not fixed): the D-120 objects are
ip-ranges/ip-addresses, object classes the sandbox loop does NOT cover
(prod-draft-dump.py ENDPOINTS and sandbox-fidelity-check.py KEY/EXPECTED_NEW
both stop at rirs/roles/regions/sites/aggregates/prefixes). So the hardened
fidelity check cannot verify the D-120 load; verifying it needs a tool
extension (+harness) or a recorded manual read-back. Confirmed no live system
config points at any NetBox URL -- documentation-only change.
Files: dc-dc-deployment-workflow.md (State + C2 row), dc-dc-netbox-buildout-scope.md
(sec 8.1 + 8.6), session-ledger.md (5 narrative spots + new deferred-item bullet;
machine-derived block re-seeded from the 2026-07-15 scan), design-decisions.md
(D-010 note + D-103 pointer), netbox/README.md (target banner), + changelog.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
D-120 ADOPTED + EXECUTED: re-IP Office1 services into the .2-.49 static band
...
office1-netbox .201->.10 and office1-tailscale .202->.11, on the wire, no
wipe. NetBox HTTP 200 at .10; tailscale still advertises 10.10.0.0/22.
Method (DC replication inherits it): MAAS refuses interface edits on a
Deployed machine and the only lift is Release->redeploy=WIPE, so the live
address is set on the guest (netplan + curtin cfg, reboot-durable) over
lxc exec (socket immune to the netplan apply), and MAAS-model drift is
accepted (reconciles at next teardown/redeploy). NetBox ALLOWED_HOSTS=*
so no restart; tailscale re-asserted automatically.
Swept SANDBOX_HOSTS .201->.10 in the three importers + test + the on-box
working copy; updated live-state docs; flipped D-120 to ADOPTED with an
Execution record; resolved the re-IP runbook's PENDING markers to the
measured method. Gauntlet ALL GREEN (58); repo-lint 0 fail / 1 legacy WARN.
Still owed: D-120 Step 6 (register the /24 child ranges + service IPs in
the sandbox, feed upstream) rides with C2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpGaiX6UvhSReDwN2ywstJ
|
D-120 PROPOSED: static-services addressing for compose /24s; re-IP plan drafted
...
Office1's persistent services (office1-netbox .201, office1-tailscale .202) sit in
MAAS's deployed-node band -- addressed like ephemeral compute nodes, by MAAS
auto-assign default, not by design. They are NOT in the DHCP dynamic range
(.100-.200), so no collision -- but no static-services sub-layout was ever ratified
for the /24 (D-115 left it "MAAS DHCP/PXE").
D-120 (PROPOSED) defines a functional band layout for EVERY MAAS-managed compose /24
(Office1 + the DCs, which inherit it):
.1 gateway (site containment VM)
.2-.49 STATIC site services (MAAS static-assign, outside the dynamic range)
.50-.99 reserved/spare
.100-.200 MAAS dynamic (unchanged)
.201-.254 deployed nodes (unchanged)
Proposed Office1 addresses: netbox .201->.10, tailscale .202->.11. Recommends
option (a): adopt + re-IP Office1 now, so the DC template is clean. Addresses AWAIT
operator ratification.
Re-IP plan: runbooks/dc-dc-office1-service-reip.md -- gated mutation on two running
services, honest PENDING VERIFICATION markers for the MAAS static-assign CLI and the
netbox-docker ALLOWED_HOSTS / tailscale re-assert (cannot confirm without the
operator-held box keys). Cascades to 20 repo refs incl. SANDBOX_HOSTS .201 in the
two pynetbox importers + their tests. NOT executed.
The layout belongs IN NetBox (the IPAM authority deciding where NetBox lives) --
recorded via the sandbox loop, not prose.
repo-lint 0 fail. No live services touched -- this is a proposal + plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Consolidate credential/env sprawl into ~/vr1-office1-creds/ (SEC-009); seed memory
...
Operator-approved. Three env files were loose in ~ outside the consolidated creds
folder -- .vr1-netbox.env, vr1-office1.env, and vr1-stage1.env. The last was mode
0664 (world-readable) and held TF_VAR_maas_api_key: a real MAAS-secret exposure,
not just untidiness. None created this session (mtimes 07-10..12).
Done (files outside the repo; recorded in the changelog):
- moved all three into ~/vr1-office1-creds/; chmod 600 on every sensitive file
(fixed the two 0664 files: vr1-stage1.env and tailscale-authkey.txt).
- no CONTENTS read -- key NAMES only, to identify each file's purpose.
- fixed a D-119 follow-up the in-repo sweep could not reach: vr1-stage1.env had
TF_VAR_dc1_pool_path/dc2 -> renamed to TF_VAR_vr1_dc0/vr1_dc1_pool_path.
Repo (committed):
- active path refs old ~/.vr1-netbox.env -> ~/vr1-office1-creds/vr1-netbox.env in
prod-draft-dump.py, the phase1 runbook Step 10b, as-built, and the ledger. Dated
changelog/incident records left as historical snapshots.
- as-built secrets table: the three env files + the convention. CORRECTED a
mid-session wrong assumption -- the SANDBOX token is NOT on vcloud; it is on
office1-netbox (/root/netbox-secrets/). vr1-netbox.env is the UPSTREAM token only.
- security-ledger SEC-009 (REMEDIATED) + the standing per-site consolidation convention.
- seeded agent memory (was EMPTY): the creds convention + the closed-test posture,
so neither is re-lost at compaction.
STANDING CONVENTION: per-site ~/<site>-creds/ (0700; files 0600); no loose env in ~;
~/vr1-dc0-creds/ and ~/vr1-dc1-creds/ follow the pattern from day one. Data-loss /
continuity control first; real cred hardening deferred to post-teardown (closed test).
GAUNTLET ALL GREEN (58). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-14 |
C4 DONE: document the NetBox sandbox loop; add the upstream-write guard it exposed
...
C4 (Stage 2 close-out) -- the sandbox loop is now documented in two places, no
rationale duplicated: docs/dc-dc-netbox-buildout-scope.md section 8 (design: why a
sandbox, the fidelity check + its 2026-07-14 hardening, the write-guard table, the
WAF trap, C2 preconditions) and runbooks/dc-dc-phase1-office1-standup.md Step 10b
(the dump->seed->apply->verify->gated-write sequence, two-host/two-token separation).
Drafted by a subagent, corrected against this session's own fixes before placement.
The guard C4 exposed: roles-aggregates-import.py and dc-dc-prefixes-import.py had NO
hostname gate -- until today the WAF 403 was the only thing stopping an accidental
--commit to the apex, and the UA fix removed that accidental safety. Both now gate a
non-sandbox --commit behind --yes-write-upstream (SANDBOX_HOSTS allowlist), matching
d115-office-carve.py -- so all three write-capable importers behave identically. This
is defence-in-depth with the whole-plan preflights (which prevent a half-write).
Tests: prefixes-import 86->91 (refused without the flag + writes nothing; allowed
with it; sandbox host needs no flag). roles-aggregates 20->24. GAUNTLET ALL GREEN (58).
Stage 2: C1/C3/C4 DONE. C2 (production write) is the only item left -- gated, with
preconditions now documented (re-verify sandbox under the hardened check, run from
office1-netbox, --yes-write-upstream, approve each mutation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119 OpenTofu rename APPLIED -- repo/state back in sync (operator-gated)
...
tofu apply: 11 added, 0 changed, 11 destroyed. tofu plan now reports "No changes".
Pre-apply LIVE re-check confirmed all 11 objects empty (0 leases, 0 volumes, no
domain attached) -- verify-before-mutate, not the earlier measurement. Fresh plan
was 11 add / 11 destroy, all "must be replaced" via the moved{} blocks.
Verified on the host: six vr1-dc0-* networks, three mesh-vr1-* links, vr1-dc0-pool
/vr1-dc1-pool; ZERO old dc1-*/dc2-*/mesh-dc* names remain. Office1 (office1-local/
office1-wan/wan/edge/voffice1) was NOT in the plan and is untouched -- edge and
voffice1 both reachable post-apply.
Applied now, while the objects are empty, so Stage 3's own tofu apply does not
silently bundle this 11-resource rename with real DC provisioning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Add the missing preflight-failure test; reconcile the stale tofu-sync ledger line
...
Two follow-ups a review caught:
1. Write-path fix #3 (whole-plan role preflight in dc-dc-prefixes-import.py) shipped
with NO test exercising it FAILING. Every existing test preseeds all roles, so
they ran the preflight only when it passes -- silently. A future edit could delete
the preflight and the harness would stay green: exactly the "green while the bug is
armed" failure this session has been killing. Added a test that runs main() with one
six-plane role missing and asserts nonzero exit AND that NOTHING was written (no
site, no prefixes). prefixes-import 82 -> 85.
2. The ledger's "Live state" bullet still claimed OpenTofu repo/state were IN SYNC
("tofu plan reports No changes"). False since D-119 -- plan is 11 to add / 11 to
destroy. Reconciled to state the gated destroying apply, contradicting nothing.
GAUNTLET ALL GREEN (58). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger: write-path bugs 1-3 fixed; flag the sandbox re-verify owed before C2
...
The sandbox was declared "proven faithful" using the fidelity check BEFORE it was
hardened -- i.e. under the version that could false-green. That verdict must be
re-established under the hardened check before C2 feeds the carve upstream. Needs
the operator-held sandbox token; folds into the C2 step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 LAN IPv6 is LIVE and proven on the wire -- Stage 2 close-out C1 CLOSED
...
The last build item on Office1. Executed against the live edge, verified against
the kernel and the wire, not by {"result":"saved"}.
scripts/opnsense-set-interface-v6.sh --commit 10.10.0.1 lan 2602:f3e2:f01 :1 64
POST /api/radvd/settings/addEntry {"entries":{...,"mode":"unmanaged"}}
POST /api/radvd/service/reconfigure
PROVEN: address on the kernel (ifconfig, not the config file); radvd advertising
2602:f3e2:f01:/64 AdvAutonomous/AdvOnLink with no M/O flags; voffice1
autoconfigured ...:5054:ff:fe6a:87e5 by SLAAC + an RA-learned route; edge->voffice1
GUA ping 0.0% loss. The neighbour MAC matches voffice1's known Kea lease hwaddr.
No collateral damage: v4 10.10.0.1 intact, Kea DHCPv4 still serving udp/67, the
compose-net route still good.
mode=unmanaged is DELIBERATELY not the model's default. `stateless` sets the RA
Other-config flag, pointing clients at a DHCPv6 server -- and Kea DHCPv6 is
enabled=0 with no daemon running. The default would have advertised a server that
does not exist. Field names, the request key (addBase('entries','entries')), the
mode values and the apply endpoint were all READ OFF THE EDGE, not recalled.
OPERATOR RULING -- the v6 NAT was NOT built. A v4-to-v6 NAT (a second OPNsense, or
nginx) was proposed to simulate the v6 the lab will not forward. Withdrawn: it
contradicts D-101 ("native GUA routing, no NAT66"), D-115's amendment says the
missing egress is the upstream's problem and not ours, and a translation box is
precisely the component Roosevelt will NOT have -- maximum delta for zero proof.
nginx is an L7 proxy, not NAT, and proves nothing about RA/ICMPv6/Ceph-over-v6.
Ruling: configure IPv6 internally now; configure the IPv6 edge AFTER deployment.
TRAP recorded: voffice1 has forwarding=1 + accept_ra=0 and STILL autoconfigured --
netplan/systemd-networkd processes RA in USERSPACE, independent of the sysctl.
WAN v6 leg stays DEFERRED: re-measured today, v6 internet is 100% loss from vcloud
while v4 is fine. Internet GUA reachability remains UNPROVABLE in VR1.
repo-lint 0 fail; opnsense-set-interface-v6 harness 40/40.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Consolidate to one session; close last night's two delivery gaps
...
Work had fragmented across FOUR concurrent Claude sessions on this one clone,
including two --permission-mode auto background agents (idle 14-15h, still
holding auto-approve rights against the live cloud) and a second interactive
session that got the same "resume last night's work" prompt six minutes before
this one and started bootstrapping in parallel. All terminated; the supervising
daemon stopped (it had respawned the agents once -- killing the child is not
enough). Working tree was clean and 3-ahead throughout; nothing lost.
Last night's three commits were made by the auto-permission agent and left
UNPUSHED and UNDOCUMENTED. Both closed:
- Pushed 6ed6b9a..522ba7d (the D-113 amendment + the v6 script existed only on
local disk; a jumphost rebuild would have lost them).
- changelog-20260714-opnsense-set-interface-v6.md -- the delivery rule wants
harness + lint + changelog-with-revert; the script shipped with the first two.
Records that it is WRITTEN AND TESTED but NEVER RUN with --commit (C1 is open).
Ledger:
- Machine block re-seeded: next-free D was STALE at 114 (actually 119; D-114..118
are all ADOPTED), and SEC-008 was missing.
- Logged a ledger-scan DEFECT: it reports D-115 as PROPOSED/OPEN, but D-115 is
ADOPTED -- the scan matches "PROPOSED/OPEN" in body prose instead of reading the
Status line. The status twin of the decoy-token rule. Logged, not fixed.
- CORRECTED pinned item 1a, which still instructed the REST API for the LAN IPv6.
That path does not exist (D-113 amendment, measured). It would have sent the
next session at a dead end.
repo-lint 0 fail; opnsense-set-interface-v6 harness 40/40.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Pin Stage 2 as the branch merge point; reconcile the Stage 2 tracker; log the OpenTofu naming collision (gap #19)
...
Operator ruling 2026-07-14: branch dc-dc-stage1-phase0-first-exec merges to
main at STAGE 2 CLOSE, not at session end. Recorded in the ledger's PINNED
section (a ruling, not an open question) and in the workflow doc's Stage 2
section, which now carries the C1..C5 close-out checklist that defines what
"closed" means. This is NEW policy: main has never taken a feature merge, so
all 52 commits of VR1 work live only on the branch.
The Stage 2 status block was STALE -- it still listed MAAS-region, LXD, the
LXD-KVM-host registration and both composed service machines as NOT DONE. All
of that landed 2026-07-13, and the route enable + the D-115 carve import have
landed since. Corrected against measured state; the doc has now drifted three
times and says so. Stage 2 is in CLOSE-OUT, not build.
GAP #19 (new, BLOCKING Stage 3): "dc1" now names two different datacenters.
D-117 moved the repo to the apex's 0-indexed convention on the IMPORT PATH
ONLY. opentofu/main.tf still says dc1_planes/dc2_storage for VR1's FIRST and
SECOND DCs, and lib-net.sh's dc0 means VR0's DC0 -- a different region, and a
RUNNING cloud. So the next DC we deploy is simultaneously apex vr1-dc0,
OpenTofu dc1_*, and a $DC value that collides with the live testcloud. A
mechanical sed would hand VR1's first DC VR0's deployed literals. Nothing in
opentofu/ gains a new DC-keyed module until this is closed.
Docs only -- no script, no tooling, no cloud state touched.
repo-lint: 0 fail (1 documented legacy warn).
Revert: git revert HEAD
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-118 ADOPTED (org ULA fd50:840e:74e2::/48); VR1 DC0 + DC1 IMPORTED; roles importer bug fixed
...
D-118: ORG_ULA_48 = fd50:840e:74e2::/48. RFC 4193 valid (fd -> L=1, 40-bit Global
ID 0x50840e74e2, /48 -> 256 x /56 for D-101's per-DC /56). The load-bearing check
is NOT the RFC conformance -- it is that it does NOT collide with Tailscale's
fd7a:115c:a1e0::/48: office1-tailscale runs --accept-routes, so an overlap would
have been a LIVE tailnet routing conflict, not a paper one.
Provenance stated honestly: the 'CSPRNG-generated' claim cannot be re-verified.
It does not change the ruling (RFC 4193's SHA1 construction is a SHOULD; the
requirement is a pseudo-random 40-bit Global ID). Regenerating would only swap in
a different random number at the cost of re-editing 7 places.
Corrects a FALSE claim: changelog-20260711 called this value 'ratified' when no
D-number ever assigned it. That phantom ratification is why it read as settled for
two days. Same failure mode as D-117: a document asserting authority it lacked.
IMPORTED into the sandbox: six-plane roles + 2 RIRs + 5 aggregates, then VR1 DC0
(18 prefixes) and VR1 DC1 (18). The D-117 fix is now LIVE-PROVEN, not just green
offline: --dc dc0 -> apex site VR1 DC0 (id=8), --dc dc1 -> VR1 DC1 (id=9). ULA
reads DC.NN in the 4th hextet per D-111 (fd50:840e:74e2:220:: .. :350::).
A REAL BUG, found by running roles-aggregates-import.py for the first time for
real: it checked existence by SLUG only, but NetBox also enforces UNIQUE NAMES.
The draft's legacy role 'Replication' (slug repl) blocks the six-plane role (slug
replication, name Replication) -- so it created 4 roles, 400'd on the 5th, and
DIED, leaving the apex HALF-POPULATED with no RIRs, no aggregates, no rollback.
It had never met a NetBox holding the real draft (only empty ones) and it SHIPPED
WITH NO HARNESS. That is why it survived.
- six-plane role renamed 'Replication Plane' (slug unchanged -- lib-net.sh
SPACES6 and the prefix importer look up 'replication')
- the script now PREFLIGHTS every name/slug and writes NOTHING unless the whole
plan is viable. An IPAM importer that can die partway corrupts what it
populates.
- tests/roles-aggregates-import/ CREATED: 16/16 PASS.
Final state PROVEN by sandbox-fidelity-check (now carrying the full planned
delta): the sandbox is the upstream draft + EXACTLY D-115 + D-117 + D-118 --
every shared object field-identical, nothing lost, nothing stray. 90 -> 134
prefixes.
repo-lint 0 fail. run-tests-all: ALL GREEN (56 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
Prove the sandbox is a FAITHFUL replica, not just a count-matching one
...
netbox/sandbox-fidelity-check.py (+4 assertions in tests/sandbox-seed, 22/22 PASS)
Counts and idempotency CANNOT prove a faithful seed. The region-scope bug proved
it: 17 of 90 prefixes silently lost their scope, every count still matched, and
re-runs were still perfectly idempotent. It was caught by LUCK on a spot-check
that happened to print 'site None'. This is that catch, made deliberate.
Method needs no new plumbing: prod-draft-dump.py is READ-ONLY, so point it at the
sandbox and diff the two JSON dumps FIELD BY FIELD.
RESULT: every shared object is field-level IDENTICAL to upstream, and the delta is
EXACTLY the expected D-115 additions (+edge role, +vr1-off1 site, +8 prefixes).
Nothing missing, nothing unexpected. The sandbox is a PROVEN baseline -- which is
what the later feed-back diff to production depends on.
Also de-misdirects docs/dc-dc-netbox-buildout-scope.md, which still told the next
person to run '--dc dc2' with DC2_V4_SUPERNET -- both of which the new guards now
REJECT by name. Its G5 recommendation (rename the APEX to vr1-dc1/vr1-dc2) is
marked SUPERSEDED: D-117 ruled the opposite. ORG_ULA_48 is now listed there as
UNASSIGNED and BLOCKING, not as a settled literal.
Disclosure: D-117 Option B is PARTIALLY executed -- the import path and the scope
doc are done; the $DC shell selector and the runbook DC prose are NOT. Recorded
in the ledger so 'D-117 ADOPTED' is not misread as 'fully renamed'.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-115 office carve IMPORTED into the Office1 NetBox; DC import blocked on ORG_ULA_48
...
netbox/d115-office-carve.py (+ tests/d115-office-carve/ 20/20 PASS)
Applied to the Office1 sandbox: site vr1-off1, roles office + edge (edge is NEW),
and 8 prefixes. Idempotent (re-run: 0 created / 11 present).
10.10.0.0/16 office container the Office v4 block, /22 per office
10.10.0.0/22 office container VR1 Off1's /22
10.10.0.0/24 office active office1-local LAN (Kea on the edge)
10.10.1.0/24 office active lxdbr0 compose net (MAAS DHCP)
172.30.0.0/16 edge container NEW Edge role; mirrors v6 2602:f3e2:fe::/48
172.30.1.0/24 edge active office1-wan, the simulated ISP uplink
2602:f3e2:f01:/56 office container mirrors VR0 Off0's e01:/56
2602:f3e2:f01:/64 office active office subnet (NOT deployed; v6 has no egress)
Every address was already RUNNING. D-115 blesses what is deployed -- zero
renumber. What it fixes is that three of them were SQUAT: on the wire with no
allocation behind them.
Writing upstream is GATED IN CODE: the tool refuses a non-sandbox target unless
given --yes-write-upstream. Feeding the production apex is an operator decision,
never a side effect of running an import.
BLOCKED, deliberately: dc-dc-prefixes-import.py --dc dc0 CANNOT run. It needs
ORG_ULA_48, which is still UNASSIGNED (gap G3; the scope doc recommends
fd50:840e:74e2::/48 but no ADOPTED decision assigns it -- grep ORG_ULA
design-decisions.md returns nothing). Using it anyway would be an inferred value
(hard rule 2). This is a DECISION gate, not a tooling gap: the tooling is fixed,
tested and ready. Recorded in the ledger with the exact commands to unblock.
Also: docs/vr1-office1-as-built.md gains the IPAM table, and records that the
Tailscale 10.10.0.0/22 route is now ENABLED (edge GUI reachable from the
workstation with no tunnel), superseding the tunnel instructions.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
| 2026-07-13 |
D-116 ADOPTED: no Office1-local GitBucket -- keep using git.baldurkeep.com
...
Operator ruling: "we've decided not to build a separate one on the test cloud. We will continue to
use the current setup and create new repos for deployment tasks as needed."
SUPERSEDES the GitBucket half of D-103 (which gave Office1 three service VMs) and removes GitBucket
from Stage 2's build list entirely. Office1's MAAS-composed service machines are now exactly TWO --
office1-netbox and office1-tailscale -- and both are LIVE.
WHAT THIS ACTUALLY CLOSES, and why it is a simplification rather than a deferral: the Stage 2 runbook
carried an OPEN QUESTION about the Office1-local GitBucket -- "what it mirrors, and under what repo
path, is not decided" -- because a SECOND GitBucket alongside the pre-existing git.baldurkeep.com
raised a real authority/mirroring question nobody had answered. That question is now MOOT. There is
one git service, it is the existing one, and it is authoritative.
Recorded explicitly in D-116 so nobody misreads the scope: D-107's per-DC ARTIFACT mirror (OS/package
artifacts for node provisioning) is a SEPARATE concern and is NOT affected. And if Roosevelt later
wants a site-local git service, that is a fresh decision against real requirements -- not something
this ruling forecloses.
Applied everywhere it was an instruction, not just recorded:
- runbooks/dc-dc-phase1-office1-standup.md: Step 11 replaced with a TOMBSTONE (kept rather than
deleted -- step numbering is referenced elsewhere, and someone WILL come looking after reading
D-103); removed from the sequence, the compose steps, and the open-questions list.
- docs/dc-dc-deployment-workflow.md: dropped from Stage 2's Build and, importantly, from its GATE --
"GitBucket serving" is no longer a gate condition.
- docs/design-decisions.md: D-103 annotated in place with the supersession, so grepping for GitBucket
lands on the amendment rather than the stale instruction.
- docs/vr1-office1-as-built.md + session ledger updated.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger: pin Office1 LAN IPv6 as the next step after the NetBox deploy (D-115 q3)
...
Operator sequenced it: NetBox first, then dual-stack. Pinned with its known limit attached so the
next session cannot mistake internal v6 for working v6 -- IPv6 does not egress the lab (measured),
so the LAN leg proves addressing/RA/services-binding and nothing about internet GUA reachability.
The WAN v6 leg is DEFERRED: it would be a path to nowhere and untestable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END.
...
MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.
vcloud (L1 -- itself a KVM guest, measured)
+- voffice1 (L2) MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
+- office1-netbox (L3) LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected
THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.
THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
10.10.0.0/24 (site LAN) -> Kea on the OPNsense edge MAAS dhcp_on=False
10.10.1.0/24 (lxdbr0) -> MAAS MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.
NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.
FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
be picked by accident.
DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.
10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.
Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.
Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
...
APPLIED, operator-approved: 5 added, 0 changed, 0 destroyed. The running OPNsense edge was NOT
touched (confirmed by plan BEFORE apply -- no in-place update, so no repeat of the 2026-07-13
guest bounce).
THE MODULE WAS SILENTLY BROKEN FOR D-114'S ENTIRE MODEL. modules/cloudinit-vm's libvirt_domain
had NO `cpu` block. With none, libvirt renders a GENERIC EMULATED CPU exposing no `svm` -- and
LXD *virtual machines* are qemu/KVM guests, so NO service VM could ever have been composed into
voffice1. D-114 would have failed at its first step with a confusing "KVM not available" that
pointed nowhere near OpenTofu. Measured proof of the masking: host is an AMD EPYC 9965; a
default-CPU guest was handed an "Opteron_G3".
Fix: `cpu { mode = "host-passthrough" }` always, plus a new `expose_nested_virt` variable with NO
default -- a real per-VM decision. true = pass `svm` through (required for D-114's containment
VMs); false = disable it, matching modules/opnsense-edge. The operator asked whether the edge's
`svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a
triple-fault fix on 2026-07-12, did NOT resolve it (the real cause was the memory unit), and was
retained on hardening grounds. A router has no business seeing nested virt. It stays disabled there.
voffice1 as built (MEASURED): Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, single NIC on
office1-local, reports the real EPYC CPU with svm on all 16 cores and /dev/kvm PRESENT, egress to
1.1.1.1 at 4.6ms THROUGH the edge. Cloud-init deliberately MINIMAL (identity + key + guest agent);
MAAS and LXD go in as separate GATED steps rather than buried in a first-boot script that either
silently works or silently does not. The SSH pubkey is read via file(var...) at plan time -- key
material never enters a command line, the repo, or an agent's context; the tfvars holding the path
stays gitignored. network_config matches the NIC by GLOB, not a guessed kernel name (an inferred
value); it came up enp1s0.
TWO GATES CLEARED:
1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item: the daemon was proven, the
SERVICE never was (office1-local had no client). voffice1 -- the first client ever on that LAN
-- took 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state
active, read back through the D-113(a2) REST API. DHCP now works end to end.
2. D-114's NESTING PROBE PASSES AT L3. Stated honestly: this proves nested KVM is AVAILABLE. The
DEFINITIVE proof is a guest actually booting inside voffice1 -- that lands with the first
composed LXD VM. Not recording L3 as fully proven until then.
FINDING, logged NOT actioned (DC1 is gated behind Office1 per D-114): modules/node-vm has the SAME
missing-cpu-block defect. DC nodes run nova-compute, which needs working KVM -- without the fix
they will come up unable to run a single instance. MUST be fixed before Stage 3.
opentofu-validate PASS, repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Fold the AppArmor/libvirt rule into scripts/prereqs -- it gated every VR1 VM boot
...
`/var/lib/libvirt/vr1/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu had
been applied BY HAND on vcloud and existed ONLY as host state. Nothing in the repo
carried it, so a rebuild or a second host walked straight back into the same wall.
Why it matters: libvirt's stock abstractions/libvirt-qemu grants qemu only the DEFAULT
pool path (/var/lib/libvirt/images). VR1 uses a custom pool parent, which is not in that
abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The domain
DEFINES fine, then fails to start with a bare qemu "Permission denied" -- and NOTHING in
the libvirt error names AppArmor. It cost a session on 2026-07-12 (DOCFIX-186), was
logged as a PREREQ candidate, and was never actioned until the ledger collapse resurfaced it.
New: scripts/prereqs/install-apparmor-libvirt.sh, wired into install-all.sh (after
install-virtualization -- it edits libvirt's own profile dir) and reported by
check-prereqs.sh.
- Idempotent and deliberately QUIET on a good host: if the rule is present it exits 0
having touched nothing -- it does NOT reload apparmor or bounce libvirtd. Those happen
only on the run that actually adds the rule.
- Writes to local/, the vendor-sanctioned override include, so it survives package upgrades.
- Pool parent is a variable (VR1_POOL_PARENT, default /var/lib/libvirt/vr1), not a literal.
- Non-AppArmor hosts report N/A and SUCCEED -- no manufactured failure on a host it
does not apply to.
Verification: tests/prereqs 32/32 PASS (was 24/24) -- the new cases drive DETECTION off a
pool parent guaranteed absent from the profile, so the negative path is deterministic on
any host: a false OK from --check IS the bug, --dry-run must plan and mutate nothing, and
the planned rule must be exactly `<parent>/** rwk,`. Full gauntlet ALL GREEN (52 harnesses),
repo-lint 0 fail. Run live on vcloud: --check reports OK and leaves the profile byte-identical,
confirming the already-satisfied path is a true no-op.
No host state was mutated by this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger collapse: close the coverage gap in the carry-forward
...
The collapse commit's guards (lint, ASCII, ledger-scan, size) validated FORM, not
COVERAGE. Two ledger regions had only been keyword-grepped, not read. Read them.
Three items surfaced; each was VERIFIED against the repo rather than trusted:
- run-tests-all.sh dirtying the tree with 37 spurious mode-change diffs: NOW FIXED
(the fakebin stubs are committed 100755 -- 36 of them, measured). Not carried.
- `scripts/opnsense-apply-config.sh` (once "wanted"): SUPERSEDED, and dangerous to
build -- its snapshot/scp/install/reboot flow IS the config.xml path D-113(a2)
deleted. Recorded as "do not build this", so nobody resurrects it from the history.
- The tcsh trap it taught is REAL and carried forward: root's shell on the OPNsense
edge is tcsh, so `$(...)` in a quoted remote command dies with "Illegal variable
name" -- QUIETLY. It already cost a silently no-op'd pre-install snapshot. Always
feed remote commands to `sh -s`.
Also carried (this one matters and was nowhere in the open lists): DHCP HAS NEVER
SERVED A REAL LEASE. kea-dhcp4 is bound to udp/67 with the subnet+pool intact, but
office1-local has no client host, so the DAEMON is proven and the SERVICE is not.
The first VM on office1-local (the Stage 2 headend VMs) is the real test -- logged
as that stage's gate rather than an assumption.
Plus a measured doc-accuracy finding: SEC-007 says "D-112(c) makes SSH the only
management path", but the edge's web GUI IS listening on the LAN (10.10.0.1:443,
HTTPS 200) and is correctly firewalled off on the WAN. Not an exposure -- OPNsense
default, and D-113 kept OPNsense *because* the GUI is an operator requirement -- but
the wording is imprecise. Logged, not actioned (hard rule 1).
Guards: repo-lint 0 fail (1 documented legacy warn), 0 non-ASCII. No live system
touched; the edge probes were read-only TCP/HTTP reachability checks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Collapse the three ledger streams into one; reconcile against repo ground truth
...
The ledger carried three separately-owned parallel stream sections (main-chat,
jumphost, shared) plus ~30 append-only session narratives -- 2189 lines, several
of them contradicting each other and the repo. Collapsed to ONE stream, 222 lines,
carrying only what is still OPEN plus the facts that live nowhere else.
The narrative is NOT lost: it is in this commit's parent (f81177a) and, durably,
in the 65 docs/changelog-*.md files, design-decisions.md, and the incident reports.
RECONCILED (repo ground truth wins over any stream's self-claim; all verified):
- machine-derived block was STALE: said D=111/DOCFIX=162/BUNDLEFIX=012 and listed
only SEC-001/003/004. Re-seeded from today's scan: D=114/DOCFIX=195/BUNDLEFIX=013,
six open SEC rows (001/003/004/005/006/007).
- main-chat called backlog items 6/7/8 "not started". FALSE -- keystone-policy-drift.sh,
cloud-snapshot.sh and tests/tenant-acceptance/ all exist. Delivered by the jumphost
stream; this was the cross-stream reconcile the ledger itself had pending.
- item 3 remainder (fold vault-kv-health into cloud-assert): VERIFIED still open.
- item 5 (tenant-cluster-create.sh): VERIFIED still absent.
- R-3/D-063: ledger claimed CLOSED, handoff doc still says PROPOSED/OPEN. D-063 is
RESOLVED/CLOSED in design-decisions.md -- so the HANDOFF DOC is the stale one.
Carried forward as an open item rather than fixed here (hard rule 1).
- D-076: not this repo's to work. D-076..D-099 is a RESERVED source-project band
(D-110), so its "uncommitted worktree draft" was never here. No work lost.
- "beta cluster left at node_count=2" was STALE -- the 2026-07-08 audit found no beta
cluster and no foil1 tenant. Retired.
CARRIED FORWARD (would have been operationally lost -- the skill loads this file):
the API-managed-DHCP edge trap; "updated in-place" still BOUNCES the guest; do not
re-implement OPNsense's key crypto, call the interface; config ISO can never be read
on a nano image; guide<->skill coupling (repo-lint L8); controller model is
admin/controller; State facts; Project-completion.
NEW, measured this session: the apparmor rule /var/lib/libvirt/vr1/** GATES EVERY VR1
VM BOOT but exists only as host state -- it is NOT in scripts/prereqs/, so a rebuild
hits the identical foundational failure. Logged as an open item.
Guards: repo-lint 0 fail (1 documented legacy warn), 0 non-ASCII, ledger-scan
reconciles against the re-seeded machine block. No live system touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
APPLIED + CORRECTION: "updated in-place" does NOT mean "no restart" -- the apply bounced the edge
...
The config_seed/cdrom removal is APPLIED to the live Office1 edge. Apply complete: 0 added,
1 changed, 1 destroyed. `tofu plan` now reports "No changes" -- repo and state are back in sync,
so Stage 3 cannot trip over the divergence.
CORRECTION, recorded because the prediction was WRONG and the mistake is reusable:
I predicted the apply would NOT interrupt the guest, on the grounds that the plan said
"libvirt_domain.vm will be updated IN-PLACE". THE APPLY RESTARTED THE GUEST. Measured: uptime went
from 8:36 to 6 secs across the apply. The dmacvicar/libvirt provider redefines the domain to apply
a disk-list change and BOUNCES it -- a ~30s outage during which the edge was neither routing nor
serving DHCP.
"updated in-place" means the RESOURCE is not replaced. It says NOTHING about whether the GUEST
keeps running. Do not read it as "no interruption" -- that is exactly what I did.
STAGE 3 CONSEQUENCE (the reason this is worth a commit of its own): any future tofu apply that
touches a libvirt_domain's devices -- on a DC edge, or on a node VM -- must be assumed to BOUNCE
THE GUEST, and scheduled/gated as an outage rather than a no-op config tidy-up.
Recovery was clean and needed no intervention. Post-boot, measured: kea-dhcp4 running and bound
udp4 10.10.0.1:67 with subnet 10.10.0.0/24 + pool .100-.199 intact; WAN 172.30.1.2, LAN 10.10.0.1,
default route 172.30.1.1; egress to 1.1.1.1 at 0.0% packet loss; 8 LAN pass rules in pf; serial
console + getty alive (DOCFIX-192 holds). The cdrom is gone from the persistent domain XML.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2): API-key bootstrap -- no GUI click, no re-implemented crypto
...
The last unknown in D-113(a2) edge provisioning. The API path was proven (read AND write), but
every new edge still needed a MANUAL GUI CLICK to mint its key -- which would have put a human in
the middle of building DC1's and DC2's edges in Stage 3. Closed.
scripts/opnsense-mint-apikey.php runs ON the edge; calls OPNsense's OWN model
(Auth\User -> apikeys->add()) -- the exact code path
the GUI's ticket button invokes
scripts/opnsense-bootstrap-apikey.sh ships it over SSH, runs it, retrieves the key, wipes
the remote copies
tests/opnsense-bootstrap-apikey/ offline harness, 8 PASS (ssh/scp stubbed)
Gauntlet ALL GREEN (54 harnesses, was 53); repo-lint 0 fail.
VERIFIED END TO END on the live Office1 edge: minted a SECOND key via the vendor model ->
GET core/firmware/status returned HTTP 200 (the vendor-minted key AUTHENTICATES) ->
POST auth/user/del_api_key/<id> -> {"result":"deleted"} -> search_api_key shows 1 row remaining
(the operator's original key, untouched throughout) -> the retired key is now REJECTED (a real
negative test, not just an absence of errors). Temp creds shredded from the jumphost. The edge is
back to its prior state.
THE DESIGN DECISION, and why the alternative was rejected: OPNsense stores API secrets as
crypt(secret,'$6$') -- SHA-512 with an EMPTY salt (verified on the live 26.1 box: the stored hash
is "$6$$" + 86 chars). We COULD mint keys offline and seed them into a bootstrap config.xml. We
deliberately DO NOT. That would couple our provisioning to a vendor hash format we would have to
keep byte-compatible forever, and a mismatch FAILS SILENTLY -- keys that simply never
authenticate, with no error at generation time. Calling the vendor's own generator has no format
to keep in sync. Same principle as D-113(a2) itself: call the interface, do not re-implement the
internals -- which is exactly what DOCFIX-191/192/193 cost us. (An offline-hash attempt was
blocked by the permission classifier mid-session; the block was correct and produced this better
design.)
GUARD CLAUSE WORTH KEEPING (harness T5): the script REFUSES to overwrite an existing creds file.
An existing file means a LIVE key on the edge; overwriting the local copy would strand it there
PERMANENTLY, because the secret is hashed on save and can never be read back. That is
unrecoverable credential loss. T5 asserts both the refusal and that the existing file is left
byte-intact. The secret is never printed -- creation is the only moment it exists in cleartext.
WHAT THIS UNBLOCKS: edge provisioning is now automatable with no human in the loop --
boot factory nano -> D-112(c) console bootstrap (SSH + key) -> opnsense-bootstrap-apikey.sh
-> opnsense-api.sh for DHCP/firewall/interfaces.
There is NO config.xml anywhere in that chain.
STILL OPEN: the template reduction itself. And the chain above raises the real question -- the
template may not need REDUCING so much as DELETING, since the console bootstrap plus the key
minter cover first contact. NOT ruled; not done unilaterally. (Note the opnsense-edge module's
config_seed volume is INSTANTIATED -- it is in tfstate -- so removing the ISO wiring is a LIVE
change, not a docs change.)
Revert: git rm the two scripts + tests/opnsense-bootstrap-apikey/ (nothing live depends on them --
the live edge's key was minted via the GUI and is unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
Correct the workflow doc's status: it was materially FALSE and would misdirect work
...
docs/dc-dc-deployment-workflow.md is the first thing a fresh session reads. Its status claims
dated from 2026-07-09 and had drifted into being wrong. Corrected against MEASURED state.
Docs only; no code, no live system touched.
gap #2 "UNVALIDATED -- no tofu binary" -> OpenTofu v1.12.3 IS installed; tree validates
(root + 10/10 modules) and is partly APPLIED
(11 resources in state). But 2 modules were
BROKEN and had never been parsed (DOCFIX-194).
gap #17 "no per-site ISP-uplink network -> CLOSED FOR OFFICE1: office1-wan is live
exists anywhere, for ANY site" (virbr11, NAT, 172.30.1.0/24); the edge's WAN
sits on it at 172.30.1.2, egress verified. The
gap's own proposed answer (a dedicated per-site
uplink, NOT a mesh leg) is what got built.
DC1/DC2 STILL have none.
opnsense-edge "BUILT, UNVALIDATED" -> BUILT, INSTANTIATED, RUNNING (routing, NAT,
egress, console, SSH, Kea DHCP on udp/67).
config.xml tmpl "BUILT, fully tested" -> SUPERSEDED by D-113(a2); a full push would now
CLOBBER live API-managed DHCP.
netem-link "BUILT, UNVALIDATED" -> WAS BROKEN; fixed 2026-07-13 (DOCFIX-194).
Stage 2 "NOT YET EXECUTED" -> PARTIALLY EXECUTED.
THE TRAP THIS REMOVES: Stage 2's gate lists "NetBox authoritative" and "GitBucket serving" -- and
NetBox and GitBucket DO answer, at baldurkeep.com. But those are PRE-EXISTING services, NOT
Office1 headend VMs. Measured: office1-opnsense is the ONLY VM on the vcloud host. It would be
very easy to tick that gate as met and walk into Stage 3 with no headend at all. The corrected
State section now says this explicitly.
What actually remains of Stage 2 (the real critical path): the three headend service VMs
(MAAS-region, NetBox, GitBucket) do not exist. Blocked on a genuine decision, not on typing:
Option A (modules/cloudinit-vm) needs user_data/meta_data/network_config designed and an image
source chosen; Option B is manual virt-install, logged as debt. Both cloudinit-vm and base-image
now at least VALIDATE (DOCFIX-194) -- they never had before.
Revert: git revert this commit (restores the stale, false status claims).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules
...
scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.
ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.
Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.
DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.
DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.
THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.
VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.
Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.
Nothing was instantiated; no live system is affected by either the bug or the fix.
Revert: git revert this commit (restores the two broken modules and the root-only gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SAFETY SWEEP: mark the config.xml push path as a LIVE HAZARD (no behaviour change)
...
After D-113(a2) was proven (Office1's config is now API-managed), every instruction in the repo
that still said "render a config.xml and push it to the edge" became ACTIVELY DANGEROUS: a full
config.xml push replaces /conf/config.xml wholesale and drops ~667 migration-populated elements
(measured), including the only 2 firewall pass rules on the box. Following the old runbook steps
against the live edge would clobber it.
Warnings and headers only. No behaviour changes, no live system touched.
Marked (5 files):
runbooks/dc-dc-phase1-office1-standup.md DANGER banner; the config.xml render + config-ISO
sub-steps removed as RUNNABLE instructions (they
would clobber the live edge). Image-prep retained;
history preserved in git + the build changelog.
runbooks/dc-dc-phase2-tofu-dc-substrate.md STOP banner on Step 4.
scripts/opnsense-render-config.sh DANGER header at point-of-use. NOT dead: under
D-113(a2) it is to be reduced to a MINIMAL bootstrap
render. Until then, safe only for a brand-new,
not-yet-booted edge.
scripts/opnsense-build-config-iso.sh RETIRED header quoting the upstream source proving
the importer can never fire on a nano image.
docs/dc-dc-deployment-workflow.md STALE-CONTENT warning atop the tooling gap register.
THE SWEEP ALSO CAUGHT A PRE-EXISTING LANDMINE, unrelated to today's work: dc-dc-phase2 Step 4
still instructs building a CONFIG ISO for DC1's edge -- but D-112 established that ISO can never
be read (opnsense-importer probes for a read-only root; on a pre-installed nano the root is
writable and a factory /conf/config.xml already exists, so it bootstrap_and_exit 0's without
enumerating a single device). That runbook has been telling anyone who follows it to build an
inert artifact and then wonder why the edge came up on factory defaults -- which is EXACTLY the
day lost on 2026-07-12. It was never corrected at the source. Now it is. Its WAN_IF/LAN_IF
"chicken-and-egg" discussion is moot for the same reason: that problem only exists if you try to
seed a full config before first boot, and D-112(c) measures vtnetN AFTER boot.
Changelogs deliberately NOT touched -- they are history, not instructions. Marking history is
noise; marking instructions is safety.
Verification: opnsense-render-config 24 PASS, opnsense-build-config-iso 2 PASS, opnsense-api
21 PASS; repo-lint 0 fail.
STILL OPEN (the actual D-113(a2) migration): reduce the template to a minimal bootstrap, retire
the config-ISO path and the opnsense-edge module's config_seed/cdrom wiring, and rewrite Stage 3's
edge steps around the API. This sweep makes the repo SAFE, not FINISHED.
Revert: git revert this commit (restores the previous, dangerous instructions verbatim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
CHECKPOINT 2026-07-13: session close (edge routing + serving DHCP; D-113(a2) proven)
...
Session-ledger checkpoint so a fresh session can resume cold.
LIVE STATE (measured): office1-opnsense is UP -- routing, NAT, egress 0.0% loss, serial console,
SSH key management, and kea-dhcp4 serving DHCP on udp/67 (pool .100-.199). 8 LAN pass rules in
pf. Nothing broken, nothing mid-flight.
LANDED TODAY:
1. DOCFIX-193 APPLIED -- DHCP live (was built-but-unapplied at last close)
2. SEC-005 -- GitBucket credential stored; unattended git push now WORKS (the branch had sat
12 commits ahead of origin for two days -- a real data-loss exposure, now closed)
3. SEC-006/007 -- the two known credential exposures put ON the ledger (prose only before)
4. D-113 RULED (a2): stay on OPNsense, move config to the REST API
5. D-113(a2) PROVEN END TO END, read AND write: API -> OPNsense -> Kea -> daemon.
scripts/opnsense-api.sh + harness (21 PASS); gauntlet 53 ALL GREEN.
THE ONE TRAP: DHCP on Office1 is now API-MANAGED. A full rendered config.xml push WOULD CLOBBER
IT. Do NOT run the old scp/install/reboot path against this edge until the template is reduced.
NEXT: reduce opnsense-config.xml.tmpl to a minimal bootstrap (sshd + root key + console) and move
DHCP/firewall/interfaces to the API. Repo work; no live mutation needed to land it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
