| 2026-07-16 |
docs(session-ledger): record D-125 push + the post-Model-B doc drift-reconciliation batch
...
Standing-ledger update (deliverable discipline): logs D-125 committed d34ec94 +
pushed, and the three doc-only reconciliations (8496c2e workflow-doc, 87f9d75
opentofu/README, 788911c teardown-rollback) with the repo-wide drift-scan
finding (remaining deleted-file/old-count refs are changelogs/decision-records/
tombstones, not live drift; deeper prose + Stage-4/5 sections still lag).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3 design gap
...
OBS-3: Model B nesting put vr1-dc0-wan (a libvirt NAT) inside vvr1-dc0, whose
only leg is the SEC-010 FORWARD-dropped transit -- so the per-DC simulated ISP
had no egress. Operator ruled bridge-in (single-NAT) over double-NAT.
- modules/wan-bridge (NEW): libvirt forward=bridge onto a pre-existing host
bridge (bridge={name}, schema measured from the provider, not guessed). No
subnet/NAT. Sibling to site-wan so every NAT user stays untouched. Covered by
opentofu-validate S3 (standalone PASS).
- OUTER root: vcloud ISP NAT vr1_dc0_uplink (the single NAT) + a 2nd IP-less
uplink NIC on vvr1-dc0 + the br-vr1-dc0-wan bridge declared in netplan (transit
stays NIC1 so SEC-010/--transit-if keep keying on it). New HELD var
vr1_dc0_uplink_cidr (no default; office1-netbox apex, hard rule 2).
- INNER root: vr1_dc0_wan site-wan(NAT) -> wan-bridge onto br-vr1-dc0-wan;
OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113, HELD).
- Bootstrap (site-headend-install.sh): --uplink-if/--wan-bridge; --check verifies
the bridge exists + uplink ENSLAVED (fail-open guard, same class as the transit
check); SEC-010 rationale corrected (bridge-in has no inner NAT) + br_netfilter
CONSTRAINT (keep the FORWARD-drop interface-scoped, NEVER global). Harness 49->54.
Records: D-125 (new); D-122 amendment (forward-pointer -- WAN realization
superseded, intent stands); SEC-010 (br_netfilter + hardened --check);
model-a-fallback B->A delta item 8 + revert; platform-traps 1g (bridge +
br_netfilter + nested-foreign-MAC trap).
NOT closed: the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in
equivalent of the depth-4 boot gate); fallback = double-NAT. HELD: vr1_dc0_uplink_cidr
+ OPNsense WAN re-address. Both roots validate; gauntlet ALL GREEN (63); repo-lint
0 fail. Present-only; NOT pushed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage-3 review sweep C+D: Model B two-root reshape + SEC-010 hardening
...
Phase C (Model B reshape, D-123 operator ruling):
- outer root (opentofu/main.tf): vvr1-dc0 resized to whole-DC containment VM
(108 vCPU / 425984 MiB / 3221225472000 bytes; expose_nested_virt=true;
single Office1 transit leg); node/plane/wan/opnsense modules removed from
vcloud level (MOVED markers), storage-04 added per R-3.
- inner root (opentofu/vr1-dc0-substrate/): NEW second root, provider
qemu+ssh to vvr1-dc0; reuses ../modules/* verbatim to build 9 nodes +
6 planes + wan + opnsense inside the containment VM. dmacvicar/libvirt
0.9.8 pinned (.terraform.lock.hcl committed). Both roots tofu-validate.
- variables.tf: vvr1_dc0_vcpu/memory_mib/disk_bytes with derivation comment.
Phase D (SEC-010, R3-F02): site-headend-install.sh --host-nodes node-host
bootstrap installs qemu-kvm+libvirt, persists+verifies nested=1, writes a
transit FORWARD-drop (nftables inet sec010) enforcing DC-LOCAL under Model B.
Gate hardened (advisor catch): --check now verifies the keyed transit
interface EXISTS, else nftables oifname loads clean but matches nothing
(fail-open). Harness 49/49.
Records: D-103/107/114/121/122/123/124 amendments (append-only);
model-a-fallback-plan synced; ledgers + Phase-C changelog.
KNOWN GAP (OBS-3, operator-confirmed 2026-07-16): under Model B the inner
vr1-dc0-wan NAT is nested inside vvr1-dc0 (single transit NIC), severing the
simulated-ISP egress that Model A had at vcloud level. DC0/DC1 ARE meant to
have ISP connections -- egress path resolution is follow-up design work, not
resolved by this commit.
Present-only; not pushed. Gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Review sweep A+B + R-3 storage-04: coherent record, whole-host calculator, R-3-compliant Model A fallback
...
Phase A (record amendments) + Phase B (scripts/dc-dc-whole-host-budget.py + harness) of the
Stage-3 adversarial-review sweep, plus the R-3 storage-04 addition (Model A layout now
3+2+4 = 9 nodes/DC). This commit is the R-3-COMPLIANT Model A state and becomes the re-cut
`model-a-fallback` anchor (the prior tag at 87a7a8a was R-3-stale -- 3 storage -- flagged by
the Model B design cross-check). NO cloud mutation; nothing applied.
- design-decisions.md: D-103/D-107/D-114/D-115/D-121/D-122/D-123/D-124 amendments (rulings R-1..R-4).
- scripts/dc-dc-whole-host-budget.py + tests/ (13/13): replaces the lost optc-calc.py (R3-F06).
- opentofu/main.tf: locals.vr1_dc0_nodes += storage-04 (R-3); tofu validate Success.
- security-ledger SEC-011 closed; session-ledger rulings + sweep progress; model-a-fallback plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
WIP: freeze surface for Stage-3 adversarial review -- SEC-010/011 + ledger
...
Plane-segregation review findings logged earlier this session, committed to
freeze the review surface (per the Stage-3 adversarial-review prompt, FIRST
ACTION). Commit is not push; nothing leaves the jumphost.
- security-ledger: SEC-010 (rack ip_forward/DC-local enforcement) + SEC-011
(node least-connectivity) as PRE-APPLY hardening rows.
- session-ledger: the 2026-07-16 plane-segregation review outcome + the
functional note (rack routes to an as-yet-unattached transit peer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
...
Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:
- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
(D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
the office1<->dc0 mesh leg; rack sizing 4/8192/80.
Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
(two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
/24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
D-121/122 folded in, stale gate corrected).
docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-15 |
Stage 2 CLOSED+MERGED; open Stage 3 branch; standing merge-at-close discipline; drop stray temp/ files
...
Post-merge housekeeping after PR #1 (e468d2e) put VR1 Stage 1+2 on main.
- Workflow doc: the Stage 2 "branch merge point" note now records the merge as DONE
(PR #1, e468d2e) and GENERALIZES it into a standing rule in the Cross-cutting
discipline section -- "branch-per-stage + MERGE AT STAGE CLOSE": every stage
develops on its own branch and merging it to main (merge commit, operator-gated)
is part of that stage's definition-of-done, then the branch is retired. main is
trunk-of-record only for merged stages; the next stage branches off post-merge main.
- session-ledger: Stage 2 CLOSED+MERGED recorded; the PINNED merge ruling graduated
into the standing discipline; NEXT = Stage 3 on dc-dc-stage3-phase2-dc-substrate,
with D-119 selectors + creds-audit (SEC-009) in place.
- Removed temp/ (3 stray screenshot PNGs that rode onto main via 95f0634).
repo-lint 0 fail. No script/harness touched (gauntlet unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
C2 sub-task (ii): extend sandbox loop to ip-ranges/ip-addresses; load + verify D-120 bands
...
Closes the logged tooling gap and the D-120 Step 6 load, verified end to end. With
this, C2 is fully closed and Stage 2 close-out (C1..C5) is COMPLETE.
Tooling:
- prod-draft-dump.py: ENDPOINTS += ipam/ip-ranges, ipam/ip-addresses (ranges get a
synthetic start-end key). Verified live.
- sandbox-fidelity-check.py: KEY + EXPECTED_NEW cover the 3 D-120 ranges + 2 service
IPs; both-bounds gives endpoint/address correctness (pure delta -- the frozen draft
has none). Success banner now D-115/D-117/D-118/D-120. Harness +T11/T12 (12->14).
- netbox/d120-compose-bands.py: stdlib, UA-aware, SANDBOX_HOSTS-guarded, dry-by-default,
whole-plan preflight (parent 10.10.1.0/24 exists, endpoints in-parent, start<=end) so
it cannot half-write. Harness 16/16 pins the band values + guards.
Load (operator-approved --commit into office1-netbox, the VR1 apex; sandbox-local):
- created 3 ip-ranges (.2-.49 / .100-.200 / .201-.254) + 2 ip-addresses (.10 netbox,
.11 tailscale), 0 already present. Re-dump -> 3 ranges, 2 IPs. Fidelity check -> exit 0,
"EXACTLY the planned delta (D-115/D-117/D-118/D-120)."
D-120 Step 6 CLOSED (no apex write -- write-back stays deferred). Gauntlet ALL GREEN
(60); repo-lint 0 fail. Workflow C2 row + session-ledger updated: Stage 2 close-out
complete -> branch->main merge point reached (merge remains operator-gated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
C2 sub-task (i): sandbox re-verified faithful; fidelity-checker scope false-positive fixed
...
Re-ran the HARDENED fidelity check against office1-netbox (the VR1 IPAM apex,
DOCFIX-195) -- read-only dump via the consolidated sandbox token + an SSH tunnel.
Result: exit 0, "upstream draft + EXACTLY the planned delta (D-115/D-117/D-118)."
C2 sub-task (i) is CLOSED; the sandbox is proven faithful.
The hardened checker false-failed on 10.10.0.0/16 and 172.30.0.0/16 being
unscoped -- but those two role-container /16s are unscoped BY DESIGN
(d115-office-carve.py), matching the 27 unscoped role containers already in the
upstream apex. A full cross-check of all 44 delta prefixes against their intended
scope was 44/44 (every DC prefix on the correct DC site; no wrong-target binding).
Fix (netbox/sandbox-fidelity-check.py): assert scope == INTENDED per delta prefix
(EXPECTED_PREFIX_SCOPE; DC half generated by _dc_prefix_scopes so the CIDR set and
scope map cannot drift). Strictly stronger than the old "has a scope?" test -- still
catches a dropped scope, and now catches a WRONG-TARGET binding (a DC0 prefix bound
to vr1-dc1, the D-117 near-miss). NOT fixed by exempting the two CIDRs.
Harness moved to dump format (scope_site), T1 sets each prefix's scope from the
checker's own map, +T9 (wrong-target fails) +T10 (legit-unscoped passes). 10->12.
Gauntlet ALL GREEN (58); repo-lint 0 fail.
SEC-009: the sandbox NetBox token had been MISSED by the consolidation (lived only
at /root/netbox-secrets/api.token on office1-netbox); now consolidated to
~/vr1-office1-creds/vr1-netbox-sandbox.env (0600), copied off the VM without
printing. Ledger addendum recorded.
No NetBox state changed -- dump and check are read-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
DOCFIX-195: office1-netbox holds the VR1 IPAM apex role; apex write-back deferred; C2 redefined
...
Operator ruling 2026-07-15: office1-netbox (10.10.1.10) HOLDS THE IPAM APEX
ROLE during VR1 -- the authoritative NetBox every consuming system (OpenTofu,
MAAS, overlays, bundle) derives its values from. It is a working draft (edited
as the buildout proceeds), but it is the apex. netbox.baldurkeep.com takes NO
write during VR1; it is the eventual production apex, frozen as a read-only v1
reference draft. Merge-back is deferred to end-of-deployment (a maybe), NOT a
Stage 2 gate.
The repo encoded the opposite roles and defined close-out C2 (the last Stage 2
gate, and the branch->main merge trigger) as an operator-gated write UPSTREAM.
Under the ruling there is no apex write in Stage 2, so C2 is redefined:
office1-netbox holds the COMPLETE validated dataset -- carve + six-plane roles
+ both DCs + the D-120 compose-/24 child ranges + the two service IPs -- and it
is VERIFIED. C2 stays OPEN.
Finding surfaced while drafting (logged, not fixed): the D-120 objects are
ip-ranges/ip-addresses, object classes the sandbox loop does NOT cover
(prod-draft-dump.py ENDPOINTS and sandbox-fidelity-check.py KEY/EXPECTED_NEW
both stop at rirs/roles/regions/sites/aggregates/prefixes). So the hardened
fidelity check cannot verify the D-120 load; verifying it needs a tool
extension (+harness) or a recorded manual read-back. Confirmed no live system
config points at any NetBox URL -- documentation-only change.
Files: dc-dc-deployment-workflow.md (State + C2 row), dc-dc-netbox-buildout-scope.md
(sec 8.1 + 8.6), session-ledger.md (5 narrative spots + new deferred-item bullet;
machine-derived block re-seeded from the 2026-07-15 scan), design-decisions.md
(D-010 note + D-103 pointer), netbox/README.md (target banner), + changelog.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
D-120 ADOPTED + EXECUTED: re-IP Office1 services into the .2-.49 static band
...
office1-netbox .201->.10 and office1-tailscale .202->.11, on the wire, no
wipe. NetBox HTTP 200 at .10; tailscale still advertises 10.10.0.0/22.
Method (DC replication inherits it): MAAS refuses interface edits on a
Deployed machine and the only lift is Release->redeploy=WIPE, so the live
address is set on the guest (netplan + curtin cfg, reboot-durable) over
lxc exec (socket immune to the netplan apply), and MAAS-model drift is
accepted (reconciles at next teardown/redeploy). NetBox ALLOWED_HOSTS=*
so no restart; tailscale re-asserted automatically.
Swept SANDBOX_HOSTS .201->.10 in the three importers + test + the on-box
working copy; updated live-state docs; flipped D-120 to ADOPTED with an
Execution record; resolved the re-IP runbook's PENDING markers to the
measured method. Gauntlet ALL GREEN (58); repo-lint 0 fail / 1 legacy WARN.
Still owed: D-120 Step 6 (register the /24 child ranges + service IPs in
the sandbox, feed upstream) rides with C2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpGaiX6UvhSReDwN2ywstJ
|
D-120 PROPOSED: static-services addressing for compose /24s; re-IP plan drafted
...
Office1's persistent services (office1-netbox .201, office1-tailscale .202) sit in
MAAS's deployed-node band -- addressed like ephemeral compute nodes, by MAAS
auto-assign default, not by design. They are NOT in the DHCP dynamic range
(.100-.200), so no collision -- but no static-services sub-layout was ever ratified
for the /24 (D-115 left it "MAAS DHCP/PXE").
D-120 (PROPOSED) defines a functional band layout for EVERY MAAS-managed compose /24
(Office1 + the DCs, which inherit it):
.1 gateway (site containment VM)
.2-.49 STATIC site services (MAAS static-assign, outside the dynamic range)
.50-.99 reserved/spare
.100-.200 MAAS dynamic (unchanged)
.201-.254 deployed nodes (unchanged)
Proposed Office1 addresses: netbox .201->.10, tailscale .202->.11. Recommends
option (a): adopt + re-IP Office1 now, so the DC template is clean. Addresses AWAIT
operator ratification.
Re-IP plan: runbooks/dc-dc-office1-service-reip.md -- gated mutation on two running
services, honest PENDING VERIFICATION markers for the MAAS static-assign CLI and the
netbox-docker ALLOWED_HOSTS / tailscale re-assert (cannot confirm without the
operator-held box keys). Cascades to 20 repo refs incl. SANDBOX_HOSTS .201 in the
two pynetbox importers + their tests. NOT executed.
The layout belongs IN NetBox (the IPAM authority deciding where NetBox lives) --
recorded via the sandbox loop, not prose.
repo-lint 0 fail. No live services touched -- this is a proposal + plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Consolidate credential/env sprawl into ~/vr1-office1-creds/ (SEC-009); seed memory
...
Operator-approved. Three env files were loose in ~ outside the consolidated creds
folder -- .vr1-netbox.env, vr1-office1.env, and vr1-stage1.env. The last was mode
0664 (world-readable) and held TF_VAR_maas_api_key: a real MAAS-secret exposure,
not just untidiness. None created this session (mtimes 07-10..12).
Done (files outside the repo; recorded in the changelog):
- moved all three into ~/vr1-office1-creds/; chmod 600 on every sensitive file
(fixed the two 0664 files: vr1-stage1.env and tailscale-authkey.txt).
- no CONTENTS read -- key NAMES only, to identify each file's purpose.
- fixed a D-119 follow-up the in-repo sweep could not reach: vr1-stage1.env had
TF_VAR_dc1_pool_path/dc2 -> renamed to TF_VAR_vr1_dc0/vr1_dc1_pool_path.
Repo (committed):
- active path refs old ~/.vr1-netbox.env -> ~/vr1-office1-creds/vr1-netbox.env in
prod-draft-dump.py, the phase1 runbook Step 10b, as-built, and the ledger. Dated
changelog/incident records left as historical snapshots.
- as-built secrets table: the three env files + the convention. CORRECTED a
mid-session wrong assumption -- the SANDBOX token is NOT on vcloud; it is on
office1-netbox (/root/netbox-secrets/). vr1-netbox.env is the UPSTREAM token only.
- security-ledger SEC-009 (REMEDIATED) + the standing per-site consolidation convention.
- seeded agent memory (was EMPTY): the creds convention + the closed-test posture,
so neither is re-lost at compaction.
STANDING CONVENTION: per-site ~/<site>-creds/ (0700; files 0600); no loose env in ~;
~/vr1-dc0-creds/ and ~/vr1-dc1-creds/ follow the pattern from day one. Data-loss /
continuity control first; real cred hardening deferred to post-teardown (closed test).
GAUNTLET ALL GREEN (58). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-14 |
C4 DONE: document the NetBox sandbox loop; add the upstream-write guard it exposed
...
C4 (Stage 2 close-out) -- the sandbox loop is now documented in two places, no
rationale duplicated: docs/dc-dc-netbox-buildout-scope.md section 8 (design: why a
sandbox, the fidelity check + its 2026-07-14 hardening, the write-guard table, the
WAF trap, C2 preconditions) and runbooks/dc-dc-phase1-office1-standup.md Step 10b
(the dump->seed->apply->verify->gated-write sequence, two-host/two-token separation).
Drafted by a subagent, corrected against this session's own fixes before placement.
The guard C4 exposed: roles-aggregates-import.py and dc-dc-prefixes-import.py had NO
hostname gate -- until today the WAF 403 was the only thing stopping an accidental
--commit to the apex, and the UA fix removed that accidental safety. Both now gate a
non-sandbox --commit behind --yes-write-upstream (SANDBOX_HOSTS allowlist), matching
d115-office-carve.py -- so all three write-capable importers behave identically. This
is defence-in-depth with the whole-plan preflights (which prevent a half-write).
Tests: prefixes-import 86->91 (refused without the flag + writes nothing; allowed
with it; sandbox host needs no flag). roles-aggregates 20->24. GAUNTLET ALL GREEN (58).
Stage 2: C1/C3/C4 DONE. C2 (production write) is the only item left -- gated, with
preconditions now documented (re-verify sandbox under the hardened check, run from
office1-netbox, --yes-write-upstream, approve each mutation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119 OpenTofu rename APPLIED -- repo/state back in sync (operator-gated)
...
tofu apply: 11 added, 0 changed, 11 destroyed. tofu plan now reports "No changes".
Pre-apply LIVE re-check confirmed all 11 objects empty (0 leases, 0 volumes, no
domain attached) -- verify-before-mutate, not the earlier measurement. Fresh plan
was 11 add / 11 destroy, all "must be replaced" via the moved{} blocks.
Verified on the host: six vr1-dc0-* networks, three mesh-vr1-* links, vr1-dc0-pool
/vr1-dc1-pool; ZERO old dc1-*/dc2-*/mesh-dc* names remain. Office1 (office1-local/
office1-wan/wan/edge/voffice1) was NOT in the plan and is untouched -- edge and
voffice1 both reachable post-apply.
Applied now, while the objects are empty, so Stage 3's own tofu apply does not
silently bundle this 11-resource rename with real DC provisioning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Add the missing preflight-failure test; reconcile the stale tofu-sync ledger line
...
Two follow-ups a review caught:
1. Write-path fix #3 (whole-plan role preflight in dc-dc-prefixes-import.py) shipped
with NO test exercising it FAILING. Every existing test preseeds all roles, so
they ran the preflight only when it passes -- silently. A future edit could delete
the preflight and the harness would stay green: exactly the "green while the bug is
armed" failure this session has been killing. Added a test that runs main() with one
six-plane role missing and asserts nonzero exit AND that NOTHING was written (no
site, no prefixes). prefixes-import 82 -> 85.
2. The ledger's "Live state" bullet still claimed OpenTofu repo/state were IN SYNC
("tofu plan reports No changes"). False since D-119 -- plan is 11 to add / 11 to
destroy. Reconciled to state the gated destroying apply, contradicting nothing.
GAUNTLET ALL GREEN (58). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger: write-path bugs 1-3 fixed; flag the sandbox re-verify owed before C2
...
The sandbox was declared "proven faithful" using the fidelity check BEFORE it was
hardened -- i.e. under the version that could false-green. That verdict must be
re-established under the hardened check before C2 feeds the carve upstream. Needs
the operator-held sandbox token; folds into the C2 step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 LAN IPv6 is LIVE and proven on the wire -- Stage 2 close-out C1 CLOSED
...
The last build item on Office1. Executed against the live edge, verified against
the kernel and the wire, not by {"result":"saved"}.
scripts/opnsense-set-interface-v6.sh --commit 10.10.0.1 lan 2602:f3e2:f01 :1 64
POST /api/radvd/settings/addEntry {"entries":{...,"mode":"unmanaged"}}
POST /api/radvd/service/reconfigure
PROVEN: address on the kernel (ifconfig, not the config file); radvd advertising
2602:f3e2:f01:/64 AdvAutonomous/AdvOnLink with no M/O flags; voffice1
autoconfigured ...:5054:ff:fe6a:87e5 by SLAAC + an RA-learned route; edge->voffice1
GUA ping 0.0% loss. The neighbour MAC matches voffice1's known Kea lease hwaddr.
No collateral damage: v4 10.10.0.1 intact, Kea DHCPv4 still serving udp/67, the
compose-net route still good.
mode=unmanaged is DELIBERATELY not the model's default. `stateless` sets the RA
Other-config flag, pointing clients at a DHCPv6 server -- and Kea DHCPv6 is
enabled=0 with no daemon running. The default would have advertised a server that
does not exist. Field names, the request key (addBase('entries','entries')), the
mode values and the apply endpoint were all READ OFF THE EDGE, not recalled.
OPERATOR RULING -- the v6 NAT was NOT built. A v4-to-v6 NAT (a second OPNsense, or
nginx) was proposed to simulate the v6 the lab will not forward. Withdrawn: it
contradicts D-101 ("native GUA routing, no NAT66"), D-115's amendment says the
missing egress is the upstream's problem and not ours, and a translation box is
precisely the component Roosevelt will NOT have -- maximum delta for zero proof.
nginx is an L7 proxy, not NAT, and proves nothing about RA/ICMPv6/Ceph-over-v6.
Ruling: configure IPv6 internally now; configure the IPv6 edge AFTER deployment.
TRAP recorded: voffice1 has forwarding=1 + accept_ra=0 and STILL autoconfigured --
netplan/systemd-networkd processes RA in USERSPACE, independent of the sysctl.
WAN v6 leg stays DEFERRED: re-measured today, v6 internet is 100% loss from vcloud
while v4 is fine. Internet GUA reachability remains UNPROVABLE in VR1.
repo-lint 0 fail; opnsense-set-interface-v6 harness 40/40.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Consolidate to one session; close last night's two delivery gaps
...
Work had fragmented across FOUR concurrent Claude sessions on this one clone,
including two --permission-mode auto background agents (idle 14-15h, still
holding auto-approve rights against the live cloud) and a second interactive
session that got the same "resume last night's work" prompt six minutes before
this one and started bootstrapping in parallel. All terminated; the supervising
daemon stopped (it had respawned the agents once -- killing the child is not
enough). Working tree was clean and 3-ahead throughout; nothing lost.
Last night's three commits were made by the auto-permission agent and left
UNPUSHED and UNDOCUMENTED. Both closed:
- Pushed 6ed6b9a..522ba7d (the D-113 amendment + the v6 script existed only on
local disk; a jumphost rebuild would have lost them).
- changelog-20260714-opnsense-set-interface-v6.md -- the delivery rule wants
harness + lint + changelog-with-revert; the script shipped with the first two.
Records that it is WRITTEN AND TESTED but NEVER RUN with --commit (C1 is open).
Ledger:
- Machine block re-seeded: next-free D was STALE at 114 (actually 119; D-114..118
are all ADOPTED), and SEC-008 was missing.
- Logged a ledger-scan DEFECT: it reports D-115 as PROPOSED/OPEN, but D-115 is
ADOPTED -- the scan matches "PROPOSED/OPEN" in body prose instead of reading the
Status line. The status twin of the decoy-token rule. Logged, not fixed.
- CORRECTED pinned item 1a, which still instructed the REST API for the LAN IPv6.
That path does not exist (D-113 amendment, measured). It would have sent the
next session at a dead end.
repo-lint 0 fail; opnsense-set-interface-v6 harness 40/40.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Pin Stage 2 as the branch merge point; reconcile the Stage 2 tracker; log the OpenTofu naming collision (gap #19)
...
Operator ruling 2026-07-14: branch dc-dc-stage1-phase0-first-exec merges to
main at STAGE 2 CLOSE, not at session end. Recorded in the ledger's PINNED
section (a ruling, not an open question) and in the workflow doc's Stage 2
section, which now carries the C1..C5 close-out checklist that defines what
"closed" means. This is NEW policy: main has never taken a feature merge, so
all 52 commits of VR1 work live only on the branch.
The Stage 2 status block was STALE -- it still listed MAAS-region, LXD, the
LXD-KVM-host registration and both composed service machines as NOT DONE. All
of that landed 2026-07-13, and the route enable + the D-115 carve import have
landed since. Corrected against measured state; the doc has now drifted three
times and says so. Stage 2 is in CLOSE-OUT, not build.
GAP #19 (new, BLOCKING Stage 3): "dc1" now names two different datacenters.
D-117 moved the repo to the apex's 0-indexed convention on the IMPORT PATH
ONLY. opentofu/main.tf still says dc1_planes/dc2_storage for VR1's FIRST and
SECOND DCs, and lib-net.sh's dc0 means VR0's DC0 -- a different region, and a
RUNNING cloud. So the next DC we deploy is simultaneously apex vr1-dc0,
OpenTofu dc1_*, and a $DC value that collides with the live testcloud. A
mechanical sed would hand VR1's first DC VR0's deployed literals. Nothing in
opentofu/ gains a new DC-keyed module until this is closed.
Docs only -- no script, no tooling, no cloud state touched.
repo-lint: 0 fail (1 documented legacy warn).
Revert: git revert HEAD
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-118 ADOPTED (org ULA fd50:840e:74e2::/48); VR1 DC0 + DC1 IMPORTED; roles importer bug fixed
...
D-118: ORG_ULA_48 = fd50:840e:74e2::/48. RFC 4193 valid (fd -> L=1, 40-bit Global
ID 0x50840e74e2, /48 -> 256 x /56 for D-101's per-DC /56). The load-bearing check
is NOT the RFC conformance -- it is that it does NOT collide with Tailscale's
fd7a:115c:a1e0::/48: office1-tailscale runs --accept-routes, so an overlap would
have been a LIVE tailnet routing conflict, not a paper one.
Provenance stated honestly: the 'CSPRNG-generated' claim cannot be re-verified.
It does not change the ruling (RFC 4193's SHA1 construction is a SHOULD; the
requirement is a pseudo-random 40-bit Global ID). Regenerating would only swap in
a different random number at the cost of re-editing 7 places.
Corrects a FALSE claim: changelog-20260711 called this value 'ratified' when no
D-number ever assigned it. That phantom ratification is why it read as settled for
two days. Same failure mode as D-117: a document asserting authority it lacked.
IMPORTED into the sandbox: six-plane roles + 2 RIRs + 5 aggregates, then VR1 DC0
(18 prefixes) and VR1 DC1 (18). The D-117 fix is now LIVE-PROVEN, not just green
offline: --dc dc0 -> apex site VR1 DC0 (id=8), --dc dc1 -> VR1 DC1 (id=9). ULA
reads DC.NN in the 4th hextet per D-111 (fd50:840e:74e2:220:: .. :350::).
A REAL BUG, found by running roles-aggregates-import.py for the first time for
real: it checked existence by SLUG only, but NetBox also enforces UNIQUE NAMES.
The draft's legacy role 'Replication' (slug repl) blocks the six-plane role (slug
replication, name Replication) -- so it created 4 roles, 400'd on the 5th, and
DIED, leaving the apex HALF-POPULATED with no RIRs, no aggregates, no rollback.
It had never met a NetBox holding the real draft (only empty ones) and it SHIPPED
WITH NO HARNESS. That is why it survived.
- six-plane role renamed 'Replication Plane' (slug unchanged -- lib-net.sh
SPACES6 and the prefix importer look up 'replication')
- the script now PREFLIGHTS every name/slug and writes NOTHING unless the whole
plan is viable. An IPAM importer that can die partway corrupts what it
populates.
- tests/roles-aggregates-import/ CREATED: 16/16 PASS.
Final state PROVEN by sandbox-fidelity-check (now carrying the full planned
delta): the sandbox is the upstream draft + EXACTLY D-115 + D-117 + D-118 --
every shared object field-identical, nothing lost, nothing stray. 90 -> 134
prefixes.
repo-lint 0 fail. run-tests-all: ALL GREEN (56 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
Prove the sandbox is a FAITHFUL replica, not just a count-matching one
...
netbox/sandbox-fidelity-check.py (+4 assertions in tests/sandbox-seed, 22/22 PASS)
Counts and idempotency CANNOT prove a faithful seed. The region-scope bug proved
it: 17 of 90 prefixes silently lost their scope, every count still matched, and
re-runs were still perfectly idempotent. It was caught by LUCK on a spot-check
that happened to print 'site None'. This is that catch, made deliberate.
Method needs no new plumbing: prod-draft-dump.py is READ-ONLY, so point it at the
sandbox and diff the two JSON dumps FIELD BY FIELD.
RESULT: every shared object is field-level IDENTICAL to upstream, and the delta is
EXACTLY the expected D-115 additions (+edge role, +vr1-off1 site, +8 prefixes).
Nothing missing, nothing unexpected. The sandbox is a PROVEN baseline -- which is
what the later feed-back diff to production depends on.
Also de-misdirects docs/dc-dc-netbox-buildout-scope.md, which still told the next
person to run '--dc dc2' with DC2_V4_SUPERNET -- both of which the new guards now
REJECT by name. Its G5 recommendation (rename the APEX to vr1-dc1/vr1-dc2) is
marked SUPERSEDED: D-117 ruled the opposite. ORG_ULA_48 is now listed there as
UNASSIGNED and BLOCKING, not as a settled literal.
Disclosure: D-117 Option B is PARTIALLY executed -- the import path and the scope
doc are done; the $DC shell selector and the runbook DC prose are NOT. Recorded
in the ledger so 'D-117 ADOPTED' is not misread as 'fully renamed'.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-115 office carve IMPORTED into the Office1 NetBox; DC import blocked on ORG_ULA_48
...
netbox/d115-office-carve.py (+ tests/d115-office-carve/ 20/20 PASS)
Applied to the Office1 sandbox: site vr1-off1, roles office + edge (edge is NEW),
and 8 prefixes. Idempotent (re-run: 0 created / 11 present).
10.10.0.0/16 office container the Office v4 block, /22 per office
10.10.0.0/22 office container VR1 Off1's /22
10.10.0.0/24 office active office1-local LAN (Kea on the edge)
10.10.1.0/24 office active lxdbr0 compose net (MAAS DHCP)
172.30.0.0/16 edge container NEW Edge role; mirrors v6 2602:f3e2:fe::/48
172.30.1.0/24 edge active office1-wan, the simulated ISP uplink
2602:f3e2:f01:/56 office container mirrors VR0 Off0's e01:/56
2602:f3e2:f01:/64 office active office subnet (NOT deployed; v6 has no egress)
Every address was already RUNNING. D-115 blesses what is deployed -- zero
renumber. What it fixes is that three of them were SQUAT: on the wire with no
allocation behind them.
Writing upstream is GATED IN CODE: the tool refuses a non-sandbox target unless
given --yes-write-upstream. Feeding the production apex is an operator decision,
never a side effect of running an import.
BLOCKED, deliberately: dc-dc-prefixes-import.py --dc dc0 CANNOT run. It needs
ORG_ULA_48, which is still UNASSIGNED (gap G3; the scope doc recommends
fd50:840e:74e2::/48 but no ADOPTED decision assigns it -- grep ORG_ULA
design-decisions.md returns nothing). Using it anyway would be an inferred value
(hard rule 2). This is a DECISION gate, not a tooling gap: the tooling is fixed,
tested and ready. Recorded in the ledger with the exact commands to unblock.
Also: docs/vr1-office1-as-built.md gains the IPAM table, and records that the
Tailscale 10.10.0.0/22 route is now ENABLED (edge GUI reachable from the
workstation with no tunnel), superseding the tunnel instructions.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
| 2026-07-13 |
D-116 ADOPTED: no Office1-local GitBucket -- keep using git.baldurkeep.com
...
Operator ruling: "we've decided not to build a separate one on the test cloud. We will continue to
use the current setup and create new repos for deployment tasks as needed."
SUPERSEDES the GitBucket half of D-103 (which gave Office1 three service VMs) and removes GitBucket
from Stage 2's build list entirely. Office1's MAAS-composed service machines are now exactly TWO --
office1-netbox and office1-tailscale -- and both are LIVE.
WHAT THIS ACTUALLY CLOSES, and why it is a simplification rather than a deferral: the Stage 2 runbook
carried an OPEN QUESTION about the Office1-local GitBucket -- "what it mirrors, and under what repo
path, is not decided" -- because a SECOND GitBucket alongside the pre-existing git.baldurkeep.com
raised a real authority/mirroring question nobody had answered. That question is now MOOT. There is
one git service, it is the existing one, and it is authoritative.
Recorded explicitly in D-116 so nobody misreads the scope: D-107's per-DC ARTIFACT mirror (OS/package
artifacts for node provisioning) is a SEPARATE concern and is NOT affected. And if Roosevelt later
wants a site-local git service, that is a fresh decision against real requirements -- not something
this ruling forecloses.
Applied everywhere it was an instruction, not just recorded:
- runbooks/dc-dc-phase1-office1-standup.md: Step 11 replaced with a TOMBSTONE (kept rather than
deleted -- step numbering is referenced elsewhere, and someone WILL come looking after reading
D-103); removed from the sequence, the compose steps, and the open-questions list.
- docs/dc-dc-deployment-workflow.md: dropped from Stage 2's Build and, importantly, from its GATE --
"GitBucket serving" is no longer a gate condition.
- docs/design-decisions.md: D-103 annotated in place with the supersession, so grepping for GitBucket
lands on the amendment rather than the stale instruction.
- docs/vr1-office1-as-built.md + session ledger updated.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger: pin Office1 LAN IPv6 as the next step after the NetBox deploy (D-115 q3)
...
Operator sequenced it: NetBox first, then dual-stack. Pinned with its known limit attached so the
next session cannot mistake internal v6 for working v6 -- IPv6 does not egress the lab (measured),
so the LAN leg proves addressing/RA/services-binding and nothing about internet GUA reachability.
The WAN v6 leg is DEFERRED: it would be a path to nowhere and untestable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END.
...
MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.
vcloud (L1 -- itself a KVM guest, measured)
+- voffice1 (L2) MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
+- office1-netbox (L3) LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected
THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.
THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
10.10.0.0/24 (site LAN) -> Kea on the OPNsense edge MAAS dhcp_on=False
10.10.1.0/24 (lxdbr0) -> MAAS MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.
NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.
FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
be picked by accident.
DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.
10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.
Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.
Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-114: voffice1 is LIVE. cloudinit-vm gains nested virt -- it was silently broken for the model
...
APPLIED, operator-approved: 5 added, 0 changed, 0 destroyed. The running OPNsense edge was NOT
touched (confirmed by plan BEFORE apply -- no in-place update, so no repeat of the 2026-07-13
guest bounce).
THE MODULE WAS SILENTLY BROKEN FOR D-114'S ENTIRE MODEL. modules/cloudinit-vm's libvirt_domain
had NO `cpu` block. With none, libvirt renders a GENERIC EMULATED CPU exposing no `svm` -- and
LXD *virtual machines* are qemu/KVM guests, so NO service VM could ever have been composed into
voffice1. D-114 would have failed at its first step with a confusing "KVM not available" that
pointed nowhere near OpenTofu. Measured proof of the masking: host is an AMD EPYC 9965; a
default-CPU guest was handed an "Opteron_G3".
Fix: `cpu { mode = "host-passthrough" }` always, plus a new `expose_nested_virt` variable with NO
default -- a real per-VM decision. true = pass `svm` through (required for D-114's containment
VMs); false = disable it, matching modules/opnsense-edge. The operator asked whether the edge's
`svm` disable was a leftover of the failed OPNsense deploy: it is NOT. It was tried as a
triple-fault fix on 2026-07-12, did NOT resolve it (the real cause was the memory unit), and was
retained on hardening grounds. A router has no business seeing nested virt. It stays disabled there.
voffice1 as built (MEASURED): Ubuntu 24.04.4 LTS, 16 vCPU / 32 GiB / 600 GiB, single NIC on
office1-local, reports the real EPYC CPU with svm on all 16 cores and /dev/kvm PRESENT, egress to
1.1.1.1 at 4.6ms THROUGH the edge. Cloud-init deliberately MINIMAL (identity + key + guest agent);
MAAS and LXD go in as separate GATED steps rather than buried in a first-boot script that either
silently works or silently does not. The SSH pubkey is read via file(var...) at plan time -- key
material never enters a command line, the repo, or an agent's context; the tfvars holding the path
stays gitignored. network_config matches the NIC by GLOB, not a guessed kernel name (an inferred
value); it came up enp1s0.
TWO GATES CLEARED:
1. KEA HAS SERVED ITS FIRST REAL DHCP LEASE. Standing open item: the daemon was proven, the
SERVICE never was (office1-local had no client). voffice1 -- the first client ever on that LAN
-- took 10.10.0.100, hwaddr 52:54:00:6a:87:e5 (matches its NIC), hostname voffice1, state
active, read back through the D-113(a2) REST API. DHCP now works end to end.
2. D-114's NESTING PROBE PASSES AT L3. Stated honestly: this proves nested KVM is AVAILABLE. The
DEFINITIVE proof is a guest actually booting inside voffice1 -- that lands with the first
composed LXD VM. Not recording L3 as fully proven until then.
FINDING, logged NOT actioned (DC1 is gated behind Office1 per D-114): modules/node-vm has the SAME
missing-cpu-block defect. DC nodes run nova-compute, which needs working KVM -- without the fix
they will come up unable to run a single instance. MUST be fixed before Stage 3.
opentofu-validate PASS, repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Fold the AppArmor/libvirt rule into scripts/prereqs -- it gated every VR1 VM boot
...
`/var/lib/libvirt/vr1/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu had
been applied BY HAND on vcloud and existed ONLY as host state. Nothing in the repo
carried it, so a rebuild or a second host walked straight back into the same wall.
Why it matters: libvirt's stock abstractions/libvirt-qemu grants qemu only the DEFAULT
pool path (/var/lib/libvirt/images). VR1 uses a custom pool parent, which is not in that
abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The domain
DEFINES fine, then fails to start with a bare qemu "Permission denied" -- and NOTHING in
the libvirt error names AppArmor. It cost a session on 2026-07-12 (DOCFIX-186), was
logged as a PREREQ candidate, and was never actioned until the ledger collapse resurfaced it.
New: scripts/prereqs/install-apparmor-libvirt.sh, wired into install-all.sh (after
install-virtualization -- it edits libvirt's own profile dir) and reported by
check-prereqs.sh.
- Idempotent and deliberately QUIET on a good host: if the rule is present it exits 0
having touched nothing -- it does NOT reload apparmor or bounce libvirtd. Those happen
only on the run that actually adds the rule.
- Writes to local/, the vendor-sanctioned override include, so it survives package upgrades.
- Pool parent is a variable (VR1_POOL_PARENT, default /var/lib/libvirt/vr1), not a literal.
- Non-AppArmor hosts report N/A and SUCCEED -- no manufactured failure on a host it
does not apply to.
Verification: tests/prereqs 32/32 PASS (was 24/24) -- the new cases drive DETECTION off a
pool parent guaranteed absent from the profile, so the negative path is deterministic on
any host: a false OK from --check IS the bug, --dry-run must plan and mutate nothing, and
the planned rule must be exactly `<parent>/** rwk,`. Full gauntlet ALL GREEN (52 harnesses),
repo-lint 0 fail. Run live on vcloud: --check reports OK and leaves the profile byte-identical,
confirming the already-satisfied path is a true no-op.
No host state was mutated by this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Ledger collapse: close the coverage gap in the carry-forward
...
The collapse commit's guards (lint, ASCII, ledger-scan, size) validated FORM, not
COVERAGE. Two ledger regions had only been keyword-grepped, not read. Read them.
Three items surfaced; each was VERIFIED against the repo rather than trusted:
- run-tests-all.sh dirtying the tree with 37 spurious mode-change diffs: NOW FIXED
(the fakebin stubs are committed 100755 -- 36 of them, measured). Not carried.
- `scripts/opnsense-apply-config.sh` (once "wanted"): SUPERSEDED, and dangerous to
build -- its snapshot/scp/install/reboot flow IS the config.xml path D-113(a2)
deleted. Recorded as "do not build this", so nobody resurrects it from the history.
- The tcsh trap it taught is REAL and carried forward: root's shell on the OPNsense
edge is tcsh, so `$(...)` in a quoted remote command dies with "Illegal variable
name" -- QUIETLY. It already cost a silently no-op'd pre-install snapshot. Always
feed remote commands to `sh -s`.
Also carried (this one matters and was nowhere in the open lists): DHCP HAS NEVER
SERVED A REAL LEASE. kea-dhcp4 is bound to udp/67 with the subnet+pool intact, but
office1-local has no client host, so the DAEMON is proven and the SERVICE is not.
The first VM on office1-local (the Stage 2 headend VMs) is the real test -- logged
as that stage's gate rather than an assumption.
Plus a measured doc-accuracy finding: SEC-007 says "D-112(c) makes SSH the only
management path", but the edge's web GUI IS listening on the LAN (10.10.0.1:443,
HTTPS 200) and is correctly firewalled off on the WAN. Not an exposure -- OPNsense
default, and D-113 kept OPNsense *because* the GUI is an operator requirement -- but
the wording is imprecise. Logged, not actioned (hard rule 1).
Guards: repo-lint 0 fail (1 documented legacy warn), 0 non-ASCII. No live system
touched; the edge probes were read-only TCP/HTTP reachability checks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
