| 2026-07-20 |
NEW: maas-node-power.sh (+harness 24/24) -- per-machine virsh power, MAC-matched
...
Operator-ruled 2026-07-20 over the pod route. Sets power_type=virsh +
power_address/power_id on already-enlisted machines, matching domains BY MAC
(MAAS renames machines at enlistment, so name matching would map nothing or
the wrong node). Dry by default; every write verified by an actual
query-power-state, since a stored parameter proves nothing about reachability.
Carries the measured reasoning: pods fail on node-vm's pool+volume disks
(domblkinfo 'missing storage backend', reproduced locally), the pod's
discovery role was already done by PXE, and per-node power is the Roosevelt
(IPMI) shape. Gauntlet 72 harnesses ALL GREEN.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
FIX false-success: opnsense-plugins.sh apply ALWAYS dry-ran (DRY_RUN:+ expands on 0)
...
Found on the first-ever live run (vr1-dc0 edge, 2026-07-20): ${DRY_RUN:+--dry-run}
expands whenever the var is set and NON-EMPTY, and DRY_RUN=0 is non-empty -- so
every apply silently dry-ran while printing 'OK: profile applied'. Nothing was
ever installed by this script. The harness missed it because every test drove
--dry-run, leaving the live path with zero coverage. Fix tests the VALUE; harness
gains two live-path tests via a stub api client (verified to FAIL 2/2 against the
buggy version before restoring). 20/20.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
NEW: opnsense-set-interface-v4 pair (+harness 53/53) -- IPv4 + WAN gateway via the vendor config model
...
Operator-ruled mechanism 2026-07-20 ('Build the v4 setter pair'). Mirrors
the proven v6 sibling: vendor Config singleton (never a config.xml push),
dry-by-default, interface-count guard, fresh read-back verify, sh -s for
the tcsh trap, kernel ground truth. Adds the gateway half (at most ONE
gateway_item, count-guarded) and MEASURES the interface's device from the
edge's own config instead of hardcoding lan->vtnet0 like the v6 sibling.
D-113 amendment RE-MEASURED on 26.7: interfaces/settings/get carries only
offload flags; interfaces/overview/list and interfaces/<if>/get both 404 --
so the amendment holds on two majors. Gauntlet 71 harnesses ALL GREEN.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
Deploy step B COMPLETE: rack bootstrapped + enrolled, SEC-010 CLOSED both ends (operator-ruled), OPNsense 26.7 base staged (operator-ruled); ledger-scan fail-open hardening
...
Bootstrap runs 1+2 (stale-lists 404 fixed by proxied apt update):
maas 3.7/stable, rack enrolled, libvirt+nested KVM, SEC-010 loaded
+ persistent on enp1s0; final --check EXIT 0 ALL items
(docs/audit/stepB-check-20260720-final.txt). voffice1-side SEC-010
pin applied per the row's follow-on clause. 26.7 nano prepped on the
rack (measured leaseweb mirror via region proxy); inner-root base
default -> 26.7. ledger-scan: FAIL-OPEN no longer counts as OPEN
(harness 45/45). CURRENT-STATE position/G10/G14 + section-5 updated
in this commit (GA-R1/C1). Open SEC count 8 -> 7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr
|
| 2026-07-19 |
Batch 6: exit runs 1/2/4/5 -- archive exclusion (operator-ruled), adjudication, plan==triple, S2
...
6.1: docs/archive/ joins docs/audit/ as inventoried-not-counted (ruling
quoted in the exit record); record-audit + harness 35/35; per-group
adjudication = ZERO contradictory status-claim pairs on counted surfaces
(the charter's criterion as stated); group floor 9 recorded as re-run
baseline; missed patch file archived; CURRENT-STATE stale-note made
token-free. 6.2: 16 top-level docs (<25). 6.4: fresh capture
outer-plan-20260719-phase6-exit.txt = 6/0/6 == section-5 triple. 6.5:
G9 row names SEC-010's step-B dependency (S2). Exit record:
docs/audit/phase6-exit-runs-20260719.md. Items 3+6 pend operator
execution. G3 cell updated same commit.
Revert: git revert this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
Batch 4 C6: docs disposition -- 24 history docs archived, 16 retained, refs rewritten
...
DISPOSITION MANIFEST (archive, one-line justifications): clientdocs-
workflow-review (v1 tenant-docs era) / D-057-DECIDED-append +
D-057-REVIEW-ITEMS + README-D057-PACK + D-058-renumber (executed v1
decision packs) / D-068-openbao-assessment-DRAFT (supplementary; the
cited analysis retained) / DOCFIX-064-phase08-changelist + docfix-draft-
20260702 (executed/draft) / handoff-20260703 + handoff-20260705
(superseded handoffs) / incident-20260712-triplefault (trap routed to
platform-traps 1b/1c long ago) / netbox-vip-queue + phase-00-maas-
standup-notes + script-quality-findings + session-findings-2026-07-02 +
tenant-cidr-overlap-PLAN + v1-pre-deploy-fixes (v1-era executed/dated) /
repo-lint-nextfree-bug-FINDING (absorbed DOCFIX-105/107; repo_lint.py
pointer updated) / upstream-bug-draft-dashboard-tls (draft, dated) /
model-a-fallback-plan (D-123 Model B adopted; fallback preserved in
archive) / stage3-adversarial-review-20260716 (R3-F register,
dispositioned in 2.10) / dc-dc-replication-DR-seed (absorbed per
workflow) / dc-dc-ipv6-charm-research + dc-dc-netem-and-ula-gua-proposal
(stage 4+ research, refs updated). RETAINED 16 top-level, justified in
the session changelog. Live-surface refs to all moved files rewritten to
archive paths (18 files). G3 cell + ledger + changelog updated same
commit (the L10 fire on a Status-line path rewrite is satisfied by the
CURRENT-STATE touch -- the check worked as designed).
Revert: git revert this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
Batch 1 open: L10 status-coupling flipped WARN -> FAIL (GA-R1/C1)
...
One-constant flip (L10_FAIL = True) per the sweep plan's Batch 0 revert
note ("WARN for one batch before becoming FAIL"). Harness rc
expectations updated for the five firing cases; 42/42 PASS. G3 cell
updated same commit: Batch 0 CLOSED, Batch 1 OPENED by operator.
Revert: git revert this commit (restores WARN severity).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
Batch 0.4: extractor status scan keyed to the legal token set (GA-F10 / vocab H1)
...
record-audit.py now knows the ratified Option A vocabulary: legal set
OPEN/BLOCKED/HELD/CLOSED (OPEN was previously unscanned -- a post-
migration blind spot where OPEN-vs-CLOSED stage contradictions would
have gone undetected); DONE/TO RUN/RUNNING added to the token scan.
New report section 6 classifies every distinct token (legal/illegal/
event) and flags illegal tokens plus event words (VOID/REOPENED/
SUPERSEDED) minted in status position (**Status: line or table row);
prose event words stay legitimate row-history vocabulary per H1.
Report-only: the 0/1/2 exit contract stays contradiction-driven, so
Batch 6.1 semantics are unchanged. Live sanity run (scratch --out, the
Phase-1 capture untouched): 12 distinct tokens, 163 flagged counted
occurrences -- Batch 2.5's migration work-list. G3 cell updated same
commit (GA-R1 rule 8).
Harness: tests/record-audit +9 checks, 33/33 PASS; repo-lint 0 fail.
Revert: git revert this commit (restores the 8-token scan and the
section-5-only report).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
Batch 0.3: ledger-scan wrap-aware next-free exclusion (GA-F15)
...
The per-line "next-free"/"reproduc*" exclusion went blind when a quoted
next-free value wrapped its hyphenated token onto the following line
(happened twice live inside the 2026-07-18 audit -- CURRENT-STATE
section 9). An excluded line now suppresses itself AND the immediately
following line; the window is exactly one line and resets at every file
boundary. Chose this over assignment-site counting: DOCFIX/BUNDLEFIX
have no single register file to key assignment sites on (unlike D
headers). Live counters re-verified unchanged post-fix: D=130,
DOCFIX=197, BUNDLEFIX=052. CURRENT-STATE section 9 marked FIXED + G3
progress cell updated same commit (GA-R1 rule 8).
Harness: tests/ledger-scan +3 checks (wrapped quote excluded, wrapped
repro example excluded, two-lines-below still counts), 44/44 PASS;
repo-lint 0 fail.
Revert: git revert this commit (restores the per-line grep exclusion
and the section-9 "recorded, NOT fixed" wording).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
Batch 0.2: ledger-scan SEC reporting repointed at security-ledger.md (GA-R4/F3)
...
Scan output now emits open COUNT + ID list + register-of-record pointer;
row-level dispositions dropped (they live in docs/security-ledger.md
ONLY). Session ledger's machine-derived SEC bullet re-seeded to pointer
+ count form. G14 evidence cell repointed at security-ledger.md same
commit, and its count corrected 9 -> 8 per GA-R1/C2 (measurement wins:
the cell's own enumeration and the scan both measure 8; divergence noted
in the cell, not silently harmonized).
Harness: tests/ledger-scan +4 checks (count+IDs surfaced, pointer
present, disposition text absent), 41/41 PASS; repo-lint 0 fail.
Revert: git revert this commit (restores per-row scan output, prior
ledger bullet, and the 2026-07-18 G14 cell incl. its count error).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
Batch 0.1: repo-lint gains L10 status-coupling check (GA-R1 rule 8 / C1)
...
L10: a change set modifying state-bearing surfaces (docs/audit/ incl.
ga-rulings.md; design-decisions.md **Status: lines; session-ledger.md
gate rows) without touching docs/CURRENT-STATE.md is flagged -- pending
working-tree set when dirty, HEAD commit when clean. WARN this batch
(L10_FAIL=False); flips to FAIL at Batch 1 open per the sweep plan.
Divergence noted: the sweep plan's paraphrase listed "CURRENT-STATE gate
rows" (vacuous -- such an edit touches CURRENT-STATE by construction);
implemented per the ratified C1 utterance, "ledger gate rows"
(docs/audit/ga-rulings.md:129). G3 evidence cell updated same commit
per the rule this check enforces.
Harness: tests/repo-lint T34-T41 added, 42/42 PASS; live lint 0 fail,
1 legacy WARN (unchanged).
Revert: git revert this commit (check + tests self-contained; restores
the prior G3 cell).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
| 2026-07-18 |
GA Phase 1: deterministic record inventory + measured evidence (freeze upheld)
...
- scripts/record-audit.py + tests/record-audit/ (24/24; gauntlet 69 GREEN):
doc census, status-claim extractor, reference integrity. Exit 0 == zero
contradiction groups (the Phase-6 re-run criterion).
- docs/audit/record-inventory.md: 179 files, 858 claim rows, 12 contradiction
groups (plan counts, version pins, stage statuses), reference-rot counts.
- docs/audit/outer-plan-20260718.txt: captured outer plan -- 7 add / 2 change
/ 7 destroy; voffice1 seed volume forces replacement. THE citable
plan-count source from here forward.
- docs/audit/env-snapshot-20260718.md: measured host/guest/toolchain state
(vcloud, -136 bookend done, autostart enable, tofu 1.12.4, MAAS 3.7.2,
LXD 5.21.5).
- docs/audit/grounding-audit-20260718.md: findings register GA-F01..GA-F14
incl. the operator provenance rulings on the two memory files and the
07-16 06:41 churn-mechanism correlation.
No fixes applied; no existing file touched. Findings only, per the charter
freeze. Phase 2 (CURRENT-STATE.md rebuild) starts from this evidence base.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ZV3uqyU9oWoFTcwiE4a21
|
D-129: build the OPNsense edge plugin profile-installer + sharpen pins
...
Autonomous processing of D-129 items that need no operator keyboard. Repo-only;
NOTHING applied to the cloud (live plugin install stays classifier-gated).
- NEW scripts/opnsense-plugins.sh + tests/opnsense-plugins/ (17/17): named-profile
installer over the D-113 firmware API. Installs ONE plugin at a time + polls
upgradestatus (the firmware-lock lesson), delegates auth to opnsense-api.sh,
warns os-qemu-guest-agent != a working agent, offline --dry-run. Closes the
"build the reusable installer" pin.
- D-129 + dc-dc-phase2 pin: installer recorded as BUILT (use it, not build it);
qga channel logged as an OPT-IN-var design (default OFF -> no office1/DC0-plan
side-effect; dmacvicar 0.9.8 mechanism-to-verify); os-git-backup operator ask.
- Gauntlet ALL GREEN (68), repo-lint 0-fail.
Revert: git rm the script+harness; revert the doc/runbook edits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JUBUyinU4sBAT4HV1b9vSz
|
| 2026-07-17 |
D-126 base-leg oneshot (site-baseleg.sh) + D-128 operating model; post-restart sweep
...
vcloud post-restart health sweep all-green (host on 6.8.0-136, patch+reboot done;
libvirt/pools/autostart clean; Office1 stack self-recovered end-to-end). Sweep
surfaced a recurring break: office1-local is an ISOLATED libvirt net, so the host's
base L3 leg (10.10.0.10/24 on the bridge + compose route 10.10.1.0/24 via 10.10.0.20)
lives nowhere in the XML and drops on every reboot -- the un-persisted foundation
D-126 assumed but never owned.
- NEW scripts/site-baseleg.sh + tests/site-baseleg/ (24/24): root Type=oneshot unit,
After=libvirtd (+virtnetworkd for modular hosts; ordering MEASURED here = monolithic
libvirtd), re-adds the leg at boot. Bridge DISCOVERED from the stable net name (never
a baked virbrN, PATTERN-1); apply idempotent + root-guarded; check = session-verify.
INSTALLED + enabled this session; boot-race UNPROVEN until an actual reboot.
- docs/design-decisions.md: D-126 amendment (base-leg durability) + NEW D-128 (operating
model -- Claude on the vcloud jumphost; Plane-1 substrate on vcloud / Plane-2 executes
on voffice1; workstation tailnet is the human path; D-107 untouched).
- Skill invariant + routing row (SKILL.md), Cross-cutting bullet (dc-dc-deployment-workflow.md),
session-ledger + changelog.
Gauntlet ALL GREEN (67 harnesses), repo-lint 0-fail. Nothing merged to main (stage-close).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMKdXCquMnh1jKH2sWXPMg
|
| 2026-07-16 |
D-126: durable rootless agent access to VR1 site VMs (shell + forward) + DC creds prep
...
Establishes durable, rootless jumphost->site-VM access so the recurring
sudo-route friction stops (and generalizes to the DCs). Operator ruled Option A
(SSH via systemd --user / ssh_config ProxyJump), advisor-conferred.
Shell layer (FOUNDATION, EXERCISED): scripts/site-ssh-config.sh generates
~/.ssh/config.d/vr1-sites ProxyJump aliases (voffice1, office1-netbox,
office1-tailscale, office1-opnsense). All four aliases proven live from the
agent's session after the operator installed with the one-key office1 env key --
key read from IdentityFile, no creds path in any command (clears the guard hook).
Forward layer (convenience, reviewed-but-unexercised): scripts/site-forward.sh
holds a systemd --user SSH local-forward -> http://127.0.0.1:8000 for office1-netbox.
Secrets boundary: the key is operator-supplied via --key (a PATH, never read/baked);
generated fragments name that path and are written locally (not committed).
DC prep: per-env-key isolation (operator ruling) -- each DC gets a DEDICATED key in
its own ~/vr1-dc<N>-creds/ folder, minted at deploy. Folders pre-created (0700);
creds-manifests/vr1-dc{0,1}.manifest declare the keypairs. DC ssh aliases deferred
until their IPs are measured post-apply.
Also: D-124 dc0 rack tfvars written locally (opentofu/d124-rack.auto.tfvars,
gitignored) -- the independent `tofu plan` unblocker. vr1_dc1_rack_* NOT written
(no ruled dc1 addressing; variables absent) -- STOP flagged for an operator ruling.
Tests: site-forward 38/38, site-ssh-config 30/30; repo-lint 0-fail (1 legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
D-125: Model B per-DC ISP egress (bridge-in, single-NAT) -- resolves OBS-3 design gap
...
OBS-3: Model B nesting put vr1-dc0-wan (a libvirt NAT) inside vvr1-dc0, whose
only leg is the SEC-010 FORWARD-dropped transit -- so the per-DC simulated ISP
had no egress. Operator ruled bridge-in (single-NAT) over double-NAT.
- modules/wan-bridge (NEW): libvirt forward=bridge onto a pre-existing host
bridge (bridge={name}, schema measured from the provider, not guessed). No
subnet/NAT. Sibling to site-wan so every NAT user stays untouched. Covered by
opentofu-validate S3 (standalone PASS).
- OUTER root: vcloud ISP NAT vr1_dc0_uplink (the single NAT) + a 2nd IP-less
uplink NIC on vvr1-dc0 + the br-vr1-dc0-wan bridge declared in netplan (transit
stays NIC1 so SEC-010/--transit-if keep keying on it). New HELD var
vr1_dc0_uplink_cidr (no default; office1-netbox apex, hard rule 2).
- INNER root: vr1_dc0_wan site-wan(NAT) -> wan-bridge onto br-vr1-dc0-wan;
OPNsense WAN re-addresses into the vcloud ISP /24 (edge config, D-113, HELD).
- Bootstrap (site-headend-install.sh): --uplink-if/--wan-bridge; --check verifies
the bridge exists + uplink ENSLAVED (fail-open guard, same class as the transit
check); SEC-010 rationale corrected (bridge-in has no inner NAT) + br_netfilter
CONSTRAINT (keep the FORWARD-drop interface-scoped, NEVER global). Harness 49->54.
Records: D-125 (new); D-122 amendment (forward-pointer -- WAN realization
superseded, intent stands); SEC-010 (br_netfilter + hardened --check);
model-a-fallback B->A delta item 8 + revert; platform-traps 1g (bridge +
br_netfilter + nested-foreign-MAC trap).
NOT closed: the L2-NAT-of-foreign-MAC path is a DEPLOY-TIME gate (bridge-in
equivalent of the depth-4 boot gate); fallback = double-NAT. HELD: vr1_dc0_uplink_cidr
+ OPNsense WAN re-address. Both roots validate; gauntlet ALL GREEN (63); repo-lint
0 fail. Present-only; NOT pushed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage-3 review sweep C+D: Model B two-root reshape + SEC-010 hardening
...
Phase C (Model B reshape, D-123 operator ruling):
- outer root (opentofu/main.tf): vvr1-dc0 resized to whole-DC containment VM
(108 vCPU / 425984 MiB / 3221225472000 bytes; expose_nested_virt=true;
single Office1 transit leg); node/plane/wan/opnsense modules removed from
vcloud level (MOVED markers), storage-04 added per R-3.
- inner root (opentofu/vr1-dc0-substrate/): NEW second root, provider
qemu+ssh to vvr1-dc0; reuses ../modules/* verbatim to build 9 nodes +
6 planes + wan + opnsense inside the containment VM. dmacvicar/libvirt
0.9.8 pinned (.terraform.lock.hcl committed). Both roots tofu-validate.
- variables.tf: vvr1_dc0_vcpu/memory_mib/disk_bytes with derivation comment.
Phase D (SEC-010, R3-F02): site-headend-install.sh --host-nodes node-host
bootstrap installs qemu-kvm+libvirt, persists+verifies nested=1, writes a
transit FORWARD-drop (nftables inet sec010) enforcing DC-LOCAL under Model B.
Gate hardened (advisor catch): --check now verifies the keyed transit
interface EXISTS, else nftables oifname loads clean but matches nothing
(fail-open). Harness 49/49.
Records: D-103/107/114/121/122/123/124 amendments (append-only);
model-a-fallback-plan synced; ledgers + Phase-C changelog.
KNOWN GAP (OBS-3, operator-confirmed 2026-07-16): under Model B the inner
vr1-dc0-wan NAT is nested inside vvr1-dc0 (single transit NIC), severing the
simulated-ISP egress that Model A had at vcloud level. DC0/DC1 ARE meant to
have ISP connections -- egress path resolution is follow-up design work, not
resolved by this commit.
Present-only; not pushed. Gauntlet ALL GREEN (63); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Review sweep A+B + R-3 storage-04: coherent record, whole-host calculator, R-3-compliant Model A fallback
...
Phase A (record amendments) + Phase B (scripts/dc-dc-whole-host-budget.py + harness) of the
Stage-3 adversarial-review sweep, plus the R-3 storage-04 addition (Model A layout now
3+2+4 = 9 nodes/DC). This commit is the R-3-COMPLIANT Model A state and becomes the re-cut
`model-a-fallback` anchor (the prior tag at 87a7a8a was R-3-stale -- 3 storage -- flagged by
the Model B design cross-check). NO cloud mutation; nothing applied.
- design-decisions.md: D-103/D-107/D-114/D-115/D-121/D-122/D-123/D-124 amendments (rulings R-1..R-4).
- scripts/dc-dc-whole-host-budget.py + tests/ (13/13): replaces the lost optc-calc.py (R3-F06).
- opentofu/main.tf: locals.vr1_dc0_nodes += storage-04 (R-3); tofu validate Success.
- security-ledger SEC-011 closed; session-ledger rulings + sweep progress; model-a-fallback plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
...
Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:
- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
(D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
the office1<->dc0 mesh leg; rack sizing 4/8192/80.
Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
(two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
/24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
D-121/122 folded in, stale gate corrected).
docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-15 |
creds-audit: enforce per-site credential consolidation (SEC-009); prevents the DC0/DC1 spread
...
SEC-009 established "all site secrets live in ~/<site>-creds/" but nothing enforced
it, and it had two blind spots: VM-minted secrets (no forcing function to consolidate
-- office1-netbox's sandbox NetBox token went un-consolidated until needed) and silent
sprawl (a loose *.env in ~ had no detector). Standing up DC0/DC1 multiplies both.
Built (operator-directed):
- creds-manifests/<site>.manifest -- declares exactly the secrets a site must hold,
each with mode + provenance (local, or <vm>:<path> for the VM-minted class).
vr1-office1.manifest seeded (11 entries); the live folder audits CLEAN.
- scripts/creds-audit.sh <site> -- verifies the live ~/<site>-creds/ against the
manifest (0700 folder, declared modes, present, non-empty), flags UNDECLARED files,
and scans the home root for SPRAWL. Reads NO secret values (metadata only; harness
T7 asserts no cat/head/tail/od/xxd). CREDS_ROOT/MANIFEST_DIR overrides for testing.
- tests/creds-audit/run-tests.sh (7/7) -- missing VM-secret, wrong mode, undeclared
file, home-root sprawl, non-0700 folder, metadata-only. Auto-discovered by gauntlet.
- dc-dc-phase3 GATE -- non-negotiable close-out: consolidate + manifest each secret AT
MINT TIME, and creds-audit $DC must be CLEAN before a DC is declared done.
- security-ledger SEC-009 ENFORCEMENT note.
Verified: harness 7/7; live creds-audit vr1-office1 -> CLEAN (also tool-confirms the
sandbox-token gap is closed); gauntlet ALL GREEN (59); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015WgVrZE2ryY9aXP7JiBuwj
|
| 2026-07-14 |
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
opnsense-set-interface-v6: script the LAN IPv6 the REST API cannot set (D-113 amendment)
...
Operator ruling 2026-07-14: take the PHP-model-over-SSH path, not a GUI click and
not the unproven VIP path. Rationale: the DC0 and DC1 edges get the same treatment
at Stage 3, so this must be a script, not a manual step paid three times.
WHY THIS EXISTS: measured on the live Office1 edge (OPNsense 26.1) -- there is NO
REST endpoint for a base interface's ipaddrv6. Interfaces > [LAN] is served by the
LEGACY page /interfaces.php?if=lan; only the Devices and Neighbors sub-trees were
ever migrated to MVC. D-113(a2)'s claim that "interfaces" are API-covered was
inferred from DHCP/firewall being API-native, never measured, and is false.
WHY IT IS NOT THE FORBIDDEN config.xml PUSH -- the distinction is the whole point:
it never AUTHORS a config. It ships opnsense-set-iface-v6.php, which loads the
edge's LIVE config through OPNsense's own Config singleton, mutates exactly two
leaf fields (ipaddrv6, subnetv6), and lets the vendor's own code serialize it --
byte-for-byte the code path interfaces.php runs on Save. Nothing is replaced
wholesale, so the API-managed Kea DHCP and the 8 firewall rules cannot be
clobbered. Same principle as opnsense-bootstrap-apikey.sh, which mints a key by
calling the vendor's own model rather than re-implementing its $6$ crypt format:
call the interface, do not re-implement the internals.
Safety properties, all pinned by the harness (40/40):
- DRY BY DEFAULT in both halves; --commit required to write.
- The interface COUNT is asserted unchanged before saving AND re-asserted on a
fresh forceReload() read-back. A silent structural drop on a live router fails
loud rather than being found later by a ping that stopped.
- Only ipaddrv6/subnetv6 may be assigned -- the harness counts the assignments.
- Values validated locally BEFORE they ever reach the router.
- Remote commands go through `sh -s`: root's shell on the edge is TCSH, where a
$(...) in a quoted remote command dies QUIETLY.
- Ground truth is the KERNEL (ifconfig), never the config file or a self-report.
NOT YET RUN against the edge -- that is the next, individually-gated step.
repo-lint 0 fail; gauntlet ALL GREEN (57 harnesses, +1).
Revert: git revert HEAD (removes both scripts + the harness; the edge is untouched)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
| 2026-07-13 |
office1-netbox DEPLOYED: NetBox 4.6.4 live on the Office1 headend, three levels deep
...
MAAS composed the machine, PXE-booted it, commissioned it, and DEPLOYED Ubuntu 24.04.4 onto it --
an LXD VM inside voffice1, inside vcloud, which is itself a KVM guest. NetBox 4.6.4 (Django 6.0.6)
runs on it and its API returns HTTP 200. The D-114 chain is complete in its final form.
FOUR FINDINGS. TWO ARE NOW GUARDED IN CODE.
1. MAAS-DISCOVERED SUBNETS HAVE NO GATEWAY (FIXED). MAAS discovers the compose subnet from lxdbr0's
interface, and that discovery carries no gateway_ip. The first deploy produced a machine that
LOOKED healthy -- it had an address and DNS even resolved (MAAS's bind9 is link-local) -- but had
NO default route and NO egress. It only surfaces when the first apt install hangs.
scripts/site-headend-install.sh now sets gateway_ip on the compose subnet; the harness asserts it
(19/19). office1-netbox was released and REDEPLOYED to prove the fix rather than hand-patching a
route onto the running box.
2. NETBOX 4.6 v2 TOKENS: THE WIRE FORMAT IS `nbt_<key>.<plaintext>`, NOT THE API's `token` FIELD.
NetBox 4.6 hashes tokens: `key` is a 12-char PREFIX, plus `plaintext` and an hmac_digest.
Authentication infers the version FROM A PREFIX (TOKEN_PREFIX = 'nbt_') and splits on '.'. So the
57-char assembled form is what authenticates. POST /api/users/tokens/provision/ returns `key` and
`token` as SEPARATE fields, and the `token` field alone is NOT usable -- present it raw and NetBox
parses it as a legacy v1 token and returns 403 "Invalid v1 token", an error that names v1 while
you are holding a v2 token. Anything scripted against this NetBox (including
netbox/dc-dc-prefixes-import.py and any pynetbox client) must ASSEMBLE the token.
Also: netbox-docker's SUPERUSER_API_TOKEN env is NOT honored on this version, and hand-building a
Token row trips the DB's enforce_version_dependent_fields constraint. Call the interface; do not
re-implement it -- the same lesson this repo already learned minting OPNsense API keys, which I
proceeded to re-learn the hard way.
3. A WRONG HYPOTHESIS, RECORDED BECAUSE THE MISTAKE IS REUSABLE. When the token kept 403-ing I
concluded that rotating SECRET_KEY after NetBox had initialised must have broken the token HMAC
pepper, and I WIPED THE DATABASE (docker compose down -v) to re-init. That was WRONG -- the 403
was purely the wire format above. The wipe was harmless because the DB was empty, but the
reasoning was not: on a populated NetBox it would have been destructive. "Invalid v1 token" is a
FORMAT error, not a crypto or key-rotation error. Read the auth code before reaching for a
destructive reset.
4. netbox-docker ships a PUBLIC default SECRET_KEY (it is in the public repo) -- session-forgery
material on any reachable instance. Rotated to a generated value. (SKIP_SUPERUSER=true is the
shipped default, so no default admin/admin exists.)
All secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored 0600 under
/root/netbox-secrets/, never printed into the session.
REACHABILITY CAVEAT, not solved: NetBox is on 10.10.1.201, behind voffice1's NAT. Reachable from
voffice1, NOT from office1-local or a workstation. Making it reachable needs a static route for
10.10.1.0/24 via 10.10.0.20 on the OPNsense edge -- a live edge change, deliberately NOT done
unilaterally.
NEXT: import the D-115 carve into this NetBox.
repo-lint 0 fail. tests/site-headend-install 19/19 PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END.
...
MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.
vcloud (L1 -- itself a KVM guest, measured)
+- voffice1 (L2) MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
+- office1-netbox (L3) LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected
THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.
THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
10.10.0.0/24 (site LAN) -> Kea on the OPNsense edge MAAS dhcp_on=False
10.10.1.0/24 (lxdbr0) -> MAAS MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.
NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.
FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
be picked by accident.
DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.
10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.
Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.
Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Fold the AppArmor/libvirt rule into scripts/prereqs -- it gated every VR1 VM boot
...
`/var/lib/libvirt/vr1/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu had
been applied BY HAND on vcloud and existed ONLY as host state. Nothing in the repo
carried it, so a rebuild or a second host walked straight back into the same wall.
Why it matters: libvirt's stock abstractions/libvirt-qemu grants qemu only the DEFAULT
pool path (/var/lib/libvirt/images). VR1 uses a custom pool parent, which is not in that
abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The domain
DEFINES fine, then fails to start with a bare qemu "Permission denied" -- and NOTHING in
the libvirt error names AppArmor. It cost a session on 2026-07-12 (DOCFIX-186), was
logged as a PREREQ candidate, and was never actioned until the ledger collapse resurfaced it.
New: scripts/prereqs/install-apparmor-libvirt.sh, wired into install-all.sh (after
install-virtualization -- it edits libvirt's own profile dir) and reported by
check-prereqs.sh.
- Idempotent and deliberately QUIET on a good host: if the rule is present it exits 0
having touched nothing -- it does NOT reload apparmor or bounce libvirtd. Those happen
only on the run that actually adds the rule.
- Writes to local/, the vendor-sanctioned override include, so it survives package upgrades.
- Pool parent is a variable (VR1_POOL_PARENT, default /var/lib/libvirt/vr1), not a literal.
- Non-AppArmor hosts report N/A and SUCCEED -- no manufactured failure on a host it
does not apply to.
Verification: tests/prereqs 32/32 PASS (was 24/24) -- the new cases drive DETECTION off a
pool parent guaranteed absent from the profile, so the negative path is deterministic on
any host: a false OK from --check IS the bug, --dry-run must plan and mutate nothing, and
the planned rule must be exactly `<parent>/** rwk,`. Full gauntlet ALL GREEN (52 harnesses),
repo-lint 0 fail. Run live on vcloud: --check reports OK and leaves the profile byte-identical,
confirming the already-satisfied path is a true no-op.
No host state was mutated by this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2): API-key bootstrap -- no GUI click, no re-implemented crypto
...
The last unknown in D-113(a2) edge provisioning. The API path was proven (read AND write), but
every new edge still needed a MANUAL GUI CLICK to mint its key -- which would have put a human in
the middle of building DC1's and DC2's edges in Stage 3. Closed.
scripts/opnsense-mint-apikey.php runs ON the edge; calls OPNsense's OWN model
(Auth\User -> apikeys->add()) -- the exact code path
the GUI's ticket button invokes
scripts/opnsense-bootstrap-apikey.sh ships it over SSH, runs it, retrieves the key, wipes
the remote copies
tests/opnsense-bootstrap-apikey/ offline harness, 8 PASS (ssh/scp stubbed)
Gauntlet ALL GREEN (54 harnesses, was 53); repo-lint 0 fail.
VERIFIED END TO END on the live Office1 edge: minted a SECOND key via the vendor model ->
GET core/firmware/status returned HTTP 200 (the vendor-minted key AUTHENTICATES) ->
POST auth/user/del_api_key/<id> -> {"result":"deleted"} -> search_api_key shows 1 row remaining
(the operator's original key, untouched throughout) -> the retired key is now REJECTED (a real
negative test, not just an absence of errors). Temp creds shredded from the jumphost. The edge is
back to its prior state.
THE DESIGN DECISION, and why the alternative was rejected: OPNsense stores API secrets as
crypt(secret,'$6$') -- SHA-512 with an EMPTY salt (verified on the live 26.1 box: the stored hash
is "$6$$" + 86 chars). We COULD mint keys offline and seed them into a bootstrap config.xml. We
deliberately DO NOT. That would couple our provisioning to a vendor hash format we would have to
keep byte-compatible forever, and a mismatch FAILS SILENTLY -- keys that simply never
authenticate, with no error at generation time. Calling the vendor's own generator has no format
to keep in sync. Same principle as D-113(a2) itself: call the interface, do not re-implement the
internals -- which is exactly what DOCFIX-191/192/193 cost us. (An offline-hash attempt was
blocked by the permission classifier mid-session; the block was correct and produced this better
design.)
GUARD CLAUSE WORTH KEEPING (harness T5): the script REFUSES to overwrite an existing creds file.
An existing file means a LIVE key on the edge; overwriting the local copy would strand it there
PERMANENTLY, because the secret is hashed on save and can never be read back. That is
unrecoverable credential loss. T5 asserts both the refusal and that the existing file is left
byte-intact. The secret is never printed -- creation is the only moment it exists in cleartext.
WHAT THIS UNBLOCKS: edge provisioning is now automatable with no human in the loop --
boot factory nano -> D-112(c) console bootstrap (SSH + key) -> opnsense-bootstrap-apikey.sh
-> opnsense-api.sh for DHCP/firewall/interfaces.
There is NO config.xml anywhere in that chain.
STILL OPEN: the template reduction itself. And the chain above raises the real question -- the
template may not need REDUCING so much as DELETING, since the console bootstrap plus the key
minter cover first contact. NOT ruled; not done unilaterally. (Note the opnsense-edge module's
config_seed volume is INSTANTIATED -- it is in tfstate -- so removing the ISO wiring is a LIVE
change, not a docs change.)
Revert: git rm the two scripts + tests/opnsense-bootstrap-apikey/ (nothing live depends on them --
the live edge's key was minted via the GUI and is unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules
...
scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.
ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.
Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.
DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.
DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.
THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.
VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.
Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.
Nothing was instantiated; no live system is affected by either the bug or the fix.
Revert: git revert this commit (restores the two broken modules and the root-only gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SAFETY SWEEP: mark the config.xml push path as a LIVE HAZARD (no behaviour change)
...
After D-113(a2) was proven (Office1's config is now API-managed), every instruction in the repo
that still said "render a config.xml and push it to the edge" became ACTIVELY DANGEROUS: a full
config.xml push replaces /conf/config.xml wholesale and drops ~667 migration-populated elements
(measured), including the only 2 firewall pass rules on the box. Following the old runbook steps
against the live edge would clobber it.
Warnings and headers only. No behaviour changes, no live system touched.
Marked (5 files):
runbooks/dc-dc-phase1-office1-standup.md DANGER banner; the config.xml render + config-ISO
sub-steps removed as RUNNABLE instructions (they
would clobber the live edge). Image-prep retained;
history preserved in git + the build changelog.
runbooks/dc-dc-phase2-tofu-dc-substrate.md STOP banner on Step 4.
scripts/opnsense-render-config.sh DANGER header at point-of-use. NOT dead: under
D-113(a2) it is to be reduced to a MINIMAL bootstrap
render. Until then, safe only for a brand-new,
not-yet-booted edge.
scripts/opnsense-build-config-iso.sh RETIRED header quoting the upstream source proving
the importer can never fire on a nano image.
docs/dc-dc-deployment-workflow.md STALE-CONTENT warning atop the tooling gap register.
THE SWEEP ALSO CAUGHT A PRE-EXISTING LANDMINE, unrelated to today's work: dc-dc-phase2 Step 4
still instructs building a CONFIG ISO for DC1's edge -- but D-112 established that ISO can never
be read (opnsense-importer probes for a read-only root; on a pre-installed nano the root is
writable and a factory /conf/config.xml already exists, so it bootstrap_and_exit 0's without
enumerating a single device). That runbook has been telling anyone who follows it to build an
inert artifact and then wonder why the edge came up on factory defaults -- which is EXACTLY the
day lost on 2026-07-12. It was never corrected at the source. Now it is. Its WAN_IF/LAN_IF
"chicken-and-egg" discussion is moot for the same reason: that problem only exists if you try to
seed a full config before first boot, and D-112(c) measures vtnetN AFTER boot.
Changelogs deliberately NOT touched -- they are history, not instructions. Marking history is
noise; marking instructions is safety.
Verification: opnsense-render-config 24 PASS, opnsense-build-config-iso 2 PASS, opnsense-api
21 PASS; repo-lint 0 fail.
STILL OPEN (the actual D-113(a2) migration): reduce the template to a minimal bootstrap, retire
the config-ISO path and the opnsense-edge module's config_seed/cdrom wiring, and rewrite Stage 3's
edge steps around the API. This sweep makes the repo SAFE, not FINISHED.
Revert: git revert this commit (restores the previous, dangerous instructions verbatim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) step 1: thin OPNsense REST API client + harness
...
First delivery under the D-113 ruling (stay on OPNsense, move config off hand-authored
config.xml and onto the documented REST API).
scripts/opnsense-api.sh -- [--dry-run] <GET|POST> <api-path> [json-body]
tests/opnsense-api/run-tests.sh -- offline harness, 21 PASS, no network, no real key
Gauntlet ALL GREEN (53 harnesses, was 52). repo-lint 0 fail.
Design decisions that the harness now enforces:
- THE SECRET NEVER REACHES ARGV. Credentials are read from a file and handed to curl via
--config on STDIN. `curl -u key:secret` would put the secret in argv, where any user on the
box can read it out of `ps`. Harness T9 stubs curl with an argv recorder and asserts the
secret is absent from argv and present on stdin -- so a future "simplification" to `curl -u`
turns the harness red. That case is the point of the file.
- The API host is NEVER inferred (hard rule 2): $OPNSENSE_API_HOST unset is a loud failure, not
a silent default of 10.10.0.1 (T3).
- --insecure is deliberate and scoped: the edge's self-signed cert is REGENERATED ON EVERY BOOT
(measured -- "Created web GUI TLS certificate" appears in the config revision trail after each
reboot), so pinning it is pointless. Acceptable ONLY on the private lab LAN leg; never to be
copied to a tenant-facing surface.
- Dry-run reads no credentials (T11), which is what lets the harness prove URL construction
offline.
NOT RUN against the edge yet, and blocked on one thing: the API is alive (lighttpd on 443
answers 401) but root has 0 API keys. Minting one is a GUI action (System > Access > Users >
root > API keys), deliberately NOT automated -- writing apikeys into config.xml by hand is the
exact anti-pattern D-113 just retired. Key lands in ~/vr1-office1-creds/opnsense-api.txt,
operator-only, never read into agent context (SEC-007 covers that directory).
Proof-of-path once the key exists: drive yesterday's DHCP subnet through the API and confirm it
round-trips. If it does, the template retires to a minimal bootstrap config (sshd + root key +
console + apikey) and DHCP/firewall/interfaces move to the API. If it cannot express it, that is
the signal to revisit D-113.
Revert: git rm scripts/opnsense-api.sh && git rm -r tests/opnsense-api/ (nothing live touched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
