| 2026-07-14 |
C4 DONE: document the NetBox sandbox loop; add the upstream-write guard it exposed
...
C4 (Stage 2 close-out) -- the sandbox loop is now documented in two places, no
rationale duplicated: docs/dc-dc-netbox-buildout-scope.md section 8 (design: why a
sandbox, the fidelity check + its 2026-07-14 hardening, the write-guard table, the
WAF trap, C2 preconditions) and runbooks/dc-dc-phase1-office1-standup.md Step 10b
(the dump->seed->apply->verify->gated-write sequence, two-host/two-token separation).
Drafted by a subagent, corrected against this session's own fixes before placement.
The guard C4 exposed: roles-aggregates-import.py and dc-dc-prefixes-import.py had NO
hostname gate -- until today the WAF 403 was the only thing stopping an accidental
--commit to the apex, and the UA fix removed that accidental safety. Both now gate a
non-sandbox --commit behind --yes-write-upstream (SANDBOX_HOSTS allowlist), matching
d115-office-carve.py -- so all three write-capable importers behave identically. This
is defence-in-depth with the whole-plan preflights (which prevent a half-write).
Tests: prefixes-import 86->91 (refused without the flag + writes nothing; allowed
with it; sandbox host needs no flag). roles-aggregates 20->24. GAUNTLET ALL GREEN (58).
Stage 2: C1/C3/C4 DONE. C2 (production write) is the only item left -- gated, with
preconditions now documented (re-verify sandbox under the hardened check, run from
office1-netbox, --yes-write-upstream, approve each mutation).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Fix the WAF User-Agent gap: the two pynetbox importers could never write upstream
...
A C2 blocker, measured 2026-07-14: upstream netbox.baldurkeep.com 403s the default
Python User-Agent (references/platform-traps.md) -- curl->200, urllib/pynetbox->403.
The stdlib tools (prod-draft-dump/sandbox-seed/d115-office-carve) already send an
accepted UA, but roles-aggregates-import.py and dc-dc-prefixes-import.py built
pynetbox.api() with the DEFAULT UA and never overrode it. So they would 403 against
upstream -- they could only ever have written to the WAF-less sandbox, never to the
apex C2 must feed. The gap was invisible because every real run so far hit the sandbox.
Fix: get_nb() sets nb.http_session.headers["User-Agent"] = "curl/8.5.0" in both
(same value/rationale as the stdlib tools; pynetbox exposes its requests.Session).
Tests: prefixes-import 85->86 (the fake now has .http_session; the UA is asserted at
RUNTIME, not by source grep). roles-aggregates 19->20. GAUNTLET ALL GREEN (58).
Still open for C2 (logged in the changelog): pynetbox is NOT on vcloud, so the two
importers must run on office1-netbox (has pynetbox 7.0.0); the sandbox re-verify
needs the operator-held token; the upstream write stays operator-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Add the missing preflight-failure test; reconcile the stale tofu-sync ledger line
...
Two follow-ups a review caught:
1. Write-path fix #3 (whole-plan role preflight in dc-dc-prefixes-import.py) shipped
with NO test exercising it FAILING. Every existing test preseeds all roles, so
they ran the preflight only when it passes -- silently. A future edit could delete
the preflight and the harness would stay green: exactly the "green while the bug is
armed" failure this session has been killing. Added a test that runs main() with one
six-plane role missing and asserts nonzero exit AND that NOTHING was written (no
site, no prefixes). prefixes-import 82 -> 85.
2. The ledger's "Live state" bullet still claimed OpenTofu repo/state were IN SYNC
("tofu plan reports No changes"). False since D-119 -- plan is 11 to add / 11 to
destroy. Reconciled to state the gated destroying apply, contradicting nothing.
GAUNTLET ALL GREEN (58). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
NetBox write-path hardening: the bugs that would have corrupted the apex at C2
...
Found by adversarial review during D-119; independent of the naming work. These
are what would have bitten during C2 (the production IPAM write Stage 2 needs).
1. roles-aggregates-import.py STILL died half-way through a production write. The
preflight added on 2026-07-13 -- after this same script committed 4 roles, 400'd
on the 5th and left the apex half-populated -- only covered role-NAME collisions.
The ARIN RIR lookup and validate_ula_48() both still ran BELOW the first write,
so an apex without ARIN, or a typo'd ORG_ULA_48, would commit 5 roles (and the
RIRs) and THEN die, with no rollback. The exact failure the preflight was added
to prevent, still armed one section lower. The RIR half of the preflight loop was
a literal no-op. Both checks moved INTO the preflight.
A preflight that does not cover every die() downstream of the first write is not
a preflight.
2. sandbox-fidelity-check.py could return a FALSE GREEN -- and it is the script we
have been citing as PROOF the sandbox is a faithful replica. It field-compared
only SHARED objects; the planned delta was checked merely as a SUBSET of
EXPECTED_NEW. With ZERO of the 36 DC prefixes created, extra={} -> unexpected={}
-> it printed "Nothing lost, nothing stray" and exited 0. EXPECTED_NEW was an
upper bound masquerading as an assertion. It is now BOTH bounds, and delta
prefixes are checked for scope and role (the 17-prefix scope-drop bug relocated
into the delta, where the old check was structurally blind).
3. dc-dc-prefixes-import.py died mid-loop on a missing role -- find_role() was
resolved lazily INSIDE the write loop. Against upstream as it stands TODAY (no
six-plane roles), a --commit creates the site + 4 prefixes then dies on the 5th:
a half-written datacenter. Now a whole-plan preflight resolves every role first.
4. The test fake was KINDER THAN REALITY at the failure point: fake_pynetbox.get()
returned matches[0] on a multi-match where real pynetbox RAISES. NetBox permits
duplicate prefixes (the importer's own docstring anticipates them vs vr0-dc0), so
this would blow up in production while the harness sailed through. It now raises.
TESTS: tests/sandbox-fidelity-check/ is NEW (10/10) -- the checker shipped with NO
harness at all, which is exactly why the false green survived; T3 reproduces it, and
T1 derives its expectation from the checker's own EXPECTED_NEW rather than re-typing
it. roles-aggregates-import 16->19, asserting BY POSITION that ARIN and the ULA are
validated before the first create (the old harness only checked the WORD "PREFLIGHT"
appeared -- green while the landmine was armed).
GAUNTLET ALL GREEN (58). repo-lint 0 fail. Zero NetBox writes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
opnsense-set-interface-v6: script the LAN IPv6 the REST API cannot set (D-113 amendment)
...
Operator ruling 2026-07-14: take the PHP-model-over-SSH path, not a GUI click and
not the unproven VIP path. Rationale: the DC0 and DC1 edges get the same treatment
at Stage 3, so this must be a script, not a manual step paid three times.
WHY THIS EXISTS: measured on the live Office1 edge (OPNsense 26.1) -- there is NO
REST endpoint for a base interface's ipaddrv6. Interfaces > [LAN] is served by the
LEGACY page /interfaces.php?if=lan; only the Devices and Neighbors sub-trees were
ever migrated to MVC. D-113(a2)'s claim that "interfaces" are API-covered was
inferred from DHCP/firewall being API-native, never measured, and is false.
WHY IT IS NOT THE FORBIDDEN config.xml PUSH -- the distinction is the whole point:
it never AUTHORS a config. It ships opnsense-set-iface-v6.php, which loads the
edge's LIVE config through OPNsense's own Config singleton, mutates exactly two
leaf fields (ipaddrv6, subnetv6), and lets the vendor's own code serialize it --
byte-for-byte the code path interfaces.php runs on Save. Nothing is replaced
wholesale, so the API-managed Kea DHCP and the 8 firewall rules cannot be
clobbered. Same principle as opnsense-bootstrap-apikey.sh, which mints a key by
calling the vendor's own model rather than re-implementing its $6$ crypt format:
call the interface, do not re-implement the internals.
Safety properties, all pinned by the harness (40/40):
- DRY BY DEFAULT in both halves; --commit required to write.
- The interface COUNT is asserted unchanged before saving AND re-asserted on a
fresh forceReload() read-back. A silent structural drop on a live router fails
loud rather than being found later by a ping that stopped.
- Only ipaddrv6/subnetv6 may be assigned -- the harness counts the assignments.
- Values validated locally BEFORE they ever reach the router.
- Remote commands go through `sh -s`: root's shell on the edge is TCSH, where a
$(...) in a quoted remote command dies QUIETLY.
- Ground truth is the KERNEL (ifconfig), never the config file or a self-report.
NOT YET RUN against the edge -- that is the next, individually-gated step.
repo-lint 0 fail; gauntlet ALL GREEN (57 harnesses, +1).
Revert: git revert HEAD (removes both scripts + the harness; the edge is untouched)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-118 ADOPTED (org ULA fd50:840e:74e2::/48); VR1 DC0 + DC1 IMPORTED; roles importer bug fixed
...
D-118: ORG_ULA_48 = fd50:840e:74e2::/48. RFC 4193 valid (fd -> L=1, 40-bit Global
ID 0x50840e74e2, /48 -> 256 x /56 for D-101's per-DC /56). The load-bearing check
is NOT the RFC conformance -- it is that it does NOT collide with Tailscale's
fd7a:115c:a1e0::/48: office1-tailscale runs --accept-routes, so an overlap would
have been a LIVE tailnet routing conflict, not a paper one.
Provenance stated honestly: the 'CSPRNG-generated' claim cannot be re-verified.
It does not change the ruling (RFC 4193's SHA1 construction is a SHOULD; the
requirement is a pseudo-random 40-bit Global ID). Regenerating would only swap in
a different random number at the cost of re-editing 7 places.
Corrects a FALSE claim: changelog-20260711 called this value 'ratified' when no
D-number ever assigned it. That phantom ratification is why it read as settled for
two days. Same failure mode as D-117: a document asserting authority it lacked.
IMPORTED into the sandbox: six-plane roles + 2 RIRs + 5 aggregates, then VR1 DC0
(18 prefixes) and VR1 DC1 (18). The D-117 fix is now LIVE-PROVEN, not just green
offline: --dc dc0 -> apex site VR1 DC0 (id=8), --dc dc1 -> VR1 DC1 (id=9). ULA
reads DC.NN in the 4th hextet per D-111 (fd50:840e:74e2:220:: .. :350::).
A REAL BUG, found by running roles-aggregates-import.py for the first time for
real: it checked existence by SLUG only, but NetBox also enforces UNIQUE NAMES.
The draft's legacy role 'Replication' (slug repl) blocks the six-plane role (slug
replication, name Replication) -- so it created 4 roles, 400'd on the 5th, and
DIED, leaving the apex HALF-POPULATED with no RIRs, no aggregates, no rollback.
It had never met a NetBox holding the real draft (only empty ones) and it SHIPPED
WITH NO HARNESS. That is why it survived.
- six-plane role renamed 'Replication Plane' (slug unchanged -- lib-net.sh
SPACES6 and the prefix importer look up 'replication')
- the script now PREFLIGHTS every name/slug and writes NOTHING unless the whole
plan is viable. An IPAM importer that can die partway corrupts what it
populates.
- tests/roles-aggregates-import/ CREATED: 16/16 PASS.
Final state PROVEN by sandbox-fidelity-check (now carrying the full planned
delta): the sandbox is the upstream draft + EXACTLY D-115 + D-117 + D-118 --
every shared object field-identical, nothing lost, nothing stray. 90 -> 134
prefixes.
repo-lint 0 fail. run-tests-all: ALL GREEN (56 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
Prove the sandbox is a FAITHFUL replica, not just a count-matching one
...
netbox/sandbox-fidelity-check.py (+4 assertions in tests/sandbox-seed, 22/22 PASS)
Counts and idempotency CANNOT prove a faithful seed. The region-scope bug proved
it: 17 of 90 prefixes silently lost their scope, every count still matched, and
re-runs were still perfectly idempotent. It was caught by LUCK on a spot-check
that happened to print 'site None'. This is that catch, made deliberate.
Method needs no new plumbing: prod-draft-dump.py is READ-ONLY, so point it at the
sandbox and diff the two JSON dumps FIELD BY FIELD.
RESULT: every shared object is field-level IDENTICAL to upstream, and the delta is
EXACTLY the expected D-115 additions (+edge role, +vr1-off1 site, +8 prefixes).
Nothing missing, nothing unexpected. The sandbox is a PROVEN baseline -- which is
what the later feed-back diff to production depends on.
Also de-misdirects docs/dc-dc-netbox-buildout-scope.md, which still told the next
person to run '--dc dc2' with DC2_V4_SUPERNET -- both of which the new guards now
REJECT by name. Its G5 recommendation (rename the APEX to vr1-dc1/vr1-dc2) is
marked SUPERSEDED: D-117 ruled the opposite. ORG_ULA_48 is now listed there as
UNASSIGNED and BLOCKING, not as a settled literal.
Disclosure: D-117 Option B is PARTIALLY executed -- the import path and the scope
doc are done; the $DC shell selector and the runbook DC prose are NOT. Recorded
in the ledger so 'D-117 ADOPTED' is not misread as 'fully renamed'.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-115 office carve IMPORTED into the Office1 NetBox; DC import blocked on ORG_ULA_48
...
netbox/d115-office-carve.py (+ tests/d115-office-carve/ 20/20 PASS)
Applied to the Office1 sandbox: site vr1-off1, roles office + edge (edge is NEW),
and 8 prefixes. Idempotent (re-run: 0 created / 11 present).
10.10.0.0/16 office container the Office v4 block, /22 per office
10.10.0.0/22 office container VR1 Off1's /22
10.10.0.0/24 office active office1-local LAN (Kea on the edge)
10.10.1.0/24 office active lxdbr0 compose net (MAAS DHCP)
172.30.0.0/16 edge container NEW Edge role; mirrors v6 2602:f3e2:fe::/48
172.30.1.0/24 edge active office1-wan, the simulated ISP uplink
2602:f3e2:f01 :/56 office container mirrors VR0 Off0's e01:/56
2602:f3e2:f01:/64 office active office subnet (NOT deployed; v6 has no egress)
Every address was already RUNNING. D-115 blesses what is deployed -- zero
renumber. What it fixes is that three of them were SQUAT: on the wire with no
allocation behind them.
Writing upstream is GATED IN CODE: the tool refuses a non-sandbox target unless
given --yes-write-upstream. Feeding the production apex is an operator decision,
never a side effect of running an import.
BLOCKED, deliberately: dc-dc-prefixes-import.py --dc dc0 CANNOT run. It needs
ORG_ULA_48, which is still UNASSIGNED (gap G3; the scope doc recommends
fd50:840e:74e2::/48 but no ADOPTED decision assigns it -- grep ORG_ULA
design-decisions.md returns nothing). Using it anyway would be an inferred value
(hard rule 2). This is a DECISION gate, not a tooling gap: the tooling is fixed,
tested and ready. Recorded in the ledger with the exact commands to unblock.
Also: docs/vr1-office1-as-built.md gains the IPAM table, and records that the
Tailscale 10.10.0.0/22 route is now ENABLED (edge GUI reachable from the
workstation with no tunnel), superseding the tunnel instructions.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
NetBox sandbox seeded from the upstream draft (129 objects) + the two-phase tooling
...
The Office1 sandbox NetBox was EMPTY, and no seeder existed. The documented loop
(upstream draft -> sandbox -> simulate -> feed back) had no first step.
netbox/prod-draft-dump.py READ-ONLY dump of the upstream draft -> JSON artifact
netbox/sandbox-seed.py applies that JSON to a SANDBOX. Dry by default.
netbox/draft/vr1-draft.json the reviewable artifact (90 prefixes, 23 roles, ...)
TWO PHASES, deliberately, rather than one source->dest script:
- the upstream token never travels (dump runs on vcloud; seed runs on
office1-netbox, where the sandbox token already lives)
- 'the sim NEVER writes upstream' becomes STRUCTURAL: the script that writes
has no upstream credential and no upstream URL, and REFUSES to write to the
instance the draft was read from (self-configuring via the draft's _source,
so it keeps working if upstream ever moves -- a hostname denylist would not)
- the JSON is diffable: you can read what is about to be written
Applied to the sandbox: 129 created, then verified idempotent (0 created / 129
present on re-run). vr1-dc0 -> f02::/48 and vr1-dc1 -> f03::/48, matching the
apex exactly, which is what D-117 exists to guarantee.
TWO BUGS FOUND AND FIXED WHILE BUILDING IT:
1. The dry run mapped only dcim.site scopes, so REGION-scoped prefixes lost
their scope -- 17 of 90 upstream, including the ENTIRE VR1 region-scoped
draft. Caught by a spot-check printing 'site None'. Fixed + reseeded
(--update), now bound to region VR1.
2. The upstream-refusal guard sat AFTER json.load() and after the banner: it
was dead code for any malformed draft and READ as safe. Guards go first.
The harness now proves it actually refuses.
tests/sandbox-seed/: 18/18 PASS. repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-117 ADOPTED (B, DC-only): fix the off-by-one site binding + make the importer dry-by-default
...
Operator ruling: rename the DC numbers to match the NetBox apex; the office
keeps its number (Office1 -> new site vr1-off1, so nothing deployed changes).
netbox/dc-dc-prefixes-import.py:
--dc dc1|dc2 -> --dc dc0|dc1, bound to the APEX slugs measured live:
dc0 -> vr1-dc0 (2602:f3e2:f02::/48) dc1 -> vr1-dc1 (f03::/48)
It previously mapped dc1 -> vr1-dc1 while treating f02::/48 as dc1, i.e. it
would have written VR1 DC0's prefixes to the OTHER DC's site.
Now DRY BY DEFAULT with --commit (it used to write with no flag at all).
A dry run against a FRESH NetBox now plans the site too (_PlannedSite) rather
than bailing with 'no site id' -- previewing an empty apex is the whole point.
DC2_V4_SUPERNET is REJECTED BY NAME (retired) rather than silently ignored.
Every 'DCn' in prose is region-qualified: 'DC0' meant VR0's rehearsal DC in
some sentences and VR1's first DC in others. No sed could have caught that.
tests/dc-dc-prefixes-import/: 73 checks ALL PASS, incl. regression guards that
the dry run creates NOTHING, that --dc dc0 never touches vr1-dc1, and that the
invented vr1-dc2 site is never created.
FOUND, RECORDED, NOT YET FIXED (D-117 amendment): scripts/lib-net.sh carries a
SECOND, colliding $DC namespace where dc0 already means VR0's DC0. Renaming
VR1's dc1->dc0 there would collide with VR0. Proposed fix: region-qualify the
shell selector to the apex slugs (vr0-dc0/vr1-dc0/vr1-dc1). Does not block the
import; the two selectors are NOT interchangeable meanwhile.
repo-lint 0 fail. run-tests-all: ALL GREEN (53 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
| 2026-07-13 |
office1-netbox DEPLOYED: NetBox 4.6.4 live on the Office1 headend, three levels deep
...
MAAS composed the machine, PXE-booted it, commissioned it, and DEPLOYED Ubuntu 24.04.4 onto it --
an LXD VM inside voffice1, inside vcloud, which is itself a KVM guest. NetBox 4.6.4 (Django 6.0.6)
runs on it and its API returns HTTP 200. The D-114 chain is complete in its final form.
FOUR FINDINGS. TWO ARE NOW GUARDED IN CODE.
1. MAAS-DISCOVERED SUBNETS HAVE NO GATEWAY (FIXED). MAAS discovers the compose subnet from lxdbr0's
interface, and that discovery carries no gateway_ip. The first deploy produced a machine that
LOOKED healthy -- it had an address and DNS even resolved (MAAS's bind9 is link-local) -- but had
NO default route and NO egress. It only surfaces when the first apt install hangs.
scripts/site-headend-install.sh now sets gateway_ip on the compose subnet; the harness asserts it
(19/19). office1-netbox was released and REDEPLOYED to prove the fix rather than hand-patching a
route onto the running box.
2. NETBOX 4.6 v2 TOKENS: THE WIRE FORMAT IS `nbt_<key>.<plaintext>`, NOT THE API's `token` FIELD.
NetBox 4.6 hashes tokens: `key` is a 12-char PREFIX, plus `plaintext` and an hmac_digest.
Authentication infers the version FROM A PREFIX (TOKEN_PREFIX = 'nbt_') and splits on '.'. So the
57-char assembled form is what authenticates. POST /api/users/tokens/provision/ returns `key` and
`token` as SEPARATE fields, and the `token` field alone is NOT usable -- present it raw and NetBox
parses it as a legacy v1 token and returns 403 "Invalid v1 token", an error that names v1 while
you are holding a v2 token. Anything scripted against this NetBox (including
netbox/dc-dc-prefixes-import.py and any pynetbox client) must ASSEMBLE the token.
Also: netbox-docker's SUPERUSER_API_TOKEN env is NOT honored on this version, and hand-building a
Token row trips the DB's enforce_version_dependent_fields constraint. Call the interface; do not
re-implement it -- the same lesson this repo already learned minting OPNsense API keys, which I
proceeded to re-learn the hard way.
3. A WRONG HYPOTHESIS, RECORDED BECAUSE THE MISTAKE IS REUSABLE. When the token kept 403-ing I
concluded that rotating SECRET_KEY after NetBox had initialised must have broken the token HMAC
pepper, and I WIPED THE DATABASE (docker compose down -v) to re-init. That was WRONG -- the 403
was purely the wire format above. The wipe was harmless because the DB was empty, but the
reasoning was not: on a populated NetBox it would have been destructive. "Invalid v1 token" is a
FORMAT error, not a crypto or key-rotation error. Read the auth code before reaching for a
destructive reset.
4. netbox-docker ships a PUBLIC default SECRET_KEY (it is in the public repo) -- session-forgery
material on any reachable instance. Rotated to a generated value. (SKIP_SUPERUSER=true is the
shipped default, so no default admin/admin exists.)
All secrets (SECRET_KEY, admin password, API token) generated ON the VM, stored 0600 under
/root/netbox-secrets/, never printed into the session.
REACHABILITY CAVEAT, not solved: NetBox is on 10.10.1.201, behind voffice1's NAT. Reachable from
voffice1, NOT from office1-local or a workstation. Making it reachable needs a static route for
10.10.1.0/24 via 10.10.0.20 on the OPNsense edge -- a live edge change, deliberately NOT done
unilaterally.
NEXT: import the D-115 carve into this NetBox.
repo-lint 0 fail. tests/site-headend-install 19/19 PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 headend LIVE: MAAS 3.7.2 + LXD 5.21 on voffice1. D-114 PROVEN END TO END.
...
MAAS composed, PXE-booted and COMMISSIONED its first VM inside voffice1. The model works.
vcloud (L1 -- itself a KVM guest, measured)
+- voffice1 (L2) MAAS 3.7.2 region+rack + PostgreSQL 16.14 + LXD 5.21.5
+- office1-netbox (L3) LXD VIRTUAL-MACHINE, MAAS-composed, MAAS-PXE-booted,
leased 10.10.1.100 from MAAS, status READY, 2c/4096MiB detected
THE L3 PROOF IS NOW DEFINITIVE. Earlier today we could only say nested KVM was AVAILABLE
(/dev/kvm + svm present). Now a guest has actually BOOTED, PXE'd, COMMISSIONED and reported its
hardware three levels deep. D-114's Office1 gate is CLEARED. This reproduces VR0's `lxd` +
`tailscale` pattern exactly -- and the Juju-created LXD CONTAINERS remain invisible to MAAS, as in VR0.
THE DHCP SPLIT IS STRUCTURAL, NOT CONVENTIONAL:
10.10.0.0/24 (site LAN) -> Kea on the OPNsense edge MAAS dhcp_on=False
10.10.1.0/24 (lxdbr0) -> MAAS MAAS dhcp_on=True
Separate L2s; lxdbr0 is never bridged onto the site LAN, so they cannot see each other. The
installer asserts after configuring DHCP that MAAS owns NO other subnet, and hard-fails otherwise.
NEW: scripts/site-headend-install.sh + tests/site-headend-install/ (18/18 PASS). It is the sequence
we ACTUALLY EXECUTED, codified -- not written from docs -- and reusable verbatim for DC1/DC2. Its
--check mode was run against the live voffice1 and reports every item OK.
FOUR TRAPS, ALL HIT FOR REAL TODAY, ALL NOW ENCODED AND HARNESS-ASSERTED:
1. `lxd init --auto` FAILS on a MAAS host. It brings up lxdbr0 with LXD's own DHCP+DNS, which starts
a dnsmasq; dnsmasq binds the WILDCARD 0.0.0.0:53 and MAAS's bind9 already holds :53. The error
names the BRIDGE address ("failed to create listening socket for 10.10.1.1"), which sends you
hunting the wrong thing. `ipv4.dhcp=false` + `dns.mode=none` are NOT enough -- LXD still spawns
dnsmasq. The fix is raw.dnsmasq="port=0". Verified after: dnsmasq holds ZERO :53 and ZERO :67.
2. `lxc` READS CONFIG FROM STDIN. Piped to `bash -s` over ssh, the first unredirected `lxc` call
EATS THE REST OF THE SCRIPT as YAML and everything after it silently vanishes. Every lxc call now
ends in </dev/null. Same class as the OPNsense tcsh trap.
3. LXD snap track changes are ONE-WAY -- install DIRECTLY onto 5.21/stable, never via `latest`.
MAAS 3.6/3.7 is incompatible with LXD >= 6.7. (Bonus: core.trust_password exists in 5.21, gone in
6.x -- the pin also preserves the scriptable registration path.)
4. MAAS DHCP on the compose network ONLY. --compose-cidr is REQUIRED with no default so it can never
be picked by accident.
DISCIPLINE: every MAAS/LXD flag was read from the tool's own --help ON THE BOX before use (maas init,
createadmin, vm-host compose), not trusted from docs. All secrets (DB, admin, API key, LXD trust) are
generated on the target, stored 0600 under /root/maas-secrets/, and never printed into the session.
10.10.1.0/24 IS AN UNBLESSED ALLOCATION -- chosen deliberately (DC1=10.12.x, DC2=10.13.0.0/19, tenant
pool=10.20.0.0/16 per D-016; the obvious 10.20.x would have COLLIDED with tenant space), but NetBox
has not assigned it. Register once NetBox exists -- it is one of the VMs this headend composes.
Also: the ops skill is now tuned to this exact stack (new references/platform-traps.md with a
verbatim-error -> cause index; the OPNsense section that still routed sessions into the DELETED
config-ISO path is replaced). Upstream corroboration found for two of our own findings: the libvirt
provider's release notes document that an "in-place" domain update undefines and re-defines the
domain (our measured guest-bounce), and there is an upstream issue for the AppArmor denial.
Gauntlet ALL GREEN (53 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Fold the AppArmor/libvirt rule into scripts/prereqs -- it gated every VR1 VM boot
...
`/var/lib/libvirt/vr1/** rwk,` in /etc/apparmor.d/local/abstractions/libvirt-qemu had
been applied BY HAND on vcloud and existed ONLY as host state. Nothing in the repo
carried it, so a rebuild or a second host walked straight back into the same wall.
Why it matters: libvirt's stock abstractions/libvirt-qemu grants qemu only the DEFAULT
pool path (/var/lib/libvirt/images). VR1 uses a custom pool parent, which is not in that
abstraction, so AppArmor blocks qemu even with perfect POSIX permissions. The domain
DEFINES fine, then fails to start with a bare qemu "Permission denied" -- and NOTHING in
the libvirt error names AppArmor. It cost a session on 2026-07-12 (DOCFIX-186), was
logged as a PREREQ candidate, and was never actioned until the ledger collapse resurfaced it.
New: scripts/prereqs/install-apparmor-libvirt.sh, wired into install-all.sh (after
install-virtualization -- it edits libvirt's own profile dir) and reported by
check-prereqs.sh.
- Idempotent and deliberately QUIET on a good host: if the rule is present it exits 0
having touched nothing -- it does NOT reload apparmor or bounce libvirtd. Those happen
only on the run that actually adds the rule.
- Writes to local/, the vendor-sanctioned override include, so it survives package upgrades.
- Pool parent is a variable (VR1_POOL_PARENT, default /var/lib/libvirt/vr1), not a literal.
- Non-AppArmor hosts report N/A and SUCCEED -- no manufactured failure on a host it
does not apply to.
Verification: tests/prereqs 32/32 PASS (was 24/24) -- the new cases drive DETECTION off a
pool parent guaranteed absent from the profile, so the negative path is deterministic on
any host: a false OK from --check IS the bug, --dry-run must plan and mutate nothing, and
the planned rule must be exactly `<parent>/** rwk,`. Full gauntlet ALL GREEN (52 harnesses),
repo-lint 0 fail. Run live on vcloud: --check reports OK and leaves the profile byte-identical,
confirming the already-satisfied path is a true no-op.
No host state was mutated by this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-113(a2) COMPLETE: DELETE the config.xml path (template, renderer, ISO builder)
...
Operator ruling ("I'll defer to your lean"): the OPNsense config.xml template is DELETED, not
reduced.
WHY DELETED RATHER THAN REDUCED: the (a2) plan was to shrink the template to a minimal bootstrap
(sshd + root key + console + a seeded API key). That proved UNNECESSARY -- every one of those is
covered without a config.xml at all:
sshd + root key -> the D-112(c) console bootstrap (proven on Office1)
an API key -> scripts/opnsense-bootstrap-apikey.sh (OPNsense's OWN model; no GUI
click, no re-implemented crypto)
DHCP/firewall/interfaces -> the REST API (proven read AND write)
So the provisioning chain has NO config.xml anywhere:
boot factory nano -> console bootstrap -> mint API key -> configure over REST
A config.xml renderer that nobody should ever run is not a safety net -- it is a loaded gun
pointed at a live router. The 2026-07-13 safety sweep is the evidence: the repo still contained
runbook steps telling an operator to render a config and push it to the edge, which by then would
have CLOBBERED live API-managed DHCP.
DELETED: opentofu/templates/opnsense-config.xml.tmpl; scripts/opnsense-render-config.sh +
tests/opnsense-render-config/; scripts/opnsense-build-config-iso.sh +
tests/opnsense-build-config-iso/; the opnsense-edge module's config_seed volume + cdrom disk and
its config_iso_path variable; the xorriso/genisoimage prereq (it existed ONLY for the ISO
builder). Gauntlet 54 -> 52 harnesses (the two deleted scripts took their harnesses with them --
expected arithmetic, not a regression). All files remain in git history.
opentofu/templates/README.md is now a TOMBSTONE explaining what happened and what to use instead.
THE LIVE CHANGE IS NOT APPLIED IN THIS COMMIT. Removing the module's ISO wiring touches an
INSTANTIATED resource (libvirt_volume.config_seed is in tfstate). Measured with `tofu plan` first:
module.office1_opnsense.libvirt_domain.vm will be updated IN-PLACE
module.office1_opnsense.libvirt_volume.config_seed will be destroyed
Plan: 0 to add, 1 to change, 1 to destroy.
NO replacement of the running domain -- confirmed explicitly. A replacement would have DESTROYED
the live, routing, DHCP-serving edge, and would have been grounds to stop.
A LINT GUARD DID ITS JOB: repo-lint's L3 (runbooks must not reference missing scripts) went RED on
the tombstone notes, because they named the deleted paths. The rule was RIGHT and it has no
opt-out (only L4 does). Rather than weaken a guard that exists to stop runbooks pointing at dead
scripts, the tombstones were reworded to name the bare filename instead of the scripts/ path. The
guard stays fully intact; the information is preserved.
Verification: repo-lint 0 fail; opentofu-validate PASS (root + 10/10 modules standalone);
gauntlet ALL GREEN (52).
Revert: git revert this commit (restores template/renderer/ISO builder/harnesses), then
`cd opentofu && tofu apply` to re-create the (inert) config_seed volume + cdrom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2): API-key bootstrap -- no GUI click, no re-implemented crypto
...
The last unknown in D-113(a2) edge provisioning. The API path was proven (read AND write), but
every new edge still needed a MANUAL GUI CLICK to mint its key -- which would have put a human in
the middle of building DC1's and DC2's edges in Stage 3. Closed.
scripts/opnsense-mint-apikey.php runs ON the edge; calls OPNsense's OWN model
(Auth\User -> apikeys->add()) -- the exact code path
the GUI's ticket button invokes
scripts/opnsense-bootstrap-apikey.sh ships it over SSH, runs it, retrieves the key, wipes
the remote copies
tests/opnsense-bootstrap-apikey/ offline harness, 8 PASS (ssh/scp stubbed)
Gauntlet ALL GREEN (54 harnesses, was 53); repo-lint 0 fail.
VERIFIED END TO END on the live Office1 edge: minted a SECOND key via the vendor model ->
GET core/firmware/status returned HTTP 200 (the vendor-minted key AUTHENTICATES) ->
POST auth/user/del_api_key/<id> -> {"result":"deleted"} -> search_api_key shows 1 row remaining
(the operator's original key, untouched throughout) -> the retired key is now REJECTED (a real
negative test, not just an absence of errors). Temp creds shredded from the jumphost. The edge is
back to its prior state.
THE DESIGN DECISION, and why the alternative was rejected: OPNsense stores API secrets as
crypt(secret,'$6$') -- SHA-512 with an EMPTY salt (verified on the live 26.1 box: the stored hash
is "$6$$" + 86 chars). We COULD mint keys offline and seed them into a bootstrap config.xml. We
deliberately DO NOT. That would couple our provisioning to a vendor hash format we would have to
keep byte-compatible forever, and a mismatch FAILS SILENTLY -- keys that simply never
authenticate, with no error at generation time. Calling the vendor's own generator has no format
to keep in sync. Same principle as D-113(a2) itself: call the interface, do not re-implement the
internals -- which is exactly what DOCFIX-191/192/193 cost us. (An offline-hash attempt was
blocked by the permission classifier mid-session; the block was correct and produced this better
design.)
GUARD CLAUSE WORTH KEEPING (harness T5): the script REFUSES to overwrite an existing creds file.
An existing file means a LIVE key on the edge; overwriting the local copy would strand it there
PERMANENTLY, because the secret is hashed on save and can never be read back. That is
unrecoverable credential loss. T5 asserts both the refusal and that the existing file is left
byte-intact. The secret is never printed -- creation is the only moment it exists in cleartext.
WHAT THIS UNBLOCKS: edge provisioning is now automatable with no human in the loop --
boot factory nano -> D-112(c) console bootstrap (SSH + key) -> opnsense-bootstrap-apikey.sh
-> opnsense-api.sh for DHCP/firewall/interfaces.
There is NO config.xml anywhere in that chain.
STILL OPEN: the template reduction itself. And the chain above raises the real question -- the
template may not need REDUCING so much as DELETING, since the console bootstrap plus the key
minter cover first contact. NOT ruled; not done unilaterally. (Note the opnsense-edge module's
config_seed volume is INSTANTIATED -- it is in tfstate -- so removing the ISO wiring is a LIVE
change, not a docs change.)
Revert: git rm the two scripts + tests/opnsense-bootstrap-apikey/ (nothing live depends on them --
the live edge's key was minted via the GUI and is unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
DOCFIX-194: the OpenTofu gate was GREEN over two BROKEN modules
...
scripts/opentofu-validate.sh printed PASS on 2026-07-13 while node-vm and netem-link were flatly
broken -- neither could have applied, and one could not even init. Both are modules Stage 3 and
Stage 4 are built on.
ROOT CAUSE OF THE MISS: root-only validation. `tofu validate` against the ROOT module only parses
modules the root actually INSTANTIATES. A module that is not called -- or is called only from a
commented-out block -- is NEVER PARSED. Root calls dc-planes, dc-storage-pool, mesh-link,
office1-network, opnsense-edge. It does NOT call base-image, cloudinit-vm, maas-vm-host,
netem-link, node-vm -- exactly the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm,
maas-vm-host, netem-link) depend on. They had never been parsed by any tool.
Only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3). The tree was
authored in sessions with no binary to self-check -- exactly the risk opentofu/README.md flagged.
DEFECT 1 -- node-vm: libvirt_volume had a top-level `format = {...}`, which is not a real
attribute ("An argument named format is not expected here"). Schema-verified via
`tofu providers schema -json` (dmacvicar/libvirt v0.9.8): libvirt_volume has NO top-level
`format` -- it nests under `target`. modules/base-image had the correct nesting all along, which
is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks); Stage 3/4
would have failed on first use. The module's own header said "VERIFY with tofu providers schema
-json before the first real apply" -- nobody could, until now.
DEFECT 2 -- netem-link: a destroy-time provisioner referenced var.*, which OpenTofu rejects AT
INIT ("Destroy-time provisioners ... may only reference attributes of the related resource, via
'self', 'count.index', or 'each.key'"). The module could not even initialize. It is the
DR/latency mechanism for Stage 3 and the Stage 6 failover drill. Fixed with the canonical
pattern: stash values in `input`, read back as self.input.*.
THE DURABLE FIX -- S3: the gate now validates EVERY module STANDALONE, in a temp copy (so it
never writes .terraform/ or a module-level lock file into the repo; only the ROOT lock file is
tracked, deliberately). The two bugs are one-line fixes; the real defect was the gate. A gate
that reports green over unparsed code is worse than no gate -- it manufactures confidence.
VERIFICATION: tests/opentofu-validate 9 PASS (T2 skipped -- cannot exercise the missing-binary
guard on a box that has the binary). T8 is the load-bearing case: a fixture whose ROOT IS VALID
and does NOT call a BROKEN module -- root-only validation reports PASS on it, so S3 must FAIL.
If T8 ever goes green, the blind spot is back. T9 proves T8 fails on the defect, not the fixture
shape. T10 covers all 10 real modules. Fixtures use terraform_data (a builtin), so no provider
download is needed.
Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, 10/10 modules PASS
standalone. Gauntlet ALL GREEN (53 harnesses); repo-lint 0 fail.
Nothing was instantiated; no live system is affected by either the bug or the fix.
Revert: git revert this commit (restores the two broken modules and the root-only gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) step 1: thin OPNsense REST API client + harness
...
First delivery under the D-113 ruling (stay on OPNsense, move config off hand-authored
config.xml and onto the documented REST API).
scripts/opnsense-api.sh -- [--dry-run] <GET|POST> <api-path> [json-body]
tests/opnsense-api/run-tests.sh -- offline harness, 21 PASS, no network, no real key
Gauntlet ALL GREEN (53 harnesses, was 52). repo-lint 0 fail.
Design decisions that the harness now enforces:
- THE SECRET NEVER REACHES ARGV. Credentials are read from a file and handed to curl via
--config on STDIN. `curl -u key:secret` would put the secret in argv, where any user on the
box can read it out of `ps`. Harness T9 stubs curl with an argv recorder and asserts the
secret is absent from argv and present on stdin -- so a future "simplification" to `curl -u`
turns the harness red. That case is the point of the file.
- The API host is NEVER inferred (hard rule 2): $OPNSENSE_API_HOST unset is a loud failure, not
a silent default of 10.10.0.1 (T3).
- --insecure is deliberate and scoped: the edge's self-signed cert is REGENERATED ON EVERY BOOT
(measured -- "Created web GUI TLS certificate" appears in the config revision trail after each
reboot), so pinning it is pointless. Acceptable ONLY on the private lab LAN leg; never to be
copied to a tenant-facing surface.
- Dry-run reads no credentials (T11), which is what lets the harness prove URL construction
offline.
NOT RUN against the edge yet, and blocked on one thing: the API is alive (lighttpd on 443
answers 401) but root has 0 API keys. Minting one is a GUI action (System > Access > Users >
root > API keys), deliberately NOT automated -- writing apikeys into config.xml by hand is the
exact anti-pattern D-113 just retired. Key lands in ~/vr1-office1-creds/opnsense-api.txt,
operator-only, never read into agent context (SEC-007 covers that directory).
Proof-of-path once the key exists: drive yesterday's DHCP subnet through the API and confirm it
round-trips. If it does, the template retires to a minimal bootstrap config (sshd + root key +
console + apikey) and DHCP/firewall/interfaces move to the API. If it cannot express it, that is
the signal to revisit D-113.
Revert: git rm scripts/opnsense-api.sh && git rm -r tests/opnsense-api/ (nothing live touched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
| 2026-07-12 |
DOCFIX-193: LAN DHCP was never implemented -- add Kea to the edge config template
...
The operator's goal for the Office1 edge was "router + DHCP for office1-local". The
router half works; the DHCP half was NEVER BUILT. The template had no DHCP section at
all -- no <dhcpd>, no <Kea> -- and nothing served DHCP (measured on the live edge: no
kea/dhcpd/dnsmasq process, nothing on udp/67). The ledger's claim of "DHCP .100-.199"
was false; corrected in place.
OPNsense 26.1 uses KEA. ISC dhcpd is gone, so an old-style <dhcpd> block would have been
inert -- which is the trap this could easily have fallen into.
The Kea shape is NOT invented: it mirrors what OPNsense itself PERSISTS on the live 26.1
box (read back from its config.xml, which its own migrations had populated with an
enabled=0 skeleton), and the field names match the upstream model
src/opnsense/mvc/app/models/OPNsense/Kea/KeaDhcpv4.xml.
Render script now derives and VALIDATES, rather than trusting inputs:
- LAN_NETWORK_CIDR derived from LAN_IPADDR/LAN_SUBNET_BITS (the subnet must be the
NETWORK, not the host address -- 10.12.8.1/22 -> 10.12.8.0/22).
- subnet4 uuid is a DETERMINISTIC uuid5 of the LAN CIDR, so re-rendering is idempotent
and does not churn the config.
- DHCP_POOL_START/END are REQUIRED and validated: a pool outside the LAN subnet, a pool
that swallows the router's own LAN address, or reversed bounds all FAIL LOUD. Each of
those is XML-valid and would serve nobody (or fight the router) -- exactly the class of
silent misconfiguration this session has been finding all day.
Harness 15 -> 24 PASS (T10-T13, incl. three negative cases). Gauntlet ALL GREEN (52);
repo-lint 0 fail.
NOT YET APPLIED to the live edge -- that is a separate gated mutation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-192: the edge config silenced its own ONLY console (no serial, no video device)
...
Third lockout-class defect in the same config, found by executing D-112(c) for real.
After the config was applied and the box rebooted, the serial console went PERMANENTLY
SILENT -- no login prompt, no output, nothing (measured: raw console read returns empty).
The edge VM has NO VIDEO DEVICE, so the serial port is its only console and its only
break-glass path.
Cause: the config set neither <primaryconsole> nor <serialspeed>, so the applied config
did not select the serial console. Boot output still appears (the loader honours the nano
image's own /boot.config -S115200), which is why this looks like a hang rather than a
config choice -- the console dies exactly when OPNsense applies our config.
The box was NOT lost: DOCFIX-191 had just enabled sshd + installed the service key, so it
stayed reachable. That is the only reason this was a finding and not an outage. Two
lockout bugs in one config, and the first one saved us from the second.
Key names VERIFIED against upstream src/www/system_advanced_admin.php:
- `primaryconsole` and `serialspeed` are STRING values.
- There is NO `enableserial` key in OPNsense -- that is a pfSense-ism. Do not add it.
115200 matches the nano image's own /boot.config.
Harness 12 -> 15 PASS. T9b asserts the serial console is retained; T9c guards against
someone "fixing" this by re-adding the bogus pfSense <enableserial> key.
(T9c initially failed on my own template COMMENT, which names enableserial while warning
against it. Tightened to match an actual element. The harness caught it.)
Gauntlet ALL GREEN (52); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-191: the rendered edge config would have LOCKED MANAGEMENT OUT (no sshd, no key)
...
Found while building the D-112(c) bootstrap. The config template had:
<ssh><group>admins</group></ssh>
and no authorized key on root. That is OPNsense's DISABLED-sshd shape. So the config
we were about to push would have left the edge with sshd off and no key installed --
reachable only from the serial console. Under D-112(c), SSH *is* the management path,
so this is a defect in the config itself, independent of how it gets delivered. Had the
Importer worked back in July, we would have shipped an unmanageable edge and blamed the
network.
Element names are VERIFIED against upstream, not guessed:
- src/www/system_advanced_admin.php: `enabled` is the literal STRING "enabled";
`permitrootlogin` / `passwordauth` are PRESENCE-checked (isset), so omitting
passwordauth leaves password auth OFF.
- src/etc/inc/auth.inc: `$keys = base64_decode($user['authorizedkeys']);` -- the
<user><authorizedkeys> value is BASE64 and is written to ~/.ssh/authorized_keys.
Changes:
- template: <ssh> gains <enabled>enabled</enabled> + <permitrootlogin>; root <user>
gains <authorizedkeys>{{ROOT_AUTHORIZED_KEYS_B64}}</authorizedkeys>.
- render script: ROOT_AUTHORIZED_KEYS is now REQUIRED (no default -- an edge rendered
without a key is a lockout waiting to happen). PUBLIC key material only; the private
key is never read by this repo. Base64 is computed AFTER the req() loop so a missing
key fails with the friendly message, not a `set -u` crash.
- Key-only auth is deliberate; the serial console stays the break-glass fallback.
Harness 9 -> 12 PASS. T7b asserts sshd is actually enabled; T7c asserts <authorizedkeys>
base64-DECODES back to the supplied key (a round-trip, not a presence check -- a
wrongly-encoded key yields a well-formed config that silently grants no access); T8
asserts a missing key FAILS LOUD.
Note: the harness's existing T1c well-formed-XML check caught a bug in this very change
(an XML comment containing "--", which is illegal). Fixed. That is the harness earning
its keep.
Gauntlet ALL GREEN (52); repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
Office1 OPNsense edge build: real-ISP-router config + first-boot fixes
...
Bringing the Office1 OPNsense edge online (router + DHCP for office1-local)
before the Office1 NetBox VM. First-ever OPNsense boot in this repo; several
first-execution findings fixed.
DOCFIX-184: opnsense-prep-image.sh decompresses the .img.bz2 via python3's
stdlib bz2 when bunzip2 is absent (no sudo install needed). Harness 3/3.
DOCFIX-185: the OPNsense edge is a normal simulated-ISP router, not an
egress-airgap. Operator clarified the transport model -- each site (DCs +
Office) has its own real ISP connection; dark fiber is East-West/replication
only; D-107 node-airgap is a separate DC concern. Stripped the WAN
egress-control rules (seq 20/21/99) + MIRROR_* tokens from the config template,
render script, and harness (8/8). Flagged a D-100/D-107 amendment note as
follow-up.
DOCFIX-186: instantiate module "office1_opnsense" in main.tf (first real
validation of modules/opnsense-edge). Two infra findings: (1) config_iso_path
must be OUTSIDE the pool dir (create.content.url collides with its own target
volume) -> stage the ISO; (2) apparmor blocks the custom /var/lib/libvirt/vr1/
pool path (foundational -- all VR1 VMs) -> needs a one-time local apparmor rule
(operator sudo). office1-wan NAT network created via virsh (D-103 debt).
repo-lint 0 fail; opnsense-prep-image 3/3; opnsense-render-config 8/8.
Boot blocked only on the apparmor fix (pending operator sudo).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-11 |
D-111: align VR1 v6 subcarve to deployed NN mnemonic; fix ULA-gen doc bug
...
Basing VR1 NetBox on the real netbox.baldurkeep.com modeling (operator
directive). A read-only live export confirmed VR0-DC0/Willamette use an NN
net-byte mnemonic with /60-per-plane + /64-active; dc-dc-prefixes-import.py
did not match it (arbitrary contiguous indices, /64-only).
DOCFIX-182 / D-111: carve_v6 replaces carve_ula/carve_gua -- provider :10
(+API-VIP :11) GUA, metal /60 shared by admin :20 / internal :21, data :30,
storage :40, repl :50, all /60+/64 ULA; the ULA /56 is indexed to the GUA site
nibble so the 4th hextet reads DC.NN in both families. Direct-math _sub_at (no
subnet enumeration -- preserves the DOCFIX-181 no-hang property). Harness
40 -> 53 PASS. D-111 ADOPTED in design-decisions.md; scope-doc sub-decision #1
ratified.
DOCFIX-183: fix the invalid-IPv6 ULA-generation sed in the netem/ULA proposal
doc (4+6 -> 2+4+4 hextets, caught by actually running it); ratified VR1 ULA is
fd50:840e:74e2::/48.
repo-lint 0 fail; tests/dc-dc-prefixes-import 53/53.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DOCFIX-181: fix carve_gua subnet-enumeration hang (gauntlet was stalling)
...
netbox/dc-dc-prefixes-import.py::carve_gua() list()-ed every /64 subnet of the
GUA prefix (2**24 = ~16.7M for D-101's /40 example) only to use the first ->
hung/OOM. Real import runs would hang identically, not just the test. Fixed to
take the first /64 lazily via next() (O(1)).
Added a faulthandler watchdog to tests/dc-dc-prefixes-import/test_logic.py so a
future blocking regression self-aborts with a traceback after 30s instead of
silently stalling the whole run-tests-all.sh gauntlet -- the false-green that
let the ledger claim "40/40" for a test that never actually completed.
tests/dc-dc-prefixes-import: 40/40 in 0.05s (was: hung indefinitely).
Full gauntlet: ALL GREEN (52 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-174: fix self-inflicted false positive in ledger-scan.sh next-free scan
...
DOCFIX-173's own changelog narrated its {3}->{3,} regex fix using a literal
illustrative example ("DOCFIX-1004"), which the widened regex then matched
as a real assignment -- inflating DOCFIX next-free from 174 to 1005.
Reworded the illustrative examples in prose, added a reproduc-line
exclusion to nextfree_mentions() as defense-in-depth, and added a
regression fixture to the test harness (37/37 PASS).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-166: $DC parameterization for reenroll-hosts/carve-host-interfaces/phase-00-maas-standup (gap #15, CLI half)
...
Each script gained an opt-in $DC env var, calling lib_net_select_dc/
lib_hosts_select_dc immediately after sourcing (unset $DC is byte-for-byte
unchanged VR0/DC0 behavior -- verified directly, no fixtures needed since
the selector calls sit before any MAAS call). DC=dc1 bash
scripts/phase-00-maas-standup.sh is now a real, correct, gated call
(D-101 makes dc1's target identical to dc0's); DC=dc1/dc2 for the other
two scripts still fails loud at the host selector until real per-DC host
data exists. New/extended test cases across three harnesses. Six of gap
#15's seven sub-findings remain open -- said plainly, not overclaimed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-165: Ceph replication tooling scripts, radosgw multisite + rbd-mirror (gap #5)
...
Extracts the Stage 6 runbook's manual command sequences into real,
parameterized scripts: scripts/dc-dc-radosgw-multisite.sh (master-init/
join-readonly/enable-two-way), scripts/dc-dc-rbd-mirror.sh
(bootstrap-primary/bootstrap-secondary --direction), scripts/dc-dc-dr-drill.sh
(failover/failback). The single most safety-critical property in the whole
drill -- demote the current primary before promoting the recovering side
during failback -- is hard-coded as fixed call order, not left to
documentation. Dry-run by default; --apply required for real execution,
plus a typed confirmation for the drill script. 19/19, 19/19, 23/23 tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
