| 2026-07-14 |
NetBox write-path hardening: the bugs that would have corrupted the apex at C2
...
Found by adversarial review during D-119; independent of the naming work. These
are what would have bitten during C2 (the production IPAM write Stage 2 needs).
1. roles-aggregates-import.py STILL died half-way through a production write. The
preflight added on 2026-07-13 -- after this same script committed 4 roles, 400'd
on the 5th and left the apex half-populated -- only covered role-NAME collisions.
The ARIN RIR lookup and validate_ula_48() both still ran BELOW the first write,
so an apex without ARIN, or a typo'd ORG_ULA_48, would commit 5 roles (and the
RIRs) and THEN die, with no rollback. The exact failure the preflight was added
to prevent, still armed one section lower. The RIR half of the preflight loop was
a literal no-op. Both checks moved INTO the preflight.
A preflight that does not cover every die() downstream of the first write is not
a preflight.
2. sandbox-fidelity-check.py could return a FALSE GREEN -- and it is the script we
have been citing as PROOF the sandbox is a faithful replica. It field-compared
only SHARED objects; the planned delta was checked merely as a SUBSET of
EXPECTED_NEW. With ZERO of the 36 DC prefixes created, extra={} -> unexpected={}
-> it printed "Nothing lost, nothing stray" and exited 0. EXPECTED_NEW was an
upper bound masquerading as an assertion. It is now BOTH bounds, and delta
prefixes are checked for scope and role (the 17-prefix scope-drop bug relocated
into the delta, where the old check was structurally blind).
3. dc-dc-prefixes-import.py died mid-loop on a missing role -- find_role() was
resolved lazily INSIDE the write loop. Against upstream as it stands TODAY (no
six-plane roles), a --commit creates the site + 4 prefixes then dies on the 5th:
a half-written datacenter. Now a whole-plan preflight resolves every role first.
4. The test fake was KINDER THAN REALITY at the failure point: fake_pynetbox.get()
returned matches[0] on a multi-match where real pynetbox RAISES. NetBox permits
duplicate prefixes (the importer's own docstring anticipates them vs vr0-dc0), so
this would blow up in production while the harness sailed through. It now raises.
TESTS: tests/sandbox-fidelity-check/ is NEW (10/10) -- the checker shipped with NO
harness at all, which is exactly why the false green survived; T3 reproduces it, and
T1 derives its expectation from the checker's own EXPECTED_NEW rather than re-typing
it. roles-aggregates-import 16->19, asserting BY POSITION that ARIN and the ULA are
validated before the first create (the old harness only checked the WORD "PREFLIGHT"
appeared -- green while the landmine was armed).
GAUNTLET ALL GREEN (58). repo-lint 0 fail. Zero NetBox writes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-119: region-qualify the VR1 DC namespace -- C3 / gap #19 CLOSED (Stage 3 unblocked)
...
THE BUG: the token `dc0` meant TWO DIFFERENT CLOUDS depending on the file --
VR0's LIVE testcloud in scripts/lib-net.sh, VR1's FIRST DC in the NetBox
importer. One string, two clouds, one of them in production.
THE RULING (operator): the repo adopts the apex's names verbatim.
vr0-dc0 VR0's DC0 the LIVE testcloud (a DIFFERENT region)
vr1-dc0 VR1's FIRST DC GUA 2602:f3e2:f02::/48
vr1-dc1 VR1's SECOND DC GUA 2602:f3e2:f03::/48
Bare dc0/dc1/dc2 are now REJECTED LOUDLY everywhere.
D-119 COMPLETES D-117, it does not reverse it: it executes D-117's own amendment
across the three surfaces D-117 never touched. ZERO NetBox writes -- the apex was
already correct and self-consistent (MEASURED: f02::/48 -> vr1-dc0, f03::/48 ->
vr1-dc1); the REPO was the only surface out of step. Renaming the apex to match
the repo was considered and REJECTED (production IPAM write; VR1 would become the
only 1-indexed region; and vr1-dc1 would mean VR1's FIRST DC while vr0-dc1 means
VR0's SECOND).
THE PRIZE: the importer's DC->site map is now an IDENTITY -- there is no offset
table left to get wrong, and an assert enforces it. The original defect was
precisely a WRONG LOOKUP TABLE. The bug class is deleted, not defended.
TWO REAL BUGS the naming fix ALONE would NOT have closed (found by review sweep):
- BREAK-1: descriptions were built with f"VR1 {dc.upper()} ..." -- under D-119 that
renders "VR1 VR1-DC0 provider-public" on all 36 prefixes. Same class as the
original bug (deriving a label by munging a token), hidden in the description
field where slug-focused review missed it. Now looked up from SITES[name].
- BREAK-2 (the important one): DC_GUA_PREFIX was NEVER cross-checked against --dc.
`--dc vr1-dc0 DC_GUA_PREFIX=<f03>` was ACCEPTED: it writes the SECOND DC's GUA,
carved with the FIRST DC's ULA nibble, scoped to the FIRST DC's site -- a
silently mis-bound datacenter assembled from two disagreeing sources. Identity-
mapping the slug leaves the ADDRESSING free-floating. Now guarded by EXPECTED_GUA.
ALSO: vdc1/vdc2 -> vvr1-dc0/vvr1-dc1 (D-114 amendment). D-114's own DR primitive is
`virsh destroy vdc1`, which read as "destroy VR1 DC1" while MEANING "destroy VR1
DC0". A mislabelled destroy command in a DR drill is not cosmetic. Neither VM is
built, so it is free. voffice1/Office1 KEEP their number (operator ruling).
Env var: DC1_/DC2_V4_SUPERNET -> VR1_DC1_V4_SUPERNET (both old names rejected).
rbd-mirror's --site-name aligned: `--dc vr1-dc0 --site-name dc1` was a re-created
two-namespace collision on one command line.
TESTS: dc-selector 21->30, prefixes-import 40->82 (identity invariant; EXPECTED_GUA
in BOTH mismatch directions; mismatched run writes NOTHING; matching pair still
succeeds; no munged description). GAUNTLET ALL GREEN (57). repo-lint 0 fail.
The old harnesses PINNED THE WRONG MAPPING -- they would have gone green while
enforcing the bug.
THE tofu apply IS NOT DONE -- IT IS GATED. Plan: 11 to add, 0 to change, 11 to
destroy (libvirt object names are ForceNew; moved{} makes the plan reviewable but
cannot suppress a replace). MEASURED SAFE: all 11 are EMPTY -- no guests, no
volumes, Stage 3 hasn't run. Office1's live objects are not in the plan.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
D-118 ADOPTED (org ULA fd50:840e:74e2::/48); VR1 DC0 + DC1 IMPORTED; roles importer bug fixed
...
D-118: ORG_ULA_48 = fd50:840e:74e2::/48. RFC 4193 valid (fd -> L=1, 40-bit Global
ID 0x50840e74e2, /48 -> 256 x /56 for D-101's per-DC /56). The load-bearing check
is NOT the RFC conformance -- it is that it does NOT collide with Tailscale's
fd7a:115c:a1e0::/48: office1-tailscale runs --accept-routes, so an overlap would
have been a LIVE tailnet routing conflict, not a paper one.
Provenance stated honestly: the 'CSPRNG-generated' claim cannot be re-verified.
It does not change the ruling (RFC 4193's SHA1 construction is a SHOULD; the
requirement is a pseudo-random 40-bit Global ID). Regenerating would only swap in
a different random number at the cost of re-editing 7 places.
Corrects a FALSE claim: changelog-20260711 called this value 'ratified' when no
D-number ever assigned it. That phantom ratification is why it read as settled for
two days. Same failure mode as D-117: a document asserting authority it lacked.
IMPORTED into the sandbox: six-plane roles + 2 RIRs + 5 aggregates, then VR1 DC0
(18 prefixes) and VR1 DC1 (18). The D-117 fix is now LIVE-PROVEN, not just green
offline: --dc dc0 -> apex site VR1 DC0 (id=8), --dc dc1 -> VR1 DC1 (id=9). ULA
reads DC.NN in the 4th hextet per D-111 (fd50:840e:74e2:220:: .. :350::).
A REAL BUG, found by running roles-aggregates-import.py for the first time for
real: it checked existence by SLUG only, but NetBox also enforces UNIQUE NAMES.
The draft's legacy role 'Replication' (slug repl) blocks the six-plane role (slug
replication, name Replication) -- so it created 4 roles, 400'd on the 5th, and
DIED, leaving the apex HALF-POPULATED with no RIRs, no aggregates, no rollback.
It had never met a NetBox holding the real draft (only empty ones) and it SHIPPED
WITH NO HARNESS. That is why it survived.
- six-plane role renamed 'Replication Plane' (slug unchanged -- lib-net.sh
SPACES6 and the prefix importer look up 'replication')
- the script now PREFLIGHTS every name/slug and writes NOTHING unless the whole
plan is viable. An IPAM importer that can die partway corrupts what it
populates.
- tests/roles-aggregates-import/ CREATED: 16/16 PASS.
Final state PROVEN by sandbox-fidelity-check (now carrying the full planned
delta): the sandbox is the upstream draft + EXACTLY D-115 + D-117 + D-118 --
every shared object field-identical, nothing lost, nothing stray. 90 -> 134
prefixes.
repo-lint 0 fail. run-tests-all: ALL GREEN (56 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
Prove the sandbox is a FAITHFUL replica, not just a count-matching one
...
netbox/sandbox-fidelity-check.py (+4 assertions in tests/sandbox-seed, 22/22 PASS)
Counts and idempotency CANNOT prove a faithful seed. The region-scope bug proved
it: 17 of 90 prefixes silently lost their scope, every count still matched, and
re-runs were still perfectly idempotent. It was caught by LUCK on a spot-check
that happened to print 'site None'. This is that catch, made deliberate.
Method needs no new plumbing: prod-draft-dump.py is READ-ONLY, so point it at the
sandbox and diff the two JSON dumps FIELD BY FIELD.
RESULT: every shared object is field-level IDENTICAL to upstream, and the delta is
EXACTLY the expected D-115 additions (+edge role, +vr1-off1 site, +8 prefixes).
Nothing missing, nothing unexpected. The sandbox is a PROVEN baseline -- which is
what the later feed-back diff to production depends on.
Also de-misdirects docs/dc-dc-netbox-buildout-scope.md, which still told the next
person to run '--dc dc2' with DC2_V4_SUPERNET -- both of which the new guards now
REJECT by name. Its G5 recommendation (rename the APEX to vr1-dc1/vr1-dc2) is
marked SUPERSEDED: D-117 ruled the opposite. ORG_ULA_48 is now listed there as
UNASSIGNED and BLOCKING, not as a settled literal.
Disclosure: D-117 Option B is PARTIALLY executed -- the import path and the scope
doc are done; the $DC shell selector and the runbook DC prose are NOT. Recorded
in the ledger so 'D-117 ADOPTED' is not misread as 'fully renamed'.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-115 office carve IMPORTED into the Office1 NetBox; DC import blocked on ORG_ULA_48
...
netbox/d115-office-carve.py (+ tests/d115-office-carve/ 20/20 PASS)
Applied to the Office1 sandbox: site vr1-off1, roles office + edge (edge is NEW),
and 8 prefixes. Idempotent (re-run: 0 created / 11 present).
10.10.0.0/16 office container the Office v4 block, /22 per office
10.10.0.0/22 office container VR1 Off1's /22
10.10.0.0/24 office active office1-local LAN (Kea on the edge)
10.10.1.0/24 office active lxdbr0 compose net (MAAS DHCP)
172.30.0.0/16 edge container NEW Edge role; mirrors v6 2602:f3e2:fe::/48
172.30.1.0/24 edge active office1-wan, the simulated ISP uplink
2602:f3e2:f01 :/56 office container mirrors VR0 Off0's e01:/56
2602:f3e2:f01:/64 office active office subnet (NOT deployed; v6 has no egress)
Every address was already RUNNING. D-115 blesses what is deployed -- zero
renumber. What it fixes is that three of them were SQUAT: on the wire with no
allocation behind them.
Writing upstream is GATED IN CODE: the tool refuses a non-sandbox target unless
given --yes-write-upstream. Feeding the production apex is an operator decision,
never a side effect of running an import.
BLOCKED, deliberately: dc-dc-prefixes-import.py --dc dc0 CANNOT run. It needs
ORG_ULA_48, which is still UNASSIGNED (gap G3; the scope doc recommends
fd50:840e:74e2::/48 but no ADOPTED decision assigns it -- grep ORG_ULA
design-decisions.md returns nothing). Using it anyway would be an inferred value
(hard rule 2). This is a DECISION gate, not a tooling gap: the tooling is fixed,
tested and ready. Recorded in the ledger with the exact commands to unblock.
Also: docs/vr1-office1-as-built.md gains the IPAM table, and records that the
Tailscale 10.10.0.0/22 route is now ENABLED (edge GUI reachable from the
workstation with no tunnel), superseding the tunnel instructions.
repo-lint 0 fail. run-tests-all: ALL GREEN (55 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
NetBox sandbox seeded from the upstream draft (129 objects) + the two-phase tooling
...
The Office1 sandbox NetBox was EMPTY, and no seeder existed. The documented loop
(upstream draft -> sandbox -> simulate -> feed back) had no first step.
netbox/prod-draft-dump.py READ-ONLY dump of the upstream draft -> JSON artifact
netbox/sandbox-seed.py applies that JSON to a SANDBOX. Dry by default.
netbox/draft/vr1-draft.json the reviewable artifact (90 prefixes, 23 roles, ...)
TWO PHASES, deliberately, rather than one source->dest script:
- the upstream token never travels (dump runs on vcloud; seed runs on
office1-netbox, where the sandbox token already lives)
- 'the sim NEVER writes upstream' becomes STRUCTURAL: the script that writes
has no upstream credential and no upstream URL, and REFUSES to write to the
instance the draft was read from (self-configuring via the draft's _source,
so it keeps working if upstream ever moves -- a hostname denylist would not)
- the JSON is diffable: you can read what is about to be written
Applied to the sandbox: 129 created, then verified idempotent (0 created / 129
present on re-run). vr1-dc0 -> f02::/48 and vr1-dc1 -> f03::/48, matching the
apex exactly, which is what D-117 exists to guarantee.
TWO BUGS FOUND AND FIXED WHILE BUILDING IT:
1. The dry run mapped only dcim.site scopes, so REGION-scoped prefixes lost
their scope -- 17 of 90 upstream, including the ENTIRE VR1 region-scoped
draft. Caught by a spot-check printing 'site None'. Fixed + reseeded
(--update), now bound to region VR1.
2. The upstream-refusal guard sat AFTER json.load() and after the banner: it
was dead code for any malformed draft and READ as safe. Guards go first.
The harness now proves it actually refuses.
tests/sandbox-seed/: 18/18 PASS. repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
D-117 ADOPTED (B, DC-only): fix the off-by-one site binding + make the importer dry-by-default
...
Operator ruling: rename the DC numbers to match the NetBox apex; the office
keeps its number (Office1 -> new site vr1-off1, so nothing deployed changes).
netbox/dc-dc-prefixes-import.py:
--dc dc1|dc2 -> --dc dc0|dc1, bound to the APEX slugs measured live:
dc0 -> vr1-dc0 (2602:f3e2:f02::/48) dc1 -> vr1-dc1 (f03::/48)
It previously mapped dc1 -> vr1-dc1 while treating f02::/48 as dc1, i.e. it
would have written VR1 DC0's prefixes to the OTHER DC's site.
Now DRY BY DEFAULT with --commit (it used to write with no flag at all).
A dry run against a FRESH NetBox now plans the site too (_PlannedSite) rather
than bailing with 'no site id' -- previewing an empty apex is the whole point.
DC2_V4_SUPERNET is REJECTED BY NAME (retired) rather than silently ignored.
Every 'DCn' in prose is region-qualified: 'DC0' meant VR0's rehearsal DC in
some sentences and VR1's first DC in others. No sed could have caught that.
tests/dc-dc-prefixes-import/: 73 checks ALL PASS, incl. regression guards that
the dry run creates NOTHING, that --dc dc0 never touches vr1-dc1, and that the
invented vr1-dc2 site is never created.
FOUND, RECORDED, NOT YET FIXED (D-117 amendment): scripts/lib-net.sh carries a
SECOND, colliding $DC namespace where dc0 already means VR0's DC0. Renaming
VR1's dc1->dc0 there would collide with VR0. Proposed fix: region-qualify the
shell selector to the apex slugs (vr0-dc0/vr1-dc0/vr1-dc1). Does not block the
import; the two selectors are NOT interchangeable meanwhile.
repo-lint 0 fail. run-tests-all: ALL GREEN (53 harnesses).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ZBxzAbmqwLW7jEmMG23Xw
|
| 2026-07-11 |
D-111: align VR1 v6 subcarve to deployed NN mnemonic; fix ULA-gen doc bug
...
Basing VR1 NetBox on the real netbox.baldurkeep.com modeling (operator
directive). A read-only live export confirmed VR0-DC0/Willamette use an NN
net-byte mnemonic with /60-per-plane + /64-active; dc-dc-prefixes-import.py
did not match it (arbitrary contiguous indices, /64-only).
DOCFIX-182 / D-111: carve_v6 replaces carve_ula/carve_gua -- provider :10
(+API-VIP :11) GUA, metal /60 shared by admin :20 / internal :21, data :30,
storage :40, repl :50, all /60+/64 ULA; the ULA /56 is indexed to the GUA site
nibble so the 4th hextet reads DC.NN in both families. Direct-math _sub_at (no
subnet enumeration -- preserves the DOCFIX-181 no-hang property). Harness
40 -> 53 PASS. D-111 ADOPTED in design-decisions.md; scope-doc sub-decision #1
ratified.
DOCFIX-183: fix the invalid-IPv6 ULA-generation sed in the netem/ULA proposal
doc (4+6 -> 2+4+4 hextets, caught by actually running it); ratified VR1 ULA is
fd50:840e:74e2::/48.
repo-lint 0 fail; tests/dc-dc-prefixes-import 53/53.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Merge origin/main (netbox uploads 057c31b) into stage1 branch -- brings the NetBox buildout scope doc + roles-aggregates tool alongside the carve_gua fix
|
|
|
| 2026-07-10 |
DOCFIX-181: fix carve_gua subnet-enumeration hang (gauntlet was stalling)
...
netbox/dc-dc-prefixes-import.py::carve_gua() list()-ed every /64 subnet of the
GUA prefix (2**24 = ~16.7M for D-101's /40 example) only to use the first ->
hung/OOM. Real import runs would hang identically, not just the test. Fixed to
take the first /64 lazily via next() (O(1)).
Added a faulthandler watchdog to tests/dc-dc-prefixes-import/test_logic.py so a
future blocking regression self-aborts with a traceback after 30s instead of
silently stalling the whole run-tests-all.sh gauntlet -- the false-green that
let the ledger claim "40/40" for a test that never actually completed.
tests/dc-dc-prefixes-import: 40/40 in 0.05s (was: hung indefinitely).
Full gauntlet: ALL GREEN (52 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-160: adversarial review pass finds and fixes 3 real issues
...
Ran a fresh-eyes review subagent across tonight's full deliverable set
(lib-net.sh/lib-hosts.sh selectors, the NetBox DC-DC pipeline, all seven
Stage 1-7 runbooks). Found and fixed:
- A real Python control-flow bug in tests/dc-dc-prefixes-import/
test_logic.py: a `return` inside `try` made the `else` clause
unreachable, silently leaving 3 happy-path assertions uncounted.
37/37 -> 40/40 after the fix (independently reproduced before trusting
the finding).
- A self-contradictory pair of comments in dc-dc-prefixes-import.py about
/19-vs-/22 subnet arithmetic (no functional bug, wrong prose).
- Stale self-referential text in the Stage 4 runbook, written before
Stage 3 existed and never reconciled once it did.
Corrected the 40/40 count in the workflow tracker and session ledger.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-152: NetBox multi-DC/dual-stack import pipeline (gap #3 mechanism)
...
Adds netbox/dc-dc-prefixes-import.py, extending the v1 single-site IPv4-only
NetBox import to VR1's two-DC dual-stack model (D-101). DC1's v4 planes are
hardcoded (D-101: inherited from DC0 unchanged, this is decision text, not
an inference); DC2's v4 supernet, the org ULA /48, and the per-DC GUA carve
are all required env vars with no defaults, failing loud rather than
inventing any of them. 37/37 tests against a fake in-memory NetBox client
(no live NetBox reachable this session). Closes the tooling half of gap #3;
the real literals still need operator/NetBox assignment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
| 2026-07-03 |
|
| 2026-05-22 |
|
