diff --git a/docs/changelog-20260715-fidelity-check-scope-correctness.md b/docs/changelog-20260715-fidelity-check-scope-correctness.md new file mode 100644 index 0000000..7a4983d --- /dev/null +++ b/docs/changelog-20260715-fidelity-check-scope-correctness.md @@ -0,0 +1,83 @@ +# 2026-07-15 -- C2 sub-task (i): sandbox re-verified faithful; fidelity checker scope false-positive fixed + +## Context + +C2 (Stage 2 close-out) sub-task (i), per DOCFIX-195: re-run the HARDENED fidelity +check against `office1-netbox` (the VR1 IPAM apex, a working draft), because the +"faithful" verdict predated the 2026-07-14 hardening. Executed this session from the +jumphost, off the newly-consolidated sandbox token (see SEC-009 note below) through +an SSH tunnel to `10.10.1.10:8000`. + +## What happened + +1. **Read-only dump** of `office1-netbox` via `netbox/prod-draft-dump.py` (token + sourced from `~/vr1-office1-creds/vr1-netbox-sandbox.env`, never printed). Sandbox: + 3 RIRs / 29 roles / 5 regions / 11 sites / 5 aggregates / 134 prefixes. + +2. **The hardened checker flagged 2 problems** -- `10.10.0.0/16` (office) and + `172.30.0.0/16` (edge) "have NO SCOPE" -- exit 1. + +3. **Investigated rather than accepted the failure.** Both are role-level container + /16s that `netbox/d115-office-carve.py` creates with `scope=None` **by design** + (lines 74/84). And the upstream apex draft ALREADY carries **27 unscoped** role + containers (`10.0.0.0/8`, `172.16.0.0/12`, `23.157.124.0/24`, ...). So unscoped + role containers are the established convention; these two conform. A full + cross-check of all 44 delta prefixes against their INTENDED scope (from the two + importers) was **44/44 exact** -- every DC prefix bound to the CORRECT DC site + (f02/nibble-2 -> vr1-dc0, f03/nibble-3 -> vr1-dc1), no wrong-target binding, no + hidden scope drop. **The sandbox is faithful.** The checker over-flagged. + +4. **The checker had a false-positive defect.** Its 2026-07-14 hardening added a + "DELTA prefix has NO SCOPE -> fail" rule to catch silent scope drops, but it + over-corrected: it failed ANY unscoped delta prefix, including the legitimately- + unscoped role containers. A checker that reds on a faithful sandbox is as useless + as one that greens on an unfaithful one, so it had to be fixed before sub-task (i) + could close on a real exit 0. + +## The fix (`netbox/sandbox-fidelity-check.py` + harness) + +- New `EXPECTED_PREFIX_SCOPE`: the INTENDED scope per delta prefix -- `("site",slug)` + / `("region",slug)` / `None` for an intentionally-unscoped org container. The DC + half is generated by `_dc_prefix_scopes()` (the D-119 DC->site identity), so the + CIDR set and the scope map cannot drift apart; `EXPECTED_NEW["ipam/prefixes"]` is + now `set(EXPECTED_PREFIX_SCOPE)`. +- The delta check now asserts **scope == INTENDED** (reading the dumper's + slug-rewritten `scope_site`/`scope_region`), not merely "has a scope." This kills + the false-positive AND is strictly STRONGER: it still catches a dropped scope, and + now also catches a **WRONG-TARGET** binding -- a DC0 prefix bound to `vr1-dc1`, the + exact D-117 near-miss the old presence-only test was blind to. +- **NOT fixed by exempting the two CIDRs** (that would re-open a false-green hole on + precisely those prefixes). Intended-None is asserted like any other intended scope. +- Harness (`tests/sandbox-fidelity-check/run-tests.sh`): fixtures moved to the dump + format (`scope_site`), T1 now sets each planned prefix's scope from the checker's + own `EXPECTED_PREFIX_SCOPE`, and two cases added -- **T9** (wrong-target scope -> + fail) and **T10** (legitimately-unscoped role container -> pass). 10 -> 12 tests. + +## Result + +- Re-run against the real sandbox dump: **exit 0** -- "SANDBOX = upstream draft + + EXACTLY the planned delta (D-115/D-117/D-118). Nothing lost, nothing stray." +- Harness **12/12**; gauntlet **ALL GREEN (58)**; repo-lint 0 fail (1 legacy WARN). +- **C2 sub-task (i) is CLOSED** -- the sandbox is faithful, established by a checker + that no longer false-fails. Sub-task (ii) (load + verify the D-120 ip-ranges + + service IPs) remains, with its logged tooling gap (dumper/checker don't yet cover + ip-ranges/ip-addresses). + +## SEC-009 note (sandbox token consolidation) + +The SEC-009 consolidation had MISSED the sandbox NetBox token: it lived only at +`/root/netbox-secrets/api.token` on office1-netbox, never in `~/vr1-office1-creds/`, +contrary to the standing convention ("ALL sensitive files for a site live in +`~/-creds/`"). Consolidated this session to +`~/vr1-office1-creds/vr1-netbox-sandbox.env` (0600), token copied off the VM without +being printed. Recorded in the SEC-009 row. + +## Revert + +- `git checkout HEAD -- netbox/sandbox-fidelity-check.py tests/sandbox-fidelity-check/run-tests.sh` + restores the pre-fix checker (which false-fails on the two role-container /16s). +- `git checkout HEAD -- docs/security-ledger.md docs/session-ledger.md` reverts the notes. +- `git rm docs/changelog-20260715-fidelity-check-scope-correctness.md`. + +No NetBox state was changed by any of this -- the dump and the check are read-only, +and the token consolidation is a jumphost-local file copy. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index b715b1b..76ed6a4 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -28,3 +28,14 @@ sessions and after compaction. Rationale is looser-security-but-strict-hygiene: this is a closed in-house test, so real credential hardening (rotation, revocation, non-shared tokens) is deferred to post-teardown redeploy cycles (test2/3/4+), but consolidation happens NOW so nothing is lost. + +**ADDENDUM (2026-07-15): the sandbox NetBox token was MISSED by the original consolidation.** It lived +only at `/root/netbox-secrets/api.token` on `office1-netbox`, never in `~/vr1-office1-creds/` -- the +prior consolidation moved the three loose `~` env files but did not pull this VM-resident secret into +the folder, contrary to the standing convention ("ALL sensitive files for a site live in +`~/-creds/`"). Found while starting C2 sub-task (i) (the fidelity re-check needed it). Now +consolidated to `~/vr1-office1-creds/vr1-netbox-sandbox.env` (mode 0600; `NETBOX_URL=http://10.10.1.10:8000` ++ the assembled `nbt_.` token), copied off the VM without being printed into the +session. The `/root/netbox-secrets/` copy remains the source-of-record on the VM; this is a working +copy on the jumphost, mirroring how `vr1-netbox.env` holds the upstream token. See +`docs/changelog-20260715-fidelity-check-scope-correctness.md`. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index ab61e41..7665842 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -199,6 +199,16 @@ REDEFINED by DOCFIX-195: `office1-netbox` is the VR1 IPAM apex (the working draft every consuming system reads); `netbox.baldurkeep.com` is the production apex, held as a READ-ONLY v1 reference draft, and the write-back to it is DEFERRED to end-of-deployment -- NOT a Stage 2 gate). + **C2 sub-task (i) DONE 2026-07-15** -- the hardened fidelity check was re-run against `office1-netbox` + (read-only dump via the consolidated sandbox token + an SSH tunnel), and the sandbox is PROVEN + faithful: exit 0, "upstream draft + EXACTLY the planned delta." Along the way the hardened checker's + own scope rule was found to FALSE-FAIL on the two intentionally-unscoped role-container /16s + (`10.10.0.0/16`, `172.30.0.0/16`); fixed to assert scope == INTENDED (now also catches a WRONG-TARGET + binding -- the D-117 near-miss), harness 10->12, gauntlet ALL GREEN (58). SEC-009 gap closed: the + sandbox token was never consolidated; it now lives at `~/vr1-office1-creds/vr1-netbox-sandbox.env`. + See `docs/changelog-20260715-fidelity-check-scope-correctness.md`. **C2 sub-task (ii) STILL OPEN** -- + load + verify the D-120 `ip-ranges`/`ip-addresses` (the logged tooling gap: dumper+checker don't yet + cover those object classes). 3. **`pynetbox` -- CLOSED 2026-07-13.** Installed (7.0.0) on `office1-netbox`, which is where the imports actually run. The stdlib tools (`prod-draft-dump`/`sandbox-seed`/`sandbox-fidelity-check`) deliberately need no dependency at all. diff --git a/netbox/sandbox-fidelity-check.py b/netbox/sandbox-fidelity-check.py index 79057fe..028e5a4 100755 --- a/netbox/sandbox-fidelity-check.py +++ b/netbox/sandbox-fidelity-check.py @@ -46,21 +46,50 @@ # D-117 VR1 DC0/DC1 six-plane (36 prefixes -- 18 per DC) # D-118 the org ULA /48 (RIRs + aggregates) # G1/G2 six-plane roles + aggregates -def _dc_prefixes(): - out = set() - for v4 in ["10.12.4.0/22","10.12.8.0/22","10.12.12.0/22","10.12.16.0/22", - "10.12.32.0/22","10.12.36.0/22", # VR1 DC0 (inherited) - "10.12.64.0/22","10.12.68.0/22","10.12.72.0/22","10.12.76.0/22", - "10.12.80.0/22","10.12.84.0/22"]: # VR1 DC1 (D-115 /19) - out.add(v4) - for dc in ("f02", "f03"): - out |= {f"2602:f3e2:{dc}:10::/60", f"2602:f3e2:{dc}:10::/64", f"2602:f3e2:{dc}:11::/64"} - for n in ("2", "3"): # DC nibble +# DC delta prefixes AND their INTENDED scope (the DC's own site). The DC->site +# binding is the D-119 identity: GUA f02 / ULA nibble 2 / the first six /22s -> +# vr1-dc0; f03 / nibble 3 / the last six -> vr1-dc1. Built in ONE place so the CIDR +# set and its scope map cannot drift apart. +def _dc_prefix_scopes(): + scope = {} + v4 = {"vr1-dc0": ["10.12.4.0/22","10.12.8.0/22","10.12.12.0/22", + "10.12.16.0/22","10.12.32.0/22","10.12.36.0/22"], # VR1 DC0 (inherited) + "vr1-dc1": ["10.12.64.0/22","10.12.68.0/22","10.12.72.0/22", + "10.12.76.0/22","10.12.80.0/22","10.12.84.0/22"]} # VR1 DC1 (D-115 /19) + for site, cidrs in v4.items(): + for c in cidrs: + scope[c] = ("site", site) + for dc, site in (("f02", "vr1-dc0"), ("f03", "vr1-dc1")): + for c in (f"2602:f3e2:{dc}:10::/60", f"2602:f3e2:{dc}:10::/64", f"2602:f3e2:{dc}:11::/64"): + scope[c] = ("site", site) + for n, site in (("2", "vr1-dc0"), ("3", "vr1-dc1")): # DC nibble for nn in ("20", "30", "40", "50"): - out.add(f"fd50:840e:74e2:{n}{nn}::/60") - out.add(f"fd50:840e:74e2:{n}{nn}::/64") - out.add(f"fd50:840e:74e2:{n}21::/64") # metal-internal shares metal /60 - return out + scope[f"fd50:840e:74e2:{n}{nn}::/60"] = ("site", site) + scope[f"fd50:840e:74e2:{n}{nn}::/64"] = ("site", site) + scope[f"fd50:840e:74e2:{n}21::/64"] = ("site", site) # metal-internal shares metal /60 + return scope + +def _dc_prefixes(): + return set(_dc_prefix_scopes()) + +# INTENDED scope per DELTA prefix: ("site", slug) / ("region", slug) / None for an +# intentionally UNSCOPED org-level role container. The two role /16s (office, edge) +# are unscoped BY DESIGN in d115-office-carve.py -- mirroring the 27 unscoped role +# containers ALREADY in the upstream apex (10.0.0.0/8, 172.16.0.0/12, ...). Checking +# scope == INTENDED (not merely "has a scope") kills the false "NO SCOPE" failure on +# those two AND still catches a DROPPED scope (the 17-prefix region loss) or a +# WRONG-TARGET binding (a DC0 prefix bound to vr1-dc1 -- the D-117 near-miss). +EXPECTED_PREFIX_SCOPE = { + "10.10.0.0/16": None, # office role container -- unscoped by design + "172.30.0.0/16": None, # edge role container -- unscoped by design + "10.10.0.0/22": ("site", "vr1-off1"), + "10.10.0.0/24": ("site", "vr1-off1"), + "10.10.1.0/24": ("site", "vr1-off1"), + "172.30.1.0/24": ("site", "vr1-off1"), + "2602:f3e2:f01:100::/56": ("site", "vr1-off1"), + "2602:f3e2:f01:100::/64": ("site", "vr1-off1"), + **_dc_prefix_scopes(), +} EXPECTED_NEW = { "ipam/roles": {"edge", "provider-public", "metal-admin", "metal-internal", @@ -69,9 +98,7 @@ "ipam/rirs": {"rfc-4193-ula", "rfc-1918"}, "ipam/aggregates": {"2602:f3e2::/36", "23.157.124.0/24", "10.0.0.0/8", "172.16.0.0/12", "fd50:840e:74e2::/48"}, - "ipam/prefixes": {"10.10.0.0/16","10.10.0.0/22","10.10.0.0/24","10.10.1.0/24", - "172.30.0.0/16","172.30.1.0/24", - "2602:f3e2:f01:100::/56","2602:f3e2:f01:100::/64"} | _dc_prefixes(), + "ipam/prefixes": set(EXPECTED_PREFIX_SCOPE), } bad=0 for ep,k in KEY.items(): @@ -102,15 +129,27 @@ print( " The planned delta was NOT fully applied. A subset-only check would") print( " have called this run clean -- that is the false green this catches.") - # DELTA FIELD-CORRECTNESS. The shared-object drift loop below cannot see these - # (they exist only in the sandbox), so a new prefix created with a MISSING scope - # or MISSING role passed clean. That is the exact shape of the bug that silently - # dropped the region scope on 17 of 90 prefixes and still matched every count. + # DELTA SCOPE-CORRECTNESS. The shared-object drift loop below cannot see these + # (they exist only in the sandbox). Assert each delta prefix's scope == its + # INTENDED scope (EXPECTED_PREFIX_SCOPE), target and all -- reading the dumper's + # slug-rewritten scope_site/scope_region (scope_id is popped on the way in). This + # catches a DROPPED scope (the 17-prefix region loss), a WRONG-TARGET binding (a + # DC0 prefix bound to vr1-dc1 -- the D-117 near-miss), and a spurious scope on an + # intentionally-unscoped container -- WITHOUT false-flagging the role /16s that + # are unscoped by design (which the old "has a scope?" heuristic wrongly failed). if ep == "ipam/prefixes": for key in sorted(extra & exp): rec = S[key] - if not rec.get("scope_type") and not rec.get("scope_id"): - bad+=1; print(f" !! DELTA prefix {key} has NO SCOPE (unscoped -- it belongs to no site/region)") + if rec.get("scope_site"): + actual = ("site", rec["scope_site"]) + elif rec.get("scope_region"): + actual = ("region", rec["scope_region"]) + else: + actual = None + if key not in EXPECTED_PREFIX_SCOPE: + bad+=1; print(f" !! DELTA prefix {key}: no intended scope encoded (extend EXPECTED_PREFIX_SCOPE)") + elif actual != EXPECTED_PREFIX_SCOPE[key]: + bad+=1; print(f" !! DELTA prefix {key}: scope {actual} != intended {EXPECTED_PREFIX_SCOPE[key]}") if not rec.get("role"): bad+=1; print(f" !! DELTA prefix {key} has NO ROLE") # FIELD-LEVEL fidelity on the shared keys -- this is the check that counts cannot make diff --git a/tests/sandbox-fidelity-check/run-tests.sh b/tests/sandbox-fidelity-check/run-tests.sh index daa68d5..0c8e2f7 100755 --- a/tests/sandbox-fidelity-check/run-tests.sh +++ b/tests/sandbox-fidelity-check/run-tests.sh @@ -30,10 +30,14 @@ } SHARED='{"prefix":"10.99.0.0/24","status":"active","role":"storage","scope_type":"dcim.site","scope_id":1,"description":"shared","is_pool":false,"mark_utilized":false}' -# an EXPECTED delta prefix (in EXPECTED_NEW via _dc_prefixes(): VR1 DC0 inherited v4) -DELTA_OK='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":"dcim.site","scope_id":8,"description":"VR1 DC0 provider-public","is_pool":false,"mark_utilized":false}' +# an EXPECTED delta prefix in DUMP FORMAT (scope_site slug, as prod-draft-dump.py +# rewrites it; scope_id is popped on the way in). 10.12.4.0/22 is intended for site +# vr1-dc0 per EXPECTED_PREFIX_SCOPE. +DELTA_OK='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":"dcim.site","scope_site":"vr1-dc0","description":"VR1 DC0 provider-public","is_pool":false,"mark_utilized":false}' # the SAME expected prefix but UNSCOPED -- the 17-prefix-scope-drop bug's shape -DELTA_NOSCOPE='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":null,"scope_id":null,"description":"VR1 DC0 provider-public","is_pool":false,"mark_utilized":false}' +DELTA_NOSCOPE='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":null,"description":"VR1 DC0 provider-public","is_pool":false,"mark_utilized":false}' +# the SAME expected prefix bound to the WRONG DC site (vr1-dc1) -- the D-117 near-miss +DELTA_WRONGSITE='{"prefix":"10.12.4.0/22","status":"active","role":"provider-public","scope_type":"dcim.site","scope_site":"vr1-dc1","description":"wrong site","is_pool":false,"mark_utilized":false}' # a stray object nobody planned STRAY='{"prefix":"192.0.2.0/24","status":"active","role":"storage","scope_type":"dcim.site","scope_id":1,"description":"stray","is_pool":false,"mark_utilized":false}' @@ -73,8 +77,14 @@ rec = {key: v} if ep == "ipam/prefixes": rec.update({"status":"active","role":"provider-public", - "scope_type":"dcim.site","scope_id":8, "description":"planned","is_pool":False,"mark_utilized":False}) + sc = ns["EXPECTED_PREFIX_SCOPE"].get(v) # INTENDED scope, harvested from the checker itself + if sc is not None: + kind, slug = sc + rec["scope_type"] = "dcim.site" if kind == "site" else "dcim.region" + rec["scope_site" if kind == "site" else "scope_region"] = slug + else: + rec["scope_type"] = None # intentionally-unscoped org container rows.append(rec) sandbox[ep] = rows json.dump(upstream, open(f"{tmp}/u1.json","w")) @@ -113,19 +123,37 @@ [ "$rc" = 1 ] && grep -q "MISSING FROM SANDBOX" "$TMP/out" && ok "T5 a lost upstream object -> MISSING, exit 1" \ || { no "T5 should fail on a lost object (rc=$rc)"; sed 's/^/ /' "$TMP/out"; } -# T6: DELTA FIELD-CORRECTNESS. An EXPECTED new prefix created with NO SCOPE must -# fail. The shared-object drift loop CANNOT see this (the object exists only in -# the sandbox) -- it is precisely the 17-prefix scope-drop bug, relocated into -# the delta where the old check was blind to it. +# T6: DELTA SCOPE-CORRECTNESS. An EXPECTED new prefix whose INTENDED scope was +# DROPPED (10.12.4.0/22 should be site vr1-dc0, arrives unscoped) must fail. The +# shared-object drift loop CANNOT see this (the object exists only in the +# sandbox) -- it is precisely the 17-prefix scope-drop bug, relocated into the +# delta where the old check was blind to it. rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_NOSCOPE")") -grep -q "has NO SCOPE" "$TMP/out" && ok "T6 an expected DELTA prefix with no scope -> caught" \ - || { no "T6 delta field-correctness is blind again"; sed 's/^/ /' "$TMP/out"; } -[ "$rc" = 1 ] && ok "T6 an unscoped delta prefix exits 1" || no "T6 unscoped delta exits $rc" +grep -q "!= intended" "$TMP/out" && ok "T6 an expected DELTA prefix with its scope DROPPED -> caught" \ + || { no "T6 delta scope-correctness is blind again"; sed 's/^/ /' "$TMP/out"; } +[ "$rc" = 1 ] && ok "T6 a scope-dropped delta prefix exits 1" || no "T6 scope-dropped delta exits $rc" -# T7: a correctly-formed expected delta prefix is NOT flagged as unscoped/roleless +# T7: a correctly-SCOPED expected delta prefix (right target) is NOT flagged rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_OK")") -grep -q "has NO SCOPE" "$TMP/out" && no "T7 a well-formed delta prefix was wrongly flagged" \ - || ok "T7 a well-formed delta prefix is NOT flagged (guard is not a blanket refusal)" +grep -q "!= intended" "$TMP/out" && no "T7 a correctly-scoped delta prefix was wrongly flagged" \ + || ok "T7 a correctly-scoped delta prefix is NOT flagged (guard is not a blanket refusal)" + +# T9: WRONG-TARGET scope. 10.12.4.0/22 is intended for vr1-dc0; bound to vr1-dc1 it +# must fail. This is the D-117 near-miss (a DC's prefixes silently bound to the +# OTHER DC's site) -- a mere "has a scope?" test was blind to it; asserting scope +# == INTENDED catches it. +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_WRONGSITE")") +[ "$rc" = 1 ] && grep -q "!= intended" "$TMP/out" && ok "T9 a WRONG-TARGET scope (DC0 prefix on vr1-dc1) -> caught, exit 1" \ + || { no "T9 wrong-target scope not caught (rc=$rc)"; sed 's/^/ /' "$TMP/out"; } + +# T10: the legitimately-UNSCOPED role container (10.10.0.0/16, intended None) present +# as unscoped must NOT be flagged. This is the exact false-positive the fix +# removes: the old "has a scope?" heuristic FAILED a faithful sandbox on this +# prefix (mirroring the 27 unscoped role containers already in the upstream apex). +DELTA_UNSCOPED_OK='{"prefix":"10.10.0.0/16","status":"container","role":"office","scope_type":null,"description":"office role container","is_pool":false,"mark_utilized":false}' +rc=$(run "$(up_json)" "$(printf '{"ipam/prefixes":[%s,%s]}' "$SHARED" "$DELTA_UNSCOPED_OK")") +grep -q "!= intended" "$TMP/out" && no "T10 an intentionally-unscoped role container was wrongly flagged (false positive is back)" \ + || ok "T10 an intentionally-unscoped role container (10.10.0.0/16) is NOT flagged" # T8: read-only -- no write verbs anywhere in the checker grep -qE '\.(create|update|delete|save)\(' "$CHECK" \