diff --git a/docs/changelog-20260710-carve-gua-hang-fix.md b/docs/changelog-20260710-carve-gua-hang-fix.md new file mode 100644 index 0000000..595ddf4 --- /dev/null +++ b/docs/changelog-20260710-carve-gua-hang-fix.md @@ -0,0 +1,48 @@ +# Changelog 2026-07-10 -- DOCFIX-181: carve_gua subnet-enumeration hang + harness watchdog + +**Context.** After the Stage-1 commit, the full `scripts/run-tests-all.sh` +gauntlet was seen to HANG on `tests/dc-dc-prefixes-import/`. Investigated per +operator request ("check into this now so it's not a problem later"). Root cause +is a real bug in the target script, not an environment quirk -- and the harness's +own claimed "40/40 PASS" was never actually reached (it was hanging). + +## Root cause + +`netbox/dc-dc-prefixes-import.py::carve_gua()` built +`list(dc_gua_prefix.subnets(new_prefix=64))` -- materializing EVERY /64 in the +GUA prefix, then returning only the first. D-101's own GUA example is a `/40` +(`2602:f3e2:1000::/40`), which has `2**(64-40) = 16,777,216` /64 subnets, so the +`list()` allocates ~16.7M `IPv6Network` objects -> effective hang / OOM. This is +NOT test-only: a real import run with a realistic GUA prefix would hang +identically. (`carve_ula` is safe -- it carves /48->/56->/64, bounded at 256.) + +Diagnosed with `faulthandler.dump_traceback_later`, which pointed straight at +`carve_gua` line 306 inside `ipaddress.subnets()`. + +## Fix + +- **`netbox/dc-dc-prefixes-import.py::carve_gua`** -- take the first /64 LAZILY + with `next(dc_gua_prefix.subnets(new_prefix=64))` (O(1)) instead of + `list(...)[0]`. Same result, no enumeration. +- **`tests/dc-dc-prefixes-import/test_logic.py`** -- added a + `faulthandler.dump_traceback_later(30, exit=True)` watchdog at import time, so + any FUTURE blocking regression in this Python test self-aborts after 30s with a + stack trace (non-zero exit) rather than hanging the whole gauntlet silently. + +## Verification +- `bash tests/dc-dc-prefixes-import/run-tests.sh` -> `ALL PASS (40 checks)` in + ~0.05s (previously: hung indefinitely, never completing). +- `bash scripts/repo-lint.sh` -> 0 fail. +- Full `scripts/run-tests-all.sh` gauntlet now runs to completion. + +## Revert +- `git checkout netbox/dc-dc-prefixes-import.py tests/dc-dc-prefixes-import/test_logic.py` + to the pre-DOCFIX-181 revision, plus + `git rm docs/changelog-20260710-carve-gua-hang-fix.md`. + (Reverting reinstates the hang -- do not, without replacing the fix.) + +## Note +- The pre-existing ledger claim of "dc-dc-prefixes-import 40/40" was optimistic: + the end-to-end path had been hanging, so those checks never actually ran to a + green count until this fix. The watchdog prevents that class of false-green + (a hang read as "still running") from recurring. diff --git a/docs/session-ledger.md b/docs/session-ledger.md index a86fde7..735ab46 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -1601,8 +1601,22 @@ init/required_providers + Step-9 -input=false and 13-resource plan. Changelog `docs/changelog-20260710-phase0-runbook-as-executed.md`. repo-lint 0 fail. - **Numbers after DOCFIX-180:** DOCFIX next-free 181 (re-scan before assigning). -- **STAGE 1 CLOSED. Deliverables DOCFIX-178/179/180 ready to commit** (operator - ruled: commit). Next: Stage 2 (dc-dc-phase1-office1-standup) when directed. +- **STAGE 1 CLOSED + COMMITTED.** Branch `dc-dc-stage1-phase0-first-exec` + (340ac58): DOCFIX-178/179/180. NOT pushed. Git identity set repo-local + (JANeumatrix/jesse.austin@neumatrix.com, matching prior commits) -- this host + had none. +- **DELIVERED -- DOCFIX-181: `carve_gua` subnet-enumeration hang FIXED.** The full + `run-tests-all.sh` gauntlet was HANGING on `tests/dc-dc-prefixes-import` (its + long-claimed "40/40" was never actually reached -- it hung). Root cause: + `netbox/dc-dc-prefixes-import.py::carve_gua()` did + `list(prefix.subnets(new_prefix=64))` -- ~16.7M /64 objects for a /40 GUA -- + when it only uses the first. Fixed with `next()` (O(1)); added a + `faulthandler.dump_traceback_later(30, exit=True)` watchdog to `test_logic.py` + so any future hang self-aborts with a traceback instead of stalling the + gauntlet. Harness 40/40 in 0.05s; full gauntlet now completes green. + Changelog `docs/changelog-20260710-carve-gua-hang-fix.md`. Next-free -> 182. +- **Next: Stage 2** (`dc-dc-phase1-office1-standup`) when directed -- watch gap + #17 (per-site ISP-uplink/WAN net; `wan` is now a candidate model). diff --git a/netbox/dc-dc-prefixes-import.py b/netbox/dc-dc-prefixes-import.py index 9bf69db..999aa84 100644 --- a/netbox/dc-dc-prefixes-import.py +++ b/netbox/dc-dc-prefixes-import.py @@ -302,9 +302,13 @@ def carve_gua(dc_gua_prefix: ipaddress.IPv6Network) -> dict: - # Only provider-public carries GUA in the current family matrix. - gua_64_candidates = list(dc_gua_prefix.subnets(new_prefix=64)) - return {"provider-public": str(gua_64_candidates[0])} + # Only provider-public carries GUA in the current family matrix. Take the + # FIRST /64 LAZILY: a realistic GUA prefix (D-101's own example is a /40) + # has 2**24 = ~16.7M /64 subnets, so list(...subnets(new_prefix=64)) would + # materialize them all and hang/OOM. next() over the generator is O(1). + # (DOCFIX-181 -- found by the harness hanging on the /40 test input.) + first_64 = next(dc_gua_prefix.subnets(new_prefix=64)) + return {"provider-public": str(first_64)} # ----------------------------------------------------------------------------- diff --git a/tests/dc-dc-prefixes-import/test_logic.py b/tests/dc-dc-prefixes-import/test_logic.py index 13821a5..67f489e 100644 --- a/tests/dc-dc-prefixes-import/test_logic.py +++ b/tests/dc-dc-prefixes-import/test_logic.py @@ -20,6 +20,15 @@ import os import sys +import faulthandler + +# Watchdog: self-abort WITH a traceback after 30s instead of hanging the whole +# run-tests-all.sh gauntlet if any test ever blocks. This is exactly how the +# carve_gua /40 subnet-enumeration hang (DOCFIX-181) was diagnosed; now a future +# blocking regression fails loudly (non-zero exit + stack dump) rather than +# stalling the gauntlet indefinitely. +faulthandler.dump_traceback_later(30, exit=True) + HERE = os.path.dirname(os.path.abspath(__file__)) REPO_ROOT = os.path.dirname(os.path.dirname(HERE)) STUB_SYSPATH = os.path.join(HERE, "stub_syspath")