diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 5019a8a..917a4bb 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2486,6 +2486,17 @@ `.6` region VM (10.12.68.6/10.12.64.6) + the new `vr1-dc1-region`, dc1 `.7` when live (10.12.68.7/10.12.64.7), the D-134 utility-band `.4-.9` assignments, and the dc0 renames. Full enumeration: changelog-20260807 Item 11. Load via `netbox/*.py --commit` (dry-run + fidelity-check).** + **^ PARTLY LANDED 2026-08-07 (changelog-20260807-dc1-region-sequence Item 1).** New tested importer + `netbox/dc-util-hosts-import.py` (+harness 19/19) closed a tool gap -- the D-134 utility RANGES were in + the apex but no importer recorded the individual utility-HOST addresses. RECORDED, measured (apex + `ipam/ip-addresses` 186->194, ids 187-194, idempotent re-run EXISTS/0): dc0 `.5` juju-01 + (10.12.8.5/10.12.4.5), `.6` maas-01 (10.12.8.6/10.12.4.6), `.7` tailscale-01 (10.12.8.7/10.12.4.7); + dc1 `.6` maas-01 (10.12.68.6/10.12.64.6). Each dns_name carries the ruled `vr1-dc--NN` name, so + the dc0 RENAMES are recorded by construction. STILL PENDING: (a) the `.4` ARTIFACT host (dc0 mirror / + dc1 cache, metal-admin-only 10.12.8.4 / 10.12.68.4) -- OUT of the importer's scope (not a lib-hosts host, + per-DC divergent, no repo name), needs an operator NAMING ruling; (b) dc1 `.5` juju-01 + `.7` tailscale-01 + -- record WHEN LIVE (juju not bootstrapped, tailscale planned). The `vr1-dc1-region` MAAS region needs no + NetBox object (apex `dcim/regions` are geographic US-West/VR0/VR1, not MAAS regions; the `.6` represents it). F-CV1: designate _admin backend DOWN -- **RESOLVED 2026-08-06 (BUNDLEFIX-056, operator-approved fix EXECUTED + VERIFIED).** Root cause (governing = D-052 + generic binding rule + the D-020 amendment's ruled .62 triple, NOT D-141): designate's bundle bindings OMITTED public + internal, diff --git a/docs/changelog-20260807-dc1-region-sequence.md b/docs/changelog-20260807-dc1-region-sequence.md new file mode 100644 index 0000000..97f1c55 --- /dev/null +++ b/docs/changelog-20260807-dc1-region-sequence.md @@ -0,0 +1,56 @@ +# changelog 2026-08-07 -- dc1-region standup sequence ("sequence and continue") + +Session scope: sequence and execute the enumerated NEXT threads (dc1 region buildout, +NetBox recording, hygiene). Status lives ONLY in `docs/CURRENT-STATE.md`; this file is +the per-session body (GA-R2). Each item states what / why (evidence) / how to revert. + +--- + +## Item 1 -- NetBox tool GAP closed + as-built utility hosts LANDED into office1-netbox + +**What.** New tested importer `netbox/dc-util-hosts-import.py` (+ harness +`tests/dc-util-hosts-import/` 19/19) that records a D-134 utility-band HOST's +ip-addresses (both planes) into the apex, then landed the four as-built hosts: + +| host | metal-admin | provider-public | ids | +|---|---|---|---| +| vr1-dc0-juju-01 (.5) | 10.12.8.5 | 10.12.4.5 | 187, 188 | +| vr1-dc0-maas-01 (.6) | 10.12.8.6 | 10.12.4.6 | 189, 190 | +| vr1-dc0-tailscale-01 (.7) | 10.12.8.7 | 10.12.4.7 | 191, 192 | +| vr1-dc1-maas-01 (.6) | 10.12.68.6 | 10.12.64.6 | 193, 194 | + +Fidelity: apex `ipam/ip-addresses` 186 -> 194 (+8); idempotent re-run reports +`EXISTS / created: 0`. Target `office1-netbox` (10.10.1.10, DOCFIX-195 sandbox apex). + +**Why.** Landing the operator-flagged NetBox-pending list (changelog-20260807 +Item 11) hit a tool gap: `dc-plane-apex-import.py` had loaded the D-134 utility +RANGES (.4-.49) and the VIP addresses (.50-.62), but NO importer recorded the +individual utility-HOST assignments within the band -- all 16 candidate host IPs +measured absent. Per CLAUDE.md hard rule 4 the gap was declared (operator ruled +"Build importer, then land") and NOT improvised as raw API POSTs. + +**Design (advisor-reviewed).** DERIVED, never hardcoded: plane CIDRs from +`lib-net.sh`, host octet from `lib-hosts.sh` (an empty/missing octet is a hard die -- +the clean-zero failure class). ONE host per invocation (liveness is operator-asserted; +no L3 path apex->DC plane, SEC-010). Whole-plan preflight (both planes) before any +create; the D-134 utility RANGE is the precondition (a missing range dies). dns_name +collision guard; SANDBOX_HOSTS + `--yes-write-upstream` gate; WAF-safe UA. dns_name +carries each ruled `vr1-dc--NN` name -> the dc0 RENAMES are recorded by +construction (there were no stale old-name records to rename). + +**Scope boundary / deferred (queued findings).** +- The `.4` ARTIFACT host (dc0 apt mirror / dc1 caching proxy, D-135) is OUT of this + importer: it is not a `lib-hosts` host, is metal-admin-ONLY (`10.12.8.4` / + `10.12.68.4`, no provider-public leg -- measured from `dc-mirror.sh`/`dc-cache-proxy.sh` + LISTEN_CIDR), per-DC divergent, and has no host NAME anywhere in the repo. Recording + it needs an operator naming ruling -- a separate finding, not this tool. +- dc1 `.5` juju-01 + `.7` tailscale-01: record WHEN LIVE (juju not yet bootstrapped; + tailscale planned, rebuilt in step E). The `vr1-dc1-region` MAAS region needs no + separate NetBox object -- the apex `dcim/regions` are geographic (US-West / VR0 / VR1), + not MAAS regions; the region VM is represented by its `.6` ip-addresses. + +**How to revert.** This is a NetBox state change, not a git revert. Delete the 8 +ip-addresses by address from `office1-netbox` (ids 187-194, or by the four host +dns_names), e.g. `DELETE /api/ipam/ip-addresses//`. The importer itself and its +harness are removable with `git rm netbox/dc-util-hosts-import.py tests/dc-util-hosts-import/` ++ `bash scripts/run-tests-all.sh --record-manifest`. diff --git a/netbox/dc-util-hosts-import.py b/netbox/dc-util-hosts-import.py new file mode 100755 index 0000000..d0bda26 --- /dev/null +++ b/netbox/dc-util-hosts-import.py @@ -0,0 +1,312 @@ +#!/usr/bin/env python3 +""" +Register a D-134 UTILITY-BAND HOST's ip-addresses in office1-netbox (the VR1 IPAM +apex). ONE utility host per invocation, both of its planes (metal-admin + +provider-public). Places the individual host assignment WITHIN the D-134 utility +RANGE that netbox/dc-plane-apex-import.py already loaded -- that range is the +PRECONDITION; a missing one is a hard STOP, never a silent create. + +WHY THIS EXISTS (measured 2026-08-07). dc-plane-apex-import.py loaded the D-134 BAND +RANGES (.4-.49) and the ruled VIP ADDRESSES (.50-.62), but NOT the individual +utility-HOST addresses within the band. Against office1-netbox this session: the four +utility ranges exist, yet none of dc0 .5/.6/.7 or dc1 .6 is an ip-address object. This +tool closes that gap so every DC standup records its utility hosts from a tool rather +than by hand (Roosevelt-delta: same at every bare-metal DC). + +SCOPE, deliberately narrow (advisor-reviewed 2026-08-07): + * IN : the lib-hosts utility-band hosts -- juju-01 (.5), maas-01 (.6), + tailscale-01 (.7). All 2-plane (metal-admin + provider-public) by the + as-built carve (scripts/lib-hosts.sh: juju-01 "only TWO planes", tailscale-01 + "metal-admin + provider-public", maas-01 "SAME 2-plane shape"). + * OUT : the .4 ARTIFACT host. It is NOT a lib-hosts host -- it is a metal-admin-ONLY + service alias on the rack bridge (dc0 apt mirror / dc1 caching proxy, D-135), + per-DC divergent, with no host NAME anywhere in the repo. Recording it needs + an operator naming ruling; it is a separate queued finding, not this tool. + +DERIVED, NEVER HARDCODED (hard rule 2/3): + * plane CIDRs <- scripts/lib-net.sh via lib_net_select_dc (site-keyed authority) + * host octet <- scripts/lib-hosts.sh via lib_hosts_select_dc (the MAC-pinned carve) + The address is plane_base + octet. The ONLY in-script tables are suffix->role-label + (for the human description) and the 2-plane set -- both cited to lib-hosts, neither an + address. An empty/missing octet from lib-hosts is a hard die (NEVER a .0 default -- a + clean zero that reads as success is this repo's recorded failure mode). + +DRY BY DEFAULT -- --commit writes, then READS BACK every object. Upstream write GATED IN +CODE (SANDBOX_HOSTS + --yes-write-upstream). WHOLE-PLAN PREFLIGHT (both planes) runs +BEFORE any create, so a half-write cannot leave one plane recorded and the other not. + +Usage (through the office1-netbox tunnel/base-leg, with the sandbox v2 token): + . ~/vr1-office1-creds/vr1-netbox-sandbox.env # NETBOX_URL=http://10.10.1.10:8000 + NETBOX_TOKEN + python3 netbox/dc-util-hosts-import.py --site vr1-dc0 --host vr1-dc0-tailscale-01 + ... add --commit to write. + +Exit: 0 ok | 2 die (bad input / failed precondition). +""" +import argparse +import ipaddress +import json +import os +import re +import subprocess +import sys +import urllib.error +import urllib.parse +import urllib.request + +UA = "curl/8.5.0" # upstream 403s the default python UA -- see references/platform-traps.md + +# A sandbox is local, or the known Office1 apex address. Anything else is treated as the +# production apex and requires the explicit upstream flag. +SANDBOX_HOSTS = {"localhost", "127.0.0.1", "10.10.1.10"} + +STATUS = "active" # a real address on the wire + +# D-134 utility octet band: .4-.9. A --host whose lib-hosts octet is outside this band is +# a ROLE node (.100+), NOT a utility host -- rejected. (.4 is the artifact host, OUT of +# scope; this tool serves .5/.6/.7 -- see the module docstring.) +UTIL_BAND_LOW = 4 +UTIL_BAND_HIGH = 9 + +# The 2-plane set of the utility hosts, as-built (scripts/lib-hosts.sh). NOT an address -- +# the addresses are derived (plane base from lib-net + octet from lib-hosts). +UTIL_PLANES = ("metal-admin", "provider-public") + +# host SUFFIX -> role label (description only; cited to lib-hosts). Membership here is also +# the whitelist: a --host whose suffix is absent is refused, so a role node or the .4 +# artifact cannot slip through even if its octet somehow landed in the band. +ROLE_LABEL = { + "juju-01": "Juju controller (D-104 dedicated controller VM)", + "maas-01": "MAAS region+rack VM (D-132 q1 per-DC region)", + "tailscale-01": "Tailscale subnet router (D-129(iii) amendment)", +} + + +def die(msg: str): + print(f"FAIL: {msg}", file=sys.stderr) + sys.exit(2) + + +def _repo() -> str: + return os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + + +def lib_net(site: str) -> dict: + """Plane name -> CIDR from lib-net.sh -- the site-keyed authority (mirrors + netbox/dc-plane-apex-import.py's derivation).""" + sh = (f'source "{_repo()}/scripts/lib-net.sh" >/dev/null 2>&1 || exit 3; ' + f'lib_net_select_dc "{site}" >/dev/null 2>&1 || exit 3; ' + 'for c in "${PLANE_CIDRS[@]}"; do printf "%s\\t%s\\n" "${PLANE_NAME[$c]}" "$c"; done') + p = subprocess.run(["bash", "-c", sh], capture_output=True, text=True) + if p.returncode != 0: + die(f"lib-net.sh did not yield planes for {site} (rc={p.returncode}): {p.stderr.strip()[:200]}") + planes = {} + for line in p.stdout.splitlines(): + if "\t" in line: + name, cidr = line.split("\t", 1) + planes[name.strip()] = cidr.strip() + if not planes: + die(f"lib-net.sh yielded NO planes for {site} -- refusing to derive an address from nothing.") + return planes + + +def lib_hosts_octet(site: str, host: str) -> int: + """The host's last octet from lib-hosts.sh HOST_OCTET (the MAC-pinned carve). An + empty/missing value is a HARD die -- never a .0 default (a clean zero that reads as + success is this repo's recorded failure mode, memory #13).""" + sh = (f'source "{_repo()}/scripts/lib-hosts.sh" >/dev/null 2>&1 || exit 3; ' + f'lib_hosts_select_dc "{site}" >/dev/null 2>&1 || exit 3; ' + f'printf "%s" "${{HOST_OCTET[{host}]:-}}"') + p = subprocess.run(["bash", "-c", sh], capture_output=True, text=True) + if p.returncode != 0: + die(f"lib-hosts.sh did not select {site} for {host} (rc={p.returncode}): {p.stderr.strip()[:200]}") + raw = p.stdout.strip() + if not raw: + die(f"lib-hosts HOST_OCTET has NO entry for {host} in {site} -- refusing to invent " + f"an octet (an empty/.0 default is the clean-zero failure this repo guards against).") + try: + return int(raw) + except ValueError: + die(f"lib-hosts octet for {host} is not an integer: {raw!r}") + + +class NB: + """Stdlib NetBox client -- same shape as netbox/dc-rack-mgmt-import.py; UA-aware so it + is not 403'd by the upstream User-Agent filter.""" + def __init__(self, base, token): + self.base = base.rstrip("/") + self.token = token + + def _req(self, method, path, body=None): + data = json.dumps(body).encode() if body is not None else None + req = urllib.request.Request(f"{self.base}/api/{path}", data=data, method=method, + headers={"Authorization": f"Token {self.token}", + "Accept": "application/json", + "Content-Type": "application/json", + "User-Agent": UA}) + try: + with urllib.request.urlopen(req, timeout=45) as r: + return json.load(r) if r.status != 204 else None + except urllib.error.HTTPError as exc: + detail = exc.read().decode(errors="replace")[:300] + if exc.code == 403 and "v1 token" in detail: + die("403 'Invalid v1 token' -- NetBox 4.6 wants the ASSEMBLED v2 token " + "nbt_., not the API's bare `token` field.") + if exc.code == 403: + die(f"403 on {path}. If curl works with this token, it is the upstream " + f"User-Agent filter, NOT the token (references/platform-traps.md).") + die(f"HTTP {exc.code} {method} {path}: {detail}") + + def one(self, path, **flt): + res = self._req("GET", f"{path}/?{urllib.parse.urlencode(flt)}&limit=1") + return res["results"][0] if res["results"] else None + + def all(self, path, **flt): + q = urllib.parse.urlencode({**flt, "limit": 1000}) + res = self._req("GET", f"{path}/?{q}") + return res.get("results", []) if res else [] + + def create(self, path, payload): + return self._req("POST", f"{path}/", payload) + + +def get_nb(base, token): + """Client factory -- the injection seam the harness overrides to drive main() against + an in-memory fake without a live NetBox.""" + return NB(base, token) + + +def range_covering(nb, addr: ipaddress.IPv4Address): + """Return the apex ip-range that contains addr, or None. The D-134 utility range is a + PRECONDITION (loaded by dc-plane-apex-import.py); containment in a real range is the + check that can actually fail against live state.""" + for r in nb.all("ipam/ip-ranges"): + try: + s = ipaddress.ip_address(r["start_address"].split("/")[0]) + e = ipaddress.ip_address(r["end_address"].split("/")[0]) + except (KeyError, ValueError): + continue + if s <= addr <= e: + return r + return None + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n", 1)[0]) + ap.add_argument("--site", choices=("vr1-dc0", "vr1-dc1"), default=os.environ.get("UTIL_SITE"), + help="REQUIRED. Which DC (env: UTIL_SITE). Explicit -- never inferred -- so " + "one DC's values cannot land scoped to another site.") + ap.add_argument("--host", default=os.environ.get("UTIL_HOST"), + help="REQUIRED. The utility host to record, e.g. vr1-dc0-tailscale-01 " + "(env: UTIL_HOST). One host per run; liveness is operator-asserted " + "(there is no L3 path from the apex host to a DC plane -- SEC-010).") + ap.add_argument("--commit", action="store_true", + help="WRITE. Default is a DRY RUN that writes nothing.") + ap.add_argument("--yes-write-upstream", action="store_true", + help="Required (with --commit) to write to a NON-sandbox NetBox.") + args = ap.parse_args() + + url = os.environ.get("NETBOX_URL") + token = os.environ.get("NETBOX_TOKEN") + if not url or not token: + die("NETBOX_URL and NETBOX_TOKEN must be set.") + + # args-or-env, so not argparse required=True (that breaks the env fallback). Hand-roll. + if not args.site: + die("--site (or UTIL_SITE) is REQUIRED -- the target DC is never inferred.") + if args.site not in ("vr1-dc0", "vr1-dc1"): + die(f"--site {args.site!r} is not a known DC (expected vr1-dc0 or vr1-dc1).") + if not args.host: + die("--host (or UTIL_HOST) is REQUIRED -- the utility host is never inferred.") + + site = args.site + host = args.host + if not re.fullmatch(r"vr1-dc[01]-[a-z0-9-]+", host): + die(f"--host {host!r} must look like vr1-dc0-<role>-NN (letters/digits/dashes only).") + if not host.startswith(site + "-"): + die(f"--host {host!r} is not in --site {site!r} -- refusing (a dc0 host must not land " + f"scoped to dc1 or vice versa).") + suffix = host[len(site) + 1:] + if suffix not in ROLE_LABEL: + die(f"--host {host!r} (suffix {suffix!r}) is not a utility host this tool records. " + f"IN scope: {', '.join('<site>-' + s for s in sorted(ROLE_LABEL))}. The .4 artifact " + f"host and the .100+ role nodes are OUT of scope (see the module docstring).") + + hostname = (urllib.parse.urlparse(url).hostname or url).lower() + is_sandbox = hostname in SANDBOX_HOSTS + print(f"Target : {url} ({'SANDBOX' if is_sandbox else 'NOT a known sandbox'})") + if args.commit and not is_sandbox and not args.yes_write_upstream: + die(f"REFUSING to --commit to '{hostname}': not a known sandbox, so treated as the " + f"PRODUCTION apex. Re-run with --yes-write-upstream if that is intended.") + + print("\n*** DRY RUN -- nothing will be written. Re-run with --commit. ***" + if not args.commit else "\n*** COMMITTING. ***") + + # ---- DERIVE the plan (both planes) from lib-net + lib-hosts -- no invented value ---- + octet = lib_hosts_octet(site, host) + if not (UTIL_BAND_LOW <= octet <= UTIL_BAND_HIGH): + die(f"{host} has octet .{octet}, outside the D-134 utility band .{UTIL_BAND_LOW}-.{UTIL_BAND_HIGH} " + f"-- this is a role node, not a utility host. Refusing.") + planes = lib_net(site) + plan = [] # (plane, addr_str, addr_obj, description) + for plane in UTIL_PLANES: + cidr = planes.get(plane) + if not cidr: + die(f"lib-net has no {plane} plane for {site} -- refusing to derive its address.") + net = ipaddress.ip_network(cidr) + addr = net.network_address + octet + addr_str = f"{addr}/{net.prefixlen}" + desc = f"D-134 utility .{octet} -- {site} {ROLE_LABEL[suffix]} ({plane} leg)" + plan.append((plane, addr_str, addr, desc)) + + planned_addr_strs = {a for _, a, _, _ in plan} + + print(f"\nHost : {host} (octet .{octet}, planes: {', '.join(UTIL_PLANES)})") + for plane, addr_str, _, _ in plan: + print(f" plan {plane:16s} {addr_str}") + + nb = get_nb(url, token) + + # ---- WHOLE-PLAN PREFLIGHT (both planes) BEFORE any create -- so a bad plane cannot + # leave the other written. ---- + for plane, addr_str, addr, _ in plan: + r = range_covering(nb, addr) + if r is None: + die(f"no apex ip-range covers {addr} ({plane}) -- the D-134 utility range is a " + f"PRECONDITION (run dc-plane-apex-import.py first). Refusing to place a host " + f"address in an unallocated band.") + + # dns_name collision: any OTHER address already carrying this dns_name (not one of our + # two planned plane addresses) is a stale/rename conflict, not an idempotent re-run. + for a in nb.all("ipam/ip-addresses", dns_name=host): + if a.get("dns_name") == host and a.get("address") not in planned_addr_strs: + die(f"dns_name {host!r} already on {a.get('address')}, which is not a planned plane " + f"address -- possible stale/rename conflict. Resolve in the apex before writing.") + + # present-state (idempotency), resolved up front as part of the whole-plan gate + present = {addr_str: (nb.one("ipam/ip-addresses", address=addr_str) is not None) + for _, addr_str, _, _ in plan} + + created = existing = 0 + for plane, addr_str, _, desc in plan: + print(f"\n{plane} leg:") + if present[addr_str]: + print(f" EXISTS {addr_str} dns={host}") + existing += 1 + elif not args.commit: + print(f" [dry-run] would CREATE {addr_str} dns={host}") + created += 1 + else: + o = nb.create("ipam/ip-addresses", + {"address": addr_str, "status": STATUS, "dns_name": host, "description": desc}) + print(f" CREATED {addr_str} (id={o['id']}) dns={host}") + created += 1 + + verb = "would create" if not args.commit else "created" + print(f"\n{'=' * 66}\n{verb}: {created} already present: {existing}") + if not args.commit: + print("DRY RUN -- nothing was written. Re-run with --commit.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/HARNESS-MANIFEST b/tests/HARNESS-MANIFEST index c1b925c..4d4e42d 100644 --- a/tests/HARNESS-MANIFEST +++ b/tests/HARNESS-MANIFEST @@ -36,6 +36,7 @@ dc-region-topology dc-selector dc-snap-proxy +dc-util-hosts-import g3-domain-manager-probe juju-spaces-check keystone-policy-drift diff --git a/tests/dc-util-hosts-import/fake_netbox.py b/tests/dc-util-hosts-import/fake_netbox.py new file mode 100644 index 0000000..353e940 --- /dev/null +++ b/tests/dc-util-hosts-import/fake_netbox.py @@ -0,0 +1,45 @@ +""" +In-memory stand-in for the NB client in netbox/dc-util-hosts-import.py. + +Implements the three methods the target calls on its client -- .one(), .all(), +.create() -- so the REAL main()/preflight can be driven end-to-end without a live +server. Injected at the get_nb() seam, exactly as tests/dc-rack-mgmt-import/ does. + +FIDELITY. .one() issues GET ...?<filter>&limit=1 and takes results[0] -- single-match +by construction, returns the first record whose fields equal the filter, or None. .all() +returns every record on a path (optionally filtered), mirroring the target's limit=1000 +list read. .create() records every write so a test can assert a rejected/dry run wrote +NOTHING. The store is keyed by path, so ipam/ip-ranges and ipam/ip-addresses both work. +""" +from __future__ import annotations + + +class FakeNB: + def __init__(self, ip_ranges=(), ip_addresses=()): + self.store = { + "ipam/ip-ranges": [dict(r) for r in ip_ranges], + "ipam/ip-addresses": [dict(a) for a in ip_addresses], + } + self.creates = [] # (path, payload) for every .create() -- the write ledger + self._next_id = 2000 + + def one(self, path, **flt): + for rec in self.store.get(path, []): + if all(str(rec.get(k)) == str(v) for k, v in flt.items()): + return rec + return None + + def all(self, path, **flt): + out = [] + for rec in self.store.get(path, []): + if all(str(rec.get(k)) == str(v) for k, v in flt.items()): + out.append(rec) + return out + + def create(self, path, payload): + self.creates.append((path, dict(payload))) + rec = dict(payload) + rec["id"] = self._next_id + self._next_id += 1 + self.store.setdefault(path, []).append(rec) + return rec diff --git a/tests/dc-util-hosts-import/run-tests.sh b/tests/dc-util-hosts-import/run-tests.sh new file mode 100755 index 0000000..b267b13 --- /dev/null +++ b/tests/dc-util-hosts-import/run-tests.sh @@ -0,0 +1,71 @@ +#!/usr/bin/env bash +# tests/dc-util-hosts-import/run-tests.sh +# +# Harness for netbox/dc-util-hosts-import.py. OFFLINE -- touches no NetBox. +# +# Two layers (mirrors tests/dc-rack-mgmt-import): +# 1. STATIC greps that PIN the guards every sandbox-loop writer must have (dry-by- +# default, upstream-write gated IN CODE, WAF-safe UA, DERIVE-not-hardcode: no baked +# plane/host address literal) + the scope decision (.4 artifact is OUT). +# 2. test_logic.py drives the REAL main()/preflight against an in-memory fake NB +# (injected at get_nb()), sourcing lib-net/lib-hosts for real -- derivation, whole- +# plan preflight, range precondition, dns-collision, idempotency, refuse-upstream. +# ASCII + LF. +set -uo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +S="$(cd "$HERE/../../netbox" && pwd)/dc-util-hosts-import.py" +pass=0; fail=0 +ok() { pass=$((pass+1)); } +bad() { fail=$((fail+1)); echo " FAIL: $1"; } + +command -v python3 >/dev/null 2>&1 || { echo "FAIL: python3 required"; exit 1; } + +python3 -c "import ast;ast.parse(open('$S').read())" 2>/dev/null && ok || bad "does not parse" +python3 "$S" --help >/dev/null 2>&1 && ok || bad "--help -> 0" + +# no env -> fail loud, not guess a target +NETBOX_URL= NETBOX_TOKEN= python3 "$S" --site vr1-dc0 --host vr1-dc0-maas-01 \ + >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing NETBOX_URL/TOKEN must fail" + +# both inputs REQUIRED (target DC + host never inferred) +NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --host vr1-dc0-maas-01 \ + >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --site must fail (DC never inferred)" +NETBOX_URL=http://10.10.1.10:8000 NETBOX_TOKEN=x python3 "$S" --site vr1-dc0 \ + >/dev/null 2>&1; [ $? -ne 0 ] && ok || bad "missing --host must fail (host never inferred)" +grep -q 'default=os.environ.get("UTIL_SITE")' "$S" && ok || bad "lost the UTIL_SITE env fallback" +grep -q 'default=os.environ.get("UTIL_HOST")' "$S" && ok || bad "lost the UTIL_HOST env fallback" + +# DERIVE, NOT HARDCODE -- no baked utility-host address literal (.4-.9 on any plane /22) +grep -qE '"?10\.12\.(4|8|64|68)\.[4-9]"?' "$S" && bad "a utility-host address literal is baked in -- must be DERIVED" || ok +grep -q 'lib-net.sh' "$S" && grep -q 'lib-hosts.sh' "$S" && ok || bad "lost the lib-net/lib-hosts derivation" +# the clean-zero guard: an empty/missing octet must die, never default +grep -q 'clean-zero' "$S" && ok || bad "lost the empty-octet (clean-zero) hard die" + +# SCOPE: the .4 artifact host is explicitly OUT (docstring records the ruling boundary) +grep -q 'artifact' "$S" && ok || bad "lost the .4-artifact out-of-scope note" + +# DRY BY DEFAULT +grep -q '"--commit", action="store_true"' "$S" && ok || bad "no --commit flag -- must be dry by default" +grep -q 'DRY RUN -- nothing will be written' "$S" && ok || bad "does not announce its dry run" + +# UPSTREAM WRITE GATED IN CODE (not by discipline) +grep -q 'yes-write-upstream' "$S" && ok || bad "lost the --yes-write-upstream gate" +grep -q 'SANDBOX_HOSTS' "$S" && ok || bad "lost the SANDBOX_HOSTS gate" +out="$(NETBOX_URL=https://netbox.baldurkeep.com NETBOX_TOKEN=x python3 "$S" \ + --site vr1-dc0 --host vr1-dc0-maas-01 --commit 2>&1)" +printf '%s' "$out" | grep -q "REFUSING to --commit" && ok \ + || bad "did NOT refuse a --commit to a non-sandbox host (would have written production)" + +# WAF: UA-aware or every upstream call 403s (looks like auth failure) +grep -q 'User-Agent' "$S" && grep -q 'curl/8.5.0' "$S" && ok || bad "lost the WAF-safe User-Agent" + +# RANGE precondition present (a host address must land in an existing D-134 range) +grep -q 'range_covering' "$S" && ok || bad "lost the utility-range precondition" + +# ---- behavioral layer ---- +python3 "$HERE/test_logic.py" +rc=$? +[ $rc -eq 0 ] && ok || bad "test_logic.py behavioral suite failed (rc=$rc)" + +echo "run-tests(dc-util-hosts-import): $pass passed, $fail failed" +[ $fail -eq 0 ] || exit 1 diff --git a/tests/dc-util-hosts-import/test_logic.py b/tests/dc-util-hosts-import/test_logic.py new file mode 100644 index 0000000..4633f1d --- /dev/null +++ b/tests/dc-util-hosts-import/test_logic.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +""" +Behavioral tests for netbox/dc-util-hosts-import.py. OFFLINE -- the NB client is a fake +injected at the get_nb() seam. lib-net.sh / lib-hosts.sh ARE sourced for real (the repo +is present), so the derived addresses are exercised against the true carve -- a drift +guard: if lib-hosts moves tailscale-01 off .7 or lib-net moves a plane base, these fail. + +Each assertion has a failing DIRECTION (the 2026-07-30 lesson: a suite that stays green +when an assertion is deleted was decoration). Run via run-tests.sh. +""" +import importlib.util +import io +import os +import sys +from contextlib import redirect_stdout, redirect_stderr + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.dirname(os.path.dirname(HERE)) +sys.path.insert(0, HERE) +from fake_netbox import FakeNB # noqa: E402 + +# import the dashed-filename module by path +_spec = importlib.util.spec_from_file_location("util_hosts_import", + os.path.join(REPO, "netbox", "dc-util-hosts-import.py")) +mod = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(mod) + +_p, _f = 0, 0 + + +def ok(): + global _p + _p += 1 + + +def bad(msg): + global _f + _f += 1 + print(f" FAIL: {msg}") + + +def dc_ranges(*bases): + """Utility ranges .4-.49 for each /22 base given (e.g. '10.12.8').""" + return [{"start_address": f"{b}.4/22", "end_address": f"{b}.49/22"} for b in bases] + + +def run(argv, fake, env_over=None): + """Drive main() with argv/env against `fake`. Returns (exit_code, stdout, stderr).""" + env = {"NETBOX_URL": "http://10.10.1.10:8000", "NETBOX_TOKEN": "nbt_x.y"} + if env_over: + env.update(env_over) + old_env = {k: os.environ.get(k) for k in ("NETBOX_URL", "NETBOX_TOKEN", "UTIL_SITE", "UTIL_HOST")} + old_argv = sys.argv + old_get = mod.get_nb + for k in ("UTIL_SITE", "UTIL_HOST"): + os.environ.pop(k, None) + os.environ.update({k: v for k, v in env.items() if v is not None}) + sys.argv = ["dc-util-hosts-import.py"] + argv + mod.get_nb = lambda base, token: fake + out, err = io.StringIO(), io.StringIO() + code = 0 + try: + with redirect_stdout(out), redirect_stderr(err): + rc = mod.main() + code = rc if rc is not None else 0 + except SystemExit as e: + code = e.code if isinstance(e.code, int) else 1 + finally: + sys.argv = old_argv + mod.get_nb = old_get + for k, v in old_env.items(): + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + return code, out.getvalue(), err.getvalue() + + +# 1. DERIVATION pinned: dc0 tailscale-01 -> 10.12.8.7 + 10.12.4.7 (dry run, 0 writes) +fake = FakeNB(ip_ranges=dc_ranges("10.12.8", "10.12.4")) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-tailscale-01"], fake) +if code == 0 and "10.12.8.7/22" in out and "10.12.4.7/22" in out and not fake.creates: + ok() +else: + bad(f"dry-run derivation dc0 tailscale-01 (code={code}, creates={fake.creates}, out={out!r})") + +# 2. COMMIT writes both planes with dns_name + reads back +fake = FakeNB(ip_ranges=dc_ranges("10.12.8", "10.12.4")) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-tailscale-01", "--commit"], fake) +addrs = {p["address"] for _, p in fake.creates} +dns = {p.get("dns_name") for _, p in fake.creates} +if code == 0 and addrs == {"10.12.8.7/22", "10.12.4.7/22"} and dns == {"vr1-dc0-tailscale-01"}: + ok() +else: + bad(f"commit should create both planes with dns_name (creates={fake.creates})") + +# 3. dc1 maas-01 -> .6 on 10.12.68 / 10.12.64 +fake = FakeNB(ip_ranges=dc_ranges("10.12.68", "10.12.64")) +code, out, err = run(["--site", "vr1-dc1", "--host", "vr1-dc1-maas-01", "--commit"], fake) +addrs = {p["address"] for _, p in fake.creates} +if code == 0 and addrs == {"10.12.68.6/22", "10.12.64.6/22"}: + ok() +else: + bad(f"dc1 maas-01 should derive .6 on both planes (creates={fake.creates})") + +# 4. IDEMPOTENT: both addresses already present -> 0 creates, EXISTS +fake = FakeNB(ip_ranges=dc_ranges("10.12.8", "10.12.4"), + ip_addresses=[{"address": "10.12.8.7/22", "dns_name": "vr1-dc0-tailscale-01"}, + {"address": "10.12.4.7/22", "dns_name": "vr1-dc0-tailscale-01"}]) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-tailscale-01", "--commit"], fake) +if code == 0 and not fake.creates and "EXISTS" in out: + ok() +else: + bad(f"idempotent re-run must create nothing (creates={fake.creates})") + +# 5. MISSING utility range -> die (exit 2), nothing written +fake = FakeNB(ip_ranges=[]) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-tailscale-01", "--commit"], fake) +if code == 2 and not fake.creates and "no apex ip-range covers" in err: + ok() +else: + bad(f"missing range must die before any write (code={code}, creates={fake.creates}, err={err!r})") + +# 6. Address OUTSIDE any range (only a VIP range seeded) -> die +fake = FakeNB(ip_ranges=[{"start_address": "10.12.8.50/22", "end_address": "10.12.8.99/22"}, + {"start_address": "10.12.4.50/22", "end_address": "10.12.4.99/22"}]) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-tailscale-01", "--commit"], fake) +if code == 2 and not fake.creates: + ok() +else: + bad(f"address outside every range must die (code={code}, creates={fake.creates})") + +# 7. Non-sandbox host + --commit without --yes-write-upstream -> REFUSE +fake = FakeNB(ip_ranges=dc_ranges("10.12.8", "10.12.4")) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-tailscale-01", "--commit"], fake, + env_over={"NETBOX_URL": "https://netbox.baldurkeep.com"}) +if code == 2 and "REFUSING to --commit" in err and not fake.creates: + ok() +else: + bad(f"non-sandbox commit must refuse without --yes-write-upstream (code={code}, err={err!r})") + +# 8. ROLE node (suffix not whitelisted) -> die, no lib-hosts guess +fake = FakeNB(ip_ranges=dc_ranges("10.12.8", "10.12.4")) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-control-01", "--commit"], fake) +if code == 2 and not fake.creates: + ok() +else: + bad(f"a role node must be rejected (code={code}, creates={fake.creates})") + +# 9. CROSS-SITE: dc0 host under --site vr1-dc1 -> die +fake = FakeNB(ip_ranges=dc_ranges("10.12.68", "10.12.64")) +code, out, err = run(["--site", "vr1-dc1", "--host", "vr1-dc0-tailscale-01", "--commit"], fake) +if code == 2 and not fake.creates: + ok() +else: + bad(f"a dc0 host under --site vr1-dc1 must die (code={code}, creates={fake.creates})") + +# 10. dns_name COLLISION: same dns_name on a non-planned address -> die +fake = FakeNB(ip_ranges=dc_ranges("10.12.8", "10.12.4"), + ip_addresses=[{"address": "10.12.8.99/22", "dns_name": "vr1-dc0-tailscale-01"}]) +code, out, err = run(["--site", "vr1-dc0", "--host", "vr1-dc0-tailscale-01", "--commit"], fake) +if code == 2 and not fake.creates and "stale/rename conflict" in err: + ok() +else: + bad(f"dns_name on a non-planned address must die (code={code}, creates={fake.creates}, err={err!r})") + +# 11. UNIT: lib_hosts_octet on a host absent from HOST_OCTET -> die (the clean-zero guard) +try: + out, err = io.StringIO(), io.StringIO() + with redirect_stdout(out), redirect_stderr(err): + mod.lib_hosts_octet("vr1-dc0", "vr1-dc0-nonesuch-99") + bad("lib_hosts_octet on a missing host should die, did not") +except SystemExit as e: + ok() if e.code == 2 else bad(f"lib_hosts_octet missing-host exit != 2 ({e.code})") + +# 12. UNIT: derivation of octet .5/.6/.7 from the REAL lib-hosts (drift guard) +try: + got = {mod.lib_hosts_octet("vr1-dc0", h) for h in + ("vr1-dc0-juju-01", "vr1-dc0-maas-01", "vr1-dc0-tailscale-01")} + ok() if got == {5, 6, 7} else bad(f"lib-hosts octets drifted: {got} != {{5,6,7}}") +except SystemExit as e: + bad(f"lib_hosts_octet raised on a real host ({e.code})") + +print(f"test_logic: {_p} passed, {_f} failed") +sys.exit(1 if _f else 0)