Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / _handoff-not-committed / 10-design-decisions-append.md
@JANeumatrix JANeumatrix 20 hours ago 15 KB Chat session
Raw Blame History

10 -- Append to docs/design-decisions.md (register tail)

Replace every D-NNN with the number ledger-scan assigns. Land in the SAME commit as item 30 (GA-R1/C1). Status stays PROPOSED -- the operator rules per GA-R5.

--- BEGIN append block ---

D-NNN: NetBox-coupled per-DC render pipeline for bundle overlays and tfvars [ARCH]

Status: PROPOSED 2026-07-25 (Chat design seat, repo HEAD bc9a2df; operator has not ruled on the mechanism). Escalates the renderer deferral in docs/audit/committee-20260724-track2-bundle-render.md (Fork 2, "hand-render dc1 now, extract the tool when dc0 forces the second render") to a NetBox-coupled generator. Supersedes nothing.

Problem. Addressing is authoritative in NetBox (office1-netbox, the VR1 working apex, DOCFIX-195) but reaches the deploy as HAND-RENDERED static literals: NetBox -> imported by netbox/*.py -> literals in scripts/lib-net.sh / lib-hosts.sh and tfvars -> hand-rendered into bundle.yaml and overlays/*.yaml. There is no live pull and no generator. Measured fragility:

  • BUNDLEFIX-052 (ceph-rbd-mirror binding) and a YAML flow-comma slip, one session.
  • The app-to-VIP set exists in FOUR live places and no two agree: D-020's ratified list (11 apps, includes vault, excludes ceph-radosgw); bundle.yaml and overlays/vr1-dc1-vips.yaml (11, vault dropped, ceph-radosgw added); overlays/dc-dc-ipv6-family-matrix.yaml (10, plus octavia deliberately commented at lines 140-152).
  • Two overlays own vip: for the same 10 applications under per-key REPLACE and last-wins, so the deploy input depends on overlay ORDER, not on a reconciled value.
  • vault carries no vip: anywhere in the tracked tree while overlays/dc-ha-scaleup.yaml scales it to 3 units and declares vault-hacluster at cluster_count: 3. HA intent the artifact does not express, and an unrecorded departure from D-020.

Under the 2026-07-25 family ruling (see the ruling record for this date) the per-app VIP becomes a six-address dual-family string, for BOTH DCs. That roughly doubles the hand-maintained literal count at the precise point where the hand-render has already failed twice.

What the pipeline PULLS (verified present in netbox/draft/vr1-office1-current-20260725.json, captured 2026-07-25):

  • Six-plane v4 /22s and v6 /60+/64 per DC, selectable by scope_site (vr1-dc0, vr1-dc1); 92 of 139 prefixes carry scope_site, and the role slugs match the plane names exactly.
  • The D-111 API-VIP /64 per DC (2602:f3e2:f02:11::/64 dc0, 2602:f3e2:f03:11::/64 dc1) and the shared metal /60 at :20, admin /64 :20, internal /64 :21.
  • Aggregates (2602:f3e2::/36 ARIN, fd50:840e:74e2::/48 org ULA, RFC1918, 23.157.124.0/24), the D-120 bands as ip-ranges, two rack-controller static IPs.

What it CANNOT pull today (scoping constraint, not a defect): VIPs -- the record holds four ip-addresses and none is a VIP. MAC pins and node inventory -- there is no dcim/devices and no dcim/interfaces in the record at all. VLANs/VIDs -- no ipam/vlans; the VLAN-group buildout is the open G4 item at docs/dc-dc-netbox-buildout-scope.md section 4e.

Where the gap actually is (per docs/audit/decision-recon-20260725.md, 4-lens, 2026-07-25). The PREFIX layer is COMPLETE and DRIFT-FREE -- all 12 plane /22s, both transit /30s, both uplinks, both GUA /48s, the ARIN /36, the full v6 subcarves and the org ULA are present and match lib-net.sh value-for-value. The gap is entirely SUB-PREFIX, and it is what this decision exists to close:

  • The D-134 bands (as amended 2026-07-23, the CONTIGUOUS table: .1-.3 infra, .4-.49 reserved utility, .50-.99 VIP, .100-.200 nodes with control .100-.119 / compute .120-.149 / storage .150-.200, .201-.254 dynamic) exist ONLY as decision prose -- in neither NetBox nor lib-net.sh. lib-net.sh carries only the dc0 VIP octet band .50-.60, UNSET for dc1.
  • The 33 VIP addresses (11 services x provider/admin/internal, .50-.60) exist ONLY as hand-maintained literals in overlays/vr1-dc1-vips.yaml. They are internally consistent with the bundle octet-for-octet, but they are not a projection of the apex.

So a generator has nothing to pull for either. Division of labour, corrected: the coupling MODELS the bands (as ip-ranges) and the VIP addresses (as scoped ip-addresses) IN the apex, then GENERATES the overlays from them, so the overlay becomes a projection of NetBox rather than a parallel hand-maintained truth. The per-DC values file is therefore TRANSITIONAL -- it carries bands and VIPs by hand only until they land in the apex, after which it shrinks to genuinely non-IPAM convention.

Options.

  • (A) Build the generator now and gate the dc1 deploy on it. The forced dual-stack reconciliation IS its first job, so hand-rendering then rebuilding pays twice. Against: an unproven tool on the critical path of an otherwise-ready deploy; its first output has nothing to diff against; MAC pins and VLANs are absent from the record, so the tool cannot be complete on this timeline.

  • (B) Hand-render both DCs dual-stack; build the generator for Roosevelt. Fork 2's lean, and "render once to learn the parameter surface" has real force. Against: roughly 170 dual-family literals by hand, at the point the hand-render has already produced two defects on the simpler v4-only artifact.

  • (C) Split at the seam: build the BACK half now, add the FRONT half later. RECOMMENDED. Reconcile once, by hand, INTO a per-DC values file carrying every plane CIDR, VIP band, app-to-VIP map (one reconciled dual-family string per app, vault included) and role tag. Render the overlays from it with a thin deterministic substitution step. Both DCs ship on it. The NetBox front half (record -> values file) is added later without touching the back half, because the interface is frozen.

Why (C). It captures the whole benefit -- one source of truth, no two lists to drift -- without absorbing the front half's unknowns (VIP representability, MAC and VLAN buildout, token handling, network reach), none of which are on this deploy's critical path. It makes the front half testable BY REPRODUCTION: once the schema is frozen, NetBox-generated values are validated by diffing against known-good hand-authored values for a DC that already deployed. Option (A) has no such gate. And the values file is not scaffolding -- it is the permanent seam between IPAM fact and deployment convention in the target operating model (pinned separately).

Honest cost of (C): the schema and substitution layer are written before the NetBox schema is fully known, so the seam may need one refactor when the front half lands.

Eliminated by construction.

  1. The VIP-overlay collision -- one reconciled vip per app per DC, emitted once; no second overlay to race, so deploy input stops depending on overlay order.
  2. Vault decorative HA -- an application scaled with an hacluster peer and no VIP becomes a GENERATOR-TIME error instead of a silent runtime condition.
  3. Family drift -- D-101's matrix applied once in code instead of by a human ~170 times. The validator asserts family-per-plane-per-SITE, not mere presence: the record currently carries GUA storage prefixes for vr0-dc0 and willamette while VR1 storage is ULA, so a generator selecting on role alone pulls the wrong family.
  4. Reserved-range skew -- per D-074 the tenant-CIDR reserved set is the six plane /22s (today sourced from scripts/lib-net.sh into scripts/net_overlap.py as argv) plus the live-resolved capi-mgmt-subnet and the metadata range. Generating the plane rows from the same values file removes the risk of onboarding a tenant validated against the WRONG DC's planes.

DC-aware gates (part of this proposal). scripts/preflight.sh validates the real per-DC deploy input via scripts/provider-bundle-check.py --overlay --dc, not the base bundle alone. scripts/cloud-assert.sh gains a per-hacluster VIP-owned check, so a scaled application with no VIP fails rather than passes. Bootstrap carries the controller-tag constraint.

Mechanism (front half, when built). Extending netbox/*.py is preferred over a third-party NetBox OpenTofu provider. The existing tools already establish the correct posture: one script per SOURCE (office1-record-dump.py vs prod-draft-dump.py), read-only, token from the environment, never embedded or printed. A provider must additionally be mirrorable into the per-DC airgap mirror (D-107) or it is disqualified on Roosevelt, and it would put a write-capable credential in the tofu run while write-back is deferred (DOCFIX-195). Reads only; no write-back under this decision.

Scope boundary.

  • IN: the six operator planes per DC, VIP bands, plane CIDRs, per-DC role tags, and the generated per-DC overlays and tfvars.
  • OUT -- TENANT SPACE, permanently. Per docs/dc-dc-netbox-buildout-scope.md:128 per-tenant /60s are Neutron-managed (D-016/D-074) and not enumerated in NetBox. Per D-074 tenant private CIDRs are TENANT-CHOSEN and MAY OVERLAP, enforced only against the reserved-range registry. NetBox models pools, aggregates and reserved ranges; it never models a tenant's own networks. A maintainer extending the generator "one more table" into tenant space would either fight Neutron or gate tenant provisioning on an operator-side IPAM write, breaking the tenant self-service model this cloud is sold on.
  • IN: WRITES TO office1-netbox. DOCFIX-195 (operator ruling 2026-07-15) rules that during VR1 office1-netbox takes ALL writes and netbox.baldurkeep.com stays a READ-ONLY v1 reference with merge-back DEFERRED to end-of-deployment. Precedent tool: netbox/d120-compose-bands.py --commit already loaded the D-120 bands into the apex. (An earlier draft of this entry scoped apex writes OUT on a misreading of DOCFIX-195; that would have scoped the coupling out of its own core job -- see the gap below.)
  • OUT: writes to netbox.baldurkeep.com (deferred, DOCFIX-195; P2 merge-back).
  • OUT: VLAN / space / fabric -- PERMANENTLY, not pending. office1-netbox holds zero VLAN objects and that is CORRECT BY DESIGN: VR1 is untagged-per-fabric on every plane in both DCs (D-133, superseding VR0's VID-103/104), so a VLAN object would document tagging that does not exist on the wire. The bundle binds by SPACE NAME -> MAAS subnet by CIDR, never by VLAN/fabric/subnet id. VLAN/space/fabric is MAAS-sourced; NetBox is IPAM only.
  • OUT: MAC pins. They already have a working source of record and are drift-free across substrate main.tf <-> lib-hosts.sh HOST_BOOT_MAC <-> bundle ovn-chassis <-> discovery. They stay substrate/lib-hosts-sourced; this is not a NetBox gap.

Roosevelt-delta (A1). Roosevelt is N bare-metal DCs. Hand-rendering is O(N) human renders of a ~170-literal dual-family artifact; a generator plus one values file per DC plus one validator is O(1) tooling and O(N) data. The transferable artifact is the triple: renderer, values file, validator. This is why the back half is worth building even while the front half waits.

Artifact shape (RULED 2026-07-25 -- see the ruling record). Symmetric per-DC overlays. bundle.yaml becomes VIP-free and holds topology only; every per-DC value, dc0's included, arrives via overlay. dc0's VIPs are extracted from base into overlays/vr1-dc0-vips.yaml. The renderer therefore has ONE output shape per DC and Roosevelt's Nth DC is the same shape as its first.

PREREQUISITE, sequenced FIRST (per the recon's sequencing insight). Extend netbox/sandbox-fidelity-check.py BEFORE the coupling is built. Today it covers only the D-115/D-117/D-120 delta and diffs the FROZEN 90-prefix netbox.baldurkeep.com draft, not the current 139-prefix office1-netbox apex; it is blind to D-124 and D-134. It gates every apex reconciliation this coupling depends on, and the coupling's whole value proposition is eliminating the drift those reconciliations chase. An unextended checker cannot verify the apex mutations in the gap list above, so building the generator first would mean writing to the apex with no trustworthy fidelity gate.

A hard tooling coupling nobody has flagged. scripts/lib-net.sh:59 sets VIP_COUNT_EXPECT=11, consumed by scripts/pre-flight-checks.sh at :79 (warn on vip line count) and :99-102, where a count mismatch is a fail, not a warn. Adding vault's VIP takes the count to 12, so VIP_COUNT_EXPECT must move to 12 in the same change or the gate fails. Note that CURRENT-STATE flags pre-flight-checks.sh as VR0-frozen (no $DC selector), so whether it gates VR1 today needs confirming -- but the constant is in the shared library and any DC-aware successor inherits it. A generator emitting the VIP set should DERIVE this count rather than carry it as a second literal.

Gate relationship. This decision does NOT gate the dc1 deploy. What the family ruling gates, and what must be resolved before deploy, is three items: (1) the vault VIP plus its band octet (a D-020 conformance repair, OPS not ARCH); (2) the octavia family sub-ruling; (3) the reconciled dual-family VIP literals for both DCs, in the symmetric overlay shape above. The subcarve is NOT among them -- D-111 (ADOPTED 2026-07-11) already ratifies the per-plane v6 subcarve and the values are imported for both DCs.

Open questions (operator).

  1. Metal VIP host addresses come straight out of the plane /64 and share it with node addresses; D-111's VIP-dedicated /64 at :11 exists only on the provider/GUA leg. Confirm no MAAS deployed-node band claims the VIP octet on the metal planes. A collision is repaired by a D-111 amendment (a metal VIP sub-range), not by values.
  2. Octavia family. D-101 places lb-mgmt in the IPv6-only ULA group, but overlays/dc-dc-ipv6-family-matrix.yaml:63-82 documents live upstream IPv6 breakage on that path (LP #1911788, #1913409). Recommendation: dual-stack octavia's API endpoint like its ten peers, keep lb-mgmt v4 as a logged D-101 exception, on the ruling's own standard -- a path with open upstream bugs is not where IPv6-only "makes sense". This is an exception to an ADOPTED decision and is the operator's.
  3. Does the reserved-range registry become a first-class NetBox object, or continue to be derived (plane rows from prefixes; capi-mgmt-subnet and metadata live-resolved per D-074/D-035)? Deriving is recommended; D-074's stated gain is that NetBox's uniqueness duty shrinks to the routed space. Operator has asked for more information before ruling.
  4. Values-file format and location; and whether generated artifacts are COMMITTED (reviewable diffs, recommended) or generated at deploy time (unreviewable).
  5. Runner and credential for any automated render. GitBucket has no Actions-style runner and SSH access is v2-deferred, so an automated push needs a host (jumphost timer, or a NetBox webhook worker) and an HTTPS credential. That is a security-ledger row, not part of this decision.

Related: D-020 (dual provider+metal VIPs -- NOT D-036, mis-cited as such in four live sites; see the DOCFIX in this session's changelog), D-100/D-101 (addressing, families, mesh), D-111 (per-DC v6 subcarve, ADOPTED), D-052/D-053 (bindings, six-plane split), D-104 + 2026-07-24 amendment (controller VM), D-107 (per-DC airgap mirror), D-108 (cross-DC), D-119 (region-qualified tags), D-121 (Option C + R-3), D-016/D-074 (tenant CIDRs), D-035 (capi-mgmt reserved), D-067 (barbican-vault on metal-internal), D-110 (repo has moved; never hardcode its path), D-133 (flat per-NIC carve, no VLAN stack -- why VLANs are correctly absent from the apex), D-134 + its 2026-07-23 AMENDMENT (the contiguous band table; the amendment is authoritative and the original table above it is superseded), D-124 (rack statics), D-131 (DNS forwarder .3), D-135 (mirror .4), DOCFIX-195 (apex takes all VR1 writes; baldurkeep merge-back deferred), BUNDLEFIX-052. Companion audit records: docs/audit/decision-recon-20260725.md, docs/audit/stage5-expansion-review-20260724.md, docs/audit/committee-20260724-track2-bundle-render.md.

--- END append block ---