# Phase 04 -- Network Carve (provider external network + IPAM reference) Create the Neutron external provider network that sources floating IPs and tenant router gateways (the FIP/ext_net leg of Option B), on top of the MAAS address carve done pre-deploy in phase-00. Also the IPAM reference for where addresses live. Decisions: D-003 (provider shared-L2: public API VIPs + FIP/ext_net, Option B; FIP pool 10.12.5.0-10.12.7.254), the IPv4 provider/internal carve (front-loaded VIP /26), KI-P3-001 (VIP/primary collision -> the reserved-range fix). Troubleshooting: appendix-A -- KI-P3-001. NOTE on what the "carve" is split across: - The MAAS ADDRESS carve (delete stale iprange 2; reserve the front-loaded VIP /26 on provider + metal) runs POST-TEARDOWN / PRE-REDEPLOY -- it is in phase-00, because you must reserve the VIP block before deploying onto it. - The bundle's `vip:` values come from that reserved block -- phase-01. - THIS phase creates the Neutron EXTERNAL provider network on top of the carve -- the only post-deploy network mutation. --- ## IPAM carve reference (design; full detail in the carve doc + design-decisions D-003) Provider 10.12.4.0/22 (role Provider; shared-L2, Option B): - 10.12.4.1 provider gateway - 10.12.4.2 - 10.12.4.63 public API HA VIPs (front-loaded /26) -- MAAS RESERVED; EXCLUDED from the neutron allocation_pool. Every bundle public `vip:` is from here. - 10.12.4.64 - 10.12.4.254 host + container primaries (MAAS auto-static) - 10.12.5.0 - 10.12.7.254 FIP pool / ext_net allocation_pool (this phase's subnet) -- MAAS RESERVED Metal-admin 10.12.8.0/22 (role metal-admin; UNTAGGED; operator/MAAS/admin API -- D-052/053): - 10.12.8.2 - 10.12.8.63 ADMIN API HA VIPs (front-loaded /26) -- MAAS RESERVED (admin endpoint only) - 10.12.8.64 - 10.12.8.254 host + container primaries (incl single-unit svc endpoints, e.g. radosgw) - 10.12.9.0 - 10.12.11.254 MAAS PXE/enlistment DHCP (dynamic; iprange id 1) Metal-internal 10.12.12.0/22 (role metal-internal; TAGGED VID 103; all service east-west -- D-052/053): - 10.12.12.2 - 10.12.12.63 INTERNAL API HA VIPs (front-loaded /26) -- MAAS RESERVED. Internal endpoints land here (e.g. keystone .12.50); confirm the certs' IP-SANs FROM a unit ON this plane, never from the jumphost (Step 4.2, DOCFIX-059). - 10.12.12.64 - 10.12.12.254 host + container primaries on the internal plane DOCFIX-060: the pre-D-052 single "Metal 10.12.8.0/22 = internal/admin VIPs" model is SUPERSEDED -- admin VIPs are on metal-admin (.8.5x), internal VIPs on metal-internal (.12.5x), split per D-052/D-053. KI-P3-001 invariant: on every space carrying juju VIPs (provider AND metal), the VIP block is MAAS-reserved and DISTINCT from the primary range and any neutron allocation_pool, so a MAAS auto-static primary can never land on a configured VIP. (Root cause of the original collision: provider had NO VIP reservation, so MAAS auto-assigned container primaries .225/.226/.227 onto the .224-.236 VIP block.) ## Prerequisites (must be true entering phase-04) - phase-01/02/03 done (deploy + vault + core verify); charms active/idle. - phase-00 MAAS carve applied: FIP pool 10.12.5.0-10.12.7.254 RESERVED on the provider subnet (iprange id 3), and the front-loaded VIP /26 reservations present. - Provider segment is FLAT on physnet1 (bundle ovn-bridge-mappings physnet1:br-ex; flat-network-providers=physnet1). The provider /22 is untagged L2 (not vlan). ## Constants and env-literals (TAG: confirm per site on rebuild) - `ENV(physnet)` physnet1 - `ENV(ext-net)` provider-ext `ENV(ext-subnet)` provider-ext-fip - `ENV(ext-cidr)` 10.12.4.0/22 (full provider /22 so .1 gateway is in-subnet + FIP ARP spans the L2) - `ENV(fip-pool)` 10.12.5.0 - 10.12.7.254 (D-003 Option-A; ~765 FIPs; full pool, not a slice) - gateway 10.12.4.1 -- READ DYNAMICALLY from MAAS, never hardcoded. ## Run-location legend - `# RUN: jumphost` -- vopenstack-jesse as jessea123, admin-openrc sourced; `openstack` + `maas admin`. --- ## Command-label convention Every command block below is bracketed by bold labels, so a command line is never mistaken for surrounding prose (these render in GitBucket and read clearly in a raw editor): - **RUN -- LOC** -- the block CHANGES state; run it at LOC (e.g. `jumphost`, `vault/0`, `jumphost -> magnum/0`). - **CHECK (read-only) -- LOC** -- a read-only verification; safe to re-run. - **GATE:** -- a hard stop; do NOT proceed past the block unless the stated condition holds. - **Expect:** -- what a passing result looks like. - `> CAUTION:` -- marks a destructive, secret-handling, or irreversible step. CANONICAL EXECUTION (D-056): the reviewed, hardened scripts are the execution path -- `scripts/phase-04-network-verify.sh` (PRE/POST gates), `scripts/phase-04-network-create.sh` (the idempotent create below), and `scripts/phase-04-internal-cert-san-verify.sh` (Step 4.2, DOCFIX-059). The inline blocks below document what they do; prefer running the scripts. ## Step 4.1 -- Create the external provider network (B29; idempotent) `--external` but NOT `--share` (usable as router gateway + FIP source, but tenants cannot attach instance ports to the provider segment -- Option B isolation). `--no-dhcp` (MAAS owns DHCP on this segment; FIPs are NAT'd). The subnet is the FULL provider /22 with the FIP pool as the allocation_pool; the VIP block and primaries are MAAS-reserved so neutron never allocates them. Read-only pre-check first (verify the FIP pool is MAAS-reserved so neutron can own it): **CHECK (read-only) -- jumphost** ```bash # RUN: jumphost (MAAS profile is 'admin'; never run 'maas list' -- it prints the API key, DOCFIX-016) maas admin ipranges read | jq -r '.[] | select(.type=="reserved") | "\(.start_ip)-\(.end_ip) subnet=\(.subnet.id) [\(.comment)]"' # expect a reserved 10.12.5.0-10.12.7.254 on the PROVIDER subnet (resolve its id BY CIDR, not literal 1 -- # subnet ids drift across cutovers, DOCFIX-047); + the front-loaded VIP /26 reservations. ``` Create (idempotent `( set -e )`; dynamic gateway; tags applied via `set`, not an inline `--tag` flag): **RUN -- jumphost** ```bash source ~/admin-openrc ( set -e PHYSNET=physnet1; EXT_NET=provider-ext; EXT_SUBNET=provider-ext-fip EXT_CIDR=10.12.4.0/22; FIP_START=10.12.5.0; FIP_END=10.12.7.254 GW=$(maas admin subnets read | jq -r '.[] | select(.cidr=="10.12.4.0/22") | .gateway_ip') # DOCFIX-047: discover BY CIDR; never 'maas admin subnet read 1' (subnet ids drift post-D-052) [ "$GW" = "10.12.4.1" ] || { echo "GATE FAIL: MAAS provider gateway='$GW' (expected 10.12.4.1)"; exit 1; } echo "[OK] gateway $GW" if openstack network show "$EXT_NET" -f value -c id >/dev/null 2>&1; then echo "[SKIP] network $EXT_NET exists" else openstack network create --external --provider-network-type flat \ --provider-physical-network "$PHYSNET" "$EXT_NET" -f value -c id openstack network set --tag role=provider "$EXT_NET" echo "[OK] network $EXT_NET created + tagged" fi if openstack subnet show "$EXT_SUBNET" -f value -c id >/dev/null 2>&1; then echo "[SKIP] subnet $EXT_SUBNET exists" else openstack subnet create --network "$EXT_NET" --subnet-range "$EXT_CIDR" \ --gateway "$GW" --no-dhcp --allocation-pool start="$FIP_START",end="$FIP_END" \ "$EXT_SUBNET" -f value -c id openstack subnet set --tag role=provider --tag "netbox-iprange=${FIP_START}-${FIP_END}" "$EXT_SUBNET" echo "[OK] subnet $EXT_SUBNET created + tagged" fi echo "=== CONFIRM ===" openstack network show "$EXT_NET" -f json | jq -c '{name, external: ."router:external", type: ."provider:network_type", physnet: ."provider:physical_network", shared, tags}' openstack subnet show "$EXT_SUBNET" -f json | jq -c '{name, cidr, gateway_ip, enable_dhcp, allocation_pools, tags}' ) ``` **GATE:** `provider-ext` external=true, type=flat, physnet=physnet1, shared=false; `provider-ext-fip` cidr=10.12.4.0/22, gateway 10.12.4.1, enable_dhcp=false, allocation_pools=[10.12.5.0-10.12.7.254]. --- ## Step 4.2 -- Internal-cert SAN check (DOCFIX-059; read-only; run FROM a unit on metal-internal) Confirm every INTERNAL keystone-catalog endpoint's TLS cert carries its own metal-internal VIP IP as an IP-SAN. Internal certs here are IP-based (no FQDN SAN -- D-019 / D-021), so service-to-service TLS on metal-internal validates only if the internal VIP IP is present in the cert's subjectAltName. VANTAGE (load-bearing -- DOCFIX-059): metal-internal (10.12.12.0/22, VID 103) is an ISOLATED service plane (D-052). The jumphost is NOT on it, so an s_client from the jumphost to 10.12.12.x TIMES OUT and reports false "missing SAN" negatives. This check probes FROM a unit ON the plane (default keystone/leader) via `juju exec`; each probe is `timeout`-bounded, and non-https endpoints (the plain-HTTP glance-simplestreams image-stream) are SKIPPED. NEVER run an internal-cert check from the jumphost. **CHECK (read-only) -- jumphost** ```bash source ~/admin-openrc bash scripts/phase-04-internal-cert-san-verify.sh # optional args: [PROBE_UNIT] [MODEL] ``` **GATE:** `Summary: PASS` -- every internal https endpoint OK (its own 10.12.12.5x IP is in the cert SAN). A `NO-SAN` (cert present, missing its IP) or `NO-CERT` (no cert even from the on-plane unit) is a HOLD -- investigate before proceeding. --- ## EXIT GATE (phase-04 complete) - `provider-ext` (external, flat/physnet1, not shared) + `provider-ext-fip` (full /22, FIP allocation pool, no-dhcp) present and tagged role=provider. - Internal-cert SAN check (Step 4.2) `PASS` -- every internal https endpoint's cert carries its own metal-internal IP-SAN (run FROM a unit on the plane, DOCFIX-059). - FIP allocation + tenant router gateways are now possible (needed by phase-06 mgmt VM FIP, phase-08 cluster FIPs + LB validation). ## As-built reference (object IDs regenerate per deploy -- old IDs are dead post-teardown, not a discrepancy) - network provider-ext = a4e1a7fa-dedf-4256-8437-36582e857d7c (2026-06-30; 06-03 snapshot: 70b34bb2-...) (external, flat, physnet1, shared=false, role=provider) - subnet provider-ext-fip = f66e5bc5-b2a3-446b-bd4c-9005515f23a8 (2026-06-30; 06-03 snapshot: e3afcbae-...) (cidr 10.12.4.0/22, gateway 10.12.4.1, enable_dhcp=false, alloc 10.12.5.0-10.12.7.254, tags role=provider + netbox-iprange=10.12.5.0-10.12.7.254) - Live MAAS reservations the IPAM draft + D-003 do NOT yet list (the DRAFT is incomplete, not the cloud -- draft <- live): 10.12.4.101-10.12.4.110 (subnet 1, provider) + 10.12.8.101-10.12.8.110 (subnet 2, metal), both "mgmt-plane reserved" (10 IPs each). Both sit OUTSIDE the FIP pool (10.12.5.0-10.12.7.254) and the VIP /26 blocks -> no conflict with provider-ext-fip. FOLD into docs/netbox-vip-queue.md + D-003 in the docs sub-pass (purpose annotation pending operator confirmation; do NOT mutate NetBox until IPAM design is confirmed -- D-010). - Transitional note: MAAS already carried the front-loaded VIP reservations (.2-.63 provider + .8.2-.63 metal; old D-020 .8.224-.254 gone) ahead of the bundle's interim .50-.60 VIPs -- harmless (a reserved range blocks future auto-assign, does not evict live VIPs). NetBox modeling DEFERRED (allocate after a clean deploy). ## Next phase-05 -- octavia enablement.