# Phase 04 -- Network Carve (provider external network + IPAM reference)

Create the Neutron external provider network that sources floating IPs and tenant
router gateways (the FIP/ext_net leg of Option B), on top of the MAAS address carve
done pre-deploy in phase-00. Also the IPAM reference for where addresses live.

Decisions: D-003 (provider shared-L2: public API VIPs + FIP/ext_net, Option B;
FIP pool 10.12.5.0-10.12.7.254), the IPv4 provider/internal carve (front-loaded
VIP /26), KI-P3-001 (VIP/primary collision -> the reserved-range fix).
Troubleshooting: appendix-A -- KI-P3-001.

NOTE on what the "carve" is split across:
- The MAAS ADDRESS carve (delete stale iprange 2; reserve the front-loaded VIP /26
  on provider + metal) runs POST-TEARDOWN / PRE-REDEPLOY -- it is in phase-00, because
  you must reserve the VIP block before deploying onto it.
- The bundle's `vip:` values come from that reserved block -- phase-01.
- THIS phase creates the Neutron EXTERNAL provider network on top of the carve --
  the only post-deploy network mutation.

---

## IPAM carve reference (design; full detail in the carve doc + design-decisions D-003)
Provider 10.12.4.0/22 (role Provider; shared-L2, Option B):
- 10.12.4.1                  provider gateway
- 10.12.4.2 - 10.12.4.63     public API HA VIPs (front-loaded /26) -- MAAS RESERVED; EXCLUDED from the
                             neutron allocation_pool. Every bundle public `vip:` is from here.
- 10.12.4.64 - 10.12.4.254   host + container primaries (MAAS auto-static)
- 10.12.5.0 - 10.12.7.254    FIP pool / ext_net allocation_pool (this phase's subnet) -- MAAS RESERVED

Metal-admin 10.12.8.0/22 (role metal-admin; UNTAGGED; operator/MAAS/admin API -- D-052/053):
- 10.12.8.2 - 10.12.8.63     ADMIN API HA VIPs (front-loaded /26) -- MAAS RESERVED (admin endpoint only)
- 10.12.8.64 - 10.12.8.254   host + container primaries (incl single-unit svc endpoints, e.g. radosgw)
- 10.12.9.0 - 10.12.11.254   MAAS PXE/enlistment DHCP (dynamic; iprange id 1)

Metal-internal 10.12.12.0/22 (role metal-internal; TAGGED VID 103; all service east-west -- D-052/053):
- 10.12.12.2 - 10.12.12.63   INTERNAL API HA VIPs (front-loaded /26) -- MAAS RESERVED. Internal endpoints
                             land here (e.g. keystone .12.50); confirm the certs' IP-SANs FROM a unit ON
                             this plane, never from the jumphost (Step 4.2, DOCFIX-059).
- 10.12.12.64 - 10.12.12.254 host + container primaries on the internal plane

DOCFIX-060: the pre-D-052 single "Metal 10.12.8.0/22 = internal/admin VIPs" model is SUPERSEDED --
admin VIPs are on metal-admin (.8.5x), internal VIPs on metal-internal (.12.5x), split per D-052/D-053.

KI-P3-001 invariant: on every space carrying juju VIPs (provider AND metal), the VIP
block is MAAS-reserved and DISTINCT from the primary range and any neutron
allocation_pool, so a MAAS auto-static primary can never land on a configured VIP.
(Root cause of the original collision: provider had NO VIP reservation, so MAAS
auto-assigned container primaries .225/.226/.227 onto the .224-.236 VIP block.)

## Prerequisites (must be true entering phase-04)
- phase-01/02/03 done (deploy + vault + core verify); charms active/idle.
- phase-00 MAAS carve applied: FIP pool 10.12.5.0-10.12.7.254 RESERVED on the provider
  subnet (iprange id 3), and the front-loaded VIP /26 reservations present.
- Provider segment is FLAT on physnet1 (bundle ovn-bridge-mappings physnet1:br-ex;
  flat-network-providers=physnet1). The provider /22 is untagged L2 (not vlan).

## Constants and env-literals (TAG: confirm per site on rebuild)
- `ENV(physnet)`    physnet1
- `ENV(ext-net)`    provider-ext        `ENV(ext-subnet)` provider-ext-fip
- `ENV(ext-cidr)`   10.12.4.0/22  (full provider /22 so .1 gateway is in-subnet + FIP ARP spans the L2)
- `ENV(fip-pool)`   10.12.5.0 - 10.12.7.254   (D-003 Option-A; ~765 FIPs; full pool, not a slice)
- gateway 10.12.4.1 -- READ DYNAMICALLY from MAAS, never hardcoded.

## Run-location legend
- `# RUN: jumphost` -- vopenstack-jesse as jessea123, admin-openrc sourced; `openstack` + `maas admin`.

---

## Command-label convention
Every command block below is bracketed by bold labels, so a command line is never mistaken
for surrounding prose (these render in GitBucket and read clearly in a raw editor):
- **RUN -- LOC** -- the block CHANGES state; run it at LOC (e.g. `jumphost`, `vault/0`, `jumphost -> magnum/0`).
- **CHECK (read-only) -- LOC** -- a read-only verification; safe to re-run.
- **GATE:** -- a hard stop; do NOT proceed past the block unless the stated condition holds.
- **Expect:** -- what a passing result looks like.
- `> CAUTION:` -- marks a destructive, secret-handling, or irreversible step.


CANONICAL EXECUTION (D-056): the reviewed, hardened scripts are the execution path --
`scripts/phase-04-network-verify.sh` (PRE/POST gates), `scripts/phase-04-network-create.sh`
(the idempotent create below), and `scripts/phase-04-internal-cert-san-verify.sh` (Step 4.2,
DOCFIX-059). The inline blocks below document what they do; prefer running the scripts.

## Step 4.1 -- Create the external provider network (B29; idempotent)
`--external` but NOT `--share` (usable as router gateway + FIP
source, but tenants cannot attach instance ports to the provider segment -- Option B
isolation). `--no-dhcp` (MAAS owns DHCP on this segment; FIPs are NAT'd). The subnet
is the FULL provider /22 with the FIP pool as the allocation_pool; the VIP block and
primaries are MAAS-reserved so neutron never allocates them.

Read-only pre-check first (verify the FIP pool is MAAS-reserved so neutron can own it):

**CHECK (read-only) -- jumphost**
```bash
# RUN: jumphost (MAAS profile is 'admin'; never run 'maas list' -- it prints the API key, DOCFIX-016)
maas admin ipranges read | jq -r '.[] | select(.type=="reserved") | "\(.start_ip)-\(.end_ip) subnet=\(.subnet.id) [\(.comment)]"'
# expect a reserved 10.12.5.0-10.12.7.254 on the PROVIDER subnet (resolve its id BY CIDR, not literal 1 --
# subnet ids drift across cutovers, DOCFIX-047); + the front-loaded VIP /26 reservations.
```
Create (idempotent `( set -e )`; dynamic gateway; tags applied via `set`, not an
inline `--tag` flag):

**RUN -- jumphost**
```bash
source ~/admin-openrc
( set -e
  PHYSNET=physnet1; EXT_NET=provider-ext; EXT_SUBNET=provider-ext-fip
  EXT_CIDR=10.12.4.0/22; FIP_START=10.12.5.0; FIP_END=10.12.7.254
  GW=$(maas admin subnets read | jq -r '.[] | select(.cidr=="10.12.4.0/22") | .gateway_ip')  # DOCFIX-047: discover BY CIDR; never 'maas admin subnet read 1' (subnet ids drift post-D-052)
  [ "$GW" = "10.12.4.1" ] || { echo "GATE FAIL: MAAS provider gateway='$GW' (expected 10.12.4.1)"; exit 1; }
  echo "[OK] gateway $GW"
  if openstack network show "$EXT_NET" -f value -c id >/dev/null 2>&1; then
    echo "[SKIP] network $EXT_NET exists"
  else
    openstack network create --external --provider-network-type flat \
      --provider-physical-network "$PHYSNET" "$EXT_NET" -f value -c id
    openstack network set --tag role=provider "$EXT_NET"
    echo "[OK] network $EXT_NET created + tagged"
  fi
  if openstack subnet show "$EXT_SUBNET" -f value -c id >/dev/null 2>&1; then
    echo "[SKIP] subnet $EXT_SUBNET exists"
  else
    openstack subnet create --network "$EXT_NET" --subnet-range "$EXT_CIDR" \
      --gateway "$GW" --no-dhcp --allocation-pool start="$FIP_START",end="$FIP_END" \
      "$EXT_SUBNET" -f value -c id
    openstack subnet set --tag role=provider --tag "netbox-iprange=${FIP_START}-${FIP_END}" "$EXT_SUBNET"
    echo "[OK] subnet $EXT_SUBNET created + tagged"
  fi
  echo "=== CONFIRM ==="
  openstack network show "$EXT_NET" -f json | jq -c '{name, external: ."router:external", type: ."provider:network_type", physnet: ."provider:physical_network", shared, tags}'
  openstack subnet show "$EXT_SUBNET" -f json | jq -c '{name, cidr, gateway_ip, enable_dhcp, allocation_pools, tags}'
)
```
**GATE:** `provider-ext` external=true, type=flat, physnet=physnet1, shared=false;
`provider-ext-fip` cidr=10.12.4.0/22, gateway 10.12.4.1, enable_dhcp=false,
allocation_pools=[10.12.5.0-10.12.7.254].

---

## Step 4.2 -- Internal-cert SAN check (DOCFIX-059; read-only; run FROM a unit on metal-internal)
Confirm every INTERNAL keystone-catalog endpoint's TLS cert carries its own metal-internal VIP IP as an
IP-SAN. Internal certs here are IP-based (no FQDN SAN -- D-019 / D-021), so service-to-service TLS on
metal-internal validates only if the internal VIP IP is present in the cert's subjectAltName.

VANTAGE (load-bearing -- DOCFIX-059): metal-internal (10.12.12.0/22, VID 103) is an ISOLATED service
plane (D-052). The jumphost is NOT on it, so an s_client from the jumphost to 10.12.12.x TIMES OUT and
reports false "missing SAN" negatives. This check probes FROM a unit ON the plane (default
keystone/leader) via `juju exec`; each probe is `timeout`-bounded, and non-https endpoints (the plain-HTTP
glance-simplestreams image-stream) are SKIPPED. NEVER run an internal-cert check from the jumphost.

**CHECK (read-only) -- jumphost**
```bash
source ~/admin-openrc
bash scripts/phase-04-internal-cert-san-verify.sh          # optional args: [PROBE_UNIT] [MODEL]
```
**GATE:** `Summary: PASS` -- every internal https endpoint OK (its own 10.12.12.5x IP is in the cert SAN).
A `NO-SAN` (cert present, missing its IP) or `NO-CERT` (no cert even from the on-plane unit) is a HOLD --
investigate before proceeding.

---

## EXIT GATE (phase-04 complete)
- `provider-ext` (external, flat/physnet1, not shared) + `provider-ext-fip` (full /22,
  FIP allocation pool, no-dhcp) present and tagged role=provider.
- Internal-cert SAN check (Step 4.2) `PASS` -- every internal https endpoint's cert carries its
  own metal-internal IP-SAN (run FROM a unit on the plane, DOCFIX-059).
- FIP allocation + tenant router gateways are now possible (needed by phase-06 mgmt
  VM FIP, phase-08 cluster FIPs + LB validation).

## As-built reference (object IDs regenerate per deploy -- old IDs are dead post-teardown, not a discrepancy)
- network provider-ext = a4e1a7fa-dedf-4256-8437-36582e857d7c  (2026-06-30; 06-03 snapshot: 70b34bb2-...)
  (external, flat, physnet1, shared=false, role=provider)
- subnet provider-ext-fip = f66e5bc5-b2a3-446b-bd4c-9005515f23a8  (2026-06-30; 06-03 snapshot: e3afcbae-...)
  (cidr 10.12.4.0/22, gateway 10.12.4.1, enable_dhcp=false, alloc 10.12.5.0-10.12.7.254,
   tags role=provider + netbox-iprange=10.12.5.0-10.12.7.254)
- Live MAAS reservations the IPAM draft + D-003 do NOT yet list (the DRAFT is incomplete, not
  the cloud -- draft <- live): 10.12.4.101-10.12.4.110 (subnet 1, provider) +
  10.12.8.101-10.12.8.110 (subnet 2, metal), both "mgmt-plane reserved" (10 IPs each). Both sit
  OUTSIDE the FIP pool (10.12.5.0-10.12.7.254) and the VIP /26 blocks -> no conflict with
  provider-ext-fip. FOLD into docs/archive/netbox-vip-queue.md + D-003 in the docs sub-pass (purpose
  annotation pending operator confirmation; do NOT mutate NetBox until IPAM design is confirmed -- D-010).
- Transitional note: MAAS already carried the front-loaded VIP reservations (.2-.63
  provider + .8.2-.63 metal; old D-020 .8.224-.254 gone) ahead of the bundle's interim
  .50-.60 VIPs -- harmless (a reserved range blocks future auto-assign, does not evict
  live VIPs). NetBox modeling DEFERRED (allocate after a clean deploy).

## Next
phase-05 -- octavia enablement.
