# Phase 05 -- Octavia Enablement (D-021)

Bring Octavia from its post-deploy BLOCKED state to fully enabled: run the
`configure-resources` action (control plane + lb-mgmt overlay), then build and tag
the amphora image. End state: octavia active/idle with an ACTIVE amphora image
whose tag matches `octavia amp-image-tag`. (The end-to-end LB build + round-robin +
failover validation is D-011 criterion 4, run in phase-08 once tenant scaffolding
exists.)

Decisions: D-021 (amphora pipeline; control-plane then image). Troubleshooting:
appendix-A -- L7 (snap cannot read /tmp), octavia-configure-resources (long-running
/ o-hm0 transient), amp-image-tag-mismatch (LP#1937003).

IP-ONLY NOTE (supersedes the 2026-05-30 octavia capture): the 05-30 `/etc/hosts`
FQDN prereq DOES NOT APPLY here. This deploy is IP-only (R18 catalog) and octavia
is multi-homed (reaches the provider VIPs over its eth1), so `configure-resources`
needs no hosts/FQDN prep -- just the action.

---

## Prerequisites (must be true entering phase-05)
- phase-04 (core/network) done; vault cert cascade complete so `octavia:certificates`
  is satisfied. octavia/0 sits BLOCKED "Awaiting ... configure-resources" (the
  expected post-deploy state, D-021) -- this phase clears it.
- Glance reachable from the jumphost (provider VIP) to seed the amphora base.
- Bundle-baked octavia config (verify in 5.2 gate): `octavia-diskimage-retrofit`
  `use-internal-endpoints=true` + `image-format=raw` + `amp-image-tag=octavia-amphora`,
  and `octavia amp-image-tag=octavia-amphora` (the two MUST match -- LP#1937003).

## Constants and env-literals (TAG: confirm per site on rebuild)
- `ENV(octavia-tag)`   octavia-amphora   (octavia + retrofit amp-image-tag; must match)
- `ENV(base-name)`     jammy-amphora-base (uploaded base; NOT amphora-tagged)
- `ENV(retrofit)`      octavia-diskimage-retrofit
- `ENV(internal-glance-vip)` 10.12.8.53  (retrofit is metal-only 10.12.8.x -> internal glance)
- run-specific: base image id, amphora image id, op/task ids (capture at run).

## Run-location legend
- `# RUN: jumphost` -- vopenstack-jesse as jessea123, admin-openrc sourced; all
  octavia work is `juju run` / `juju config` / `openstack` from the jumphost.

---

## Command-label convention
Every command block below is bracketed by bold labels, so a command line is never mistaken
for surrounding prose (these render in GitBucket and read clearly in a raw editor):
- **RUN -- LOC** -- the block CHANGES state; run it at LOC (e.g. `jumphost`, `vault/0`, `jumphost -> magnum/0`).
- **CHECK (read-only) -- LOC** -- a read-only verification; safe to re-run.
- **GATE:** -- a hard stop; do NOT proceed past the block unless the stated condition holds.
- **Expect:** -- what a passing result looks like.
- `> CAUTION:` -- marks a destructive, secret-handling, or irreversible step.


## Step 5.1 -- configure-resources (D-021 Phase 1; control plane + lb-mgmt overlay)
Read-only pre-check, then the argument-free action with a bound
wait, then authoritative completion via show-operation (NOT the streamed log).

**RUN -- jumphost**
```bash
( {
  source ~/admin-openrc
  echo "=== pre-check (verify-before-mutate) ==="
  juju status octavia -m openstack | grep -E 'octavia/0' || true   # expect BLOCKED Awaiting configure-resources
  juju actions octavia -m openstack | grep -i configure-resources  # action exists; takes NO params
  echo "--- idempotency: charm-octavia-tagged resources should be EMPTY pre-run ---"
  openstack network list --tags charm-octavia -f value -c Name        # expect empty
  openstack security group list --tags charm-octavia -f value -c Name # expect empty
  openstack loadbalancer provider list                                # expect amphora present (API reachable)
} )
```
Run the action (long-running; juju's default wait may time out but the hook keeps
going -- use a bound `--wait` and tee; do NOT re-fire on a wait-timeout -- appendix-A:
octavia-configure-resources):

**RUN -- jumphost**
```bash
juju run octavia/leader configure-resources -m openstack --wait=20m 2>&1 | tee ~/octavia-configure-resources.out
```
Authoritative completion + A/B/C verify:

**CHECK (read-only) -- jumphost**
```bash
( {
  source ~/admin-openrc
  echo "=== 0. authoritative status (use show-operation, not the streamed log) ==="
  # juju show-operation <N>  -> operation status: completed AND its task: completed
  echo "=== A. octavia active/idle 'Unit is ready' (blocked cleared) ==="
  juju status octavia -m openstack | grep -E 'octavia/0'
  echo "=== B. resources created by the action (were empty pre-run) ==="
  openstack network list --tags charm-octavia -f value -c Name        # lb-mgmt-net
  openstack subnet list  --tags charm-octavia -f value -c Name        # lb-mgmt-subnetv6 (IPv6 geneve design)
  openstack security group list --tags charm-octavia -f value -c Name # lb-mgmt-sec-grp
  echo "=== C. o-hm0 up (IPv6-ULA on lb-mgmt prefix; a br-int port) ==="
  juju exec --unit octavia/0 -m openstack -- 'ip -br addr show o-hm0; sudo ovs-vsctl get Interface o-hm0 external_ids' </dev/null
} )
```
**GATE:** octavia/0 active/idle; lb-mgmt-net + lb-mgmt-subnetv6 + lb-mgmt-sec-grp present;
o-hm0 has an fc00::/.. IPv6-ULA addr and is a br-int port. (NORMAL, not faults: the
lb-mgmt-net is IPv6-ULA by design; a "Virtual network ... down" transient during
o-hm0 bring-up self-heals; the lb-mgmt `network:distributed` port shows DOWN.)

## Step 5.2 -- Amphora image pipeline (D-021 Phase 2; canonical block)
This is the verified canonical block (06-03). One `( set -e )`
subshell: config GATE -> idempotent seed (base staged in `$HOME`, NOT /tmp -- the
openstack snap cannot read /tmp, appendix-A: L7) -> retrofit build -> confirm.
Fully idempotent (amphora present -> skip to confirm; base present -> retrofit only;
fresh -> download+checksum+upload+retrofit). For a FIRST live run in a new
environment you may stop after the seed to eyeball before the multi-minute build.

SEED METHOD (canonical): stage-and-verify (download + sha256-vs-published-SHA256SUMS +
`openstack image create --file`) is CANONICAL here -- it carries provenance verification,
works for any source, and unifies with the phase-08 kube-image seed (FINDING-3). This
SUPERSEDES the 2026-06-16 "web-download canonical" ruling: web-download cannot checksum-verify
the fetched file and is infeasible for the azimuth CDN (urllib UA 403). Web-download is retained
as a TESTED ALTERNATIVE in appendix-A. Note the staged base lands QCOW2 (legacy `--file` does
NOT run glance's import conversion -- CORRECTION-1); that is fine, the retrofit consumes the
qcow2 base and emits the raw `octavia-amphora` OUTPUT (the config gate's image-format=raw is on
the retrofit OUTPUT, not the base).

**RUN -- jumphost**
```bash
# Tunables (operator-confirm the first two for your environment):
BASE_IMG_URL="https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
BASE_SUM_URL="https://cloud-images.ubuntu.com/jammy/current/SHA256SUMS"
BASE_IMG_FILE="jammy-server-cloudimg-amd64.img"
BASE_NAME="jammy-amphora-base"                     # ENV(base-name); NOT amphora-tagged (only the retrofit OUTPUT is)
VERSION_NAME="$(date -u +%Y%m%d)"                  # cosmetic (D-021): feeds the retrofit OUTPUT name
PRODUCT_NAME="com.ubuntu.cloud:server:22.04:amd64" # cosmetic (D-021): metadata only
RETRO=octavia-diskimage-retrofit                   # ENV(retrofit)
STAGE="$HOME/amphora-base"                          # snap-READABLE (home iface); NOT /tmp (L7)

( set -e
  source ~/admin-openrc

  # ---- Phase 0: config GATE (abort if the cloud is not in the expected state) ----
  UIE=$(juju config $RETRO use-internal-endpoints)
  IMGFMT=$(juju config $RETRO image-format)
  RTAG=$(juju config $RETRO amp-image-tag)
  OTAG=$(juju config octavia amp-image-tag)
  [ "$UIE" = true ]   || { echo "GATE FAIL: $RETRO use-internal-endpoints=$UIE (need true; retrofit is metal-only)"; exit 1; }
  [ "$IMGFMT" = raw ] || { echo "GATE FAIL: $RETRO image-format=$IMGFMT (need raw; Ceph RBD fast-clone)"; exit 1; }
  [ -n "$RTAG" ] && [ "$RTAG" = "$OTAG" ] || { echo "GATE FAIL: amp-image-tag mismatch retrofit='$RTAG' octavia='$OTAG' (LP#1937003)"; exit 1; }
  echo "[OK] config gate: use-internal-endpoints=true image-format=raw amp-image-tag=$OTAG"

  # ---- Phase 1: idempotency + seed the jammy base (only if no amphora AND no base) ----
  AMPH=$(openstack image list --tag "$OTAG" -f value -c ID | head -1)
  if [ -n "$AMPH" ]; then
    echo "[SKIP] image already tagged $OTAG ($AMPH) -- pipeline complete; jumping to confirm"
  else
    BASE_ID=$(openstack image list --name "$BASE_NAME" -f value -c ID | head -1)
    if [ -z "$BASE_ID" ]; then
      mkdir -p "$STAGE"; LOCAL="$STAGE/$BASE_IMG_FILE"
      EXP=$(curl -fsSL "$BASE_SUM_URL" | awk -v f="$BASE_IMG_FILE" '$2=="*"f || $2==f {print $1}')
      [ -n "$EXP" ] || { echo "GATE FAIL: no published checksum for $BASE_IMG_FILE"; exit 1; }
      if [ -f "$LOCAL" ] && [ "$(sha256sum "$LOCAL" | awk '{print $1}')" = "$EXP" ]; then
        echo "[OK] staged base present + checksum-valid; skipping download"
      else
        echo "[..] downloading jammy base to $LOCAL (snap-readable; NOT /tmp)"
        wget -q -O "$LOCAL" "$BASE_IMG_URL"
        GOT=$(sha256sum "$LOCAL" | awk '{print $1}')
        [ "$EXP" = "$GOT" ] || { echo "GATE FAIL: checksum mismatch exp='$EXP' got='$GOT'"; exit 1; }
        echo "[OK] checksum verified ($GOT)"
      fi
      echo "[..] uploading base to glance (qcow2; 5 retrofit props; NO amphora tag on the base)"
      BASE_ID=$(openstack image create "$BASE_NAME" \
        --file "$LOCAL" --disk-format qcow2 --container-format bare \
        --property architecture=x86_64 --property os_distro=ubuntu --property os_version=22.04 \
        --property version_name="$VERSION_NAME" --property product_name="$PRODUCT_NAME" \
        -f value -c id)
    fi
    [ -n "$BASE_ID" ] || { echo "GATE FAIL: base image id empty after seed"; exit 1; }
    echo "[OK] base image: $BASE_ID"

    # ---- Phase 2: retrofit (long-running build; bounded wait; tee the result) ----
    echo "-- retrofit-image action schema (informational; confirm source-image is honored) --"
    juju actions $RETRO --schema --format yaml 2>&1 | sed -n '/retrofit-image:/,/^[a-zA-Z]/p' | head -30 || true
    echo "[..] running retrofit-image (multi-minute build)"
    juju run $RETRO/leader retrofit-image source-image="$BASE_ID" --wait=30m 2>&1 | tee ~/retrofit-image.out
  fi

  # ---- Phase 3: confirm (amphora present + active + tagged == octavia's tag) ----
  echo "=== CONFIRM: images tagged $OTAG ==="
  openstack image list --tag "$OTAG" -f value -c ID -c Name -c Status
  ACT=$(openstack image list --tag "$OTAG" -f value -c Status | grep -xc active || true)
  [ "$ACT" -ge 1 ] || { echo "CONFIRM FAIL: no ACTIVE image tagged $OTAG"; exit 1; }
  echo "[OK] amphora present + active + tagged $OTAG (matches octavia amp-image-tag) -- D-021 complete"
)
```
**GATE:** an ACTIVE image tagged `octavia-amphora` whose tag matches `octavia amp-image-tag`.

---

## EXIT GATE (phase-05 complete)
- octavia/0 active/idle; lb-mgmt-net / lb-mgmt-subnetv6 / lb-mgmt-sec-grp present; o-hm0 up.
- An ACTIVE amphora image tagged `octavia-amphora`, tag matching `octavia amp-image-tag`.
- Octavia is fully enabled (control plane + amphora). The end-to-end LB validation
  (build -> listener -> pool -> health-monitor -> 2 members -> FIP; round-robin;
  admin-scope failover) is D-011 criterion 4 -- run in phase-08 (needs tenant
  scaffolding + the external provider network from phase-04).

## As-built reference (current rebuild 2026-06-30; per-deploy values regenerate -- old IDs are not discrepancies)
- octavia/0: octavia 14.0.0, charm rev 441 2024.1/stable, on 3/lxd/3, data leg 10.12.12.1;
  multi-homed (reaches provider VIPs over eth1).
- configure-resources op 35 / task 36 completed (2026-06-30; --wait=20m; prior: op 9/task 10).
  Created lb-mgmt-net / lb-mgmt-subnetv6 (IPv6 geneve) / lb-mgmt-sec-grp; o-hm0 UP with a fresh
  IPv6-ULA on the lb-mgmt prefix (fc00::/64; exact addr not captured this run -- the ULA regenerates per deploy).
- amphora: the retrofit (use-internal-endpoints=true) reaches the glance INTERNAL VIP 10.12.12.53
  (DOCFIX-061 / D-052; the old ".8.53" predates the metal-admin/metal-internal split -- .8.53 is now the
  ADMIN glance VIP). base jammy-amphora-base = 15a14c5a-... (untagged; sha256 070de108...; prior: da757cb1-...);
  retrofit op 39/task 40 (prior: op 13/task 14) built amphora-haproxy-x86_64-ubuntu-22.04-20260701
  = 807e3f5b-... ACTIVE, tag octavia-amphora (matches octavia amp-image-tag), image-format raw,
  owned by the services project (prior OUTPUT: ca5552a5-...).
- mgmt VM image ubuntu-24.04-noble is NOT pre-staged in phase-05 this rebuild; it is seeded in phase-06
  Step 6.0-BOOT (stage-and-verify). (Prior rebuilds pre-staged it here as a convenience; 06-16 id 899b4b5c-...)
- SEED METHOD this rebuild (2026-06-30): the amphora base was seeded via STAGE-AND-VERIFY -- download to
  $HOME + sha256 vs published SHA256SUMS + `openstack image create --file` -- per the canonical Step 5.2
  script (phase-05-amphora-pipeline.sh), NOT web-download. The staged --file base landed qcow2
  (CORRECTION-1); the retrofit consumed it and emitted the raw octavia-amphora OUTPUT. Web-download
  remains a tested alternative (appendix-A).
- Charm gap (parked): glance-simplestreams-sync is metal-only and cannot reach glance
  on a no-DNS deploy (use-internal-endpoints steers keystone auth but not the
  glance/swift client) -> gss does NOT seed the base. The base is seeded per Step 5.2
  and the amphora BUILD stays charm-native via the retrofit over internal
  endpoints. Roosevelt root-fix: cloud DNS + FQDN-valid certs (also fixes gss).

## Next
phase-06 -- in-cloud management cluster (D-035).
