Newer
Older
openstack-caracal-ipv4 / runbooks / d011-batch3-window-DRAFT.md

d011 batch-3 window -- combined do-document (DRAFT, not yet executed)

One logged operator window that closes out the D-011 batch-3 queue AND live-validates the onboarding workflow, in the operator-ruled interleave order (2026-07-06): D-073 live apply -> foil-tenant onboarding (= workflow validation) -> d011-04 -> d011-05 -> close-out. Every mutation is individually gated; read-only evidence precedes each. Standing authorizations already ruled (do not re-ask):

  • A2 (2026-07-06): d011-04 --include-disruptive is PRE-APPROVED conditional on the live N+1 amphora headroom check passing -- same window, no second ask.
  • D-073 (adopted 2026-07-06): list_trusts zip staged in repo (addendum 24); live attach in this window per the amendment.

Operator inputs needed at window open: foil client short-name (examples below use foil1), a non-colliding /24 for it (stage-4 guard fails closed regardless).

0. Window open (operator)

git -C ~/openstack-caracal-ipv4 pull
bash scripts/repo-lint.sh                  # expect 0 fail / 1 legacy warn
bash scripts/run-tests-all.sh              # ALL GREEN expected
bash scripts/run-logged.sh d011-batch3
bash scripts/cloud-assert.sh               # baseline MUST PASS before any mutation

1. D-073 live apply (GATED mutation: keystone policy resource)

# stage under $HOME -- the openstack/juju snaps cannot read /tmp
cp ~/openstack-caracal-ipv4/policies/overrides.zip ~/overrides-d073.zip
juju attach-resource -m openstack keystone policyd-override=~/overrides-d073.zip
# wait for keystone workload message to show "PO:" (NOT "PO (broken):")
juju status -m openstack keystone

Behavioral verify (never trust "PO:" alone):

# (a) tenant token MUST now be denied trust enumeration (403):
#     use the beta tenant's -svc app cred context ->  openstack trust list
# (b) admin context MUST still succeed:  openstack trust list
# (c) magnum trust machinery unaffected -- implicitly re-proven by d011-05 below.

Rollback if broken: git show HEAD~:policies/overrides.zip > ~/overrides-prev.zip then re-attach that file (same command shape). HOLD the window on any (a)/(b) miss.

2. Foil onboarding = live workflow validation

Stages 0-4 (foil needs an app cred for d011-05; stage 4 L3 enables the canary):

TENANT_CIDR=<ruled /24> bash scripts/tenant-onboard.sh foil1 stage0   # read-only
bash scripts/tenant-onboard.sh foil1 stage1                           # gated
bash scripts/tenant-onboard.sh foil1 stage2                           # gated
bash scripts/tenant-onboard.sh foil1 stage3                           # gated
TENANT_CIDR=<ruled /24> bash scripts/tenant-onboard.sh foil1 stage4   # gated

Then the H1 verifier -- this run ALSO clears its two VERIFY-LIVE items (keypair list --user, application credential list --user as admin):

bash scripts/tenant-assert.sh foil1        # expect: PASS (topology matches contract)

H2 guard live check (read-only in effect -- re-run stage3, expect double SKIP rc0):

bash scripts/tenant-onboard.sh foil1 stage3

H3 canary (gated; self-cleaning; VERIFY-LIVE: CANARY_SSH_USER vs image login user):

bash scripts/tenant-onboard.sh foil1 stage7    # expect: CANARY PASS + teardown done

Horizon checks (contract section 5.1/5.3; manual, browser):

  • login as foil1-domain-admin (domain-scoped) -- expect success;
  • GUI identity probe: create + delete a probe user via Horizon as the manager. If the GUI path fails but the API path passed (tenant-assert), LOG it and the contract's "CLI is the identity path" fallback stands -- not a window blocker.

3. d011-04 octavia (non-disruptive first, then the pre-approved disruptive half)

bash scripts/validate.sh --checks d011-04-octavia-lb                       # RR + member failover
# headroom evidence prints from amphora_headroom(); on PASS the A2 pre-approval applies:
bash scripts/validate.sh --checks d011-04-octavia-lb --include-disruptive  # amphora failover
# on headroom HOLD: STOP the disruptive half entirely (the pre-approval is conditional).

VERIFY-LIVE items this clears: amphora_headroom() field parsing OK-path; OCCM LB-name-contains-service-name assumption; agnhost /hostname round-robin behavior.

4. d011-05 magnum e2e (+ P3 isolation via the foil)

VR_FOIL_APPCRED=~/tenant-foil1/foil1-svc-appcred.txt bash scripts/validate.sh --checks d011-05-magnum-e2e

(Check ids per bash scripts/validate.sh --list; the full-d011 profile runs the whole suite if preferred once both new checks stand individually.)

5. Close-out

bash scripts/cloud-assert.sh               # MUST PASS
# evidence: keep the run-logged transcript; note headroom numbers + canary FIP
# in the ledger; mark cleared VERIFY-LIVE items in the handoff + contract 5.3.

Repo updates at close (jumphost lane): changelog addendum (as-executed corrections to THIS draft included -- it is a DRAFT until executed once), session-ledger jumphost section, handoff section 2/6 item states, D-011 acceptance status. If all of d011-01 through d011-06 now stand (06 stays MANUAL pending SEC-003), record the D-011 batch-3 close and queue the v1-close checklist (handoff section 6).

Abort discipline

Any cloud-assert FAIL, any tenant-assert HOLD on the foil, or any D-073 behavioral miss HOLDs the window at that boundary: capture evidence, revert only the failing step per its rollback line, close the log. No forward progress past a failed gate.