# v1 Redeploy -- Running Change Log

**Purpose:** Living log of design decisions, doc fixes, and runbook edits discovered
DURING the v1 redeploy rehearsal that must be folded into `docs/design-decisions.md`
and the phase runbooks UPON COMPLETION. This is the staging list for the completion
consolidation -- nothing here is applied to the runbooks or design-decisions yet.

**Status:** OPEN -- accumulating. Append-only. ASCII + LF.

**Session opened:** 2026-06-26 (redeploy from clean teardown; D-052/D-053 plane set).

**Next free numbers at session open:** design decision D-054; doc fix DOCFIX-039.
(Verified by grep of design-decisions.md: max D-053, max DOCFIX-038.)

---

## Verified-state checkpoint (measured this session -- authoritative as-built)

`scripts/pre-flight-checks.sh` @ commit 40e3f9e -- ALL PASS, exit 0, 2026-06-26:

Six MAAS planes resolved BY CIDR (subnet IDs are post-D-052-cutover, NOT the old map):

    provider-public  10.12.4.0/22   id=1   vid=0    gw=10.12.4.1   dns=[10.12.4.1]
    metal-admin      10.12.8.0/22   id=2   vid=0    gw=10.12.8.1   dns=[10.12.8.1]
    metal-internal   10.12.12.0/22  id=10  vid=103  gw=none        dns=[10.12.8.1]  (bridged br-internal)
    data-tenant      10.12.16.0/22  id=6   vid=0    gw=none        dns=[10.12.8.1]
    storage          10.12.32.0/22  id=7   vid=0    gw=none        dns=[10.12.8.1]
    replication      10.12.36.0/22  id=8   vid=0    gw=none        dns=[10.12.8.1]

Per-host data/storage NIC links by CIDR, octets .40-.43, all four hosts:
br-internal -> .12, enp8s0 -> .16, enp9s0 -> .32, enp10s0 -> .36.

Nodes openstack0-3 (4na83t / qdbqd6 / h8frng / tmsafc): all Ready, power off.
OSD secondary disks (`osd-blank-check.sh`): all four 512 GiB / 200 KiB blank, RC=0.
Bundle VIPs: 11 triple-column VIPs, aligned, .50-.60 band, OK=11 bad=0.
octavia-pki overlay: present, 5 lb-mgmt-* keys, ASCII clean.

---

## Pending design-decisions.md appends

### D-054 -- Reusable tested scripts in scripts/; runbooks reference them (ADOPTED in practice; formal append pending)

**What:** Repeated discovery/verify logic lives in `scripts/`, authored and tested in a
sandbox against synthetic fixtures, committed to the repo, and referenced by the runbooks.
Runbooks document expected output and remain the gate authority; the scripts are the
executable truth. All pinned network values live once in `scripts/lib-net.sh` (single
source of truth), resolved BY CIDR (subnet IDs drift across cutovers).

**Delivery workflow:** author + test in sandbox -> publish file + sha256 -> commit from
Windows -> jumphost `git pull` -> `sha256sum` match -> run via `bash scripts/X.sh`.

**Convention:** ASCII + LF (`.gitattributes` `*.sh eol=lf`); `set -euo pipefail` +
`shopt -s inherit_errexit` + `IFS=$'\n\t'`; `fail`/`warn`/`pass`/`note` helpers with
exit 0 (pass) / 1 (fatal) / 2 (warning) for gate scripts; read-only discovery kept
separate from gated mutation; `lib-net.sh` is sourced, never executed (direct-run guard).

**Why:** Eliminates the paste-corruption failure class (see Findings below) and turns
repeated discovery -- polled every redeploy cycle -- into a one-liner with a byte-identity
guarantee (sha256) instead of a fragile copy-paste block.

**Scripts added this session:** `lib-net.sh` (new), `pre-flight-checks.sh` (implemented the
placeholder), `juju-spaces-check.sh` (new), `osd-blank-check.sh` (new). All tested
end-to-end against mock `maas`/`juju` + fixtures (positive + 7 negative fault injections
for pre-flight; 4 scenarios for spaces). Committed at 40e3f9e.

---

## Pending DOCFIX entries

### DOCFIX-039 -- phase-01-bundle-deploy.md gate reconciliation (PROPOSED)

The phase-01 pre-deploy GATES encode the OLD plane layout (pre-D-052 CIDR->role map); the
deploy COMMANDS are fine. Superseded by `scripts/pre-flight-checks.sh`. Five stale items:

1. Constants: hardcoded subnet ids `1 2 6 7 8 9` + old CIDR->role map -> resolve BY CIDR
   (now in `lib-net.sh`; metal-internal is id=10 post-cutover, not id=6).
2. CHECK 1 / Step 1.3 deploy guard: provider-column-only VIP check -> triple-column
   validator (provider/admin/internal, aligned, .50-.60).
3. CHECK 2: `enp8s0` + `10.12.12.0/22` (old "data") -> links BY CIDR; `enp8s0` now carries
   `10.12.16.0/22` (data-tenant), metal-internal is on `br-internal`.
4. CHECK 3: hardcoded ids/DNS -> subnets BY CIDR.
5. EXIT GATE binding plane map (old: ceph->.16 / octavia->.12.1 / nova->.12.4x / vault->.8)
   -> corrected per D-052: ceph public/osd/mon->storage(.32); octavia overlay->data-tenant
   (.16); nova-compute neutron-plugin->data-tenant(.16); vault default->metal-admin(.8) +
   cluster->metal-internal(.12).

**Action at completion:** replace the inline CHECK blocks in phase-01 with
`bash scripts/pre-flight-checks.sh` (document expected PASS output) and add a post-add-model
`bash scripts/juju-spaces-check.sh openstack` as the per-model space gate (the old inline
CHECK 5 ran `juju spaces` pre-model and failed "model not found"; spaces are per-model).

---

## Pending runbook / file edits (apply at completion)

1. `runbooks/phase-01-bundle-deploy.md` -- DOCFIX-039 (above): swap inline pre-flight blocks
   for `bash scripts/pre-flight-checks.sh`; add post-add-model `bash scripts/juju-spaces-check.sh
   openstack`; fix the 5 stale gate items; document expected output.
2. `scripts/validate.sh` -- convert UTF-8 to ASCII when implementing the D-011 runner
   (phase-08). `file` reports "Unicode text, UTF-8 text" (em-dashes from the placeholder);
   violates the ASCII-only convention. Currently a placeholder, not yet run.
3. Teardown runbook -- reference `scripts/osd-blank-check.sh` for the OSD-blank verification
   step (replaces the inline qemu-img loop).
4. `runbooks/` README / pre-flight references -- point at the new scripts where the old
   inline discovery blocks were described.

---

## Findings / process learnings (this session)

- **Paste-corruption failure class.** A hand-built base64 pre-flight block shipped two
  transcription defects: `[:space:]` (single bracket, must be `[[:space:]]`) on the grep
  count line, and `ENV{` instead of `END{` on the awk tally (so the summary silently never
  printed). Root cause: the base64 was hand-edited AFTER testing a clean version -- the
  bytes sent were never round-tripped through the sandbox. Mitigation is now standard
  practice (D-054): tested scripts committed to the repo, verified by sha256 on the jumphost.

- **Juju spaces are per-model.** `juju spaces` / `juju reload-spaces` cannot run until after
  `juju add-model`; the old phase-01 CHECK 5 ran pre-model and failed with "model not found".
  Split into `juju-spaces-check.sh`, gated to run post-add-model.

- **Default-space globally poisons network-get (deploy root cause).** The full D-052
  binding deploy failed universally (`network-get ... ERROR space "metal" not found`,
  install hook dies on nearly every charm). Every static layer was correct -- bundle,
  model bindings, MAAS spaces/VLANs/per-NIC space tags all read `metal-internal`. The
  single stale value was controller `model-defaults default-space = metal` (a dead
  pre-D-052 name). An INVALID default-space poisons `network-get` for ALL endpoints
  regardless of their explicit binding. Fix: set `juju model-defaults
  default-space=metal-admin` (a live space) before add-model. A `default-space`-resolves-
  to-a-live-space gate is to be added to `pre-flight-checks.sh`.

- **Teardown --destroy-storage on virsh DELETES machine objects (does NOT release).**
  The phase-00 teardown (`juju destroy-model openstack --force --destroy-storage` then
  per-host `maas machine release`) assumes release-to-Ready. On a virsh/KVM MAAS,
  `--destroy-storage` DECOMPOSES (deletes) the VM-backed machine objects. All four
  openstack hosts were removed from MAAS. Recoverable only because the libvirt domains
  + disks (incl the blank OSD vdb) survived. See D-055.

---

## Pending design-decisions.md appends (continued)

### D-055 -- virsh teardown defect + host re-enrollment procedure (ADOPTED)

**Defect:** `juju destroy-model --destroy-storage` against virsh-power MAAS machines
deletes (decomposes) the machine objects rather than releasing them to Ready. The
phase-00 teardown must NOT pass `--destroy-storage` for virsh hosts; release to Ready
without it.

**Recovery (now a reusable procedure):** the libvirt domains survive, so re-enroll via
`maas admin machines create` per host with virsh power + the boot NIC MAC (NOT add-chassis
-- it would re-grab juju/lxd/tailscale). `machines create` auto-commissions
(New->Commissioning->Ready) by PXE off the 2_metal boot NIC. Then re-tag `openstack`,
then reconstruct the host interface tree (Strategy-B carve, from the captured as-built),
then verify (pre-flight), then redeploy with the default-space fix.

**Artifacts:** `scripts/lib-hosts.sh`, `scripts/reenroll-hosts.sh`,
`docs/maas-as-built-reference.md`. Proven live on openstack0 (2026-06-26): created
virsh, commissioned, Ready, all six NICs discovered, boot NIC on 2_metal.

### DOCFIX-040 -- host identity must be hostname-keyed, not system_id-keyed

`lib-net.sh` lines 45-47 key the host maps (`SYSIDS`, `SYSID_HOST`, `SYSID_OCTET`) on
the system_ids 4na83t/qdbqd6/h8frng/tmsafc -- which DIED on re-enrollment (new random
ids). Any script keyed on them silently breaks. New `scripts/lib-hosts.sh` keys all host
identity on hostname (stable) and resolves system_id at runtime (`host_sysid`). At
completion: retire the SYSID-keyed maps from lib-net.sh (or repoint them to lib-hosts).

---

## Security note (action required)

The libvirt SSH password (`logxen@10.12.64.1`) was printed in plaintext on 2026-06-26 by
`maas admin machine power-parameters` during virsh power-template discovery. Treat as
exposed: **rotate the libvirt SSH credential after the rebuild** and scrub terminal
scrollback. Runbook rule added: never use `machine power-parameters` for templating; read
`power_type` and reconstruct the address pattern instead. `reenroll-hosts.sh` reads the
password interactively (never a CLI arg, never logged, never in the repo).

---

## Scripts / docs added (this batch)

- `scripts/lib-hosts.sh` -- hostname-keyed host identity + virsh power constants (no secret).
- `scripts/reenroll-hosts.sh` -- gated/idempotent re-enrollment (auto-commission, poll Ready,
  boot-NIC-on-2_metal verify; --check read-only mode). Tested: bash -n, shellcheck clean,
  mock-maas behavior test of --check (discover-by-hostname, NOT-ENROLLED detection, exit 0).
- `docs/maas-as-built-reference.md` -- captured MAAS substrate + per-host NIC inventory +
  interface-carve target + virsh template, for DC-DC replay.
- Pending next artifact: the Strategy-B interface-carve script (built once all four are Ready;
  bridge_type pulled verbatim from captured release JSON) -> then consolidate into
  `runbooks/phase-00b-host-reenrollment.md`.

### DOCFIX-041 -- as-built reference: br-ex is charm-built, not a MAAS bridge

Correction to `docs/maas-as-built-reference.md` (first committed this session). The
bundle's ovn-chassis `bridge-interface-mappings` maps `br-ex:<provider-MAC>` for all
four hosts -> **br-ex is built by the ovn-chassis charm at deploy (OVS), enslaving the
provider NIC by MAC; it is NOT a MAAS interface.** The MAAS carve therefore:
- provider plane = **raw enp1s0 + static 10.12.4.N** (MAAS leaves it raw; the charm
  enslaves it into br-ex at deploy). MAAS does NOT create br-ex.
- storage/replication = raw enp9s0/enp10s0 + statics; Juju auto-bridges them
  (br-enp9s0/br-enp10s0, Linux) at deploy.
- the ONLY MAAS-built bridges are the metal-internal stack:
  enp7s0 -> br-metal -> br-metal.103 (VID 103) -> br-internal.

bridge_type: br-internal = standard (confirmed, D-052 command). br-metal = standard
(RECOMMENDED, reasoned-not-measured -- original bring-up predates the repo and the
capture did not preserve bridge_type; pending confirm before carve). The
deployed-host `ip`-level read that showed br-metal/br-internal "OVS" was taken during
the FAILED deploy and is reclassified UNRELIABLE.

### Carve script added + MAAS interface CLI confirmations

- `scripts/carve-host-interfaces.sh <hostname> [--apply]` -- Strategy-B per-host
  interface carve. Default DRY-RUN (resolves every id live, prints each mutation it
  WOULD run, changes nothing); --apply executes. Idempotent (skips existing
  bridge/vlan/link), resolves system_id by hostname / interface id by name / subnet
  id + VLAN object id by CIDR, asserts metal-internal is VID 103, requires Ready.
  Builds: enp1s0 raw+static (provider); enp7s0 -> br-metal(std) -> br-metal.103(VID
  103) -> br-internal(std); enp8/9/10 raw+static (data/storage/repl); enp11s0 idle.
  Does NOT create br-ex (charm-built). Tested: bash -n, shellcheck clean, mock-MAAS
  dry-run (full id resolution + command preview), input guards.

- MAAS 3.7 interface CLI confirmed (canonical.com/maas/docs/3.7 reference):
  create-bridge takes `bridge_type=standard|ovs parent=<ifid> vlan=<vlan-obj-id>`;
  create-vlan takes `vlan=<VLAN-OBJECT-ID> parent=<ifid>` (NOT the VID tag -- resolve
  the object id via the metal-internal subnet); link-subnet `mode=STATIC
  subnet=<id> ip_address=<ip>`; a NIC is moved to a plane's fabric via `interface
  update <sid> <ifid> vlan=<vlan-obj-id>` before link-subnet (re-enrolled raw NICs
  sit on transient auto-fabrics).

- FINDING (teardown runbook bug): `runbooks/phase-00-teardown-maas-reset.md`
  "Phase 3" link-subnet block uses PRE-D-052 CIDRs
  (`enp8s0=10.12.12.0/22 enp9s0=10.12.16.0/22 enp10s0=10.12.20.0/22`) and dead
  system_ids -- it would link NICs to the WRONG subnets (10.12.12 is now
  metal-internal, 10.12.16 is now data-tenant, 10.12.20 no longer exists). Must be
  rewritten to current planes + hostname-keyed before that runbook is trusted. Note:
  the normal release-to-Ready path PRESERVES host interfaces, so that block only ran
  on a normal teardown; the full carve (this script) is needed only after a
  decompose, which is why the bridges were never scripted before.

### Carve hardening: self-discovered metal IP blocks br-metal static (KI)

Root cause (cost several diagnostic rounds): after re-enrollment each host PXE-leases
its own metal IP (10.12.8.4N) at commission. MAAS records this as a StaticIPAddress
of **alloc_type 6 (DISCOVERED)** tied to the node via its boot NIC. This is a
SEPARATE object from the network-discovery table (`discoveries clear-by-mac-and-ip`
does NOT clear it) and from user allocations (`ipaddresses read` user-scope does NOT
show it). It causes `link-subnet ... ip_address=10.12.8.4N` to fail with the
misleading "IP address is already in use".

Authoritative read (the lesson): `maas admin subnet ip-addresses <subnet_id>` reports
every in-use IP WITH its alloc_type and owning node -- this is the single correct
"who holds this IP and why" query. Lead with it; do not probe ipaddresses/discovery/
leases piecemeal.

Release: `maas admin ipaddresses release ip=<ip> force=true discovered=true` (BOTH
flags required; force alone returns "does not exist" for a discovered address).

Script fix (carve-host-interfaces.sh): `release_self_discovered()` runs before every
STATIC link -- releases an alloc_type-6 record for the target IP ONLY when its owning
node == this host (node_summary.system_id), and REFUSES (fatal) if a different node
discovered it (a real conflict). Plus `emit` now captures and prints the MAAS error on
a failed mutation instead of discarding it to /dev/null (the discard hid the real
message and prolonged diagnosis). Only the metal plane (dhcp_on=true) is affected;
the no-DHCP planes never produced a self-lease. Verified: mock self-release path +
foreign-node refuse gate.

NOTE (design consistency, not a blocker): host statics .40-.43 sit inside the
metal-admin/provider/internal VIP+mgmt reserve band (.2-.100). A reserved range blocks
AUTO assignment, not explicit STATIC, so it did not break the carve -- but host octets
arguably belong outside the VIP band. Log for the reserve-layout review.

### DC-DC script audit (post-carve hardening batch)

Reviewed all MAAS scripts against what this session actually hit, so the DC-DC build
replays cleanly instead of re-deriving the metal-IP archaeology.

- **carve gate rewrite (the big one).** `release_self_discovered` keyed on
  `node_summary.system_id`, which is EMPTY on a fresh discovered record -> it silently
  no-op'd and the metal static (.8.41/.42/.43) had to be released by hand on three
  hosts. Replaced with `release_self_indexed`: the target is this host's
  architecturally-indexed metal IP (10.12.8.<octet> from HOST_OCTET), so a DISCOVERED
  observation on it is this host's own commissioning ghost. SAFETY: refuses if the
  record's system_id (when present) OR the discoveries-table MAC (when present)
  identifies a DIFFERENT host; releases otherwise. Removed the (unneeded) release call
  from carve_raw -- the no-DHCP planes never produce discovered records. Tested: 5
  branches (foreign-sysid refuse, foreign-MAC refuse, indexed-basis release, MAC-basis
  release, no-record no-op).

- **missing step added: openstack tag.** `reenroll-hosts.sh` now ensures the
  `openstack` tag exists and applies it to all four hosts after the Ready/boot-NIC
  gate (idempotent; --check-aware). Without it the bundle cannot place units
  (constraint tags=openstack). Was a manual step every rebuild.

- **DOCFIX-040 COMPLETE.** `pre-flight-checks.sh` and `osd-blank-check.sh` both looped
  over the dead system_ids (4na83t...) via lib-net's SYSID maps -- broken for any
  rebuilt/DC-DC cluster. Migrated both to hostname-keyed (lib-hosts HOSTS / HOST_OCTET
  / host_sysid). Retired the SYSID/SYSID_HOST/SYSID_OCTET maps from lib-net.sh and
  added its sourced-library shellcheck directive. osd-blank verified via mock
  (iterates the four hostnames, RC=0).

- **validate.sh**: em-dashes -> ASCII (the silent-UnicodeDecodeError class; ASCII-only
  rule for all scripts). Still a placeholder body otherwise.

REMAINING DC-DC scope (done MANUALLY this session; scripting them would make the
bring-up fully hands-off -- NOT yet built):
1. A multi-host carve-verify wrapper (assert all four hosts show the six expected
   static links on the right fabrics) -- currently an ad-hoc jq loop.
2. A redeploy-prep wrapper: set model-defaults default-space=metal-admin, add-model,
   verify the MODEL's effective default-space (the value that poisoned the last
   deploy), reload-spaces, run juju-spaces-check. Currently manual steps R1-R3.

---

## Phase-02 vault bring-up (as-executed -- COMPLETE 2026-06-27)

**Session:** 2026-06-27. Origin/jumphost HEAD at phase-02 start: 1a103f5 ("Create
phase-02-vault-preflight.sh"; was 68a0bd5 at the redeploy handoff). Model: openstack.
Next free numbers at section open: design decision D-056; doc fix DOCFIX-042 (verified
by grep: changelog max D-055 / DOCFIX-041; design-decisions max D-053 / DOCFIX-038).

### Pre-flight gate (Step 2.1 verify-before-mutate) -- PASS

Manual A-E audit on the jumphost cleared all gates; the new
`scripts/phase-02-vault-preflight.sh` then reproduced it identically with REAL jq:

    A auth      jessea123 / juju-controller / model openstack; no macaroon EOF.
    B machines  4/4 started.
    C mysql     mysql-innodb-cluster 3/3 -- /0 R/W, /1 R/O, /2 R/O, all
                "Cluster is ONLINE and can tolerate up to ONE failure." (vault backend OK)
    D vault     vault/0 [blocked] "Vault needs to be initialized" -- FRESH
                (irreversibility guard satisfied).
    E census    units=63  workload-error=0  agent-error(hook)=0
                blocked=2 (vault + octavia "Awaiting configure-resources")
                waiting=9  active=51  unknown=1 (glance-simplestreams-sync).

Census 63 vs the handoff's 31 is NOT a discrepancy: the handoff counted PRINCIPALS
(active=25/blocked=2/waiting=3/unknown=1); the script recurses into subordinates.
waiting 3->9 reconciles against the handoff prose (ovn-central x3 principals + ovn-chassis
x3 + ovn-chassis-octavia + neutron-api-plugin-ovn + nova-compute certs); active 25->51 is
the hacluster/mysql-router/filebeat subordinate layer. blocked=2 and unknown=1 match exactly.

### FINDING -- live machine IDs are 0-3, NOT the handoff's 8-11   [-> DOCFIX-043]

Committed bundle.yaml declares SYMBOLIC machine IDs "8"/"9"/"10"/"11" (machines: section,
constraints tags=openstack). Juju treats bundle machine keys as PLACEHOLDERS, not real IDs;
deployed into a fresh model they map in order to real IDs 0/1/2/3. Live (confirmed by the
preflight script's machine display lines):

    real m0 = openstack0 = 10.12.12.40   (bundle "8")   control-only, 7 LXD: 0/lxd/0..6
    real m1 = openstack1 = 10.12.12.41   (bundle "9")
    real m2 = openstack3 = 10.12.12.43   (bundle "10")
    real m3 = openstack2 = 10.12.12.42   (bundle "11")  holds vault/0 (juju-f5a310-3-lxd-5)

The openstack2/openstack3 <-> m3/m2 "swap" is MAAS tag-based allocation (hosts pinned by
tag=openstack, NOT by system_id), so the host->machine binding floats per deploy.
nova-compute `to: ["9","10","11"]` (symbolic) therefore landed on real m1/m2/m3 =
openstack1/openstack3/openstack2, leaving m0/openstack0 control-only -- CONSISTENT with the
handoff's intended role split. ceph `to: ["8","9","10","11"]` -> all four real machines.

IMPACT: zero on phase-02 (vault/0 resolves by unit name). The handoff text "= bundle
machines 8/9/10/11" is stale on LIVE ids. RULE to fold into the runbook: resolve everything
by unit name / hostname / CIDR, NEVER by machine ID; document the bundle-symbolic vs
live-real mapping so a future operator does not mistake it for a deploy fault. Phase-03
host-role verify (open-item 2) confirms which units run on which machine definitively.

### DELIVERABLE -- scripts/phase-02-vault-preflight.sh   (committed 1a103f5)   [-> DOCFIX-042]

Read-only verify-before-mutate gate packaging the A-E audit into one re-runnable command.
Mutates NOTHING; the vault init/unseal/authorize MUTATIONS stay gated human steps (item-8
principle: scripts own the deterministic/read-only/repeated; the human gate owns the
consequential mutation + secret custody). Gates: B all machines started; C mysql 3 units /
all active+ONLINE / exactly 1 R/W; D vault fresh ([blocked] "needs to be initialized" --
REFUSES and escalates if not, since a non-fresh vault may already hold keys); E zero
workload-error AND zero agent-error(hook), subordinates included. Exit 0 PROCEED / 1 HOLD /
2 precondition. Sources lib-net.sh (need_jq); whoami-direct-first so a stale-macaroon prompt
reaches the tty before captured calls; single juju-status snapshot; one jq metrics pass
(eval'd key=value); dynamic lookups, nothing host/IP/ID hardcoded. ASCII+LF, bash -n clean.

Testing: shellcheck + jq both ABSENT from Claude's sandbox -> behavior-tested with juju+jq
shims across 5 fixtures (1 healthy + 4 single-fault: vault-already-initialized D, mysql
OFFLINE C, hook-failure E, machine-down B); each produced the correct exit code and gate
attribution. jq metrics algorithm mirrored/validated in Python. REAL-jq/REAL-data
confirmation on the jumphost first run reproduced the manual audit EXACTLY (units=63,
errors 0, PROCEED, EXIT=0); Windows -> GitHub Desktop -> push -> jumphost-pull preserved
LF/ASCII/parse. FOLD INTO phase-02 do-doc: invoke this script as the Step 2.1 pre-flight gate.

### DELIVERABLE -- tests/phase-02/ regression harness   (staged; commit pending)

run-tests.sh + make_fixtures.py + fakebin/{juju,jq} shims. Offline regression for the
preflight script: drives the REAL script's decision/exit logic against the 5 generated
fixtures; touches NO live infra (fake juju emits fixtures, fake jq mirrors the metrics in
Python); runs anywhere with python3 + bash (no real jq needed); re-asserts shim exec bits so
the Windows -> git round trip dropping them will not break it. Sandbox run: ALL PASS / exit 0.
Target paths: tests/phase-02/{run-tests.sh, make_fixtures.py, fakebin/juju, fakebin/jq}.

### Step 2.1 delivery split   [-> DOCFIX-042]

The do-doc presents Step 2.1's in-session block as one paste (env-setup; vault status;
vault operator init | tee; grep -c; grep -q). Split at the verify/mutate boundary into two
gated pastes:
  2.1a (read-only verify): `export VAULT_ADDR...; umask 077; mkdir -p ~/vault-init` +
       `vault status 2>&1 | grep -E 'Initialized|Sealed|Storage Type|HA Enabled' || true`
  2.1b (irreversible)     : `vault operator init -key-shares=5 -key-threshold=3 2>&1 | tee
       ~/vault-init/init.txt` + `grep -c '^Unseal Key' ...` + `grep -q '^Initial Root Token:' ...`
Rationale: the `vault status` line exists to be OBSERVED before the irreversible init; a
single paste runs init before it can be read, defeating verify-before-mutate. Commands are
verbatim/unchanged -- only the paste boundary moves. Amend phase-02 do-doc Step 2.1 to
present 2.1a/2.1b as two gated pastes.

### Step 2.1a verify -- FRESH confirmed (2026-06-27)

Session opened on vault/0: `juju ssh -m openstack vault/0` -> ubuntu@juju-f5a310-3-lxd-5
(= real machine 3 = openstack2, LXD container 5). 2.1a output:

    Initialized        false      uninitialized; safe to init
    Sealed             true
    Storage Type       mysql      vault-on-mysql backend (mysql-innodb-cluster)
    HA Enabled         false      CORRECT for vault-on-mysql (R3); NOT a defect

Vault's own status agrees with the Juju workload-status. Cleared for 2.1b (vault init).

### Step 2.1b vault init -- EXECUTED 2026-06-27 (irreversible one-shot done)

`vault operator init -key-shares=5 -key-threshold=3 2>&1 | tee ~/vault-init/init.txt` ran
once. Token gate: `grep -q '^Initial Root Token:'` -> TOKEN_OK (root token line captured in
init.txt). Unseal-Key count gate (`grep -c '^Unseal Key'` MUST = 5): = 5 (operator confirmed); not
inferred. Operator confirmed all key material (5 shares + root token) saved OFF cloud/host;
~/vault-init/init.txt on the unit is the only on-unit copy (dies with the unit).

Post-init expected state: vault Initialized true / Sealed true (init does NOT unseal a
vault-on-mysql; unseal is the separate 2.2 step).

### Step 2.2 unseal -- EXECUTED 2026-06-27
3-of-5 via `vault operator unseal` (no arg, vault's own hidden prompt; keys never on argv/
history -- L4). Final `vault status`: Initialized true / Sealed false / Storage Type mysql /
HA Enabled false (HA false correct for single-unit vault-on-mysql -- R3). Vault is now
initialized AND unsealed. v1 policy: MANUAL unseal is the v1 standard -- re-run 3-of-5 at
the hidden prompt after any vault-unit reboot (auto-unseal via transit/KMS not configured
in v1; D-011.6 re-confirms in phase-08).

### Step 2.3 authorize-charm + generate-root-ca -- EXECUTED 2026-06-27
Short-lived child token (10m TTL) minted in vault/0 via hidden `read -s` root token +
`vault token create -ttl=10m -field=token` (NOT the root token -- juju op-log persists
action params; DOCFIX-011 param=`token`). `juju run vault/leader authorize-charm token=...`
then `juju run vault/leader generate-root-ca` (REQUIRED -- DOCFIX-014) both completed; child
token entered via hidden `read -s` on the jumphost too (narrows, does not eliminate, op-log
exposure). Root CA PEM emitted ("Vault Root Certificate Authority (charm-pki-local)") and
copied OFF cloud.

Result (juju status vault): vault 1.8.8 active "Unit is ready (active: true, mlock:
disabled)"; vault/0 active/idle on 3/lxd/5 (= machine 3 = openstack2; container 10.12.12.106);
vault-mysql-router/0 active. The "Missing CA cert" block cleared STRAIGHT to active --
validates DOCFIX-014. `mlock: disabled` is expected/benign for container vault (no IPC_LOCK).

### PHASE-02 EXIT GATE -- MET (2026-06-27)
- Vault Initialized true / Sealed false; 5 shares + root token saved OFF cloud/host.      [DONE]
- vault/0 active/idle; root CA generated (the cloud's PKI anchor); PEM saved off-cloud.    [DONE]
- Narrow cert cascade to consumers (ovn-central x3, ovn-chassis x3, ovn-chassis-octavia,
  neutron-api-plugin-ovn, barbican-vault) ACTIVE/proceeding -- watched + accepted phase-03. [IN PROGRESS]
- POST-INIT SWEEP -- cascade SETTLED 2026-06-27 (full juju status, two-image capture):
  all apps active EXCEPT octavia (blocked "Awaiting ... configure-resources" -- EXPECTED,
  phase-05) and glance-simplestreams-sync (unknown -- expected sync state). Cert consumers
  now active: ovn-central x3 (leader ovnnb_db/ovnsb_db; northd active), ovn-chassis x3,
  ovn-chassis-octavia, neutron-api-plugin-ovn, barbican + barbican-vault. NO errors / NO
  hook failures.
  * magnum/0 active "Unit is ready" (1/lxd/2, 10.12.12.115; public port 9511/tcp) -- the
    phase-01 pre-vault 9501 loopback BLOCK self-resolved at the TLS cutover, as predicted.
    Definitive *:9501 not-loopback bind check via read-only `ss` on magnum/0 (juju exec):
    9501 -> `*:9501` (all-interfaces; NOT 127.0.0.1); 9511 -> `0.0.0.0:9511` + `*:9511`.
    NOT loopback -> escalation condition NOT met; the phase-01 9501 line was the expected
    pre-vault posture, NOT a defect. Settle also confirmed at principal level via
    deploy-watch.sh: active=29 / blocked=1 (octavia) / unknown=1 (glance-ss-sync) = 31
    principals -- reconciles with the handoff's original 31. **PHASE-02 EXIT GATE CLOSED.**
  * keystone/0 "PO (broken): Unit is ready" -- UNCHANGED (FINDING-1; expected; no regression).
  * Host-role confirm (open-item 2): nova-compute on machines 1/2/3 = openstack1/openstack3/
    openstack2; openstack0 (m0) carries NO nova-compute / NO ovn-chassis (control-only, 7
    LXD: 0/lxd/0..6). CONFIRMS the bundle-symbolic->live-real machine-ID remap (DOCFIX-043)
    and the intended 3-compute/1-control split. Open design Q remains (Jesse's call, not
    phase-02): openstack0's provider MAC is still in ovn-chassis bridge-interface-mappings
    though no chassis runs there -- trim it (3-compute/1-control) vs add openstack0 to
    nova-compute `to:`.

PHASE-02 COMPLETE -- discrete vault mutations done. Cascade-settle + the post-init sweep are
the opening activities of phase-03 (runbooks/phase-03-core-verify.md).

### Next-free numbers after this append
Design decision: D-056. Doc fix: DOCFIX-044.
  DOCFIX-042 = phase-02 Step 2.1 split (2.1a verify / 2.1b init) + invoke preflight script.
  DOCFIX-043 = document bundle-symbolic vs live-real machine-ID remap + MAAS tag-allocation
               host swap; resolve by unit/hostname/CIDR, never by machine ID.

---

## Phase-03 core verify (as-executed -- IN PROGRESS)

**Session:** 2026-06-27 (continues). Next free numbers at section open: D-056; DOCFIX-044.

### CORRECTION (DOCFIX-044) -- phase-02 preflight hook-error key wrong (agent-status -> juju-status)

scripts/phase-02-vault-preflight.sh (committed 1a103f5) computed the agent/hook-error count
as `select(."agent-status".current=="error")`. In `juju status --format json` a UNIT carries
`workload-status` + `juju-status` (the agent state: idle/executing/error); there is NO
`agent-status` key on units. Confirmed against two authoritative consumers: deploy-watch.sh:43
(`.value."juju-status".current=="error"` for units) and the phase-03 do-doc acceptance walk
(`u.get('juju-status')`). So the `ae` (hook-failed) half of the E gate was INERT -- it read a
nonexistent key and always returned 0; a real hook failure would NOT have been caught.
- Decision impact this run: NONE. The cloud had zero errors of either kind, so 0/0 was the
  correct verdict regardless; the workload-error half (workload-status, correct key) worked.
  The defect is a latent false-negative, not a wrong decision.
- Why the harness missed it: the phase-02 mock fixtures + jq shim used the SAME wrong key, so
  the regression validated internal consistency against a fiction, not the real schema.
  LESSON (fold into conventions): mock fixtures MUST mirror the real `juju status` JSON schema;
  a fixture that agrees with the script's bug hides the bug. The phase-03 harness surfaced this
  because its fixtures (built from the do-doc's juju-status walk) disagreed with the bad key.
- FIX: phase-02 preflight `ae` -> `select(."juju-status".current=="error")` + an anti-regression
  header note. Harness corrected (fixtures + shim now juju-status); the FAIL-E case now sets
  juju-status.current=error and only passes because the key is right. RE-COMMIT REQUIRED over
  1a103f5. Re-running on the (healthy) cloud still yields PROCEED; the fix matters for catching
  FUTURE hook failures.

### Step 3.1 core verify -- PASS (2026-06-27)
3.1a acceptance walk: 2 non-active/idle, BOTH expected -- glance-simplestreams-sync/0 (unknown,
image-sync state) + octavia/0 (blocked "Awaiting configure-resources", D-021). No TLS consumer
stuck. 3.1b haproxy backend-health sweep (D-045/DOCFIX-031): ZERO DOWN across all principal
units -- the plaintext-vs-SSL backend failure did NOT recur this cycle (cert cascade + haproxy
reload state healthy). No remediation needed.

### DELIVERABLE -- scripts/phase-03-core-verify.sh + scripts/phase03_accept_walk.py + tests/phase-03/
Read-only Step 3.1 gate packaging 3.1a (acceptance walk) + 3.1b (haproxy sweep). HARDENED beyond
the do-doc's bare count gate: phase03_accept_walk.py gates on IDENTITY -- only octavia
(blocked/configure-resources) and glance-simplestreams-sync (unknown/waiting) may be
non-active/idle; a different app blocked also yields count==2 yet correctly FAILS. The do-doc's
inline python-in-bash acceptance walk is moved to its own tested .py (convention); the haproxy
sweep's unit list comes from jq on the captured snapshot (no second juju call, no inline python).
Mutations stay gated: a DOWN backend's `haproxy -c` + `systemctl reload` is a per-unit human step;
Step 3.2 (admin-openrc) and 3.3 (Horizon) too. tests/phase-03/: unit-tests the .py
(pass/unexpected-blocked) + behavior-tests the .sh with juju+jq shims (settled / unexpected-unit /
injected haproxy-DOWN). ALL PASS, offline, no real jq. Real-jq/real-data: 3.1a+3.1b already ran by
hand this session and PASSED; the script reproduces them.

### Next-free numbers after this append
Design decision: D-056. Doc fix: DOCFIX-045.
  DOCFIX-044 = phase-02 preflight hook-error key agent-status -> juju-status (+ harness fix).

### Artifact validation -- all four confirmed (2026-06-27, post-commit, on jumphost)
1. tests/phase-02/run-tests.sh: ALL PASS / 0 (corrected FAIL-E now drives juju-status.current=error).
2. tests/phase-03/run-tests.sh: ALL PASS / 0 (accept-walk + haproxy gate incl injected-DOWN).
3. scripts/phase-03-core-verify.sh LIVE: PROCEED / 0 -- accept walk 2 expected (glance-ss unknown,
   octavia blocked); haproxy sweep ZERO DOWN across 31 principal units. Reproduces manual 3.1.
4. scripts/phase-02-vault-preflight.sh LIVE: HOLD / 1 on gate D ONLY (vault/0 now [active],
   units=1 fresh=0) -- irreversibility guard correctly refusing re-init of a live vault. B/C/E pass.
   DOCFIX-044 closed with REAL-DATA confirmation: the corrected `ae` ran live and reported
   agent-error(hook)=0 via juju-status (post-settle census: units=63, workload-error=0,
   agent-error=0, blocked=1 [octavia], waiting=0, active=61, unknown=1).

### Step 3.2 build admin-openrc -- PASS (2026-06-27)
Vault root CA pulled via get-root-ca --format json + jq (DOCFIX-021 path): CN=Vault Root
Certificate Authority (charm-pki-local), valid 2026-06-27 -> 2036-06-24. Admin password from
get-admin-password --format json; admin project DISCOVERED via the scope-test loop (DOCFIX-022;
value recorded in ~/admin-openrc OS_PROJECT_NAME, not captured this turn). ~/admin-openrc written
(chmod 600); `openstack endpoint list` authenticated and returned the full catalog -> confirms a
SCOPED token (the gate). Endpoints IP-only on the three D-052 planes:
  public   -> provider VIP    10.12.4.5x
  internal -> metal-internal  10.12.12.5x
  admin    -> metal-admin     10.12.8.5x    (keystone admin on :35357)
VIP octets match bundle: keystone .50, barbican .51, cinderv3 .52, glance .53, magnum .54,
neutron .55, nova .56, octavia .57, placement .59, radosgw/s3/swift .60:443.

### DOCFIX-045 -- phase-03 do-doc 3.2 gate text is pre-D-052 (internal plane)
The 3.2 GATE text reads "internal+admin on the metal VIP .8.5x" -- predates D-052's dedicated
metal-internal plane. LIVE (correct) shows INTERNAL on metal-internal 10.12.12.5x and ADMIN on
metal-admin 10.12.8.5x (bundle triple-VIP "10.12.4.5x 10.12.8.5x 10.12.12.5x" + D-052 internal
binding). Amend the 3.2 gate to: public provider .4.5x; internal metal-internal .12.5x; admin
metal-admin .8.5x; keystone admin :35357.
ALSO (value drift, non-blocking): gss image-stream endpoint is HTTP on metal 10.12.8.226 this
deploy (do-doc note said .172) -- the simplestreams image-stream IP is per-deploy; note as
dynamic, do not hardcode. s3/swift on radosgw VIP .60:443 -- re-check vs radosgw :80 listener
during any Swift/S3 smoke (carried-forward do-doc note).

### Step 3.3 Horizon nginx reverse proxy -- PASS (2026-06-27)
v1 Horizon = PLAIN-HTTP reverse-proxy leg per D-044 (authoritative, adopted 2026-06-17). NO nginx
edit was needed: the existing /etc/nginx/sites-available/openstack vhost on the nginx host
(10.12.4.7) already proxies `listen 81` -> `proxy_pass http://10.12.4.58:80` at the CURRENT
dashboard provider VIP (.58 confirmed vs bundle), with `proxy_set_header Host $http_host` (B5
ALLOWED_HOSTS) + X-Forwarded-*. No proxy_ssl_* applied (that is the Roosevelt root-fix, not v1).
The vhost's "Main LXD UI" comment is a stale mislabel (it is the Horizon proxy) -- cosmetic,
flag for consolidation cleanup; left untouched to avoid mutating a working MAAS-fronting host.
Live scheme probes (decisive, verify-before-mutate, from both jumphost and nginx host):
  jumphost->.58  https rc=000 FAIL(35) | http rc=200
  nginx->.58     https rc=000 FAIL(35) | http rc=200
  s_client .58:443 -> CONNECTED but "no peer certificate available" (certless :443 listener)
  => dashboard serves Horizon over HTTP :80; :443 is an unused, certless haproxy frontend.
The certless :443 is EXPECTED under D-044 (v1 does not use dashboard HTTPS). The bundle's
openstack-dashboard:certificates<->vault:certificates relation provisions a cert, but the v1
plain-HTTP leg never serves it. NOT a v1 defect; the Roosevelt DNS + FQDN-cert workstream is the
end-to-end HTTPS root-fix. The earlier dashboard-SAN probe was therefore moot (proxy_ssl_name is
a Roosevelt concern, not v1).
Steps executed:
  A (nginx host, read-only): curl -sI http://127.0.0.1:81/horizon/ -> HTTP/1.1 302 Found (login
    redirect). GATE A met.
  B (jumphost, the one v1 mutation, PER-REBUILD, verbatim do-doc): juju ssh
    openstack-dashboard/leader wrote _99_internal_http_cookies.py (CSRF_COOKIE_SECURE=False +
    SESSION_COOKIE_SECURE=False, ASCII-only) + systemctl reload apache2. Clean.
  C (jumphost, verify adapted https->http per DOCFIX-046): csrftoken Set-Cookie present, no Secure
    attribute -> "OK: csrftoken not Secure". GATE C met.
  D: external browser login over http://10.17.11.246:81/horizon/ SUCCEEDED -- Horizon Overview
    renders as admin_domain/admin, fresh-cloud quotas 0-of-N. "Not secure" address bar = expected
    (plain-HTTP client leg, D-044). GATE D met.
PHASE-03 EXIT GATE MET: 3.1 PASS (accept walk 2-expected + haproxy ZERO DOWN across 31 principals),
3.2 PASS (admin-openrc + scoped catalog), 3.3 PASS (Horizon reachable + login).

### DOCFIX-046 -- phase-03 do-doc 3.3 carries an abandoned HTTPS-upstream edit set (contradicts D-044)
The phase-03 do-doc Step 3.3 body contains BOTH (a) an HTTPS-upstream edit set -- proxy_pass
https://10.12.4.58:443, proxy_ssl_verify on, proxy_ssl_trusted_certificate, proxy_ssl_name + a
dashboard-cert SAN discovery -- AND (b) the real "the upstream stays PLAIN HTTP (as-built)" line.
These contradict. D-044 (authoritative) resolves it: v1 is the plain-HTTP leg; the proxy_ssl_name
/ HTTPS-upstream handling is the ROOSEVELT root-fix, not v1. The (a) block, if applied on the
testcloud, would repoint nginx at the certless :443 and BREAK Horizon (curl 35) -- exactly what
the live probes confirmed would happen.
Also: the do-doc's D-044 VERIFY command uses `curl --cacert ... https://10.12.4.58/...` -- same
HTTPS assumption; it fails (rc=000/35) against the v1 HTTP dashboard. Adapt to
`curl ... http://10.12.4.58/horizon/auth/login/`.
FIX (for completion consolidation): rewrite 3.3 to the v1 plain-HTTP path (verify the existing
vhost points at the current dashboard VIP over http:80; no proxy_ssl_*; apply the cookie override;
verify over http); move the proxy_ssl_*/SAN block verbatim into a clearly-marked "Roosevelt
root-fix (DNS+FQDN certs)" subsection so a future operator does not apply it on the testcloud; fix
the verify command https->http. Cross-ref D-044. Also fix the stale "Main LXD UI" vhost comment.

### Phase-04 prep -- network-carve verify deliverable + DOCFIX-047 (2026-06-27, pre-execution)
New read-only deliverable staged ahead of running phase-04 (network carve):
  scripts/phase-04-network-verify.sh -- verify-before-mutate + EXIT-GATE check for the
    Neutron external provider network. PRE gate: discovers the MAAS provider subnet BY CIDR
    (10.12.4.0/22) -- lib-net PATTERN-1, never a hardcoded subnet id -- asserts its gateway ==
    pinned PLANE_GW (10.12.4.1) and that the FIP pool 10.12.5.0-10.12.7.254 is a RESERVED
    iprange on it (KI-P3-001). POST gate (auto-detected if provider-ext exists): external/flat/
    physnet1/NOT-shared + subnet cidr/gateway/no-dhcp/FIP-pool. Sources lib-net.sh + need_jq;
    requires admin-openrc sourced + the 'admin' MAAS profile; never calls 'maas list' (DOCFIX-016).
    Exit 0 PROCEED|PASS / 1 HOLD|FAIL / 2 precondition. Mutates nothing.
  tests/phase-04/ -- offline regression (real jq + fake maas/openstack data shims; no live
    infra). 7/7 green: PRE PROCEED (net absent); POST PASS for BOTH allocation_pools shapes
    (list-of-objects AND list-of-strings -- tolerance proven, not assumed, so the live client's
    shape cannot silently break the gate); and four failure variants (FIP pool not reserved;
    wrong gateway; provider subnet absent-by-CIDR; provider-ext shared=true). bash -n clean;
    shellcheck 0.9.0 clean (no warnings) on script + harness + shims; ASCII + 0 CR on all five.
  NOTE: fixtures put the provider subnet at id=7 (NOT 1) on purpose, to prove CIDR discovery is
  id-independent (the exact failure mode DOCFIX-047 guards against).

DOCFIX-047 -- phase-04 do-doc hardcodes the provider MAAS subnet id (violates PATTERN-1).
  runbooks/phase-04-network-carve.md reads the provider gateway via `maas admin subnet read 1`
  and its CHECK prose says "subnet id 1 (provider)" / "subnet id 2 (metal)" -- the PRE-D-052
  two-plane numbering. lib-net.sh:9 records that the D-052 cutover renumbered subnets (metal-
  internal moved id 6 -> 10), so a hardcoded `read 1` may now read the WRONG subnet. FIX (for
  completion consolidation): replace `subnet read 1` / the "subnet id N" prose with CIDR-based
  discovery (select(.cidr=="10.12.4.0/22")), exactly as scripts/phase-04-network-verify.sh does;
  cross-ref the verify script from the do-doc's CHECK block. Not yet applied to the do-doc.

### Phase-04 EXECUTED -- network carve COMPLETE (2026-06-27)
Step 4.1 create block ran clean (do-doc idempotent `( set -e )`, with the DOCFIX-047
CIDR-discovery correction for the gateway; the `[ GW = 10.12.4.1 ]` gate retained as belt+braces):
  network provider-ext     = bb386c86-d646-4c71-b6b7-550f5c691bfb  (created + tagged role=provider)
  subnet  provider-ext-fip = 544afa6a-b0cf-486b-89be-2b8e36983072  (created + tagged)
  (object IDs regenerate per deploy; the do-doc's As-built IDs are dead post-teardown, not a discrepancy.)
CONFIRM: provider-ext external=true type=flat physnet=physnet1 shared=false;
  provider-ext-fip cidr=10.12.4.0/22 gateway=10.12.4.1 enable_dhcp=false
  allocation_pools=[{start:10.12.5.0,end:10.12.7.254}] tags=[role=provider, netbox-iprange=10.12.5.0-10.12.7.254].
phase-04-network-verify.sh POST gate: PASS -- EXIT GATE met (all network+subnet assertions green;
  fip-pool-match=true). Live allocation_pools came back as the list-of-OBJECTS shape -- the real
  client emits {start,end} objects; the harness string-shape case is confirmed safety-margin only.
  PRE re-run also PASS (provider subnet by CIDR id=1 this deploy; gateway pinned; FIP reserved).
PHASE-04 EXIT GATE MET. FIP allocation + tenant router gateways now possible (needed by phase-06
mgmt-VM FIP; phase-08 cluster FIPs + LB validation).

DOCFIX-047 CONFIRMED LIVE: provider resolved to subnet id=1 THIS deploy, so the do-doc's
`subnet read 1` would have worked by luck -- but CIDR discovery is the correct id-independent
pattern (lib-net.sh:9: cutover moved metal-internal 6->10) and ran clean. Do-doc fix still pending
at consolidation.

DOCFIX-048 -- phase-04 do-doc IPAM reference VIP-reserve width drift.
  The do-doc "IPAM carve reference" lists the provider VIP reserve as 10.12.4.2-10.12.4.63 (front-
  loaded /26). LIVE MAAS shows the WIDER reserve 10.12.4.2-10.12.4.100 (comment "supersedes
  .224-.236") -- the D-052 "VIP reserve ceilings" correction. Both sit entirely in .4.x, OUTSIDE
  the FIP pool (10.12.5.0-10.12.7.254) -> no conflict; provider-ext created cleanly. The live
  mgmt-plane reserve 10.12.4.101-10.12.4.110 is also present (already in the do-doc As-built note).
  FIX (consolidation): update the do-doc IPAM reference VIP-reserve from .2-.63 to .2-.100 to match
  live + D-052. Non-blocking.

NOTE (repo hygiene, operator decision pending): all scripts on origin are committed mode 100644
(the Windows/GitHub-Desktop path strips +x), so the jumphost must invoke them as `bash scripts/X.sh`
(`./scripts/X.sh` -> Permission denied). Two durable fixes offered: (a) standardize do-docs on
`bash scripts/...`; or (b) one-time `git update-index --chmod=+x scripts/*.sh tests/*/run-tests.sh
tests/*/fakebin/*` from Git Bash + commit (writes 755 into the tree). Not yet actioned.

### Phase-05 prep -- octavia verify deliverable + entry audit + DOCFIX-049 (2026-06-27, pre-execution)
Entry audit GREEN (read-only verify-before-mutate): octavia/0 blocked "Awaiting configure-resources",
agent idle, on 3/lxd/3 addr 10.12.12.112 (metal-internal data leg); configure-resources action present;
config gate holds (retrofit use-internal-endpoints=true / image-format=raw / amp-image-tag=octavia-amphora;
octavia amp-image-tag=octavia-amphora -- match); charm-octavia net/subnet/sg all EMPTY; amphora provider
present; no pre-existing octavia-amphora image; no configure-resources operation yet (not fired). PROCEED to 5.1.

New deliverable staged:
  scripts/phase-05-octavia-verify.sh -- read-only auto-detecting verify. CONFIG GATE always
    (use-internal-endpoints=true / image-format=raw / amp-image-tag match -- LP#1937003 + Ceph-RBD).
    Then by octavia/0 workload status: blocked -> PRE (resources empty) -> PROCEED; active -> POST
    (lb-mgmt net/subnet/sg present + o-hm0 UP w/ fc00::/ ULA) then amphora image tagged+ACTIVE? ->
    CONTROL-PLANE-DONE (run 5.2) or PASS. Sources lib-net.sh + need_jq; requires admin-openrc + juju.
    Exit 0 PROCEED|CONTROL-PLANE-DONE|PASS / 1 HOLD|FAIL / 2 precondition. Mutates nothing.
  tests/phase-05/ -- offline regression (real jq + fake juju/openstack shims). 8/8 green: PROCEED,
    CONTROL-PLANE-DONE, PASS, and five failures (amp-image-tag mismatch; image-format!=raw; active-but-
    resources-missing; o-hm0 down; blocked-but-resources-present anomaly). bash -n + shellcheck clean;
    ASCII + 0 CR on all five.

DOCFIX-049 -- phase-05 do-doc internal-glance-vip is pre-D-052 (.8.53 -> .12.53).
  The do-doc's ENV(internal-glance-vip) reads 10.12.8.53 and prose says "retrofit is metal-only
  10.12.8.x -> internal glance". Post-D-052 the glance INTERNAL endpoint is 10.12.12.53 (metal-internal);
  .8.53 is now the ADMIN VIP (bundle glance vip "10.12.4.53 10.12.8.53 10.12.12.53"; same class as
  DOCFIX-045). Reference-text ONLY: the 5.2 block does NOT hardcode the glance VIP (the retrofit
  discovers it via use-internal-endpoints=true), and connectivity holds -- octavia/0 is on the
  metal-internal leg (10.12.12.112), so it reaches .12.53. FIX (consolidation): update the do-doc
  ENV + prose .8.53 -> .12.53 (metal-internal) per D-052. Watch the retrofit --wait=30m output to
  confirm it reaches internal glance. Non-blocking.

### Phase-05 verify CORRECTION (2026-06-27) -- o-hm0 operstate is UNKNOWN, not UP
First live POST run of phase-05-octavia-verify.sh FALSE-NEGATIVED on o-hm0: the check grepped for
operstate 'UP', but o-hm0 is an OVS INTERNAL port -- the kernel reports operstate UNKNOWN (no
carrier reporting). The live line was:
  o-hm0  UNKNOWN  fc00:7bab:3c0b:ae43:f816:3eff:fe2a:f741/64  fe80::.../64
i.e. HEALTHY -- the fc00::/ IPv6-ULA is present, which is the do-doc's actual GATE C signal.
DOCFIX-044 class: the harness fixtures used UP/DOWN and never the real-world UNKNOWN, so the test
validated a fiction. FIXED: accept UP or UNKNOWN, reject only DOWN/absent; fc00::/ ULA is the
positive signal. Harness now drives the real UNKNOWN value (+ an absent case) -> 9/9 green;
shellcheck + bash -n clean. The do-doc itself was CORRECT (GATE C checks the fc00:: ULA + br-int
port, not UP) -- NO DOCFIX. LEARNING: OVS internal ports (o-hm0 etc.) report operstate UNKNOWN;
liveness checks must accept UNKNOWN, never require UP.

### Step 5.1 EXECUTED -- configure-resources COMPLETE (2026-06-27)
juju run octavia/leader configure-resources --wait=20m: operation 11 / task 12 returned clean (no
wait-timeout). Result: octavia/0 active "Unit is ready"; charm-octavia net/subnet/sec-grp = 1/1/1
(lb-mgmt-net / lb-mgmt-subnetv6 / lb-mgmt-sec-grp); o-hm0 present, operstate UNKNOWN (normal),
fc00:7bab:3c0b:ae43:f816:3eff:fe2a:f741/64 IPv6-ULA. A transient "ovs-vsctl: no row o-hm0" appeared
mid-run (before o-hm0 was created) and self-cleared -- the expected bring-up transient. lb-mgmt
control plane UP. State = CONTROL-PLANE-DONE; proceeding to Step 5.2 (amphora pipeline).

### Step 5.2 EXECUTED -- amphora pipeline COMPLETE; PHASE-05 EXIT GATE MET (2026-06-27)
Canonical 5.2 block ran (config gate OK -> stage-and-verify seed -> retrofit -> confirm):
  base    jammy-amphora-base = 6ed034f7-10b0-4231-adcd-89e36ff79901 (qcow2; sha256
          ec332b438a87c3cc1e38318ced110568392484cc2f33e29c8f4eaf9ecc9de297 verified vs published SHA256SUMS)
  retrofit op 15 / task 16 on octavia-diskimage-retrofit/0
  amphora amphora-haproxy-x86_64-ubuntu-22.04-20260627 = c8f9f53a-980b-4224-9949-9e141708dd6f
          ACTIVE, tag octavia-amphora (matches octavia amp-image-tag), image-format raw.
DOCFIX-049 connectivity CONFIRMED: the retrofit (use-internal-endpoints=true) built the amphora
successfully, i.e. it reached the INTERNAL glance endpoint (post-D-052 = 10.12.12.53, metal-internal)
over octavia's metal-internal leg. The build succeeding IS the proof; the do-doc's .8.53 is stale
reference text only (DOCFIX-049, consolidation).
Paste cosmetics: the 5.2 block's terminal echo wrapped oddly (closing ')' + tee target + final echo),
but the shell executed in order -- CONFIRM found the active amphora and an independent verify counts
exactly 1 active octavia-amphora image. Not a partial run.

PHASE-05 EXIT GATE MET (all four):
  - octavia/0 active/idle "Unit is ready"
  - lb-mgmt net/subnet/sec-grp present (1/1/1)
  - o-hm0 up (operstate UNKNOWN -- normal for an OVS internal port -- with fc00::/ ULA)
  - ACTIVE amphora image tagged octavia-amphora, tag == octavia amp-image-tag
NOTE: the live verify run still FALSE-NEGATIVED on o-hm0 because the jumphost copy of
phase-05-octavia-verify.sh was the PRE-FIX version (message "not UP with an fc00::/ ULA"); the
corrected script (this session, 9/9 harness) reports PASS. Pull the corrected script (commit/push
-> jumphost pull) for the green confirmation; the gate is met on the evidence regardless.
End-to-end LB build / round-robin / admin-scope failover validation is D-011 criterion 4, deferred
to phase-08 (needs tenant scaffolding). Octavia STANDALONE failover N+1 amphora headroom remains a
Roosevelt sizing note (a cloud at scheduler ceiling cannot self-heal its LBs).

### Phase-05 amphora pipeline scripted -- phase-05-amphora-pipeline.sh + D-056 PROPOSED (2026-06-27)
Per operator request the Step 5.2 amphora pipeline is now a script, scripts/phase-05-amphora-pipeline.sh,
encapsulating the do-doc 5.2 canonical block. RATIONALE: the inline 5.2 block is large and MANGLED on
paste this session (terminal wrapped the closing ')' + tee target + final echo); a worse wrap could
drop --wait or the subshell close and half-run a mutating pipeline. Scripting removes the command-size-
overrun risk and makes it reviewable/version-controlled.
  Behavior identical to the do-doc block: config GATE; idempotent (amphora present -> skip; base
  present -> retrofit only; fresh -> download + sha256-verify + upload + retrofit); confirm.
  Hardened: env-overridable tunables (jammy/Caracal defaults), explicit -m MODEL on juju config,
  preconditions (openstack/juju/sha256sum/OS_AUTH_URL), exit 0/1/2. ASCII + LF; shellcheck clean.
  tests/phase-05-amphora/ -- offline LOGIC regression (fake juju/openstack/curl/wget/sha256sum; the
  fake retrofit "builds" via a marker so confirm sees the image appear). 6 cases + 1 assertion ALL
  PASS: config-gate fails x2; idempotent skip; fresh full path; retrofit-only (proven to NOT
  download); checksum-mismatch abort. (Two initial [XX] were line-spanning grep regexes in the
  HARNESS, not script faults -- fixed to single-line markers; the script was correct throughout.)

D-056 (PROPOSED) -- consequential-mutation runbook blocks may be scripted.
  Extends the operating model. Prior rule: "scripts own read-only/repeated verification; humans gate
  consequential mutations." Refinement: a consequential MUTATION may live in a script WHEN it is fully
  IDEMPOTENT + self-gating; the human still gates by CHOOSING to invoke it, and re-running is safe.
  Net hardening (removes large-paste overrun; adds review + version control). First instance:
  phase-05-amphora-pipeline.sh. CANDIDATES to convert similarly (operator decision): the phase-04
  network-carve create block and the phase-05 Step 5.1 configure-resources block (both multi-line
  mutating pastes). Status: PROPOSED -- ratify or amend.

### Completed-phase mutation blocks scripted -- phase-00/03/04 (D-056 ratified) (2026-06-27)
D-056 RATIFIED by operator (was PROPOSED): consequential mutation blocks may be scripted when fully
idempotent + self-gating. Following a survey of all nine do-docs, the mutation blocks from the
already-completed phases are now scripts (the 06/07/08 blocks will be built as those phases execute):

scripts/phase-04-network-create.sh -- Step 4.1 provider-ext network + FIP subnet create. Idempotent
  (skip-if-exists on network + subnet); provider gateway discovered BY CIDR (DOCFIX-047), never the
  do-doc's hardcoded subnet read. CONFIRM gate asserts external/flat/physnet1/not-shared. Exit 0/1/2.
  tests/phase-04-create/ -- 4/4 PASS (fresh create, idempotent skip, gateway-gate fail, confirm fail).

scripts/phase-03-admin-openrc.sh + scripts/extract_admin_password.py -- Step 3.2 admin-openrc build:
  vault root CA + keystone admin password + DOCFIX-022 project-scope discovery loop -> 0600 openrc.
  SECRET-handling but human-gated; the password stays inside the subshell + the 0600 file (umask 077),
  never echoed. The password parser is a tested .py helper (per "Python helpers in their own files"),
  not inline python-in-bash. tests/phase-03-openrc/ -- extractor unit 4/4 + integration 4/4 + 0600
  assertion PASS (fake juju/openstack/openssl; real python3/jq; no real secrets in the harness).

scripts/phase-00-maas-carve.sh -- the MAAS VIP/FIP carve (rebuild foundation; KI-P3-001). Idempotently
  reserves the provider + metal-admin API-VIP /26 and the provider FIP pool. Idempotency anchored on
  the START ip so a pre-existing .2-.63 OR .2-.100 reserve is left untouched (no overlap-create). Stale
  .8.224-.254 is detected + reported; deleted only with DELETE_STALE=1 on an exact start/end match
  (destructive, gated). Two correctness fixes vs the do-doc 4b block -- see DOCFIX-050/051. Exit 0/1/2.
  tests/phase-00-carve/ -- 8/8 PASS (fresh create x3, idempotent skip, .2-.63 width-drift tolerated,
  stale report-only, stale DELETE_STALE=1 delete, provider-subnet-missing fail).

All four shell scripts: bash -n + shellcheck -S warning clean; ASCII + 0 CR. The .py: ast.parse clean,
ASCII. Invoke as `bash scripts/X.sh` (origin mode 100644; Windows strips +x).

DOCFIX-050 -- phase-00 do-doc 4b hardcodes `ipranges create subnet=1 ... subnet=2` (PATTERN-1
  violation; same class as DOCFIX-047 but a distinct location). Subnet IDs drift across cutovers; the
  script resolves provider/metal subnet ids BY CIDR. Fix the do-doc before Roosevelt.
DOCFIX-051 -- phase-00 do-doc 4b creates the two VIP /26 reservations but NOT the provider FIP pool
  (.5.0-.7.254), which 4a lists as "want present". Harmless while the reservation persists across
  teardown, but a fresh-MAAS rebuild needs it -- phase-00-maas-carve.sh creates it idempotently.

### tests/phase-03-openrc renamed -> tests/phase-03-adminrc; DOCFIX-052 (2026-06-27)
The phase-03 admin-openrc harness did NOT commit with 63f5832: the .gitignore credential glob
`*-openrc` (intended for the admin-openrc secret file) also matched the directory NAME
tests/phase-03-openrc, so GitBucket/GitHub-Desktop never offered it. Per operator decision the
directory is RENAMED to tests/phase-03-adminrc -- sidesteps the glob with zero erosion of the
credential-ignore net (the negation alternative was declined to keep `*-openrc` at full breadth).
Harness re-run green under the new path (extractor 4/4 + integration 4/4 + 0600 assertion).
CONVENTION: do not name non-secret repo paths `*-openrc`; the glob will silently ignore them.
DOCFIX-052 -- .gitignore `*-openrc` matches directory names as well as the credential file; the
  only colliding path was tests/phase-03-openrc (now renamed). Glob left as-is (correct for secrets).

### Phase-06 entry-audit -- interactive-paste hazards found (no D/DOCFIX; block-authoring notes)
Running the read-only entry audit surfaced THREE block-authoring bugs (fixed in the re-issued
checks; recorded so they do not recur in scripted phase-06 blocks):
  - `!` inside a double-quoted echo triggers bash history expansion on interactive paste
    ("event not found"); broke the provider-ext line. AVOID `!` in pasted blocks (akin to the
    DOCFIX-021 heredoc-stdin traps).
  - `openstack <obj> show | jq ... || echo absent` is wrong for presence: jq on EMPTY stdin exits
    0, so the `||` never fires. Test existence by the show exit code (`show -c id >/dev/null 2>&1`)
    BEFORE piping to jq.
  - oslo.config writes `key = value` (spaces); a `^key=` grep gives false negatives. Match
    `key[[:space:]]*=` (the magnum [trust] count of 0 was this false negative, not a real absence).

### Phase-06 START -- entry audit GREEN + 6.0-BOOT scripted (2026-06-27)
Entry audit (read-only) on the fresh redeploy: provider-ext PRESENT
(id bb386c86-d646-4c71-b6b7-550f5c691bfb, matches phase-04 as-built); noble image ABSENT
(6.0-BOOT seeds); domain capi / project capi-mgmt / the three admin roles / all five flavors
ABSENT (expected fresh); capi-mgmt-net ABSENT (expected); magnum [trust] POPULATED. Cleared to 6.0-BOOT.

scripts/phase-06-bootstrap.sh -- Step 6.0-BOOT as a D-056 script (verify-or-create domain/project/
  roles/flavors + noble image stage-and-verify: download-to-$HOME / sha256-vs-SHA256SUMS / image
  create --file --import; image-conversion lands raw, D-021). Idempotent (all [SKIP] on an existing
  cloud); env-overridable; preconditions (openstack/jq/curl/wget/sha256sum/awk + scoped token; sources
  ~/admin-openrc if OS_AUTH_URL unset); exit 0/1/2. bash -n + shellcheck clean; ASCII + 0 CR.
  tests/phase-06-bootstrap/ -- 7 cases ALL PASS (fresh create-all + image active; idempotent skip-all;
  staged-present skips download; checksum mismatch abort; no-published-checksum abort; poll-never-active
  abort; missing-auth precondition exit 2).

DOCFIX-053 -- phase-06 prereq text names the trust keys trustee_domain_id / trustee_domain_admin_id,
  but the magnum charm populates the NAME variants (trustee_domain_name=magnum,
  trustee_domain_admin_name=magnum_domain_admin). The first entry-audit grep keyed on the _id names
  (plus a `^key=` no-spaces anchor) and falsely reported 0; the corrected grep confirmed [trust] is
  populated. Fix the prereq wording before Roosevelt.

D-046 reminder (phase-08 entry check, NOT phase-06): the [trust] section references Keystone domain
  `magnum` + user `magnum_domain_admin`, which do NOT survive teardown -- the magnum charm
  `domain-setup` action must be re-run before workload-cluster mint (magnum reports ready regardless).

### Phase-06 6.0-BOOT verified + 6.0/6.1 scripted (2026-06-27)
6.0-BOOT ran clean (as-executed): domain capi / project capi-mgmt / 3 roles / 5 flavors created;
ubuntu-24.04-noble imported + active, disk_format=raw, checksum 5fa5b05e...1a44 (matches the
published noble SHA256SUMS). GATE met.

scripts/phase-06-net-setup.sh -- Steps 6.0 + 6.1 (keypair + capi-mgmt-sg ingress tcp/22+6443;
  network + subnet 10.20.0.0/24 dns 1.1.1.1,1.0.0.1; router + external gateway on provider-ext) as
  one D-056 verify-or-create script. Idempotent; preconditions (openstack/jq/scoped-token/EXT-present);
  router gate polls ACTIVE + ext-gw (ROUTER_POLL_TRIES/SLEEP); exit 0/1/2. shellcheck clean; ASCII+0CR.
  tests/phase-06-net-setup/ -- 5 cases ALL PASS (fresh create-all; idempotent skip-all; router-not-
  ACTIVE gate fail; missing-pubkey exit 2; missing-auth exit 2). The SG-rule syntax is the do-doc's
  standard openstack-client form (LIVE-REVIEW note resolved: the run's verify print is the confirmation).

DOCFIX-054 -- do-doc Step 6.2 FIP allocation is NOT idempotent: `openstack floating ip create` runs
  unconditionally, so a re-run allocates + attaches a SECOND floating IP to capi-mgmt-v2 (FIP-pool
  leak + ambiguous apiserver endpoint). scripts/phase-06-mgmt-vm.sh already FIXES this -- it looks up
  the VM's neutron port and reuses an attached FIP, allocating only when none is present. (The number
  DOCFIX-054 was coined in that script in a prior session; this changelog now records it. Fix the
  do-doc 6.2 block before Roosevelt.)

scripts/phase-06-mgmt-vm.sh + scripts/resolve_tenant_ip.py -- the Step 6.2 VM+FIP mutation + tenant-IP
  resolver. Both were untracked from earlier phase-06 prep (present in the working tree, NOT in origin);
  committed now WITH a new harness. tests/phase-06-mgmt-vm/ -- resolver unit 5/5 (list shape /
  {addr}-dict shape / comma-string shape / only-FIP-empty / no-FIP-first) + integration 6/6 (fresh
  create+poll+allocate; idempotent FIP reuse; VM-ERROR abort; never-ACTIVE abort; tenant-unresolvable
  abort; missing-auth exit 2) + net.env both-keys-0600 assertion. bash -n + shellcheck clean; ASCII+0CR.

### Phase-06 6.2 fix -- re-scope to capi-mgmt (DOCFIX-055) + ERROR-VM handling (2026-06-27)
First live 6.2 run FAILED: capi-mgmt-v2 -> scheduler ERROR, fault "Security group <id> not found".
Root cause (measured): admin-openrc scopes the token to the ADMIN project (a11cf69...); `server
create` has no --project flag, so the VM + port landed in admin, where capi-mgmt-sg / capi-mgmt-net
(owned by project capi-mgmt 7072d66e...) are not visible. The do-doc 6.2 block never re-scopes.

DOCFIX-055 -- do-doc Step 6.2 must re-scope to the capi-mgmt project (the resources are owned there;
  6.0/6.1 create them with --project capi-mgmt). Fix the do-doc 6.2 block before Roosevelt.

scripts/phase-06-mgmt-vm.sh revised:
  - DOCFIX-055: export OS_PROJECT_NAME=capi-mgmt OS_PROJECT_DOMAIN_NAME=capi before the server/FIP
    calls (admin USER retains the keypair; image/flavor are --public; FIP allocates via the
    capi-mgmt-router gateway). Re-scope validated up front (exit 2 if the D-039 member grant absent).
  - Pre-create VISIBILITY preconditions: keypair + sg + network must resolve in the capi-mgmt scope
    BEFORE create -> turns the old post-hoc VM ERROR into an upfront exit 2.
  - Existing ERROR-state VM: clear abort + delete instruction by default; RECREATE_ON_ERROR=1 to
    delete+recreate in-scope. NOTE the failed instance is in the ADMIN project (different scope), so
    the fixed script will NOT see it -- it must be deleted manually under admin-openrc first.
  tests/phase-06-mgmt-vm/ expanded to 12 integration + 5 resolver-unit cases ALL PASS (adds: re-scope
  denied, keypair/sg/network not-visible, existing-ERROR abort, existing-ERROR+RECREATE).
  shellcheck clean; ASCII+0CR.

### Phase-06 6.3 BLOCKED -- provider FIP range dark cloud-wide; root-caused as D-057 (2026-06-27)

Phase-06 Step 6.3 (SSH to capi-mgmt-v2 via FIP 10.12.7.107) is blocked: every
provider-ext FIP (pool 10.12.5.0-.7.254) is unreachable -- "No route to host"
(EHOSTUNREACH; L2/ARP fails before TCP) from the jumphost, and TCP-FAIL /
ARP-INCOMPLETE from the cloud nodes themselves. The 10.12.4.234 Horizon HA VIP
is dark in the same class. The .4.x control VIPs answer normally.

Root cause CONFIRMED (measured, not inferred): on all three ovn-chassis nodes
the provider NIC enp1s0 is untagged + enslaved to a Linux bridge br-enp1s0 that
carries the host provider static L3 (openstack2: inet 10.12.4.42/22 on
br-enp1s0), while also being a port of OVS br-ex (carrier DOWN on all 3). The
provider Neutron net is flat/untagged (physnet1, no segmentation), so the FIP
range is untagged provider traffic -- which the L3-carrying Linux bridge
intercepts at the kernel before it reaches OVS br-ex, where OVN's gateway
ARP-responder lives. This violates the Canonical charm-guide rule that the OVS
uplink must carry NO L3 and host/container workloads must ride a separate VLAN
sub-interface (docs.openstack.org/charm-guide/latest/admin/networking/interface-config.html).

The OpenStack control plane, OVN logical config (port_binding up, gateway
scheduled, FIP NAT programmed), Neutron provider segment, ovn-chassis
bridge-mappings, and the MAAS provider subnet are ALL correct/healthy --
verified this session. carve-host-interfaces.sh is correct per the as-built
reference; the L3 + LXD container auto-bridge land on top of the raw NIC at
deploy. A .5.x FIP (10.12.5.19) was inbound-reachable on the PRIOR cloud (chat
4586faa7, 2026-06-05) before the D-052/D-053 teardown/re-IP/rebuild, so
reachability is achievable; this rebuild never re-established it.

Logged as design decision D-057 (PROPOSED/OPEN): separate the host/API plane
from the flat FIP plane per the Canonical shared-NIC VLAN pattern; raw untagged
enp1s0 reserved for OVS br-ex with NO L3. NIC-limited by design (no spare NIC;
ex-lbaas enp11s0 to be REMOVED). OPEN blocking question: the interaction with
D-003B (API VIPs must stay tenant-reachable) -- next session must choose between
(a) API VIPs on a tagged routed VLAN + re-validate Option-B, or (b) remove the
host provider static from enp1s0 (host may not need provider L3; API VIPs live
in containers) and confirm that alone clears the interception. Full root-cause
chain in docs/fip-edge-diagnosis-checkpoint-20260627.md. NO live ip/ovs-vsctl/
ovn-nbctl writes -- remediation is a MAAS-carve + bundle + redeploy change.

DIAGNOSTIC DISCIPLINE (carry forward; cost many turns this session): read
source-of-truth/history before live probing; query OpenStack objects BY ID not
name across scopes; never filter diagnostics through 2>/dev/null+jq (it hid
scope/empty errors); ICMP from nodes is a weak signal (test the real open port);
compare across all nodes before declaring a one-host anomaly.

(No mutations to the cloud this session -- all diagnosis was read-only. Phase-06
6.0-BOOT / 6.0 / 6.1 / 6.2 remain as-built from the prior session: capi-mgmt-v2
ACTIVE, FIP 10.12.7.107, tenant 10.20.0.84, persisted to ~/capi-mgmt-net.env.)

### CORRECTION -- real project state: rebuild after destructive delete; NIC-enslavement is the target (2026-06-29)

Supersedes the framing of the 2026-06-27 "Phase-06 6.3 BLOCKED ... D-057" entry
below. That entry cast D-057 as an OPEN phase-06 FIP blocker on the running cloud;
that is NOT the project state and it misled later work (including a stale-summary
assistant). Accurate history:

- The cloud was fully deployed and FUNCTIONAL. Additional network spaces were created
  and several subnets segregated (D-052 / D-053); the live deployment existed to
  validate those network-segregation changes -- and they were validated.
- A destructive command (provided by Claude) DELETED openstack0-3. The hosts were
  rebuilt from scratch.
- The rebuild surfaced the real defect: the hosts came up with INCORRECT NIC
  configuration -- the D-057-class enslavement cascade (provider NIC enp1s0 untagged
  + enslaved to a Linux bridge carrying the host provider static L3, starving OVS
  br-ex of carrier, killing OVN's gateway ARP responder, darkening the provider FIP
  plane). This is a REBUILD defect, not a blocker on the original cloud. D-057's root
  cause is understood; the open work is making a rebuild produce CORRECT host NICs.

What the current MAAS-reconfigure work targets (remediation in scope):
  - D-058 -- full plane renumber (clean fabric-grouped /22 scheme; Roosevelt fidelity).
  - provider-vip plane (VID 104) -- public API/VIP plane rides a tagged sub-interface so
    enp1s0 stays raw + L3-less for OVS br-ex (defeats the enslavement cascade).
  - carve-host-interfaces.sh -- produces the correct per-host NIC tree on rebuild
    (enp1s0 raw via ensure_raw_unlinked; enp1s0.104 -> br-prov-api; metal stack).
  - D-059 -- host NIC budget: build on FIVE physical NICs now; four-NIC collapse
    (fold storage+replication) PARKED pending Roosevelt hardware sheets.

Tooling built for the reconfigure (tested, committed):
  - scripts/phase-00-maas-standup.sh -- single MAAS-address authority (topology +
    VIP/FIP/mgmt reserves); audit + create-if-absent.
  - scripts/phase-00-maas-recidr.sh -- gated D-052/053 -> D-058 re-CIDR, reuse-in-place.
  - runbooks/phase-00-maas-reconfigure.md -- teardown -> re-CIDR -> standup -> verify ->
    jumphost bridges -> deploy sequence.
  - phase-00-maas-carve.sh RETIRED (folded into the standup).

Live state at correction time (measured 2026-06-29): the openstack model is DEPLOYED
and running (28 machines, 63 units, capi-mgmt Ready) on the pre-D-058 D-052/053 scheme;
the standup dry-run reports the three expected DRIFT planes (.8/.12/.16). Teardown has
NOT been run -- the cloud was kept up for information-gathering, now complete.

### Next-free numbers
Design decision: D-060. Doc fix: DOCFIX-056.

### 2026-06-30 -- Pattern A redeploy executed: phases 00->03 PASS (mysql, Vault, core-verify, Horizon)

Full Pattern-A (D-060) redeploy executed end to end through phase-03. Net: a clean,
core-verified cloud -- mysql clustered, Vault PKI up, cert cascade settled, 31/31 haproxy
backends serving, admin-openrc built, Horizon reachable. Phases 04-08 remain.

PHASE-00 (teardown/recarve/redeploy prep):
- CARVEFIX-001 (committed adf2890, prior) VALIDATED LIVE: every host shows `enp1s0 -> VLAN
  5001` before `create br-ex`; br-ex lands fabric 0 (1_provider) with 10.12.4.4N. The explicit
  VLAN move IS required (MAAS link-subnet does not auto-move VLAN).
- carve x4 clean (octets .40-.43), fabric-prune clean (auto-fabrics self-reclaim; fabric-4 kept
  = LXD substrate), standup OK (consistent D-052/D-053).
- OSD wipe: vdb images carried 50-63 MiB LVM/ceph-volume residue post-decompose (NOT data, but
  enough signature to warrant the wipe). qemu-img map (read-only) confirmed before the gated
  rm+recreate -> four blank 512G (root:root 600). ceph-osd accepted the wiped disks cleanly.

THREE MACHINE-DECOMPOSE INCIDENTS (Claude-caused; root-caused + fixed):
- destroy-model decomposed openstack0-3 three times (twice via Claude-advised destroy, incl. one
  run with NO storage flag). Root cause + fix = D-061: machine retention is `remove-machine
  --keep-instance`, NOT a destroy-model storage flag. Two replacement scripts written
  (phase-00-teardown-release.sh / -destroy.sh); old phase-00-teardown.sh deprecated (DOCFIX-057).
  Each incident cost a reenroll+carve cycle (libvirt domains survived every time -- the saving grace).

PHASE-01 (deploy):
- BUNDLEFIX (this session): a single-mysql seed edit (num_units 1) was committed (bf0f3fa) then
  found WRONG -- see D-062. Reverted to num_units 3 / to:[lxd:8,9,10].
- Deployed; converged to the Vault gate (vault blocked "needs initialization"; ovn/neutron/barbican
  waiting on certs -- all expected). openstack0 = control-only confirmed (no nova-compute).

PHASE-01b (mysql clustering -- D-062):
- single-unit mysql parked blocked `'cluster' missing, Instance not yet configured for clustering`.
  Root-caused: the reactive charm bootstraps from the `cluster` PEER-relation handler, which never
  fires at num_units 1. NOT the Canonical Charmed MySQL charm. Fix: deploy at 3.
- Corrected in place: remove-unit the stuck mis-placed /1, then add-unit -n 2 --to lxd:1,lxd:2.
  cluster-status: OK, 1 R/W + 2 R/O, instanceErrors=[] on all three. Clean cold bootstrap.

PHASE-02 (Vault) -- PASS, no DOCFIX, proven path held:
- init (2>&1|tee, 5 shares/threshold 3) -> keys saved off-host; unseal 3-of-5 -> Sealed false,
  Storage Type mysql, HA Enabled false (correct); authorize-charm (10m child token) + generate-root-ca
  -> vault active/idle, root CA valid to 2036. Cert cascade settled fast: ovn-central x3, ovn-chassis
  x3, ovn-chassis-octavia, neutron-api-plugin-ovn, barbican-vault all -> active. magnum self-resolved
  to active at TLS cutover (FINDING-2). Tally 29 active / 1 unknown (gss) / 1 blocked (octavia) / 0 error.
- REFINEMENT: keystone landed `active` ("PO (broken): Unit is ready" at unit level) pre-policy-attach,
  not the PO-broken-app the runbook anticipated -- the empty policyd-override is a clean no-op on this
  charm rev. Not a regression; D-051 zip attaches in phase-08.

PHASE-03 (core verify) -- PASS:
- 3.1 phase-03-core-verify.sh: acceptance walk PASS (only gss + octavia non-active, both expected);
  haproxy backend sweep PASS (all backends UP across 31 principal units, zero DOWN).
- 3.2 admin-openrc built: scoped token OK; endpoint catalog confirms the D-052 three-plane VIP scheme
  (public .4.5x / internal .12.5x / admin .8.5x; keystone admin :35357). admin project = `admin`.
  NOTE: gss image-stream live VIP is 10.12.8.166 (do-doc 3.2 text says .8.172 -- stale reference,
  DOCFIX-class, no action).
- 3.3 Horizon: the dashboard VIP 10.12.4.58 serves PLAIN HTTP (haproxy pass-through, backend :433/:70,
  no crt) -- the do-doc's proxy_ssl_name/DNS-SAN upstream-TLS machinery is INAPPLICABLE here. The live
  nginx vhost (proxy host 10.12.4.7, listen 81) was ALREADY correct HTTP-upstream
  (proxy_pass http://10.12.4.58:80, Host preserved, no proxy_ssl_*); no edit needed. D-044 cookie
  override reapplied (per-rebuild). Browser login page renders end-to-end via 10.17.11.246:81/horizon
  with the Domain field (multidomain enabled). DOCFIX-058: phase-03 do-doc 3.3 must document the
  HTTP-upstream reality (the as-built proxy is right; the runbook prose is stale -- DOCFIX-046 adjacent).

CORRECTION to an earlier sweep note: the bundle `to: [lxd:8,9,10]` is NOT a bug -- the bundle declares
its own `machines:` 8-11, so 8/9/10 are internally consistent. The live 0-3 numbering was a deploy-time
artifact; the bundle correctly uses 8-11.

### 2026-06-30 -- Phase-04 executed (network carve + internal-cert SAN gate); DOCFIX-059/060

PHASE-04 (network carve) -- PASS:
- Step 4.1 provider-ext + provider-ext-fip created idempotently (phase-04-network-create.sh); EXIT gate
  PASS via phase-04-network-verify.sh (external/flat/physnet1/not-shared; subnet 10.12.4.0/22, gw
  10.12.4.1, no-dhcp, FIP pool 10.12.5.0-10.12.7.254). As-built this deploy: provider-ext = a4e1a7fa-...,
  provider-ext-fip = f66e5bc5-... (runbook as-built refreshed).

DOCFIX-059 -- internal-cert SAN gate + the VANTAGE correction (the substantive finding):
- The prompt's phase-04 item "confirm internal certs carry 10.12.12.5x SANs" was NOT implemented by any
  committed artifact. Added scripts/phase-04-internal-cert-san-verify.sh (+ tests/phase-04-internal-cert-san/)
  and runbook Step 4.2.
- VANTAGE (load-bearing): metal-internal (10.12.12.0/22, VID 103) is an ISOLATED service plane (D-052); the
  jumphost is NOT on it. An s_client from the jumphost to 10.12.12.x TIMES OUT / conn-errors, and an
  un-hardened check mislabels that as "no IP-SAN" -- a FALSE negative (observed live this session). The gate
  must probe FROM a unit ON the plane (keystone/leader) via juju exec. Confirmed live: all 11 internal https
  endpoints carry their own 10.12.12.5x IP-SAN (keystone/glance/nova/neutron/cinderv3/placement/barbican/
  octavia/magnum/swift/s3). Internal TLS is correct; the earlier failures were purely vantage.
- HARDENING in the gate: (a) every probe is timeout-bounded (an unbounded s_client hangs ~127s on a filtered
  VIP -- proven at 6.02s vs 127s), classified TIMEOUT/CONN-ERR distinctly from a real NO-SAN; (b) non-https
  endpoints (the plain-HTTP glance-simplestreams image-stream) are SKIPPED (no cert). Test covers PASS /
  SKIP-http / NO-SAN / NO-CERT (fake openstack/juju + real jq; run on the jumphost).

DOCFIX-060 -- phase-04-network-carve.md drift (the script was right; the md lagged):
- Inline Step 4.1 used the hardcoded `maas admin subnet read 1` -- a post-D-052 landmine (subnet ids drift
  across cutovers). Corrected to gateway-by-CIDR, matching phase-04-network-create.sh (DOCFIX-047).
- IPAM reference carried the pre-D-052 single "Metal 10.12.8.0/22 = internal/admin VIPs" model. Corrected to
  the D-052/053 split: metal-admin 10.12.8 (admin VIPs .8.5x + PXE), metal-internal 10.12.12 (VID 103,
  internal VIPs .12.5x + all service east-west).
- Added a CANONICAL EXECUTION note (D-056) pointing to the three phase-04 scripts; refreshed as-built IDs.

PROCESS lessons (recorded; no code change):
- PASTE SAFETY: a delivered block whose BEGIN/END label lines (which contain parentheses) were left inside
  the fenced code region broke on paste -- bash rejected the parenthesis as an unexpected token. Label lines
  carrying parens are NOT comments. RULE: put only valid bash inside a fenced block; labels go as # comments
  or prose -- and run bash -n on EVERY delivered block first (that was the miss). The runbooks' own
  bold-label convention (labels OUTSIDE the fence) is correct and unaffected.
- NETWORK-PROBE TIMEOUT: any s_client/curl/nc in a runbook step must be timeout-bounded with an explicit
  timeout branch -- an unbounded probe is not acceptable in a deterministic gate.

PHASE-05 (octavia) -- IN PROGRESS at handoff: config gate clear (retrofit use-internal-endpoints=true,
image-format=raw, amp-image-tag=octavia-amphora on both sides); octavia blocked, charm-octavia resources
0/0/0; PRE gate PROCEED. Step 5.1 configure-resources running (--wait=20m; do NOT re-fire on wait-timeout).

### 2026-06-30 -- Phase-05 executed (Octavia enablement, D-021) -- PASS; DOCFIX-061

PHASE-05 (octavia) -- PASS (scripts ran clean; no script defects):
- 5.1 configure-resources (op 35/task 36, --wait=20m) cleared octavia's blocked -> active; lb-mgmt-net /
  lb-mgmt-subnetv6 / lb-mgmt-sec-grp created; o-hm0 UP with an fc00:: ULA (state=UNKNOWN is normal for an
  OVS internal port). Benign in-progress noise confirmed harmless: `ovs-vsctl: no row o-hm0` (queried
  before the action creates the port) and the systemd-networkd stop/socket warning.
- 5.2 amphora pipeline (phase-05-amphora-pipeline.sh): config gate clear; base seeded via STAGE-AND-VERIFY
  (sha256 070de108...); retrofit op 39/task 40 built amphora-haproxy-x86_64-ubuntu-22.04-20260701
  (807e3f5b-...) ACTIVE, tag octavia-amphora, image-format raw. phase-05-octavia-verify.sh -> PASS.

DOCFIX-061 -- phase-05 as-built reconciliation (runbook drift; no script change):
- Retrofit's internal glance target corrected .8.53 -> 10.12.12.53: under D-052 the INTERNAL glance VIP is
  on metal-internal (.12.53, confirmed live this session); the doc's ".8.53" predates the metal-admin/
  metal-internal split and is now the ADMIN VIP.
- Seed-method note corrected: this rebuild used STAGE-AND-VERIFY (the canonical Step 5.2 script), not the
  06-16 web-download expedient. Object IDs / op numbers refreshed to 2026-06-30. noble is seeded in
  phase-06 6.0-BOOT this rebuild (not pre-staged in phase-05). o-hm0 ULA not captured this run (regenerates).

DISCIPLINE (operator-directed 2026-06-30): reconcile scripts + commands + this changelog at the SUCCESSFUL
completion of EACH phase, before starting the next. Deliver the per-phase reconciliation as a repo-relative ZIP.

### 2026-07-01 -- Phase-06 executed + swept (in-cloud CAPI mgmt cluster, D-035) -- PASS; DOCFIX-062

PHASE-06 progress this session (all steps executed clean; full sweep completed at phase-06 completion per the
per-phase discipline above):
- 6.0-BOOT (phase-06-bootstrap.sh) -- domain capi / project capi-mgmt / roles member+load-balancer_member+
  reader on admin@admin_domain (D-039) / 5 flavors / image ubuntu-24.04-noble active raw public. PASS.
- 6.0+6.1 (phase-06-net-setup.sh) -- keypair capi-mgmt-key, SG capi-mgmt-sg (22+6443), net capi-mgmt-net,
  subnet capi-mgmt-subnet 10.20.0.0/24, router capi-mgmt-router ACTIVE gw set. PASS.
- 6.2 (phase-06-mgmt-vm.sh) -- VM capi-mgmt-v2 ACTIVE (gp.large/noble). FIP 10.12.7.222, TENANT 10.20.0.207
  -> ~/capi-mgmt-net.env (per-rebuild, non-deterministic; DOCFIX-038). PASS.
- 6.3 GATE 1 (inline ssh egress probe) -- VIP-OK (Keystone VIP 10.12.4.50:5000) + NET-OK (1.1.1.1:443). PASS.
- 6.4 k8s bootstrap (inline ssh nested-heredoc) -- k8s v1.32.13 (1.32-classic/stable); bootstrap-config.yaml
  with cluster-config block (DOCFIX-024) + extra-sans 10.12.7.222 / 10.20.0.207; cluster ready, 1 voter node,
  network+dns enabled. PASS.
- 6.5 GATE 2 THE D-035 GATE (inline) -- agnhost pod-egress probe -> Keystone VIP exitCode:0 (Succeeded).
  Single-NIC pod egress proven (the exact test the dual-homed D-033 node failed). PASS.
- 6.6a-6.6d (inline ssh-to-VM) -- CAPI provider stack on the mgmt VM: tooling pinned from
  capi-helm-charts@0.25.1 dependencies.json (D-034: CAPI v1.13.2 / CAPO v0.14.4 / CERT v1.20.2 / ORC v2.5.0 /
  CAAPH 0.12.0 / JANITOR 0.11.0 / HELM v3.17.3); cert-manager v1.20.2 (crds.enabled=true, DOCFIX-025a);
  ORC v2.5.0 server-side apply (images.openstack.k-orc.cloud CRD present) BEFORE clusterctl init;
  clusterctl init core+kubeadm+CAPO all condition met (capo-system Available first pass -- ORC-first order
  correct). PASS through 6.6d.
- 6.6e (inline) -- CAAPH (cluster-api-addon-provider 0.12.0) + janitor (cluster-api-janitor-openstack 0.11.0)
  via azimuth helm charts; both Running (addon-provider took one benign first-boot restart while cert-manager
  minted its webhook cert, then stable 1/1). PASS.
- 6.6f (inline) -- verify: clusterctl v1.13.2; all controllers 1/1 Running (cert-manager x3, capi core/
  bootstrap/control-plane, capo, orc, addon, janitor); all 4 key CRDs present (clusters / openstackclusters /
  kubeadmcontrolplanes / images.openstack.k-orc.cloud). Phase-06 EXIT GATE green. PASS.

SWEEP DONE (phase-06 reconciliation; delivered as a repo-relative ZIP, committed at the sweep):

DOCFIX-062 -- 6.5 kubeconfig-pull defect (ASSIGNED this sweep; grep-before-assign confirmed next-free):
- `sudo k8s config server=<url>` does NOT override the emitted apiserver URL on k8s-snap 1.32.13; it writes
  the node tenant IP (10.20.0.207:6443), unroutable from the jumphost, so `kubectl get nodes` i/o-timed-out.
- FIX (applied live, now baked into the runbook + script): pull the RAW admin config (`sudo k8s config
  </dev/null`), then rewrite the endpoint locally -- `CLUSTER=$(kubectl config view -o
  jsonpath='{.clusters[0].name}'); kubectl config set-cluster "$CLUSTER" --server="https://$MGMT_FIP:6443"`
  (cluster name read dynamically; FIP is in the cert extra-sans from 6.4, so TLS holds). Runbook Step 6.5
  rewritten to the pull-then-rewrite form; appendix-A DOCFIX-062 entry added.

SCRIPT-CONVERSION (D-056 encapsulation; operator-directed 2026-07-01) -- three new scripts, each ASCII+LF,
`bash -n` + shellcheck -e SC1091 clean, with fixture tests (fake ssh/kubectl; real bash):
- scripts/phase-06-k8s-bootstrap.sh   (6.3 GATE 1 egress + 6.4 k8s-snap install/bootstrap). Keystone target is
  the as-run literal 10.12.4.50:5000 (ENV(keystone-vip); KEYSTONE_HOSTPORT-overridable); one-shot, matching the
  as-run 6.4 block verbatim (no idempotency guard; retry = purge). tests/phase-06-k8s-bootstrap/ -- happy +
  literal-default + override + GATE1 VIP/NET fail + 6.4 fail + 3 preconditions + a no-dynamic-discovery
  fidelity assertion. ALL PASS.
- scripts/phase-06-kubeconfig-gate.sh (6.5 with the DOCFIX-062 fix baked in so it cannot be skipped: pull +
  set-cluster server-rewrite + verify-the-rewrite-took gate + node check + agnhost GATE 2 + trap cleanup).
  tests/phase-06-kubeconfig-gate/ -- 14 cases + a dedicated DOCFIX-062 assertion (fake ssh emits the tenant-IP
  server; fake kubectl actually rewrites it; test asserts FIP present + tenant IP gone + mode 600). ALL PASS.
- scripts/phase-06-capi-stack.sh      (6.6a-f, hardened ORC-before-init order; each sub-step gated on the
  remote block's own exit via PIPESTATUS; 6.6f is an explicit all-Running + 4-CRD gate). Versions read from
  dependencies.json @ CHART_TAG (D-034; pins gated non-null). tests/phase-06-capi-stack/ -- 9 cases + an
  ORC-before-init ordering assertion; the C_FAIL case proves that if ORC fails, clusterctl init never runs.
  ALL PASS.

RUNBOOK: Step 6.5 rewritten (pull-then-set-cluster); header now lists the three canonical scripts (D-056) and
DOCFIX-062; appendix-A gained the DOCFIX-062 entry under a new "Mgmt-cluster bootstrap (phase-06)" section.

RECONCILIATION (operator-directed 2026-07-01) -- the encapsulation scripts were reconciled to encode EXACTLY the
as-run 6.3-6.6 blocks (the code walked through step-by-step, confirmed by paste-back), per "confirm exact code
used successfully." Two beyond-as-run additions were removed: (1) dynamic Keystone-VIP discovery via `openstack
endpoint list` -> reverted to the as-run literal 10.12.4.50:5000 (still KEYSTONE_HOSTPORT-overridable per site,
matching the runbook's ENV(keystone-vip) convention); (2) an untested ready-skip idempotency guard in 6.4 ->
removed (the as-run block ran the bootstrap unconditionally; retry = purge-and-re-run). DOCFIX-062 (6.5
kubeconfig server-rewrite) is KEPT -- it was applied live and confirmed. The unused fake `openstack` test stubs
were dropped and a no-dynamic-discovery fidelity assertion added to both affected suites. capi-stack.sh was
already faithful (pins from dependencies.json; no discovery). All three suites re-pass.

### 2026-07-01 -- Phase-07 conductor graft EXECUTED (Pattern-A rebuild); DOCFIX-063 + D-063
Grafted the magnum-capi-helm CAPI driver onto the charm-managed conductor `magnum/0` (now at
10.12.12.107 on metal-internal, D-052) via gated copy-paste, paste-back confirmed each step:
- 7.0 domain-setup (D-046) PASS: domain `magnum` (d9d0a4a8...) + user `magnum_domain_admin`
  (0885dca3...); `coe service list` = 1 row `up`, no 403.
- 7.1 SG authorize: NO-OP -- the phase-06 capi-mgmt-sg already opens tcp/6443 to 0.0.0.0/0;
  conductor->FIP:6443 = TCP-OK with no new rule (the hardcoded 10.12.4.76/32 rule was stale AND
  unnecessary -> DOCFIX-063 verify-first).
- 7.2 kubeconfig -> conductor PASS: /etc/magnum/kubeconfig 0600 magnum:magnum, sha256 match both
  sides (26ed1091...6c11); helm auth-proof (run post-7.4) = 6 mgmt releases deployed, versions
  match D-034 dependencies.json.
- 7.3 served versions PASS: `kubectl api-versions` shows v1beta1 SERVED for all core CAPI groups
  (alongside preferred v1beta2); empty `api_resources={}` correct -> D-042 premise confirmed.
- 7.4 driver + helm PASS: helm v3.17.3 on the restricted init PATH (/usr/bin/helm, DOCFIX-035);
  magnum-capi-helm 1.4.0; entry point k8s_capi_helm_v1.
- 7.6 [capi_helm] drop-in PASS (after dir-create); 7.7 conductor --config-dir PASS; 7.7b
  keystone-v3 drop-in (D-047) PASS (v3 URLs derived from live config: public 10.12.4.50:5000/v3,
  admin 10.12.8.50:35357/v3).
- 7.8 restart PASS: magnum-conductor + magnum-api both active, both live cmdlines carry
  --config-dir; `magnum-driver-manage list-drivers` lists k8s_capi_helm_v1 (enabled).
- HEALTHY poll (7.8 tail) + 7.9 regression DEFERRED to phase-08 per fresh-deploy routing (no
  cluster / capi-k8s-v1-34 template exists yet).

DOCFIX-063 (phase-07 runbook reconciliation, six fixes folded from the as-run):
(1) Step 7.1 -> verify-first (drop the stale/unnecessary hardcoded 10.12.4.76/32 SG rule-add;
    reachability check is the gate; a measured source rule is a fallback only);
(2) Step 7.2 helm auth-proof MOVED after Step 7.4 (helm is installed there; absent on a fresh
    conductor -- integrity sha256 + 7.1 TCP already gate 7.2 without it);
(3) Step 7.3 probe -> `kubectl api-versions | grep cluster.x-k8s.io/` (api-resources shows only
    the PREFERRED version, a FALSE "v1beta1 not served" when core groups prefer v1beta2);
(4) Step 7.6 -> `install -d /etc/magnum/magnum.conf.d` before the tee (the .conf.d dir is absent
    on a fresh rebuild; the deb ships magnum.conf only; also the 7.7 --config-dir target);
(5) Step 7.5/7.6 ASCII checks -> `sudo` the grep (a non-sudo read of the root-owned /etc/magnum
    path gave a FALSE "ASCII clean" on Permission-denied);
(6) Step 7.4 helm egress pre-check -> hit a real asset (bare get.helm.sh/ 404s misleadingly).
As-built refreshed: conductor magnum/0 10.12.4.76 -> 10.12.12.107; mgmt FIP example
10.12.5.103 -> 10.12.7.222 (per-rebuild; commands stay dynamic-from-env).

D-063 (PROPOSED/OPEN): the phase-06 capi-mgmt-sg opens tcp/6443 AND tcp/22 to 0.0.0.0/0
(CAPI default; fine for a single-DC rehearsal where the FIP is the access point) -- tighten for
Roosevelt (commercial hard-isolation). Recorded, no action on v1.

NEW (phase-07 now matches the per-phase encapsulation pattern):
- scripts/phase-07-conductor-graft.sh -- encapsulates 7.0-7.8 with DOCFIX-063 baked in
  (verify-first 7.1; auth-proof after helm install; api-versions probe 7.3; install -d before
  the tee; sudo ASCII; real-asset helm pre-check). [SENSITIVE] base64-pipes the FIP kubeconfig
  to a 0600 magnum-owned file with a sha256 both-sides gate; every step is fail-loud (exit 1
  gate / exit 2 precondition). All as-run values are env-tunable but DEFAULT to the measured
  values (MODEL, CONDUCTOR unit, DRIVER_VERSION 1.4.0, HELM_VERSION v3.17.3, CHART_VERSION
  0.25.1, ENVFILE ~/capi-mgmt-net.env). Health poll + regression are phase-08 (not in-script).
- tests/phase-07-conductor-graft/ -- fakebin-stubbed suite (juju/openstack/kubectl/base64/sha256sum)
  asserting: 7.1 verify-first is a no-op when 6443 is 0.0.0.0/0; the v3-URL derivation (unversioned
  and /v2.0 inputs -> /v3 in both sections); install -d precedes the tee; api-versions probe; the
  sha256-mismatch gate fails loud; k8s_capi_helm_v1 enabled gate; precondition failures. ALL PASS.

## 2026-07-01 -- phase-08 keystone-policy blocker -> D-064 (reconcile D-051 to scs-0302)

phase-08 8.1 cluster create failed at create_trustee_and_trust: identity:create_user 403
(magnum_domain_admin has Admin on the magnum domain, authenticated -> authorization gap, not
auth). Root cause: charm-rendered old-style policy.json defines the create-op helper
admin_and_matching_user_domain_id with legacy domain_id:%(user.domain_id)s templating that does
not resolve for create_user on Caracal (keystone populates target.user.domain_id). Same defect
blocked the 2026-06-22 northwind tenant self-service rehearsal. D-051 as-staged would NOT have
fixed Magnum (its create_user admin fallthrough inherited the broken helper; only its manager
branch worked).

Reconciled the staged policies/domain-manager-policy.yaml to scs-0302 (D-064). 7 rules changed:
- create_user / create_project / create_group: broken %(<x>.domain_id)s helper ->
  %(target.<x>.domain_id)s helper (BUG-FIX; unblocks Magnum via documented admin-on-domain path).
- add_user_to_group: -> admin_and_matching_target_group_domain_id (ALIGN; restore live default;
  D-051 drift).
- list_users / list_projects / list_groups: admin_required (conservative, broader than live) ->
  cloud_admin or admin_and_matching_domain_id (ALIGN to live default; discharges D-051
  [LIVE-READ PENDING]). list_roles kept at admin_required (its live default).
Manager persona (is_domain_manager=role:manager; is_domain_managed_role=member +
load-balancer_member) unchanged. Consequence: NO manager grant on magnum_domain_admin needed.
Validated: oslo.policy parses all 37 rules (charm checks YAML only); YAML+ASCII+connector lint
clean; every fallthrough tail diffed vs the live effective policy. Behavioral gate pending live
attach: Magnum 8.1 re-run + tenant G3. New reference: runbooks/appendix-C-identity-rbac.md
(role/policy/account assignment tables + attach + G3 procedure). Known SCS limitation carried:
manager can enumerate domain + role names cloud-wide (list_domains/list_roles); remove override
on any 2024.2+ upgrade.

## 2026-07-02 -- multi-tenant trust blocker -> D-065 (create_trust template fix)

Multi-tenant validation (tenant acme, manager persona) reached cluster-create and failed at
identity:create_trust 403 -- AFTER D-064 unblocked create_user (trustee user created). Reproduced
identity-independently: admin (password) and a direct `openstack trust create` with trustor==self
both 403. Root cause: base charm policy.json ships create_trust with the non-resolving
user_id:%(trust.trustor_user_id)s (Caracal populates target.trust.trustor_user_id) -> evaluates
false for everyone. Same defect class as D-064. Causation proven by toggling use-policyd-override=false
(still 403 -> base policy owns it, override exonerated), then re-enabling (PO: restored).

Fix (D-065): added identity:create_trust = user_id:%(target.trust.trustor_user_id)s (keystone's
shipped default) to the override. create_trust is the ONLY broken trust rule (list_trusts_for_* already
target-prefixed; get/delete/list/roles-for-trust are empty but code-guarded -> left unchanged --
evidence-gated, not a blanket sweep). oslo.policy parses all 38 rules; ASCII+LF+connector clean.
appendix-D D.3 (role-delegation hypothesis) REFUTED and to be revised. list_trusts="" info-disclosure
recorded PROPOSED/OPEN, not actioned.

## 2026-07-02 -- multi-tenant tenant->cluster buildout: trust path PROVEN; Barbican/Vault substrate blocker

Ran a full multi-tenant buildout (tenant `acme`) validating the SCS Domain Manager persona end to
end and the tenant cluster-create path. Outcomes:
- IDENTITY/TRUST PATH DONE + PROVEN: D-064 (create_user) + D-065 (create_trust template fix) + the
  D-066 Option-3 password model carried cluster-create through create_user, create_trust, and INTO
  cert generation. Manager self-service (create_project/user/grant) validated via CLI (G3); tenant
  isolation validated (anti-escalation DENY; cross-domain DENY; domain-enumeration own-domain-only,
  tighter than appendix-C's SCS worst-case -- appendix-C already corrected).
- KEY CONSTRAINT (D-066): keystone blocks trust creation from app-cred tokens regardless of
  unrestricted (this build's _check_application_credential). Cluster-creator MUST be password-auth.
  Adopted Option-3 3-account model (domain-admin / cluster / svc). allow_insecure_...escalation REJECTED.
- BLOCKER (D-067): cluster-create dies at cert-gen -- Barbican 500 -> Vault AppRole login rejected by
  CIDR restriction ("source address 10.12.8.176 unauthorized"). barbican->Vault uses metal-admin;
  should be metal-internal (D-052/D-053). Bundle is CORRECT (all secrets endpoints metal-internal);
  LIVE drifted. Fix = live rebind (gated, next session), NOT CIDR-widen. refresh-secrets rotated the
  secret_id but did not fix it (CIDR, not expiry -- "TTL" hypothesis refuted).
- D-068 PROPOSED: Vault substrate hardening (1.16 pin [bundle done], TLS, AppRole lifecycle).

Package this session: bundle vault 1.16/stable pin; D-066/067/068; appendix-D corrected (D.3 refuted);
appendix-C account table -> Option-3; tenant-onboarding-v2 updated to Option-3 + password-create +
substrate note; scripts/tenant-onboard.sh (DRAFT); session-findings doc (incl. probe-discipline lessons).

PROBE-DISCIPLINE lessons logged (recurred this session; now runbook conventions): (1) validate raw
output WHOLE, never extract-then-check (a `tr -dc 0-9` MARK guard synthesized a fake number from an
error string); (2) whitelist-print secrets, never blacklist-redact (an `approle_secret_id` leaked past
a `secret`-keyed redact); (3) no `exit`/bare-`return` in interactive paste -- subshell-wrap (an `exit`
logged the operator out); (4) privileged reads over juju ssh use `sudo cat`, never `sudo cmd < file`
(unprivileged redirect -> Permission denied); (5) read the deployment's DECLARED endpoint/scheme, not
the conventional one (assumed Vault https; it serves http).

### Next-free numbers
Design decision: D-069. Doc fix: DOCFIX-065. (D-064 ASSIGNED above = reconcile D-051 to scs-0302
+ create-op templating fix. DOCFIX-064 RESERVED = phase-08 runbook sweep (image --public; seed
retry/timeout + poll hard-gate + post-active property re-verify; image-absent guard; template
capi-mgmt scope preamble + flavor floor; 8.1 D-039 role + keypair pre-checks; octavia prereq
real-exit capture), to be written at phase-08 close. D-063 = capi-mgmt-sg 0.0.0.0/0 hardening,
PROPOSED/OPEN. DOCFIX-063 = phase-07 reconciliation, six fixes.)

## 2026-07-02 (session 2) -- D-067 FIXED live + CLOSED; root cause corrected to bundle+charm defect

Fixed the Barbican/Vault cert-gen blocker with ONE gated mutation:
`juju bind vault external=metal-internal`. Root cause CORRECTED from "live drift" to bundle+charm:
the bundle omitted vault's `external` binding (-> '' default metal-admin); charm-vault publishes
vault_url (access) then vault_url_external (external), and the vault-kv interface ignores
remote_binding (LP#1895185, verified at vendored commit 6f7848c), so the external URL clobbers
vault_url on all kv consumers while the AppRole stays CIDR-bound to the consumer's metal-internal
/32. Diagnosis was code-grounded (charm-vault stable/1.8 + build.lock-pinned interface), gated:
Gate 1 (config/network-get/hacluster preflight) -> Gate 2 (bind + relation-data + conf-render
polls; base64-to-file delivery after a raw-paste truncation) -> Gate 3 (AppRole login 200 from
authentic source .110 + admin secret store/get/payload/delete round-trip). No refresh-secrets
needed; no CIDR widening. BUNDLEFIX-007 adds `external: metal-internal` permanently (double-publish
present on charm-vault master). D-052's "vault external -> metal-admin operator path" premise
corrected (endpoint's only functional use is the kv URL advertisement). D-067 amendment appended;
tenant-onboard.sh stage-6 gating note updated to RESOLVED.

Also this session: handoff commit 22a1eef verified 7/8 (bundle.yaml was DELETED by the loose-file
commit pattern; restored + verified in 10e9186 -- diff vs pre-delete is exactly the vault 1.16 pin).
Observation (logged, not actioned): barbican Secret href renders https://None:9312/... (host_href
None) -- cosmetic for UUID-extracting clients; watch during stage-6 cert refs.

### Next-free numbers
Design decision: D-069. Doc fix: DOCFIX-065 (unchanged). Bundle fix: BUNDLEFIX-008 (007 ASSIGNED
above; repo tree grep showed 002-004 but 001-006 exist in history, so 007 is the first safe free).

### 2026-07-02 (session 2, addendum) -- tenant-onboard.sh Option-3 keypair defect fixed pre-run

Code-review catch before the first clean-room run: stage3 created the nova keypair as -svc while
stage6 creates the cluster as -cluster. Nova keypairs are USER-scoped; magnum validates the keypair
in the REQUEST context (magnum 18.0.0 attr_validator.validate_keypair -> cli.nova().keypairs.get,
invoked from cluster.py:545 at cluster create) and node boot resolves it as the TRUSTOR (the
per-cluster trust app-cred impersonates the trustor). A -svc-owned key is invisible to -cluster ->
deterministic KeyPairNotFound 400 at stage6. Option-3-specific: all prior validations used a single
identity that owned its own key. Fix: keypair created by -cluster (stage3, second subshell);
template create (stage5) drops --keypair (keypair_id optional, default=None -- verified in source);
stage6 already supplies --keypair as -cluster. Rejected alternative: creating the template as
-cluster (blurs the Option-3 svc/cluster division for no gain).

### 2026-07-02 (session 2, addendum 2) -- beta ACCEPTANCE PASS; offboard + vault-kv-health delivered

TENANT ACCEPTANCE (gates t2-02 + t3-01, tenant beta, clean-room Option-3, zero admin fallback):
- Cluster beta-cluster (a360df32) CREATE_COMPLETE in ~13.5 min as beta-cluster (PASSWORD);
  health_status=HEALTHY (D-042 1.4.0 pin confirmed on a tenant-created cluster).
- D-066 evidence: per-cluster child app cred magnum-a360df32-... OWNED BY beta-cluster (trust
  impersonates the trustor); one trust, impersonation=True. TRUSTEE CORRECTION: the trustee is
  the PER-CLUSTER user <cluster_uuid>_<project_id> in the magnum domain -- magnum_domain_admin
  is the identity that CREATES these trustees (the D-064 create_user op), not the trustee itself.
  tenant-acceptance.sh P0 now asserts the real invariant (name == <cluster_uuid>_<project_id>,
  domain == magnum).
- P1 kubeconfig fetched BY THE TENANT; 2 Ready nodes; 31/31 pods Running. P2 OCCM ->
  Octavia LB (FIP 10.12.6.97) serving in ~2 min; kubeapi + service LBs ACTIVE/ONLINE.
  P3 isolation: acme blind to every beta resource (cluster 404, net not-found, empty lists).
- host_href=None observation: did NOT affect cert-gen, cluster config fetch, or cert refs.

NEW DELIVERABLES (mock-tested; harnesses committed under tests/):
- scripts/tenant-offboard.sh: full tenant offboarding, audit-by-default, --apply gated by typed
  client name at /dev/tty, hard domain blocklist (Default/admin_domain/service_domain/magnum/capi),
  dependency-ordered teardown (clusters-as-tenant first -> trusts -> LB/FIP/server -> L3 ->
  users/projects/domain -> local creds retired). 6-branch harness ALL PASS incl. zero-mutation
  audit proof and pty-driven confirmation tests. Immediate consumer: acme cleanup.
- scripts/vault-kv-health.sh + vault-kv-inner-probe.sh: D-068 item 3 proactive probe. Dynamic
  consumer discovery from vault:secrets relations; per consumer: relation vault_url plane vs
  METAL_INTERNAL_CIDR (lib-net.sh), conf render == relation data, AppRole login 200 from the
  authentic source; plus vault-unsealed and the BUNDLEFIX-007 live invariant (external==access;
  reports "D-067 clobber condition LIVE" on drift). Unknown consumer = loud coverage gap, exit 2.
  Probe encoded at RUNTIME from the sibling file (no embed drift). 7-branch harness ALL PASS.

PROBE/PORTABILITY LESSONS (added to conventions):
- mawk (Ubuntu default awk) does not support backslash-s -- use POSIX [[:space:]] classes in awk
  and grep; never rely on gawk/GNU extensions in repo scripts.
- Never append an inline # comment inside a $( ... ) command substitution or pipeline -- it
  comments out the closing paren (caught by bash -n in harness).
- Edit-plan discipline for scripted file surgery: overlapping replace targets must be ordered by
  dependency or made disjoint (two fail-loud aborts this session, zero corrupted writes).

SCRIPT BACKLOG (maintained; build at the appropriate step):
1. DONE scripts/tenant-offboard.sh   2. DONE scripts/vault-kv-health.sh
3. LARGELY DONE by cloud-assert.sh (DOCFIX-075, committed 2026-07-03: D-045/D-046/D-051/D-062/
   vault-sealed behavioral checks + harness). REMAINING: fold vault-kv-health.sh in as a
   cloud-assert section (D-067/BUNDLEFIX-007/AppRole coverage is additive today), and use the
   combined gate as the backbone for regenerating the stale restart-procedure runbook.
4. scripts/validate.sh -- draft the real D-011 runner (still PLACEHOLDER at f7bb36f); compose
   from tenant-acceptance.sh + phase-03/04/05 verifies; D-019 dropped the DNS criterion.
5. scripts/tenant-cluster-create.sh -- generalize the t2-02 stage6+watch+D-066-evidence wrapper.
6. scripts/keystone-policy-drift.sh -- D-051 base_* alignment (LIVE-READ PENDING) via
   oslopolicy-policy-generator diff; guards the 2024.2 upgrade-removal caution.
7. scripts/cloud-snapshot.sh -- pre/post-maintenance state capture.
8. HALF DONE: tests/tenant-onboard/ landed 2026-07-03; tests/tenant-acceptance/ still pending
   (mock suite exists from the 2026-07-02 session).
NOTE (tooling gate): scripts/run-tests-all.sh (committed 2026-07-03) now runs every
tests/*/run-tests.sh; both harnesses in this package were verified under it pre-commit.

### 2026-07-03 (addendum 3) -- DOCFIX-082/083 hardening; acme offboarded live; handoff reconciliation

ACME OFFBOARDED (first live run of tenant-offboard.sh, pre-hardening build): audit -> typed-gate
apply, exit 0. CREATE_FAILED cluster 76aea330 deleted by magnum in ~30s (the delete path through
the now-healthy barbican); Phase B printed no residual-trust delete, i.e. magnum's cluster delete
cleaned trust 8592ee35 itself; L3 + both users + project + domain removed; four legacy loose cred
files flagged for manual handling. Post-apply verification (domain/trust/orphan-trustee/beta
untouched) recorded separately when run.

DOCFIX-082 -- tenant-offboard.sh error-regime hardening (handoff H-1).
  WHAT: set -uo pipefail; all mutations via run() (capture-then-test, loud FAILED lines,
  per-phase counters); sweep failures now exit 22 (continue-and-report regime, stated in
  header); identity failures exit 23 with count; is_id tightened to ^[0-9a-f]{32}$;
  capture-then-grep converted to here-strings (SIGPIPE discipline under pipefail).
  WHY: the pre-082 build lost mutation exit codes in display pipelines -- concretely, Phase E's
  `... | sed ... || FAILED=1` bound the || to sed, making the failure flag DEAD CODE. The acme
  run above was protected only by the domain-gone assertion (sufficient there: disabled-domain
  delete cascades users/projects in keystone; but sweep-phase failures could strand resources
  silently). Found on re-inspection prompted by handoff H-1; the register was right.
  ACCEPT: tests/tenant-offboard 7/7 incl. new sweep-failure-injection case (exit 22 + FAILED
  line asserted). REVERT: git checkout of scripts/tenant-offboard.sh +
  tests/tenant-offboard/run-tests.sh at the parent of this commit.

DOCFIX-083 -- vault-kv-inner-probe.sh: credentials off argv (handoff H-2).
  WHAT: AppRole login body now fed to curl via stdin (--data @-); nothing secret-bearing on
  curl argv. WHY: `-d "$BODY"` exposed role_id+secret_id in /proc/*/cmdline (ps) on the unit
  for the request duration; 60s token TTL bounded it but the house rule is absolute.
  ACCEPT: tests/vault-kv-health 8/8 incl. new source-level argv-hygiene assertion on the login
  invocation. REVERT: git checkout of scripts/vault-kv-inner-probe.sh +
  tests/vault-kv-health/run-tests.sh at the parent of this commit.

HANDOFF H-3 (identifiers): the 2026-07-02/03 script deliveries are recorded in addenda 2-3 of
this file; vault-kv-health implements D-068 item 3 (cross-linked there by the D-068 text);
DOCFIX-082/083 above are their follow-up identifiers with reverts. Next-free after this
commit: D-071, DOCFIX-084, BUNDLEFIX-009.

HANDOFF D-3 (D-042 status): CONFIRMED RESOLVED BY PIN. The magnum-capi-helm 1.4.0 pin (D-042)
was validated live 2026-07-03: tenant-created beta-cluster reports health_status=HEALTHY (the
1.3.x false-negative would have shown UNHEALTHY/None persistently). The seven-stage contract-ref
graft runbook staged for 1.3.x is OBSOLETE for v1 -- recommend closing D-042 with a
resolved-by-upgrade note rather than executing the graft.

OFFBOARD v2 GAP (logged, not built): if magnum's cluster-delete cleanup is ever incomplete, an
orphaned PER-CLUSTER TRUSTEE USER (<cluster_uuid>_<project_id>) survives in the MAGNUM domain --
outside tenant-offboard.sh's user sweep, which covers only the client domain. Post-apply verify
blocks check for this orphan; a --sweep-magnum-orphans mode is the v2 item.

HARNESS LESSON (this session): assertion selectors need the same rigor as production greps --
two consecutive mis-anchored source assertions (first-line-with-curl; regex that could not
cross || at line end) were caught by running the harness before delivery. Parameter expansion
beats regex for line-splitting assertions.


### 2026-07-03 (addendum 4) -- 8d verified; V-1 ruled; D-063 measured + adopted (a-amended)

GATE 8d (acme offboard post-apply verification): ALL FOUR PASS -- domain gone, trust 8592ee35
gone, NO orphan per-cluster trustee in the magnum domain (magnum's cluster delete cleaned the
trustee; the offboard-v2 orphan gap stays theoretical), beta untouched at
CREATE_COMPLETE/HEALTHY. This completes the acme offboard record opened in addendum 3.

V-1 RULING (octavia controller cert SAN, handoff section 2): RISK-ACCEPTED for v1.
Deployed SAN reads DNS:octavia-controller.omega.dc0.vr0.cloud.neumatrix.local,
DNS:octavia.omega.dc0.vr0.cloud.neumatrix.local, IP:10.12.4.233 (stale literal from the prior
carve). Octavia is demonstrably green (beta kubeapi LB + lbtest service LB built to
ACTIVE/ONLINE this session). Mechanism note for the record: controller_cert.pem is presented
as a TLS CLIENT credential to the amphora agent (:9443); client-cert verification performs no
hostname/IP SAN matching, so the stale IP-SAN never enters a verification path in the working
direction. Regenerate at next redeploy; NO hot-rotation. Local generation copies exist under
~/octavia-pki/ (2026-05-29) as corroborating evidence.

D-063: measured (route + 170s SYN window with operator trigger) and ADOPTED as (a-amended) --
full measurement record and rule set in the D-063 amendment (design-decisions.md). Key
finding: upstream masquerade collapses ALL routed-in sources (incl. the jumphost) to
10.12.4.1, so the operator rule is documented-coarse; magnum /32s + capi-mgmt-subnet hairpin
complete the set. SG APPLY IS PENDING as its own gated mutation. Probe-method lesson recorded:
transient clients need window capture, not ss snapshots; and one-liner probes now get fixture
tests like scripts (three parse bugs in one measurement chain this session -- ss column drop
under state filters, cut on mapped-v6 colons, regex vs || line tails -- all caught, none live).

Next-free: D-071, DOCFIX-084, BUNDLEFIX-009 (unchanged; D-063 amendment reuses its number).

### 2026-07-03 (addendum 5) -- DOCFIX-084: scripts/d063-apply.sh (D-063 a-amended applier)

DOCFIX-084 -- scripts/d063-apply.sh + tests/d063-apply/.
  WHAT: repo-deliverable applier for the D-063 (a-amended) rule set on capi-mgmt-sg.
  Audit-by-default; --apply gated by typing d063 at /dev/tty. Magnum unit /32s derived LIVE
  per unit via `ip route get $FIP` (the same mechanism the adopting measurement used --
  PATTERN-1, enumerates units, Roosevelt-safe); wide 0.0.0.0/0 and ::/0 rules for 22/6443
  removed by rule IDs MEASURED in the same run. Safety ordering: all adds created AND read
  back present (exit 27 if any missing) BEFORE any remove -- no window where the conductor
  poll is cut off. Duplicate adds (409) are idempotent-present. Derivation failure aborts
  with zero mutations (exit 21). set -uo pipefail + run() capture-then-test throughout.
  WHY: D-063 amendment mandates per-environment re-derivation, never rule copy-forward; a
  harnessed script is the reusable form. ACCEPT: tests/d063-apply 7/7 incl. ordering
  assertion (last create strictly before first delete), zero-mutation proofs on
  confirm-mismatch and derive-fail, and a readback-gate case. Harness lesson: two fixture
  bugs (non-hex mock IDs rejected by the tightened is_id; a 409 mock that lied about rule
  existence) were both caught BY THE SCRIPT'S OWN GATES -- the mock world must be truthful.
  REVERT: delete the added /32 and hairpin rules by readback and recreate the two wide rules
  (openstack security group rule create --ingress --protocol tcp --dst-port {22,6443}
  --remote-ip 0.0.0.0/0 <sg>); or git checkout both files at this commit's parent.

Next-free: D-071, DOCFIX-085, BUNDLEFIX-009.

### 2026-07-03 (addendum 6) -- DOCFIX-085: d063-apply.sh structured-capture fix (live audit fail-closed)

First live run of d063-apply.sh --audit failed CLOSED at exit 20 with zero mutations (SG and
project resolution were clean). Measured mechanism (streams separated in a follow-up probe):
`--long` is deprecated in the deployed openstackclient and prints its warning ON STDERR; the
script captured JSON with 2>&1, so the warning landed at char 0 and killed the parse.

DOCFIX-085 -- scripts/d063-apply.sh + tests/d063-apply/.
  WHAT: drop --long everywhere (deprecated; fields present without it); NEVER merge stderr
  into structured-output captures -- both JSON rule-list captures now write stderr to a
  tempfile, surfaced as informational on success and printed in full diagnostics (stdout head
  + stderr head) on parse failure, fixing the diagnostic gap where PARSE-ERROR swallowed the
  evidence. WHY: any stderr warning (deprecations today, SDK notices tomorrow) must never be
  able to corrupt a parse; silencing with 2>/dev/null was rejected as error-masking.
  ACCEPT: tests/d063-apply 7/7 with the mock made FAITHFUL to the real CLI (rule-list now
  always warns on stderr), so every scenario regression-tests the stream separation.
  CONVENTION (add to script-authoring): structured-output (-f json/value) captures take
  stdout ONLY; stderr goes to a tempfile and is reported, never merged, never discarded.
  REVERT: git checkout both files at this commit's parent.

Next-free: D-071, DOCFIX-086, BUNDLEFIX-009.


### 2026-07-03 (addendum 7) -- D-063 RESOLVED/CLOSED: applied + functionally verified

d063-apply.sh --apply executed: 4 tight rules added (readback-gated) then 2 wide 0.0.0.0/0 rules
removed by measured ID; final SG = conductor /32 + operator /32 + hairpin /24 on 6443, operator
/32 on 22. VERIFIED through the tightened SG: (R1) TCP handshake magnum/0 -> mgmt 10.12.7.222:6443
= TCP-OK, source measured 10.12.4.154; (C3, gold standard) forced beta resize 1->2 reached
UPDATE_COMPLETE/HEALTHY/2 -- a magnum-capi-helm resize cannot complete without the conductor
helm-upgrading on the mgmt apiserver through the SG; (negative) beta POD -> mgmt:6443 TIMED OUT
(tenant SNAT not in allow set = the intended isolation); (operator) readyz ok via 10.12.4.1.
Doc-backed by StackHPC magnum-capi-helm deep-dive + upstream install guide (conductor is an
event-driven kube CLIENT of the mgmt apiserver, not a continuous poller -- correcting my earlier
mischaracterization; the idle path is why SYN/established snapshots showed nothing).

Process notes (best-practice, for the record): I mischaracterized the conductor->mgmt flow as a
continuous poll TWICE before as-built + upstream docs corrected it to event-driven client traffic;
the correction is what made the empty idle-captures obviously benign. Packet-capture (SYN) was the
wrong instrument for a bursty per-op flow 3x running -- functional outcome (a successful resize)
and TCP-from-source are the right proofs. Repeated hand-built one-liner parse bugs this thread (ss
column-drop under state filters; cut on mapped-v6; grep greedy over `src IP.`; journald vs
/var/log/magnum) reinforce the standing rule: fixture-test probe parsing like script code, even
for throwaway measurement. Beta left at 2 workers (bonus resize acceptance coverage).

Next-free: D-071, DOCFIX-086, BUNDLEFIX-009.

### 2026-07-03 (addendum 8) -- validate.sh modular foundation: lib-validate + orchestrator

D-011 validation runner rebuilt as a MODULAR suite (approved design: modular-from-start,
PASS-PENDING-MANUAL verdict tier, amphora failover in item 4, per-check --disruptive gate +
orchestrator --include-disruptive pass-through, machine-readable per-check result lines).
This addendum delivers the FOUNDATION; the six d011-0N checks compose on it next.

scripts/lib-validate.sh (sourced library; harnessed 35/35):
  - STANDARD EXIT CONTRACT for every check + the orchestrator:
      0 PASS | 1 FAIL | 2 HOLD | 3 PASS_PENDING_MANUAL | 4 SKIPPED
  - emit() one parseable RESULT line (id/word/code/elapsed/msg); vr_begin() timer.
  - vr_json(): structured-output capture that NEVER merges stderr into the JSON
    stream (the DOCFIX-085 invariant, now library-enforced so no future check can
    reintroduce it); stderr goes to a tempfile surfaced via vr_err_tail.
  - run(): capture-then-test mutation helper (DOCFIX-082 regime; no pipeline on the
    mutating command -> status is the command's, SIGPIPE-safe).
  - vr_admin_env / vr_tenant_env: scrub-then-scope OS_* (missing fields -> HOLD, never
    a silent half-scope); vr_is_hex32/vr_is_ipv4/vr_need validators; vr_disruptive_ok gate.
  - sourced-library guard (refuses direct exec, exit 2), matching lib-net.sh.

scripts/validate.sh (orchestrator; harnessed 26/26; replaces the PLACEHOLDER):
  - PROFILES (data, not code): full-d011 | post-restart | lb-health | isolation; plus
    --checks a,b,c explicit list. --list shows profiles + discovered checks.
  - Verdict aggregation worst-wins with a custom precedence FAIL>HOLD>MANUAL>PASS/SKIP;
    SKIP never lowers the verdict (a lone skipped disruptive check => OVERALL PASS).
  - --stop-on-fail (halt after first FAIL/HOLD) vs default continue-and-report-all.
  - --include-disruptive pass-through to checks (VR_DISRUPTIVE); missing check -> HOLD;
    a check emitting no RESULT line -> exit code trusted, line synthesized.
  - Single structured report: per-check RESULT lines + totals + OVERALL exit.

HARNESS LESSON (recorded): one orchestrator "failure" during the build was a WRONG TEST
ASSERTION (expected a lone skipped check to yield overall exit 4, contradicting the
SKIP-doesn't-lower-verdict contract just specified). Fixed the test, added a stronger
check-level assertion (read the check's own RESULT line). The mock world / test
expectations must match the specified contract -- same lesson as the 409 and stderr mocks.

REMAINING (checks compose on the foundation; each harnessed, each standalone-runnable):
  d011-01-charms (wrap cloud-assert A0), d011-02-vip-jumphost (author: catalog VIP TLS/HTTP),
  d011-03-vip-tenant (author: agnhost on a beta node -> keystone VIP, D-035 pattern),
  d011-04-octavia-lb (author: 2-backend RR -> member failover -> AMPHORA failover w/ N+1
  headroom HOLD-guard -> recovery -> self-cleanup; --disruptive-gated), d011-05-magnum-e2e
  (wrap tenant-acceptance), d011-06-vault-unseal (attest SEC-003 in security-ledger -> MANUAL).

Next-free: D-071, DOCFIX-086, BUNDLEFIX-009.

### 2026-07-03 (addendum 9) -- validate checks batch 1: d011-01-charms, d011-06-vault-unseal

Two checks composed on the lib-validate foundation (addendum 8); harnessed 18/18; gauntlet 30.

scripts/checks/d011-01-charms.sh -- D-011 item 1 (all charms active/idle). FOCUSED check, NOT
a cloud-assert wrapper: cloud-assert is all-or-nothing (A0-A8, no per-section invocation), so
wrapping it would over-scope item 1 and double-count other items. Asserts workload=active +
agent=idle across all units + subordinates via a fixture-tested jq walk; tolerates carried gss
units in workload 'unknown' (documented in cloud-assert A0). Model unreachable / non-JSON ->
HOLD (cannot determine, not a charm failure); a unit not active/idle -> FAIL (names it).

scripts/checks/d011-06-vault-unseal.sh -- D-011 item 6 as amended by D-069 (second-person
unseal). NOT scriptable (needs a different human), so an ATTESTATION GATE: reads the SEC-003
row in docs/security-ledger.md. OPEN/PENDING -> PASS_PENDING_MANUAL(3); CLOSED/REHEARSED/DONE
-> PASS(0); missing row/file or unrecognized status -> HOLD(2). Never auto-passes an undone
safety item (the D-070 failure mode). Currently SEC-003 is OPEN, so this returns MANUAL until
R-1 (custodian assignment) + the rehearsal are done and the ledger row is closed.

DESIGN ADJUSTMENT (flagged): plan said "wrap cloud-assert A0"; on inspection cloud-assert has no
per-section mode, so item 1 is a focused reimplementation (thin, exact D-011 semantics).
Complementary to cloud-assert, not duplicative -- different granularity.
HARNESS LESSON: integration test initially copied checks to an isolated temp, breaking their
relative lib-validate resolution ($HERE/../lib-validate.sh). Real layout (scripts/checks/ +
scripts/lib-validate.sh sibling) is mandatory; integration now runs from the real checks dir.

NUMBERING NOTE: "next-free D-071" in prior addenda is CONTENDED -- a parallel Claude Code
workstream (jumphost) is filing D-071 (Juju controller update cadence / patch policy). D-071
is relinquished to that stream; this repo's next-free advances to D-072+ AFTER that push lands
and is re-grepped. These validate checks consume NO new D/DOCFIX/BUNDLEFIX number (they
implement the addendum-8 approved design).

REMAINING: batch 2 (d011-02-vip-jumphost, d011-03-vip-tenant); batch 3 (d011-04-octavia-lb w/
amphora failover + N+1 headroom HOLD-guard, d011-05-magnum-e2e wrapping tenant-acceptance).

### 2026-07-03 (addendum 10) -- session-ledger continuity mechanism + skill hardening rules

Compaction-continuity: long sessions get compacted repeatedly and scrollback-only work is lost.

NEW docs/session-ledger.md -- durable in-flight record (active build, backlog, operator-gated,
open decisions, numbering state incl. the D-071 parallel-stream contention, state facts).
Standing practice: read at session start, update at every deliverable (like the changelog).

NEW scripts/ledger-scan.sh (+ tests/ledger-scan 16/16) -- the DRIFT CHECK that keeps the
narrative honest. Derives from repo ground truth: PROPOSED/OPEN decisions (by the LAST Status
line per block, so an amendment closes it), OPEN security-ledger rows, and next-free D/DOCFIX/
BUNDLEFIX (D from decision HEADERS; next-free-pointer prose excluded so a "next-free D-072"
mention cannot inflate the count). Read-only, exit 0. Its FIRST real run caught two real bugs
in itself (first-status vs last-status open detection; prose mentions inflating next-free) AND
one repo drift (D-063 header still said PROPOSED/OPEN after closure) -- all three fixed here.

D-063 header/first-status reconciled to RESOLVED/CLOSED (the closure had been appended as an
amendment but the header still read PROPOSED/OPEN -- exactly the drift the scan exists to catch).

SKILL HARDENING (repo .claude/skills/openstack-cloud-ops + repackaged .skill) -- generalizable
lessons from this session promoted to standing discipline so they carry to all future work:
  operating-discipline.md: parallel-work numbering coordination (declare consumed ids; re-grep
    after a parallel push; scan derives next-free from headers); Session continuity (the ledger
    practice + drift check); adds-before-removes readback gate for boundary mutations; attestation
    gates for unautomatable safety items (never auto-pass; gate on a ledger row).
  script-authoring.md: structured-output captures take stdout ONLY (never merge stderr into
    -f json/-f value -- the DOCFIX-085 class); fixture-test throwaway probe one-liners like real
    code (the ss column-drop / mapped-v6 cut / greedy-src-grep / journald-vs-/var/log family);
    truthful mock worlds + test expectations must match the specified contract.
  troubleshooting.md: prove event-driven flows by FUNCTIONAL OUTCOME, not packet/snapshot
    forensics (a successful resize outranks a captured packet).
  SKILL.md: session-start step (read ledger + run scan) + a routing-table row.
  .skill package rebuilt from the updated references.

Next-free (per scan, header-authoritative): D-071 (CONTENDED -- parallel stream), DOCFIX-086, BUNDLEFIX-010.


### 2026-07-03 (addendum 11) -- validate checks batch 2: d011-02-vip-jumphost, d011-03-vip-tenant

Two checks on the lib-validate foundation; harness now 39/39 across all checks; gauntlet 31.
The `post-restart` profile is now fully populated (d011-01/02/03 all exist).

scripts/checks/d011-02-vip-jumphost.sh -- D-011 item 2 (public API VIPs respond on hostname
from the jumphost). Enumerates PUBLIC endpoints from the keystone catalog DYNAMICALLY
(openstack endpoint list --interface public), reduces to unique scheme://host:port, probes
each root over TLS (vault CA). Healthy set {200,201,300,301,302,401,403,404}=responds;
5xx=reachable-but-erroring->FAIL; transport error->FAIL, and a TLS-verify failure is retried
with -k ONLY to classify cert-broken vs unreachable (still FAILs -- never passes an
unverifiable endpoint). No admin scope / no endpoints -> HOLD.

scripts/checks/d011-03-vip-tenant.sh -- D-011 item 3 (API reachability from a tenant VM,
Option B / the D-035 pod-egress proof). Derives the keystone VIP:port from the tenant cred's
auth_url (dynamic); reuses ~/tenant-<client>/kube/config or fetches via `coe cluster config`
AS THE TENANT; runs ONE ephemeral agnhost pod (`/agnhost connect VIP:port`) and deletes it
(trap, even on failure). Succeeded=Option-B proven->PASS; Failed/TIMEOUT/REFUSED->FAIL; no
kubectl/creds/kubeconfig/VIP->HOLD. Additive+self-cleaning, so NOT --disruptive-gated.

LEDGER-SCAN limitation found + fixed: a "Next-free:" pointer that WORD-WRAPPED onto a second
line escaped ledger-scan's line-based `next-free` exclusion, falsely showing BUNDLEFIX next-free
as 011. No real assignment -- unwrapped the pointer here; CONVENTION added: keep "Next-free:"
pointer lines on ONE line (or rely on ledger-scan, which is now the next-free authority).
Real next-free unchanged: D-071 (contended), DOCFIX-086, BUNDLEFIX-010.

REMAINING: batch 3 -- d011-04-octavia-lb (2-backend round-robin -> member failover -> AMPHORA
failover with N+1 scheduler-headroom HOLD-guard -> recovery -> self-cleanup; --disruptive-gated)
and d011-05-magnum-e2e (wrap tenant-acceptance.sh + timing).

### 2026-07-03 (addendum 12) -- validate checks batch 3: d011-04-octavia-lb, d011-05-magnum-e2e

DRAFTED, mock-tested only (no live octavia/magnum in the sandbox); check harness 62/62,
gauntlet 31. The full-d011 profile is now COMPLETE (all six items exist). REQUIRES LIVE
VALIDATION before being trusted for an acceptance gate -- see the verify-live notes below.

scripts/checks/d011-05-magnum-e2e.sh -- D-011 item 5. Thin wrapper over tenant-acceptance.sh
with timing; maps 0->PASS, 11/12->FAIL, 13->FAIL(CRITICAL isolation), 14->HOLD, other->HOLD.
Additive+self-cleaning; not disruptive. DEPENDENCY SURFACED: P3 isolation needs a SECOND tenant
as the foil; post-acme-offboard only beta exists, so without VR_FOIL_APPCRED it HOLDs with
"onboard a 2nd tenant" -- honest, not hidden. Scope note: verifying the standing HEALTHY cluster
is a legitimate item-5 check; a FRESH create-verify-teardown is deferred to
tenant-cluster-create.sh (backlog #5) as a future --full mode.

scripts/checks/d011-04-octavia-lb.sh -- D-011 item 4 (Bobcat v3 LB pattern). Stands up its own
2-replica agnhost backend + LoadBalancer in the tenant cluster, self-cleans via trap. Two tiers:
  ALWAYS (additive): round-robin (>=2 distinct backends over N curls to /hostname) + member
    failover (delete a backend pod, assert continuity) + recovery (assert 2 backends again).
  --disruptive ONLY: AMPHORA failover, guarded by an N+1 scheduler-headroom pre-check. Octavia
    STANDALONE failover transiently needs room for one MORE amphora; at/near ceiling => HOLD
    (never FAIL, and never trigger a failover that would strand the LB in ERROR). On headroom OK:
    `loadbalancer failover`, wait ACTIVE, assert serving again.
  Verdict: FAIL if any assertion broke; HOLD if octavia unready / setup or headroom undetermined;
  PASS otherwise (with an explicit amphora SKIPPED note when --disruptive is absent).

VERIFY-LIVE QUEUE (highest-risk assumptions in d011-04, confirm on the real cloud before trust):
  1. amphora_headroom() parses `server show -f json` .flavor and `hypervisor list --long -f json`
     (.vcpus/.vcpus_used/.memory_mb/.memory_mb_used). Field shapes vary by microversion; .flavor
     may render as a dict. SAFE DEFAULT: any parse miss -> UNKNOWN -> HOLD (never failover on bad
     data), so a mismatch degrades to a conservative HOLD, not a wrong action -- but confirm the
     OK path live.
  2. LB-name match assumes the OCCM Octavia LB name CONTAINS the k8s service name. Confirm live.
  3. agnhost netexec /hostname round-robin distinctness + member-failover continuity behave as
     assumed. Confirm with a live 2-replica service.
  Run NON-disruptively first (RR+member) on beta, then --include-disruptive once headroom confirmed.
  beta sits at node_count=2 (adequate for a 2-replica service).

Next-free: D-071 (CONTENDED -- parallel stream), DOCFIX-086, BUNDLEFIX-010.

### 2026-07-04 (addendum 13, jumphost stream) -- ops-update-procedure runbook (DOCFIX-086) + D-071 filed

Authored by the parallel jumphost workstream (controller-update); identifiers consumed
per the addendum-10 contention note: D-071 and DOCFIX-086. No BUNDLEFIX consumed.

Trigger: live measurement 2026-07-04 -- controller 3.6.24 (single, non-HA) with client
snap already 3.6.25 (3/stable), and ~20 apps showing can-upgrade-to newer revisions
WITHIN their pinned channels. The repo had no update runbook and no update policy.

NEW runbooks/ops-update-procedure.md (DOCFIX-086) -- routine update window in three
gated layers: controller patch -> model agents -> in-channel charm refreshes (keystone
first, control-plane APIs, octavia, dashboards, nova-compute LAST; full cloud-assert at
group boundaries, per-app settle gates between). Vault EXPLICITLY excluded (D-068
PROPOSED; bundle pins 1.16/stable while live is 1.8/stable -- the runbook carries the
CAUTION so a naive bundle-sync cannot trip the major upgrade). Anti-fabrication gate:
upgrade-command flags are read from live `juju help` before composing mutations; the
2026-07-04 magnum can-upgrade-to anomaly (target named ch:amd64/magnum-dashboard-122,
a DIFFERENT charm) is generalized into a charm-name self-consistency GATE that excludes
and logs any mismatched app. Not yet as-executed; carries [REVALIDATE] markers.
REVERT: git checkout HEAD~ -- runbooks/ops-update-procedure.md (plus the README index
line below).

NEW docs/design-decisions.md D-071 (PROPOSED) -- routine update cadence + controller
patch policy: monthly-review cadence trigger, PATCH-only controller jumps within 3.6,
standing controller->agents->charms order, in-channel-only refreshes. Carries an
explicit Risk section: single non-HA controller, no supported backup in Juju 3.6, no
in-band downgrade -- acceptance rests on patch-only jumps + proven-healthy pre-state
(committed pre-change BOM) + D-070 rebuild posture; rehearsal-phase acceptance only,
Roosevelt must revisit with HA + a real backup story. Operator to rule.
REVERT: git checkout HEAD~ -- docs/design-decisions.md

runbooks/README.md -- indexed ops-update-procedure.md AND fixed a found gap:
ops-restart-procedure.md was never added to the runbook index (only ops-capi-recovery
was listed). Folded under DOCFIX-086 delivery, no separate number consumed.
REVERT: git checkout HEAD~ -- runbooks/README.md

FINDING 1 (logged, not fixed -- hard rule 1): repo-lint L5 "next-free identifiers"
and ledger-scan.sh disagree (L5 claimed BUNDLEFIX 051 / D 073 / DOCFIX 100, written
here WITHOUT the hyphenated token shape so this very line cannot inflate the scan).
Cause read from repo_lint.py: L5 derives next-free from ANY textual mention of an
identifier across all live docs (max+1), so prose forward references inflate it;
ledger-scan derives D from headers and drops lines containing "next-free". The scan
is authoritative per operator ruling. DOCFIX candidate for a future window: make
L5's next-free output header-based or drop it in favor of the scan.

FINDING 2 (CONVERGED with the main stream at merge): this stream independently hit
the same ledger-scan limitation the main stream found and fixed in addendum 11 --
a "Next-free:" pointer that word-wraps escapes the scan's line-based exclusion and
falsely inflates the count (the addendum-10 wrapped pointer briefly showed BUNDLEFIX
one higher than truth). The main stream unwrapped that pointer and set the
convention: keep "Next-free:" pointer lines on ONE line. This entry follows it;
no scan over-report remains after this merge. Still open as a DOCFIX candidate:
make the exclusion paragraph-aware so a future wrap cannot regress this.

Next-free after this push (per scan, keep this line unwrapped): D-072, DOCFIX-087, BUNDLEFIX-010.

### 2026-07-05 (addendum 14, jumphost stream) -- cloud-assert A7 false-negative fix (DOCFIX-087)

Identifier consumed: DOCFIX-087. Found during the ops-update-20260705 window pre-flight
(ops-update-procedure Section 1.4, first execution): cloud-assert FAILed A7 "keystone
shows NO policy override" on a cloud where the override is demonstrably live
(use-policyd-override=true; keystone/0 message "PO: Unit is ready"). Root cause: A7
grepped `juju status keystone --format=line`, and juju 3.6 line format omits workload
MESSAGES entirely, so the PO: marker can never appear -- an unconditional false FAIL
(and a violation of the house rule: --format json for anything captured).

scripts/cloud-assert.sh -- A7 now reads keystone status via --format=json and greps the
application-status message plus every unit workload message (jq join; jq/parse failure
degrades to the fail-closed NO-override branch). Semantics unchanged: PO (broken) ->
FAIL, PO: -> ok, neither -> FAIL.

tests/cloud-assert/run-tests.sh -- fake juju now serves ks-status.json for the keystone
JSON call (the line-format fixture is gone); T5 PO(broken) rewritten as JSON; NEW T10
asserts the regression (healthy keystone, no PO: marker -> FAIL). Harness 10/10, gauntlet
31/31, repo-lint 0 fail. HARNESS LESSON (addendum-10 "truthful mock worlds" instance):
the old ks-line.txt fixture encoded the line-format assumption, so the harness validated
a fiction the real client does not produce; fixtures must replicate measured output.

Window note: executed under Option B ruled by the operator mid-window (fix the verifier,
re-gate on a true PASS) -- the update window is HELD at Section 1.4 until this lands.
REVERT: git checkout HEAD~ -- scripts/cloud-assert.sh tests/cloud-assert/run-tests.sh

Next-free after this push (per scan, keep this line unwrapped): D-072, DOCFIX-088, BUNDLEFIX-010.

### 2026-07-05 (addendum 15, jumphost stream) -- G2/G3 complete + dashboard-TLS RCA + vault bundle revert (BUNDLEFIX-010)

Identifier consumed: BUNDLEFIX-010 (vault bundle revert; scan-verified next-free).
D-068 AMENDMENT recorded (operator ruling; no new D number -- amends D-068 item 1).

WINDOW PROGRESS (ops-update-20260705): G2 octavia 441->542 (LBs 2/2 ACTIVE/ONLINE,
cloud-assert PASS incl. A6). G3 dashboards openstack-dashboard 728->750,
magnum-dashboard 59->122, octavia-dashboard 120->168 (settled; HTTP + D-044 login
path healthy; D-044 override INTACT at its canonical /usr/share path). G3 stands --
NO revert -- per the RCA below and operator ruling.

RCA FINDING (dashboard VIP HTTPS dead SINCE DEPLOY -- not a G3 regression): charm
diff 728 vs 750 contains NO vhost/template/binding change; the charmhelpers
LP-1952414 trail log shows 2827 identical renders since 2026-06-30 -- the
internal-plane SSL vhost NEVER existed. Chain: dashboard charm (no os-*-network
options, public-only extra-binding) renders SSL vhosts only for 10.12.4.157/.8.182
while charm-rendered haproxy targets its 443 backend at the cluster-binding address
10.12.12.112:433; apache answers that socket as PLAINTEXT main-server (proven:
plain HTTP 200 on :433) and haproxy's L4-only check stays green -- TLS clients die
in handshake (curl 000) on every dashboard VIP. Deploy gate phase-03 3.3 never
caught it because its https CHECK FAILS OPEN (empty curl capture -> "OK: csrftoken
not Secure" printed on transport failure).

Logged, not actioned (candidates for post-window work): (1) DOCFIX: phase-03 3.3
check fail-closed rewrite; (2) DOCFIX: ops-update-procedure G3 probe corrected to
the D-044 http-leg + login (https-VIP assumption wrong; also fold the create-backup
and format=line findings from earlier in the window); (3) upstream bug to OpenStack
Charmers (haproxy TLS backend on a vhost-less address; L4 check masks it) --
evidence package: this entry + unit forensics; (4) long-term fix ruling: rebind
(BUNDLEFIX-class) vs Roosevelt edge-TLS root-fix (D-044 trajectory).

VAULT BUNDLE REVERT (BUNDLEFIX-010, coordinated with the main stream / Claude
chat research): bundle.yaml vault channel 1.16/stable -> 1.8/stable + comment
replaced with the D-068-amendment rationale. docs/design-decisions.md gains
## D-068 -- AMENDMENT (2026-07-05) (operator text verbatim): 1.16 is an
INCOMPATIBLE different charm for this reactive cloud (certs V0->V1 vs 18 V0
requirers, Raft-only storage vs MySQL backend, no upgrade path, Ceph breakage
reports, BUSL licensing review); deployable bundles MUST pin 1.8/stable; "get off
EOL 1.8" stays OPEN with 1.16 ruled out; the D-002-amendment item-3 supersession
note is withdrawn (vault row = 1.8/stable). Evidence:
docs/D-068-vault-1.8-vs-1.16-analysis.md (committed by operator). Gates: bundle
invariants PASS, channel assert PASS (29 pins), repo-lint 0 fail.
REVERT: git checkout HEAD~ -- bundle.yaml docs/design-decisions.md

Next-free after this push (per scan, keep this line unwrapped): D-072, DOCFIX-088, BUNDLEFIX-011.

### 2026-07-05 (addendum 16, jumphost stream) -- ops-update-20260705 WINDOW CLOSED (DOCFIX-088)

Identifier consumed: DOCFIX-088 (runbook as-executed corrections). D-071 AMENDMENT
recorded (no new number).

WINDOW RESULT: COMPLETE, all gates green. Controller 3.6.24 -> 3.6.25 (single,
non-HA; blind window ~1 min); all 91 model agents + controller machine at 3.6.25;
17 apps refreshed in-channel (G0 keystone 817; G1 placement 154, n-c-c 823,
neutron-api 710, n-a-p-ovn 215, glance 681, gss 152, o-d-retrofit 232, cinder 820,
cinder-ceph 568, barbican 265, barbican-vault 99; G2 octavia 542; G3 dashboards
750/122/168; G4 nova-compute 894). Every refresh individually settle-gated;
cloud-assert PASS at every group boundary and at close; guests byte-identical
pre/post (6). Post BOM asbuilt/20260705-194617 committed; appendix-B B.1 fully
re-baselined from it (also absorbing stale pre-window rows -- hacluster 131->166,
ceph-mon 268->491, mysql-innodb 159->164, router rows -- the post-rebuild
re-baseline had been missed; that gap is exactly what the 1.3 divergence check
caught). BOM diff vs pre: 17 expected revision pairs + ONE explained metadata
delta (cinder rev 820 adds an optional `secrets-storage` endpoint; unbound ->
inherits default space metal-admin in the export; no relation, no live change;
if ever related, bind deliberately).

EXCLUSIONS HELD: vault untouched (1.8/stable rev 372; its in-channel 372->714
offer logged only -- D-068/BUNDLEFIX-010 posture). magnum untouched (name-mismatch
gate + s390x-only channel state at pre-flight; NOTE the Charmhub target CHANGED
mid-window to a name-consistent magnum-96 -- republish in flight upstream;
re-evaluate at the NEXT window's pre-flight, do not chase mid-window).

DOCFIX-088 (runbook ops-update-procedure, as-executed fold-back): STATUS set
as-executed; [REVALIDATE] markers cleared; 1.2 version sweep corrected to
`juju-status` (units do NOT carry `agent-status.version` on juju 3.6); 2.1
REWRITTEN -- `juju create-backup` EXISTS on 3.6 and is now the STANDARD gated
pre-upgrade step (902MB/~35s measured; admin/controller namespace note); 3
corrected (no separate upgrade-model for the controller model; explicit
--agent-version); G3 probe corrected to the D-044 http leg + canonical override
path + browser login, with the VIP-TLS since-deploy defect cross-referenced
(addendum 15 RCA); 0b gains the mid-window-target-drift expectation row.
REVERT: git checkout HEAD~ -- runbooks/ops-update-procedure.md

D-071 AMENDMENT (2026-07-05): risk-section premise corrected (backup exists;
four compensating controls); controller-restore rehearsal added to Roosevelt
open questions; policy still PROPOSED.
REVERT: git checkout HEAD~ -- docs/design-decisions.md

STILL QUEUED POST-WINDOW (logged, not actioned): upstream bug to OpenStack
Charmers (dashboard haproxy TLS backend / L4 mask, addendum 15 evidence);
DOCFIX candidate phase-03 3.3 fail-open https check; operator ruling on the
dashboard-TLS long-term fix (rebind vs Roosevelt edge-TLS); magnum channel
re-check next window; Vault off-EOL-1.8 path (D-068 amendment, 1.16 ruled out).

Next-free after this push (per scan, keep this line unwrapped): D-072, DOCFIX-089, BUNDLEFIX-011.

### 2026-07-05 (addendum 17, jumphost stream) -- follow-on refreshes: magnum 96 + vault 714; fleet fully current

No identifiers consumed. Operator-ruled follow-on to the closed window, after
download-and-diff analysis of both targets (the method that RCA'd the dashboards).

ANALYSIS FIRST: the Charmhub churn is a publisher-side MASS REBUILD campaign
(new charmcraft, py3 wheels, noble series, per-arch batches) -- explains the
partial-arch channel heads, the vault 46M->12M size shrink (wheel format), and
the transient magnum metadata corruption our name-gate caught. magnum 70->96:
noble series added, ZERO functional code change. vault 372->714: reactive
charm CONFIRMED (tls-certificates V0 provider, mysql-shared storage --
explicitly NOT the 1.16 family; D-068 amendment fence intact),
reactive/vault_handlers.py BYTE-IDENTICAL, snap channel default unchanged +
live-pinned 1.8/stable so no payload change; base 22.04/amd64 matches.

EXECUTION: magnum refreshed 70->96 (settle clean; conductor OK; clusters
unchanged; A8 graft SURVIVED -- D-037 intact; cloud-assert PASS). vault
refreshed 372->714 as its own gated step with unseal keys on hand and per-poll
seal detection: NEVER SEALED (service not restarted, as the diff predicted);
daemon-confirmed Initialized/unsealed at 714; final cloud-assert PASS.
RESIDUAL can-upgrade-to: NONE -- fleet fully current. Post BOM
asbuilt/20260705-201118 committed; appendix-B B.1 re-baselined from it.
NOTE: the vault refresh is an operator-ruled exception to ops-update-procedure
Section 0 (vault excluded from ROUTINE windows); the exclusion stands for
routine windows -- this entry is the template for gated vault-refresh
exceptions (keys on hand + seal-detect + solo + A1 gates).
REVERT: juju refresh magnum --revision 70 ; juju refresh vault --revision 372
(then juju refresh <app> --channel <pinned> to resume tracking); docs:
git checkout HEAD~ -- runbooks/appendix-B-asbuilt-version-lock.md

Next-free after this push (per scan, keep this line unwrapped): D-072, DOCFIX-089, BUNDLEFIX-011.

### 2026-07-05 (addendum 18, jumphost stream) -- D-072 dashboard TLS repair executed + DOCFIX-089 + BUNDLEFIX-011

Identifiers consumed: D-072, DOCFIX-089, BUNDLEFIX-011.

D-072 (ADOPTED, as-executed): operator ruled in-place fix for the since-deploy
dashboard VIP TLS defect (addendum 15 RCA). Mechanism: cluster binding moved to the
default-binding space -- live `juju bind openstack-dashboard cluster=metal-admin`
(haproxy backends re-rendered to 10.12.8.182 :70/:433) + durable bundle.yaml
bindings edit (BUNDLEFIX-011, with in-bundle rationale comment; never
provider-public). VERIFIED: https 200 on 10.12.4.58 + 10.12.8.58 for the FIRST
TIME on this build, vault-CA-verified with csrftoken cookie present; http leg
unchanged; D-044 override intact; cloud-assert PASS; post-fix BOM
asbuilt/20260705-202637 committed (bindings delta = the D-072 change, expected).
REVERT: juju bind openstack-dashboard cluster=metal-internal ; git checkout HEAD~
-- bundle.yaml docs/design-decisions.md

DOCFIX-089: phase-03 3.3 https/cookie check rewritten FAIL-CLOSED (transport, HTTP
status, cookie presence, cookie attribute gate separately; no WARN degrade). The
old form printed OK on a dead endpoint -- the exact mechanism that let the D-072
defect pass the deploy gate. NEW appendix-A entry "Horizon VIP https handshake
death" (symptom -> cause -> fix -> diagnostic ladder).
REVERT: git checkout HEAD~ -- runbooks/phase-03-core-verify.md runbooks/appendix-A-troubleshooting.md

NEW docs/upstream-bug-draft-dashboard-tls.md -- Launchpad-ready draft (sanitized)
for charm-openstack-dashboard: multi-space deployments render the https backend on
the cluster-binding address which never gets an SSL vhost; L4 check masks it.
OPERATOR ACTION: submit + record the LP number back into D-072.
REVERT: git rm docs/upstream-bug-draft-dashboard-tls.md

Driver note (process): the rebind driver false-HALTed on an over-narrow haproxy
readback grep (juju-ssh quoting); the render had in fact succeeded -- verified
manually before proceeding. Gate philosophy held: it stopped on missing evidence.

### 2026-07-05 (addendum 19, main-chat stream) -- ledger delimited-section fences + fence-validator + D-068 appendix pointer (DOCFIX-090)

Reconciliation after the jumphost ops-update window closed. Four coordinated doc/tooling changes:
- scripts/ledger-scan.sh: added a `--fences` MODE that validates session-ledger delimited-section
  fence integrity (matched/ordered/non-overlapping/uniquely-named SECTION/END pairs; unknown-name
  and missing-OWNER are WARN). DEFAULT output is byte-for-byte unchanged (separate mode). Harness
  tests/ledger-scan/run-tests.sh extended (35 checks, all green); wired into the gauntlet.
- docs/session-ledger.md: applied the delimited-section convention -- machine-derived / main-chat /
  jumphost (pre-existing) / shared fences. Additive only: original body byte-identical; validated
  by `ledger-scan.sh --fences` (4 sections, 0 errors). Added a shared-section "Cross-session pins"
  block (D-071 sweep outcome; pre-DC-DC controller HA/backup planning).
- runbooks/appendix-B: vault row note now points to docs/D-068-vault-1.8-vs-1.16-analysis.md
  (1.8/stable is a deliberate compatibility pin, not currency; 1.16 ruled out; EOL is an open D-068 item).
- D-071 NOT ratified here (operator authority); its status stays PROPOSED, recorded in the pins.
REVERT: git checkout HEAD~ -- scripts/ledger-scan.sh tests/ledger-scan/run-tests.sh docs/session-ledger.md runbooks/appendix-B-asbuilt-version-lock.md

Next-free after this push (per scan, keep this line unwrapped): D-073, DOCFIX-091, BUNDLEFIX-012.

### 2026-07-06 (addendum 20, jumphost stream) -- operator ruling sweep: D-050 closed, D-073 adopted, SEC re-triggers, D-072 bug withdrawn

Operator issued a batch of rulings in the 2026-07-06 session; recorded in canonical
homes, nothing applied live yet:
- docs/design-decisions.md: D-050 RESOLVED via D-051 option (b) (operator ruling; the
  policyd zip is bundle-native with fail-closed drift gate + behavioral G3 acceptance).
- docs/design-decisions.md: D-073 filed ADOPTED -- tighten identity:list_trusts to the
  modern keystone default via the D-051 zip mechanism. IMPLEMENTATION + TESTING PENDING;
  no live change in this addendum. Proposed sequencing: land before batch-3 d011-05.
- docs/design-decisions.md: D-072 amendment -- upstream dashboard-TLS bug WITHDRAWN
  (operator judges it site misconfiguration, not a charm defect); draft retained in-repo.
- docs/security-ledger.md: SEC-001 trigger re-ruled to v1 close; SEC-003 deferred
  (explicitly noted it keeps d011-06 MANUAL); SEC-004 deferral reaffirmed.
- Pinned (no action): D-071 ratification and D-068 off-EOL path choice held until
  A1/A2 (batch-3 entry) complete; magnum can-upgrade-to upstream report pinned.
REVERT: git checkout HEAD~ -- docs/design-decisions.md docs/security-ledger.md docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-091, BUNDLEFIX-012.

### 2026-07-06 (addendum 21, jumphost stream) -- DOCFIX-091: tenant onboarding contract doc (A1 rulings)

Operator commissioned (A1 rulings 1-4): NEW docs/tenant-onboarding-contract.md --
authoritative operator/tenant boundary statement: explicit role table (manager
persona, NEVER admin, exact grants per identity), operator-only surface with
rationale, tenant intake list (7 items incl. two named credential custodians),
stage-by-stage creation steps/requirements/dangers (keypair trap, app-cred/trust
block, CIDR fail-closed), onboarding confirmation set (Horizon login, tri-identity
API auth, Horizon GUI identity probe VERIFY-LIVE, canary access-proof PROPOSED as
the "SSH access" confirmation), and a hardening register: existing guards + H1-H5
proposals (tenant-assert.sh verifier, stage-3 idempotency guard, canary stage,
policy-drift periodic, no-ad-hoc rule) -- proposals LOGGED, none implemented.
Also: tenant-onboarding-v2-DRAFT.md banner gains point 4 deferring boundary/role/
intake content to the contract doc.
REVERT: git rm docs/tenant-onboarding-contract.md; git checkout HEAD~ -- runbooks/tenant-onboarding-v2-DRAFT.md docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-092, BUNDLEFIX-012.

### 2026-07-06 (addendum 22, jumphost stream) -- H1 tenant-assert.sh + clientdocs/ package (DOCFIX-092); A2 + sequencing rulings

Three operator rulings recorded: (1) client-facing docs live in TOP-LEVEL `clientdocs/`
(hard internal/deliverable separation; keeps client prose out of the docs/+runbooks/
identifier scans); (2) onboarding-package build INTERLEAVES with batch-3 (foil-tenant
onboarding doubles as the live end-to-end workflow validation); (3) A2: d011-04
disruptive half (amphora failover) PRE-APPROVED conditional on the live N+1 headroom
check passing -- one window, no second ask; the headroom HOLD remains the safety gate.

Delivered:
- NEW scripts/tenant-assert.sh (H1) -- read-only post-onboard topology verifier:
  exact triad assignment sets (IDs not names), explicit admin-absence check
  (escalation wording), keypair-trap check (owner must be -cluster), app-cred
  ownership, handover-file mode 600 (existence+mode only, content never read).
  Exit 0 PASS / 1 HOLD / 2 precondition. Mutates NOTHING.
  VERIFY-LIVE flagged in header: `keypair list --user` + `application credential
  list --user` as admin are harness-mocked; confirm on first real run.
- NEW tests/tenant-assert/run-tests.sh -- 7 cases (happy, escalation, keypair trap,
  missing user, precondition, bad handover mode, absent handover dir = skip-not-fail);
  app-cred fake emits stderr noise to exercise the stdout-only JSON capture rule.
  ALL PASS; auto-discovered by the gauntlet.
- NEW clientdocs/ (DOCFIX-092): README.md (directory rules: no internal identifiers,
  client language, packaging map), intake-form.md (custodians, quota, CIDR,
  workload profile), welcome.md (boundary + house rules), self-service-guide.md
  (triad usage rules incl. the two platform traps in client language, build order,
  when-to-call-us). All drafts/living per ruling.
REVERT: git rm -r clientdocs scripts/tenant-assert.sh tests/tenant-assert; git checkout HEAD~ -- docs/handoff-20260705-open-items.md docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-093, BUNDLEFIX-012.

### 2026-07-06 (addendum 23, jumphost stream) -- H2 stage-3 re-run guards + H3 canary stage 7 (tenant-onboard.sh)

Operator-away autonomous queue work (repo-only; operator pre-approved drafting).
- scripts/tenant-onboard.sh stage3 (H2): app-cred and keypair creates are NOT
  idempotent and their secrets are one-shot. New guards: exists remotely + local
  secret intact -> SKIP rc0 (idempotent re-run); exists remotely WITHOUT usable
  local secret -> STOP rc1 with explicit operator guidance (delete-and-recreate
  rotates a live credential / breaks node access -- an operator decision, never
  an implicit re-run side effect).
- scripts/tenant-onboard.sh stage7 NEW (H3, contract 5.4, A1.4 ruling): optional
  canary access-proof as -cluster: SG + server on <c>-net + FIP + SSH with the
  -cluster key, then ALWAYS teardown (trap EXIT), pass or fail. Explicit
  invocation only -- never part of 'all'. VERIFY-LIVE: CANARY_SSH_USER (default
  ubuntu) vs the public image's login user; CANARY_FLAVOR (default gp.mid) vs
  quota. Image resolve uses the stderr-safe capture (stderr to file, surfaced on
  parse failure) -- NOT the stage0/stage5 "-f json 2>&1" merge pattern (those two
  remain as-is pending the DOCFIX-085-class sweep, logged).
- tests/tenant-onboard/run-tests.sh: extended 2 -> 12 cases (guards: skip/stop
  branches both credentials; canary: happy, dead-ssh, teardown-always incl.
  teardown.log evidence). ALL PASS; gauntlet green.
REVERT: git checkout HEAD~ -- scripts/tenant-onboard.sh tests/tenant-onboard/run-tests.sh docs/handoff-20260705-open-items.md docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-093, BUNDLEFIX-012.

### 2026-07-06 (addendum 24, jumphost stream) -- D-073 implementation staged: list_trusts hardening in the policy zip

- policies/domain-manager-policy.yaml: identity:list_trusts at the modern keystone
  default verbatim (38 -> 39 rules); full provenance + the documented one-off
  exception to the no-new-style invariant (branch inert under old-style enforcement,
  narrows only) + tooling-compat note in the file itself.
- policies/overrides.zip rebuilt (python3 zipfile -- no zip(1) on jumphost);
  provider-bundle-check content-compare PASS; yaml/ASCII/LF validated.
- NOT done here (deferred to the gated live window): oslo.policy parse (module absent
  on jumphost) and behavioral acceptance (tenant 403 / admin ok / magnum unaffected).
- See the D-073 amendment in design-decisions for the gated live-apply block.
REVERT: git checkout HEAD~ -- policies/domain-manager-policy.yaml policies/overrides.zip docs/design-decisions.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-093, BUNDLEFIX-012.

### 2026-07-06 (addendum 25, jumphost stream) -- DOCFIX-093 batch-3 window do-doc; OpenBao research draft; H5 embed; stderr-sweep inventory

- NEW runbooks/d011-batch3-window-DRAFT.md (DOCFIX-093): one logged window covering
  D-073 gated live apply (+behavioral verify +rollback), foil onboarding stages 0-4
  as the live workflow validation (tenant-assert incl. its 2 VERIFY-LIVE calls, H2
  re-run SKIP check, H3 canary, Horizon manual checks), d011-04 with the A2
  conditional pre-approval spelled out, d011-05 with VR_FOIL_APPCRED, close-out +
  abort discipline. Check ids verified against validate.sh (--checks, --list).
- NEW docs/D-068-openbao-assessment-DRAFT.md: web-research input for the B2
  discussion (research only, nothing ruled). Headlines: no OpenBao charm exists
  anywhere; OpenBao REMOVED the MySQL backend; no V1-certificates plan for legacy
  charms (V1 investment goes to Sunbeam); reactive 1.8/stable still rebuilt
  (2026-07-03); 1.8.8 CVE posture "manageable today, unauditable tomorrow";
  research recommendation = deadline-bound risk acceptance (2027-04 proposed) +
  snap-channel-headroom measurement + Sunbeam parity watch. UNVERIFIED items
  flagged inline.
- tenant-onboarding-v2-DRAFT.md banner point 5: H5 standing no-ad-hoc rule embedded.
- LOGGED (DOCFIX candidate, fixes NOT actioned -- handoff immediate item, sweep done):
  stderr-merged structured captures ("-f json ... 2>&1" into a parser), full
  inventory: scripts/cloud-assert.sh:120,125,130 (capture-then-parse; evaluate
  whether parse guards make these fail-closed A7-class); scripts/tenant-onboard.sh:46,178
  (image resolvers; stage7 already uses the stderr-safe form); scripts/tenant-acceptance.sh:43;
  scripts/tenant-offboard.sh:73; runbooks/tenant-onboarding-v2-DRAFT.md:123,130,359
  (paste blocks). Fix pattern: lib-validate.sh vr_json / stderr-to-file.
REVERT: git rm runbooks/d011-batch3-window-DRAFT.md docs/D-068-openbao-assessment-DRAFT.md; git checkout HEAD~ -- runbooks/tenant-onboarding-v2-DRAFT.md docs/handoff-20260705-open-items.md docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-094, BUNDLEFIX-012.

### 2026-07-06 (addendum 26, jumphost stream) -- DOCFIX-094 stderr-separation fixes; vault snap-track measurements; juju auth expiry

Operator-away batch, part 2.
- **DOCFIX-094 (fixes the addendum-25 inventory):** structured captures now take
  stdout ONLY, stderr to a tempfile surfaced on failure.
  - scripts/cloud-assert.sh A5/A6 (3 sites): the A5 compute-service variant was
    genuinely FAIL-OPEN (jq ... 2>/dev/null || true -> any stderr noise or API
    error silently printed "all enabled compute services up"). Now fail-closed
    with the stderr head as evidence. Harness +2 cases (T11 noise->PASS proves
    separation; T12 erroring service-list->FAIL proves the fail-open is closed);
    12/12 green.
  - scripts/tenant-onboard.sh stage0 + stage5 image resolvers (stage7 pattern).
  - scripts/tenant-offboard.sh trust capture: parse failure now PRECOND exit 20
    ("sweep would be blind") instead of silently emptying the trust sweep;
    stderr NOTE line added. Harness 7/7 green (no fixture change needed).
  - runbooks/tenant-onboarding-v2-DRAFT.md: 3 paste blocks drop the 2>&1 merge
    (stderr goes to the operator terminal, parser gets clean stdout).
  - NOT fixed: scripts/tenant-acceptance.sh:43 -- no harness exists yet (script
    backlog item 8, main-chat); fix rides that harness's delivery. LOGGED.
- **Read-only live measurements (before auth expiry; evidence in
  ~/openstack-baseline/ vault-config-20260706.yaml + vault-snap-tracks-20260706.txt):**
  reactive vault charm HAS `channel` config (1.8/stable; description warns a
  change SEALS all units -- planned-window event under our manual-unseal norm);
  Canonical vault snap tracks measured on the live unit: 1.1-1.12, 1.15-1.20,
  2.0 -- **NO 1.13 or 1.14 track**; max MPL-licensed payload reachable in-charm
  is **1.12.11**, not 1.14.x. Listener TLS options verified present and empty:
  ssl-ca/ssl-cert/ssl-chain/ssl-key (the D-068 item 2 mechanism, names now
  measured not assumed). docs/D-068-openbao-assessment-DRAFT.md updated with
  these measurements (follow-up (a) is now MEASURED).
- **BLOCKER (operator action):** the juju controller macaroon EXPIRED mid-session;
  every juju command now prompts for the jumphost user's controller password
  (interactive). Operator: run `juju login -u jessea123` at next presence.
  Blocked until then: keystone policy.json [PENDING-LIVE-READ] trio read;
  barbican host_href config look. Both queued.
REVERT: git checkout HEAD~ -- scripts/cloud-assert.sh scripts/tenant-onboard.sh scripts/tenant-offboard.sh tests/cloud-assert/run-tests.sh runbooks/tenant-onboarding-v2-DRAFT.md docs/D-068-openbao-assessment-DRAFT.md docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-095, BUNDLEFIX-012.

### 2026-07-06 (addendum 27, jumphost stream) -- d011 batch-3 window EXECUTED: D-073 live, foil1 onboarded, d011-04/05 PASS; DOCFIX-095/096/097

The d011-batch3 window (runbooks/d011-batch3-window-DRAFT.md, DOCFIX-093) ran
end-to-end this session, logged per-command to the as-executed log (label
d011-batch3; see logs/as-executed-index.md). All gates green; two checker
defects found live and fixed in-window (each harness-proven before re-run).

- **D-073 LIVE APPLY (gated mutation, operator-present):** policies/overrides.zip
  (39 rules) attached to keystone via policyd-override resource; settled
  active/idle "PO: Unit is ready". Behavioral verify: (a) beta -svc app-cred
  context DENIED trust enumeration (403 identity:list_trusts, verbatim); (b)
  admin trust list OK (1 trust); (c) magnum trust machinery re-proven by
  d011-05 PASS below. Bonus proof: first live run of the (worktree-staged) H4
  drift detector reported PASS -- live overlay parsed IDENTICAL to repo (39
  rules), base trio + trust family at recorded values. D-073 status:
  ADOPTED -> APPLIED. REVERT (cloud): git show <pre-D073>:policies/overrides.zip
  > ~/overrides-prev.zip; juju attach-resource -m openstack keystone
  policyd-override=~/overrides-prev.zip.
- **foil1 ONBOARDED (stages 0-4 + H2 + H3, live workflow validation):**
  10.20.29.0/24; stage0 preflight PASS; stages 1-4 clean first-try (domain,
  triad, anti-escalation OK, app cred secret len 86, keypair, full L3).
  H2 guard live: stage3 re-run = double SKIP rc 0. H3 canary live: CANARY PASS
  (booted foil1-net, SSH OK via FIP 10.12.7.177 as 'ubuntu' with foil1-key,
  teardown verified server/FIP/SG removed). CANARY_SSH_USER verify-live item
  CLEARED. Horizon manual checks (contract 5.1/5.3) still pending operator --
  logged as open, non-blocking per the do-doc.
- **DOCFIX-095 -- tenant-assert.sh app-cred check reworked to owner-context
  probe.** First live run HELD on the queued VERIFY-LIVE item: admin-side
  `application credential list --user` is ALWAYS 403 on this cloud -- the
  legacy base-policy `admin_or_owner` rule keys on target.token attrs the
  app-cred API never populates (live policy.json line 7, measured), so it
  degrades to owner-only cross-domain. NOT a D-073 regression (overlay has no
  application_credential rules -- verified both sides). Fix (operator-ruled):
  the check now authenticates AS the handover app cred (id/secret exported
  from the 0600 file, never printed; identity public endpoint discovered from
  the catalog) and lists own creds -- stronger (proves the cred works). Harness
  7->11 checks (owner-context PASS/auth-fail/wrong-name/double-skip; truthful
  mock answers ONLY under owner context). Re-run: TENANT-ASSERT foil1 PASS
  15/15. Also clears the keypair-list VERIFY-LIVE item (confirmed under admin).
  REVERT: git checkout HEAD~ -- scripts/tenant-assert.sh tests/tenant-assert/run-tests.sh
- **DOCFIX-096 -- d011-04 amphora_headroom() live-shape fixes.** Disruptive
  half correctly self-HELD (headroom UNKNOWN, fail-closed as designed) on the
  queued VERIFY-LIVE item. Two live-vs-assumed CLI shape misses, measured then
  fixed: (1) `loadbalancer amphora list` omits compute_id WITHOUT --long
  (default columns id/lb/status/role/ips only) -- --long added; (2)
  `hypervisor list --long -f json` emits DISPLAY-NAME keys ("vCPUs",
  "vCPUs Used", "Memory MB", "Memory MB Used"), not snake_case -- jq now
  accepts both shapes. Mocks made truthful to both measurements (compute_id
  only with --long; display-name keys; live flavor-string shape
  "charm-octavia (uuid)"). Harness: checks suite ALL PASS (62).
  REVERT: git checkout HEAD~ -- scripts/checks/d011-04-octavia-lb.sh tests/checks/run-tests.sh
- **d011-04 PASS (both halves).** Non-disruptive: RR 2 distinct backends;
  member-loss continuity 6/6 200s; recovery to 2 backends. Clears the OCCM
  LB-name and agnhost round-robin verify-live items. Disruptive (A2
  pre-approval, conditional gate honored): headroom OK -- evidence: flavor
  charm-octavia 1 vcpu/1024MB; hypervisors (vCPUs used/total | free MB):
  4/8|11610, 4/8|17754, 4/8|7514 -- all three N+1 capable; amphora failover
  triggered, recovered ACTIVE + 6/6 200s. RESULT PASS 255s.
- **d011-05 PASS.** Against beta with foil1 as isolation foil
  (VR_FOIL_APPCRED): P1 kube >=2 Ready nodes, kubeconfig fetched by TENANT;
  P2 OCCM->Octavia->FIP 10.12.6.97->pod; P3 foil sees NOTHING of beta
  (cluster list empty, cluster show 404, beta-net not found, server/LB lists
  empty). RESULT PASS 33s. Full-d011 batch-3 set now stands live.
- **DOCFIX-097 -- as-executed corrections to the do-doc + logging convention.**
  (1) run-logged.sh's interactive script(1) subshell does not map to the
  Claude Code one-shot-shell lane; operator-ruled adaptation: per-command wrap
  `script -aqe ~/as-executed/<date>-<label>.log -c '<cmd>'` (same location,
  0600/0700, verbatim capture). Recorded in
  docs/as-executed-log-convention.md (Claude Code lane paragraph) and the
  do-doc. (2) do-doc corrected: tenant-assert HOLD/fix loop and d011-04
  headroom HOLD/fix loop appended as executed; headroom evidence + canary FIP
  recorded; DRAFT banner updated to as-executed-once (2026-07-06).
  REVERT: git checkout HEAD~ -- docs/as-executed-log-convention.md runbooks/d011-batch3-window-DRAFT.md
- **Queued-read closures (both agents' evidence in-session):** (1) keystone
  policy.json PENDING-LIVE-READ trio CONFIRMED explicit in live base policy as
  "rule:cloud_admin or rule:admin_and_matching_domain_id" (D-064 correct; the
  2026-06-24 oslopolicy dump was what lacked them). The [PENDING-LIVE-READ]
  comment tags in policies/domain-manager-policy.yaml are now dischargeable --
  DOCFIX candidate, not actioned. (2) barbican host_href observation CLOSED
  with corrected mechanism: charm template renders literal "https://None:9312"
  from unset os-admin-hostname under use-internal-endpoints=true, which
  SUPPRESSES barbican 18.0.0's request-derived href fallback (utils.py
  measured on-unit); functionally benign for all catalog-dialing consumers
  (stage-6 proven), only literal href dereference breaks; os-*-hostname
  "fix" rejected (re-registers keystone endpoints; wrong plane for tenants);
  charm-level fix = future BUNDLEFIX/upstream item needing a D-NNN. D-067
  entry phrasing ("host_href unset/None") correction = DOCFIX candidate.
- **Findings LOGGED, not actioned:** capi-test-1 cluster CREATE_FAILED
  (capi project) -- candidate first target for the orphan-trustee sweep audit;
  leftover lbtest deployment/service in beta-cluster (AlreadyExists reused by
  acceptance P2) -- cleanup candidate; repo-lint L1 walks .claude/worktrees/
  (5 spurious FAILs while session worktrees exist; exclusion = DOCFIX
  candidate); tenant-acceptance foil_env hardcoded OS_AUTH_URL (house rule 3)
  -- already logged by the harness draft, rides its integration.
- **Close-out gates:** baseline cloud-assert PASS (pre-mutation) and close-out
  cloud-assert PASS; tenant landscape: beta healthy (cluster UPDATE_COMPLETE,
  node_count=2 deliberate), foil1 complete/verified, acme absent (offboarded),
  devteam PINNED next (onboarding HELD on operator snapshot ruling).

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-098, BUNDLEFIX-012.

### 2026-07-06 (addendum 28, jumphost stream) -- operator-away batch INTEGRATED: DOCFIX-098..104 (six worktree drafts -> main)

Six subagent-drafted deliverables, each harness-green in an isolated worktree,
integrated after the addendum-27 window close. Operator rulings applied at
integration: orphan-sweep audit rc 0 report-only (as drafted); clientdocs L7
receipt keeps all 14 pinned files incl. policies/ + appendices (as drafted);
H4 design defaults accepted (staged-D-073 = PASS-with-distinct-text). Full
gauntlet on the integrated tree: ALL GREEN (35 harnesses, 3 new).

- **DOCFIX-098 -- scripts/keystone-policy-drift.sh + keystone_policy_compare.py
  + tests/keystone-policy-drift/ (H4).** Read-only drift detector: repo policy
  artifacts vs live keystone (base D-064 trio + helpers verbatim; cloud_admin
  structural with IDs wildcarded; D-065/D-073 trust-family premise; overlay by
  PARSED rule sets; PO: workload marker; local zip==yaml). Leader discovered
  dynamically; juju-ssh stdout pipes only; no OS_* scope needed. Exit 0 PASS /
  1 DRIFT / 2 HELD. Harness 10/10 (mock leader deliberately keystone/1; ssh
  stderr noise). Already live-proven: first real run in the addendum-27 window
  reported PASS (39 rules aligned). Closes main-chat backlog item 6 (operator
  assigned it to the jumphost stream). REVERT: delete the three paths.
- **DOCFIX-099 -- tenant-offboard.sh --sweep-magnum-orphans [--audit|--apply].**
  Classifies magnum-domain trustees ORPHAN / PROTECTED / UNPROVEN, fail-closed
  (non-convention names, ambiguous lookups, and admin-scope cluster-show
  failures are UNPROVEN, never touched; magnum_domain_admin always PROTECTED;
  exact keystone not-found message = orphan proof). --apply deletes proven
  orphans only (trusts first, post-delete verified) behind a typed
  confirmation; audit always rc 0 (operator-ruled). Verify-live before first
  --apply: exact not-found message shape; admin-scope coe cluster show
  behavior on alive tenant clusters; app-cred list --user admin behavior.
  First audit target logged: capi-test-1 CREATE_FAILED leftover.
- **DOCFIX-100 -- tenant-offboard.sh Phase E0 app-cred inventory + idempotency
  guard (H2 counterpart).** Explicit per-user app-cred inventory before
  identity teardown; H2-style SKIP rc 0 on re-runs against already-cleaned
  tenants. Ledger said "stage-3" but offboard has phases (onboard vocabulary
  drift) -- implemented as Phase E0; removal mechanism remains the user-delete
  cascade (osc has no admin-side app-cred delete). Harness (099+100):
  tenant-offboard 7->14 cases. FOUND-NOT-FIXED (DOCFIX candidate): Phase B
  residual-trust sweep increments SWEEP_FAIL inside a pipeline subshell --
  failed trust deletes cannot raise exit 22. REVERT: fenced blocks in
  tenant-offboard.sh + drop the 7 new harness cases/mock arms.
- **DOCFIX-101 -- clientdocs/ handover pack (3 reusable templates + README
  packaging rules).** handover-pack.md (identifiers/roles/endpoints/credential
  rules/boundary acknowledgements from the onboarding contract),
  ci-integration-guide.md (app-cred-only CI auth, clouds.yaml placeholders,
  least-privilege per-pipeline creds, Magnum/CI boundary, quota pacing, worked
  API sequence), acceptance-checklist.md (matrix derived from bundle-proven
  services; expected-refusal boundary rows; object storage confirm-first --
  radosgw tenant enablement UNVERIFIED). {{PLACEHOLDER}}-driven; first
  instantiation = devteam tenant (HELD on snapshot). Intake questions for the
  dev team logged in the ledger. REVERT: git rm the 3 files + checkout README.
- **DOCFIX-102 -- scripts/cloud-snapshot.sh + snapshot_scrub.py +
  tests/cloud-snapshot/ (backlog item 7, D-070 baseline).** One read-only
  command -> asbuilt/<UTC>/: bundle-exported.yaml, juju-status.json,
  model-config.yaml, checksummed manifest; credential-shaped values scrubbed
  (WARN rc 2 + count; benign-placeholder whitelist; scrub re-verified by
  --check). Flag-gated --with-controller-backup -> juju create-backup -m
  admin/controller to ~/openstack-baseline/ (jumphost-local, NEVER committed;
  D-071 amendment). juju model-config --format=yaml confirmed live this
  session. Harness 20/20. Open design questions logged (controllers.json w/
  CA cert; fingerprintless redaction; line-oriented scrub; shared asbuilt/;
  exit-2 contract reconciliation) -- defaults stand until ruled. REVERT:
  delete the three paths.
- **DOCFIX-103 -- tests/tenant-acceptance/ harness (backlog item 8) +
  acceptance:43 stderr fix (completes the DOCFIX-094 10-site inventory).**
  16 checks / 11 cases (happy, health-lag WARN, preconditions, kube/LB fails,
  isolation violations; truthful mock emits trust-list stderr noise). The one
  deferred stderr-merge site fixed in the DOCFIX-094 idiom (stdout-only JSON
  capture, stderr tempfile surfaced as NOTE); regression-proven by reverting
  (exactly the 4 guard checks fail pre-fix -- the hazard silently downgraded
  trustee checks to WARN). FOUND-NOT-FIXED (DOCFIX candidates): foil_env
  hardcoded OS_AUTH_URL (house rule 3); two single-column 2>&1 captures
  (lines ~38/52); P3 C5 foil-LB grep false-VIOLATION if the foil runs its own
  cluster. REVERT: git checkout HEAD~ -- scripts/tenant-acceptance.sh; git rm
  -r tests/tenant-acceptance.
- **DOCFIX-104 -- repo-lint rule L7: clientdocs sweep-on-change ENFORCED.**
  clientdocs/sweep-receipt.txt hash-pins every tenant-facing file (manifest =
  the path list; scripts/tenant-*.sh, clientdocs/*.md, onboarding contract
  auto-covered; tenant runbooks, appendices C/D, policies overlay+zip manual
  lines -- operator kept all). Any drift FAILs lint with the one-line remedy
  until reviewed + re-recorded via `bash scripts/repo-lint.sh
  --record-clientdocs-sweep` (the flag's ONLY mutation). Skips when
  clientdocs/ absent (fixture repos unaffected); content-hash based (immune
  to the L5 prose-token weakness). Receipt recorded this session (17 files --
  14 + the 3 new templates) after a genuine full-directory review. Namespace
  note: lint L7 is distinct from appendix-A lesson L7. tests/repo-lint 9->17
  cases. Horizon-facing behavior has no file surface -- stays conventional.
  REVERT: checkout repo_lint.py, repo-lint.sh, tests/repo-lint/run-tests.sh,
  clientdocs/README.md, script-authoring ref; git rm clientdocs/sweep-receipt.txt.

Integration notes: clientdocs/README.md hand-merged (DOCFIX-101 + DOCFIX-104
both touch it); receipt recorded AFTER all tenant-surface integrations so the
hashes pin final states. Session worktrees pruned post-commit (their L1 lint
FAILs disappear with them; standing exclusion = DOCFIX candidate). Main-chat
backlog items 6/7/8 delivered by the jumphost stream -- main chat to reconcile
its section against this addendum.

- **DOCFIX-105 (appended in-batch) -- ledger-scan.sh numbering blind past 099.**
  Assigning DOCFIX-100..104 above tripped a latent scanner defect: the
  next-free patterns were `DOCFIX-0[0-9]{2}` / `BUNDLEFIX-0[0-9]{2}` -- blind
  to 100+, so next-free would have frozen at 100 forever. Patterns widened to
  `[0-9]{3}`; harness gains 2 boundary cases (35->37 checks, ALL PASS). The
  numbering authority is correct again (verified: highest=104 -> next-free=105
  before this entry consumed it). REVERT: git checkout HEAD~ --
  scripts/ledger-scan.sh tests/ledger-scan/run-tests.sh.

Next-free after this push (per scan, keep this line unwrapped): D-074, DOCFIX-106, BUNDLEFIX-012.

### 2026-07-06 (addendum 29, jumphost stream) -- D-074 ADOPTED+EXECUTED: overlap-allowed tenant CIDRs; DOCFIX-106/107

Work order authored by the main-chat stream (docs/tenant-cidr-overlap-
correction-PLAN.md + docs/repo-lint-nextfree-bug-FINDING.md, pushed this
session); operator rulings same-day: D-074 ADOPT; client-choice CIDR with
10.100.0.0/24 fallback; capi-mgmt 10.20.0.0/24 stays reserved; L5 next-free
print REMOVED. Phase 0 gates honored before any implementation shipped.

- **Phase 0 (read-only gates, logged d074-phase0):** 0.1 allow_overlapping_ips
  = True read from the neutron-api leader's rendered neutron.conf. 0.2
  overlap+Magnum proof via foil1 (operator-ruled vehicle; the PLAN's
  onboard-a-foil step was stale -- foil1 existed and the old guard blocks
  exact-overlap onboarding): EXACT-overlap subnet (10.20.28.0/24 ==
  beta-subnet) + cluster CREATE_COMPLETE ~14.5 min + 2 Ready nodes via the
  API-LB while beta served on the same CIDR. Full detail + evidence path in
  the D-074 Phase-0.2 amendment. Teardown clean; post cloud-assert PASS.
- **D-074 decision record + D-016 amendment** appended to design-decisions
  (rationale, rulings, Phase-0 evidence, forward-only; corrections recorded:
  the PLAN's "northwind 10.20.20.0/24" example was fictional -- live tenants
  are beta/foil1).
- **DOCFIX-106 -- stage-4 guard rewrite (scripts/tenant-onboard.sh +
  scripts/net_overlap.py).** Replaces the exact-string cloud-wide check
  (over-enforced tenant uniqueness the contract never required; MISSED
  partial overlaps) with real containment math: reject only non-RFC1918
  (explicit 10/8+172.16/12+192.168/16 -- python is_private would wrongly
  accept TEST-NET) or overlap with PLANE_CIDRS (sourced from lib-net.sh),
  the capi-mgmt net (resolved live by stable name, FAIL-CLOSED on lookup
  miss), and metadata. tenant-onboard harness 12->17 cases: tenant-vs-tenant
  exact overlap ALLOWED; partial-vs-reserved REJECTED; exact-reserved
  REJECTED; capi partial REJECTED (dynamic); non-RFC1918 REJECTED; capi
  lookup miss fails closed. Doc alignment same delivery:
  docs/tenant-onboarding-contract.md section 3 item 4 (optional,
  overlap-permitted, fallback default); runbooks/tenant-onboarding-v2-DRAFT.md
  block 4.0 (new guard invocation; NOTE the PLAN named a nonexistent
  "tenant-onboarding-runbook.md"); clientdocs/intake-form.md section 5
  (optional, overlap-safe wording); L7 receipt re-recorded after the review.
  REVERT: git checkout HEAD~ -- scripts/tenant-onboard.sh
  tests/tenant-onboard/run-tests.sh docs/tenant-onboarding-contract.md
  runbooks/tenant-onboarding-v2-DRAFT.md clientdocs/intake-form.md
  clientdocs/sweep-receipt.txt; git rm scripts/net_overlap.py; re-attach the
  old guard only with a new D-NNN (D-074 governs).
- **DOCFIX-107 -- repo_lint.py L5 numbering print REMOVED** (the unreliable
  next-free line; per the FINDING
  doc's recommended option; all three defects confirmed: band-limited regex,
  mention-counting without pointer-exclusion, tests/-fixture ingestion).
  The duplicate-heading collision guard KEPT and its own regex widened past
  the 099 boundary (same defect class as DOCFIX-105). ledger-scan is the
  sole numbering authority. repo-lint harness green (T8 collision case
  intact). REVERT: git checkout HEAD~ -- scripts/repo_lint.py.
- **FINDING-doc de-fang (rides this addendum):** the main-chat FINDING doc's
  own prose carried a BUNDLEFIX token above high-water, inflating the
  ledger-scan BUNDLEFIX counter to 052 the moment it was pulled -- de-fanged
  in place with a note (the standing identifier-pollution rule; scan verified
  restored to 012).
- **Gates:** tenant-onboard 17/17; repo-lint harness 17/17; gauntlet ALL
  GREEN (35); repo-lint 0 fail / 1 legacy warn (worktree artifacts excluded);
  baseline + post-phase-0 cloud-assert PASS. Phase 3 (devteam onboarding on
  the corrected guard, TENANT_CIDR fallback 10.100.0.0/24) stays NEXT SESSION
  per operator ruling.

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-108, BUNDLEFIX-012.

### 2026-07-06 (addendum 30, jumphost stream) -- client handover package COMPLETE: DOCFIX-108 starter kit + DOCFIX-109 tenant AI skill

Two operator-approved client-facing deliverables integrated from worktree
drafts (union-merged: both touched the L7 coverage, README, and receipt; the
skill's second delivery added three scripts overlapping the kit's -- deduped
at integration, skill routing re-pointed).

- **DOCFIX-108 -- clientdocs/scripts/ tenant starter kit.** smoke-test.sh
  (read-only health: token/catalog/limits, checklist rows A1/A4/A5, triage
  hints); acceptance-run.sh (automated self-service checklist rows,
  ci-accept-* naming, EXIT-trap reverse teardown, expected-refusal rows must
  fail-to-PASS, scoreboard); ci-cleanup-sweep.sh (leftover sweep: --prefix/
  --hours, DRY-RUN default, --apply gated); tenancy-audit.sh (read-only
  tenancy snapshot + quota envelope -- from the skill delivery, kept in the
  dedupe); clouds.yaml.template; Jenkinsfile.example (credentials()
  injection, smoke -> acceptance -> always-runs sweep). All client-voice,
  fully generic (client's own OS_* env, catalog discovery, zero internal
  values). Harness tests/clientdocs-scripts/run-tests.sh: 47 checks (incl.
  teardown-on-midway-failure trap proof, dry-run-vs-apply, prefix isolation,
  stderr-noise truthful mocks). DEDUPE: the skill delivery's
  pipeline-smoke.sh and leftover-check.sh were DROPPED as duplicates of
  smoke-test.sh / ci-cleanup-sweep.sh (dry-run); skill + guide references
  re-pointed. Known adaptations documented in-script: D2 boots a throwaway
  keypair (handover key is -cluster-owned; app-cred cannot use it -- the
  checklist text carries the same latent nuance, polish-pass candidate); C2
  uses an unrouted default range with CI_ACCEPT_CIDR override.
- **DOCFIX-109 -- clientdocs/tenant-skill/ (Claude Agent Skill
  omega-cloud-tenant) + clientdocs/ai-assistant-guide.md.** SKILL.md
  (three-account model, by-design-refusal boundary table, auth patterns,
  routing table incl. starter-kit rows, 8-point live-tenancy discipline for
  an AI assistant) + references/ (day2-operations, kubernetes,
  ci-automation, troubleshooting). Client install: copy as
  .claude/skills/omega-cloud-tenant/. Placeholder set exactly
  {{TENANT_SHORT_NAME}} {{AUTH_URL}} {{DASHBOARD_URL}} {{REGION}}
  {{ACCOUNT_CONTACT}}. Harness tests/clientdocs-skill/run-tests.sh: 8 gates
  incl. internal-term LEAKAGE scan (operator tooling, component code names,
  internal hosts/CIDRs, identifier shapes -- self-tested against planted
  leaks) and routing reachability.
- **L7 auto-coverage extended (rides 108/109):** repo_lint.py sweep_auto now
  covers clientdocs RECURSIVELY (*.md anywhere incl. tenant-skill/, *.sh,
  scripts/*); repo-lint harness 17->22 cases (subdir .md/.sh
  covered/recorded/drift). Receipt re-recorded: 29 files pinned.
- **Gates:** clientdocs-scripts 47/47; clientdocs-skill 8/8; repo-lint
  harness 22/22; gauntlet ALL GREEN (37 harnesses, 2 new); lint 0 fail /
  1 legacy warn (session worktrees pruned post-commit).
- REVERT (per item): DOCFIX-108: git rm -r clientdocs/scripts
  tests/clientdocs-scripts + drop the README kit sections + re-record
  receipt. DOCFIX-109: git rm -r clientdocs/tenant-skill
  tests/clientdocs-skill clientdocs/ai-assistant-guide.md + drop the README
  skill sections + re-record receipt. L7 recursion: git checkout HEAD~ --
  scripts/repo_lint.py tests/repo-lint/run-tests.sh.

Next up (queued): operator-requested polish pass over the FINAL clientdocs
set before handover; then devteam onboarding (next session; D-074 guard,
10.100.0.0/24 fallback) with handover-pack instantiation.

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-110, BUNDLEFIX-012.

### 2026-07-07 (addendum 31, jumphost stream) -- DOCFIX-110: clientdocs pre-handover polish + RELEASED banner ruling

- **DOCFIX-110 -- clientdocs polish sweep (operator-requested) + banner
  ruling.** Coherence pass over the complete client package (worktree
  polish/clientdocs-sweep, integrated): reading-order section in welcome;
  starter-kit scripts cross-referenced from the CI guide / checklist /
  self-service guide / handover pack; CIDR wording in self-service guide,
  CI guide, and skill references aligned with D-074 (tenant-chosen,
  overlap-allowed); acceptance-checklist probe names aligned to ci-accept-*
  and a D2 note recording the throwaway-keypair adaptation (script
  unchanged); SKILL.md read-only claim corrected per-script; account-prefix
  and audience-sentence consistency. Leakage scan across ALL clientdocs
  files: zero hits. OPERATOR RULING (2026-07-07): DRAFT banners removed
  from welcome/intake/self-service/ai-assistant-guide; the three
  instantiated templates keep (TEMPLATE), drop DRAFT; README status line
  updated to RELEASED-for-handover. Receipt re-recorded (29 files).
  Verified-correct non-changes logged by the pass: -svc-appcred.txt
  filename vs -svc-cred credential name is real (onboard stage 3);
  Jenkinsfile client-side path placeholder deliberate. Gates: clientdocs
  harnesses 8/8 + 47/47; gauntlet ALL GREEN (37); lint 0 fail / 1 legacy
  warn. NOTE: the polish agent was killed mid-run by a session limit and
  resumed; its original worktree had NOT survived (coordinator assumption
  wrong -- flagged per divergence rule) but no edits were lost (pre-crash
  it had only read); it re-based onto main and verified its reads against
  HEAD before editing. REVERT: git checkout HEAD~ -- clientdocs/ then
  bash scripts/repo-lint.sh --record-clientdocs-sweep.

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-111, BUNDLEFIX-012.

### 2026-07-07 (addendum 32, jumphost stream) -- DOCFIX-111 lint excludes .claude/; orphan-sweep first live audit

- **DOCFIX-111 -- repo_lint all_text() excludes .claude/.** Session-local
  agent worktrees are full repo copies whose docs/design-decisions.md
  mirrors trip the L1 legacy carve-out (it keys on the exact repo-relative
  path) -- every concurrent-agent session produced spurious L1 FAILs until
  its worktrees were pruned (observed repeatedly 2026-07-06/07). Lint the
  repo, not the session. Harness 22->23 (planted non-ASCII under
  .claude/worktrees/ must NOT fail lint). REVERT: git checkout HEAD~ --
  scripts/repo_lint.py tests/repo-lint/run-tests.sh.
- **Orphan-trustee sweep (DOCFIX-099) FIRST LIVE AUDIT (read-only, rc 0):**
  magnum domain resolved dynamically; 3 trustees classified: ORPHAN=0,
  PROTECTED=3 (magnum_domain_admin service identity; beta-cluster trustee
  alive; capi-test-1 trustee alive -- CREATE_FAILED clusters still EXIST as
  records, so their trustees are correctly NOT orphans until the cluster
  record is deleted). VERIFY-LIVE items cleared on the OK paths: admin-scope
  coe cluster show succeeds for alive tenant clusters (no policy-denial
  noise -> no spurious UNPROVEN); dynamic domain/user resolution correct.
  HONEST GAP: the orphan-classification path (exact keystone not-found
  proof) remains harness-proven only -- no real orphan exists to exercise
  it live; first exercised the day a cluster record is deleted out from
  under a trustee. capi-test-1 cleanup (coe cluster delete of the
  CREATE_FAILED record, capi project) = operator-gated candidate; after it,
  a re-audit would classify that trustee for reaping.

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-112, BUNDLEFIX-012.

### 2026-07-07 (addendum 33, jumphost stream) -- script-quality fix batch: DOCFIX-112..117 (sweep E/R findings, operator-ruled scope)

A read-only repo-wide quality sweep (110 scripts; bash -n and py_compile
all-clean; shellcheck/pyflakes absent on the jumphost, manual passes for
their high-value classes) surfaced 4 runtime errors + robustness items.
Operator ruled the fix scope: errors + coupled robustness now; S-class
consolidation deferred. All fixes harness-proven with discriminating
regression cases (each proven failing against the pre-fix code where a
deterministic mock can express it).

- **DOCFIX-112 -- tenant-onboard.sh stage2 anti-escalation guard fail-closed
  (SECURITY).** The `role assignment list | grep -qw admin` live pipe under
  pipefail failed OPEN both ways: SIGPIPE on the very match it exists to
  catch, or a failed list, both printed "anti-escalation OK". Now
  capture-then-test, stdout-only, STOP on unverifiable (the same form the
  file already used 30 lines up). Harness +3 stage2 cases; fail-closed case
  proven discriminating vs pre-fix. REVERT: git checkout HEAD~ --
  scripts/tenant-onboard.sh tests/tenant-onboard/run-tests.sh.
- **DOCFIX-113 -- tenant-offboard.sh hardening cluster.** (a) stdout-only
  inventory captures via new inv() helper (stderr to tempfile, surfaced as
  NOTE); (b) every Phase C/D/E row shape-validated (row_ok/is_id) before
  becoming delete argv -- a merged deprecation warning previously word-split
  into deletes (mock replay: 70 bogus delete calls pre-fix); invalid rows
  SKIP + count; (c) Phase B residual-trust loop printf|while subshell ->
  here-string, so SWEEP_FAIL survives (failed trust delete now exits 22;
  found-not-fixed since addendum 28, now CLOSED); (d) LB-gone wait counts
  stdout only and fails LOUDLY on budget exhaustion or list error (was a
  silent fall-through into FIP/server deletes); (e) discarded is_id no-op
  enforced; (f) tenant_env reads auth_url= from the cred file, hardcoded VIP
  demoted to KEYSTONE_VIP-overridable fallback. Harness 14->20, mock made
  CLI-truthful (UUID rows, stderr warnings); all six new cases fail pre-fix.
  REVERT: git checkout HEAD~ -- scripts/tenant-offboard.sh
  tests/tenant-offboard/run-tests.sh.
- **DOCFIX-114 -- validate.sh non-contract exit codes count as HOLD.** A
  check dying 127/141/139 previously yielded OVERALL PASS exit 0 (note_worst
  matched nothing; the nonstd annotation was dead code ordered after
  REPORT+=). Default arm now annotates then forces HOLD. Proven: old rc=0,
  new rc=2. REVERT: git checkout HEAD~ -- scripts/validate.sh
  tests/validate/run-tests.sh.
- **DOCFIX-115 -- juju-spaces-check.sh model gate capture-then-test.** The
  live `juju models | jq | grep -qx` pipe masked juju auth failure (and the
  SIGPIPE race) as "model not present"; auth failure now exits 1 with the
  existing macaroon hint. NEW harness tests/juju-spaces-check (5 cases;
  script previously had none). REVERT: git checkout HEAD~ --
  scripts/juju-spaces-check.sh; git rm -r tests/juju-spaces-check.
- **DOCFIX-116 -- pipefail upgrades + vault-kv-health rc-test.**
  tenant-acceptance.sh and vault-kv-health.sh set -u -> set -uo pipefail
  (pipelines audited); 7 harnesses upgraded likewise;
  vault-kv-inner-probe.sh stays set -u with the paste-block exemption
  documented in-line; vault-kv-health C2 no longer misreports a juju
  show-unit failure as "no vault_url" FAIL -- rc tested first, juju failure
  warns UNVERIFIED on the HOLD path. Harness 8->10. REVERT: git checkout
  HEAD~ -- scripts/tenant-acceptance.sh scripts/vault-kv-health.sh
  scripts/vault-kv-inner-probe.sh tests/checks/run-tests.sh
  tests/d063-apply/run-tests.sh tests/ledger-scan/run-tests.sh
  tests/lib-validate/run-tests.sh tests/validate/run-tests.sh
  tests/vault-kv-health/run-tests.sh.
- **DOCFIX-117 -- guard-destructive.py secret-read blocklist extension.**
  Reader set gains python3/openssl (quote-blind) and grep/awk/sed
  (quote-sensitive: bare-path reads block, quoted prose mentions pass);
  path set gains home-anchored tenant-*/ and .pem (repo tests/tenant-*/
  stays readable). Precision-tested; harness 13->25. REVERT: git checkout
  HEAD~ -- .claude/hooks/guard-destructive.py tests/claude-guard/run-tests.sh.
- **docs/script-quality-findings-20260707.md (rides the batch):**
  deferred-findings register -- R5/R6/R7/R9/R11 verified with locations,
  S1-S8 consolidation proposals transferred VERBATIM at integration (the
  fix agent flagged it could not see the sweep session's text and refused
  to fabricate -- correct call), fixed-this-batch markers, plus new
  found-while-fixing items (tenant-offboard Phase A coe-capture stderr
  class; benign-warning aborts on fail-closed preamble captures; Phase E0
  display pollution). Sweep clean bill recorded: DOCFIX-094 stderr class
  fully closed repo-wide, no /tmp-to-snap bugs, libs correctly set -e-free.
- **Gates:** gauntlet ALL GREEN (38 harnesses, 1 new); repo-lint 0 fail /
  1 legacy warn; L7 receipt re-recorded (tenant scripts changed; clientdocs
  reviewed -- client-visible contract unchanged by this batch).

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-118, BUNDLEFIX-012.

### 2026-07-07 (addendum 34, jumphost stream) -- devteam onboarding window EXECUTED (ops-devteam-onboard; no numbers consumed)

Operator-present gated window, logged per DOCFIX-097 lane (per-command wrap,
~/as-executed/20260707-ops-devteam-onboard.log; index entry added). First
CLIENT-FACING execution of the onboarding workflow and first live use of the
D-074 fallback CIDR path.

- **Rulings collected in-window:** TENANT_CIDR = 10.100.0.0/24 (dev team
  expressed no preference -> D-074 ruled fallback); onboarding proceeds ahead
  of intake answers (pack instantiation HELD on them); capi-test-1
  CREATE_FAILED record delete = YES at window close.
- **Executed (each stage individually gated):** open cloud-assert PASS
  (baseline asbuilt/20260706-224851 reused per ruling, not stale) ->
  tenant-onboard.sh devteam stage0..4 -> tenant-assert.sh devteam 15/15
  PASS -> stage7 canary PASS (boot + SSH via FIP 10.12.7.43 with
  devteam-key, teardown clean) -> close cloud-assert PASS. ALL first-try;
  ZERO as-executed corrections -- the runbook path is validated as written
  on a second tenant and a fresh CIDR.
- **Security notes:** stage-2 anti-escalation guard (DOCFIX-112 batch) first
  client-facing live run -- admin grant denied as designed. Handover creds
  in ~/tenant-devteam (0600; never read into context; delivery out-of-band
  to named custodians only, pending intake).
- **capi-test-1:** ruled delete NOT executed agent-side -- the permission
  classifier refuses agent deletes of resources the agent did not create
  (working as intended); exact command (uuid
  fae7ec70-712c-4e58-88dd-da0d7b8c89b2, measured in-window) handed to the
  operator. Next orphan-audit reaps its trustee after the record delete.
- **Remaining to close item 1:** dev-team intake answers -> clientdocs pack
  instantiation ({{PLACEHOLDER}} fill, TEMPLATE NOTE strip) -> delivery.
REVERT (full tenant removal if devteam is abandoned): bash scripts/tenant-offboard.sh devteam; git checkout HEAD~ -- docs/session-ledger.md logs/as-executed-index.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-118, BUNDLEFIX-012.

### 2026-07-07 (addendum 35, jumphost stream) -- devteam intake + quota + FIRST clientdocs pack instantiation (no numbers consumed)

Continuation of the ops-devteam-onboard window (same log file).

- **Intake collected (operator walkthrough, internal test):** org "Dev Team";
  placeholder custodians ACCEPTED by operator ruling ("internal test");
  requesters = custodians only; quota = runbook standard; K8s yes; generated
  key kept. Filed jumphost-local (~/devteam-intake-record.md, 0600) -- client
  answers NEVER enter the repo while it is public (SEC-004).
- **Quota:** `openstack quota set devteam-prod --instances 10 --cores 20
  --ram 51200` applied + re-read verified. Values equal the nova class
  defaults; set explicitly anyway (runbook stage-1: "documents the record").
- **Pack instantiated (FIRST instantiation of the RELEASED package):**
  17 files -> ~/devteam-package per the README packaging map (CI client +
  AI skill variant). 9 tokens filled; endpoints measured in-session
  (identity VIP + region from `endpoint list`; dashboard VIP
  https://10.12.4.58 carried from the D-072 as-executed record after the
  classifier correctly refused both a ~/vault-init-CA curl and a curl -k
  fallback). TEMPLATE NOTE banners + " (TEMPLATE)" title suffixes stripped;
  verification: zero unfilled tokens (one whitelisted -- see below), zero
  banners, zero CRLF; clientdocs/ templates untouched (L7 receipt intact,
  lint clean).
- **LOGGED-NOT-ACTIONED (DOCFIX candidate):** tenant-skill/references/
  day2-operations.md explains the double-brace notation in DELIVERED prose
  ("fields were filled at handover"), so one literal token survives every
  instantiation; reword at the next clientdocs sweep (L7 re-record).
- **Remaining (operator-manual):** copy the vault CA into the package as
  omega-cloud-ca.pem; deliver ~/devteam-package + ~/tenant-devteam creds
  in-person to the custodians. capi-test-1 delete still staged to operator
  (addendum 34).
REVERT: rm -rf ~/devteam-package ~/devteam-intake-record.md (quota set was value-identical to defaults; no cloud-side revert needed); git checkout HEAD~ -- docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-118, BUNDLEFIX-012.

### 2026-07-07 (addendum 36, jumphost stream) -- devteam package URL correction (/horizon) + dashboard root-autoindex FINDING (no numbers consumed)

Operator browser check of the delivered dashboard URL hit Apache "Index of /"
at https://10.12.4.58/ -- two outcomes:

- **As-executed correction (package fixed in place):** the client-facing
  Horizon URL requires the webroot path: https://10.12.4.58/horizon
  (webroot measured via `juju config openstack-dashboard webroot`). All 4
  delivered occurrences in ~/devteam-package corrected by sed + re-verified
  (no double-path, tokens still zero). NOTE for future instantiations: the
  D-072 as-executed record's "https 200" at the VIP root was the AUTOINDEX
  page, not Horizon -- a 200-at-root is a weaker signal than believed; the
  {{DASHBOARD_URL}} fill value is <VIP>/horizon (or webroot at the time).
  DOCFIX candidate: record the full URL-with-webroot in the instantiation
  guidance (clientdocs/README.md) at the next sweep.
- **FINDING (UNRULED, logged not actioned):** the tenant-facing VIP root
  serves an Apache directory listing. Options presented to operator:
  (a) redirect / -> /horizon; (b) disable autoindex at the root; (c) both
  (recommended). Any fix is a change to a built surface (D-072/D-044
  adjacent) -- gated on an operator ruling; PROPOSED D entry to be drafted
  from the next-free number at that time.
REVERT (package correction only): sed 's|/horizon||' on the 4 corrected lines in ~/devteam-package; git checkout HEAD~ -- docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-075, DOCFIX-118, BUNDLEFIX-012.

### 2026-07-07 (addendum 37, jumphost stream) -- D-075 ADOPTED+EXECUTED: dashboard VIP root redirect + no-autoindex; DOCFIX-118 runbook/appendix delivery

Operator confirmed /horizon works (devteam contract 5.1 reachability) and RULED
option (c) on the root-autoindex finding (addendum 36). Same gated window
(ops-devteam-onboard log).

- **Measured before mutating:** charm exposes NO config option for root
  behavior (full option list read); vhost map + docroots read on-unit;
  "Index of /" REPRODUCED from the unit's own http backend (:70).
- **D-075 EXECUTED (2 gated mutations + verify):**
  /etc/apache2/conf-available/99-omega-root.conf (charm never manages
  conf-enabled/), a2enconf, configtest, reload. FIRST ATTEMPT (RedirectMatch)
  caught by the verify step emitting Location http://10.12.4.58:70/horizon --
  mod_alias absolutizes from the backend port, which haproxy does not expose;
  REPLACED with per-dir mod_rewrite echoing the client Host header + scheme.
  Final verify: VIP http root -> 302 http://10.12.4.58/horizon -> 200 at
  /horizon/auth/login/; on-unit listing grep zero. TLS-leg browser confirm
  pending operator (http leg proven; rule is scheme-agnostic).
- **DOCFIX-118:** phase-03 step 3.3 gains the D-075 PER-REBUILD block (RUN +
  fail-closed CHECK that specifically FAILs on a backend-port-leaking
  redirect); appendix-A gains the "Index of /" symptom entry incl. the two
  traps (root-200 masquerades as healthy dashboard -- the D-072 root-200 was
  this listing; RedirectMatch port leak). D-075 entry appended to
  design-decisions (ADOPTED+EXECUTED, mechanism, revert, Roosevelt sunset
  under the D-044 edge trajectory).
- **Hygiene note:** a scratchpad dump of the full dashboard charm config
  (secret-bearing values, taken for option-list reading) was deleted
  immediately after key-name extraction; values never entered context.
- Numbers consumed: D-075, DOCFIX-118.
REVERT: on-unit `sudo a2disconf 99-omega-root && sudo systemctl reload apache2`; git checkout HEAD~ -- docs/design-decisions.md runbooks/phase-03-core-verify.md runbooks/appendix-A-troubleshooting.md docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-076, DOCFIX-119, BUNDLEFIX-012.

### 2026-07-07 (addendum 38, jumphost stream) -- Horizon three-account walkthrough (contract live evidence); DOCFIX-119/120; dashboard override first apply REVERTED

Operator-driven browser walkthrough of all three devteam accounts -- the first
live execution of the contract section-5 confirmation set. Same logged window.

- **Walkthrough verdicts (all contract-correct):** `-domain-admin`: login OK,
  Identity-only view, three popups = correct nova refusals + one cosmetic
  upstream wart; Users panel ABSENT + Projects empty = the VERIFY-LIVE weak
  point confirmed (Horizon local policy predates the manager persona).
  `-cluster`: popup-free full project view; keypair visible; tenant + shared
  resources only -- GUI isolation confirmed. `-svc`: as `-cluster` but keypair
  page empty (user-scoped asymmetry as documented) and its app cred visible.
  **Object storage: container create/delete succeeded as `-cluster` -- FIRST
  live proof of tenant radosgw; the acceptance-checklist "confirm-first"
  caveat is closed by evidence.** D-075 TLS-leg redirect also confirmed.
- **DOCFIX-119 -- clientdocs aligned to observed GUI reality** (drafted by a
  background agent in a worktree from live findings, reviewed + integrated):
  self-service-guide "What to expect on first login" + CLI user-creation
  commands; tenant-skill troubleshooting popup signatures + expected-today
  identity rewording; day2-operations CLI lead-in; acceptance-checklist A3
  popup note, section-B CLI note, section I de-hedged to expected-pass;
  acceptance-run.sh manual-row instructions aligned (no behavior change).
  Gates: clientdocs-skill 8/8, clientdocs-scripts 47/47, L7 receipt
  re-recorded after review. Plus README instantiation guidance:
  DASHBOARD_URL is the FULL URL incl. webroot (the addendum-36 lesson).
- **DOCFIX-120 -- contract section 5 live verdicts:** 5.1 VERIFIED (devteam);
  5.3 FAILS-as-anticipated, CLOSED via its own CLI-path clause (API
  capability live-proven by stage 2); item 4 canary marked IMPLEMENTED as
  stage 7 (was stale "not yet implemented").
- **Dashboard policy override -- first live apply FAILED acceptance,
  REVERTED (evidence for the PROPOSED decision draft, next-free D claimed
  at its commit):** background agent drafted a 10-rule identity override
  (worktree, harness 14/14); gated apply: attach + enable -> "PO: Unit is
  ready". Browser walk FAILED: panels rendered (panel-gate fix effective)
  but Domains/Projects/Users/Groups pages refused at view level; Roles
  panel raised two popups then FORCED SESSION LOGOUT. Reverted
  (use-policyd-override=false); baseline re-confirmed by operator. Rework
  in flight: per-view rule/target mapping, logout root cause, scope-down
  to Users+Projects, offline per-view semantic tests. Client impact: none
  (CLI identity path is the documented behavior).
- **Package:** ~/devteam-package re-instantiated from the updated templates
  (all 9 values incl. the /horizon URL; operator CA copy preserved);
  operator re-copies + delivers.
- Numbers consumed: DOCFIX-119, DOCFIX-120.
REVERT: git checkout HEAD~ -- clientdocs docs/tenant-onboarding-contract.md docs/session-ledger.md docs/v1-redeploy-changelog.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep (re-pin the receipt); cloud-side already reverted in-window.

### 2026-07-08 (addendum 39, jumphost stream) -- devteam cluster-template 403 diagnosed (Public flag, by design); DOCFIX-121/122; package re-render

Incident: devteam (and the operator, reproducing) hit Horizon "Error:
Unable to create cluster template." as `devteam-cluster`. Diagnosed
read-only; NO cloud change made or needed.

- **Root cause (measured):** magnum-api werkzeug log shows POST
  /v1/clustertemplates -> 403 x3 (2026-07-07 23:37/23:38/23:59) then 201
  (00:06) for the SAME user+project (devteam-cluster/devteam-prod);
  dashboard unit error.log carries the masked verbatim reply "Not
  authorized to set public or hidden flag for cluster template (HTTP
  403)"; /etc/magnum/policy.json: `clustertemplate:create` =
  deny_cluster_user (tenant-open) but `clustertemplate:publish` =
  admin_api, enforced additionally when public/hidden is set. Operator
  confirmed by A/B test (00:10 retest with Public checked -> 403;
  unchecked -> 201). VERDICT: anti-isolation flag correctly refused --
  policy stands, do NOT relax.
- **DOCFIX-122 -- appendix-A symptom entry** ("Horizon 'Error: Unable to
  create cluster template.' (Magnum 403)"): symptom/cause/trap (generic
  toast masks the policy message; tracks the checkbox, not time; localize
  at dashboard error.log or magnum-api, not keystone)/fix/gate.
- **DOCFIX-121 -- clientdocs template-creation guidance:** kubernetes.md
  new "Creating your own cluster template" section (Public/Hidden hard
  rule with the verbatim refusal + the masked-toast tell; known-good CLI
  starting point mirroring stage5 flags with flavors left to
  `openstack flavor list` -- the T3 leak gate correctly refused the
  internal flavor literal; no --keypair in templates, keypair rides
  cluster create) + delivery claim softened to "standard onboarding
  delivers... if missing, create your own"; self-service-guide.md
  Kubernetes section gets the same one-paragraph warning. Gates:
  clientdocs-skill 8/8, clientdocs-scripts 47/47, repo-lint 0 fail,
  L7 receipt re-recorded (29 files).
- **FINDING (logged, unruled): devteam has NO delivered template** --
  `devteam-k8s` does not exist; the onboarding window ran stages 0-4 only
  and stage5 (template, as -svc appcred) was never executed for devteam,
  while clientdocs promised "delivered at handover" (claim now softened
  by DOCFIX-121). OPTION for operator: run
  `bash scripts/tenant-onboard.sh devteam 5` (gated, tenant-scoped
  mutation) to deliver `devteam-k8s` per contract; else devteam
  self-creates per the new docs (their GUI `test1` already exists,
  tenant-owned).
- **Dashboard-override side evidence (draft-only):** operator's second
  GUI identity test create_grant 403 root-caused -- Horizon submitted a
  `load-balancer_admin` grant (keystone access log + role-ID resolution);
  domain-manager policy refused per is_domain_managed_role AS DESIGNED;
  evidence + rework TODOs folded into the D-076 draft (agent worktree,
  uncommitted). Operator-side leftovers: role-less test user in the
  devteam domain (complete member grant or delete); why the Role
  dropdown carried load-balancer_admin (answer pending).
- **Package:** ~/devteam-package re-rendered for the two changed files
  (kubernetes.md, self-service-guide.md; values carried from the
  delivered copies, zero tokens verified); operator pushes tonight.
- Numbers consumed: DOCFIX-121, DOCFIX-122.
REVERT: git checkout HEAD~ -- clientdocs runbooks/appendix-A-troubleshooting.md docs/session-ledger.md docs/v1-redeploy-changelog.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep (re-pin the receipt); re-render the two package files from the reverted templates (or restore from the operator's previous copy); no cloud-side change to revert.

### 2026-07-08 (addendum 40, jumphost stream) -- devteam cluster NoValidHost diagnosed (capacity, not quota); decommission audit + recommendations (no numbers consumed)

Read-only diagnosis + audit; the only mutation this window was the operator's
own delete of the wedged cluster. Nothing agent-executed.

- **TestCluster CREATE stall root-caused:** devteam 1+3 gp.mid cluster
  (created 00:18, CAPI driver); control plane + 1 worker ACTIVE, workers
  2-3 ERROR with nova fault "No valid host was found" (verbatim, by UUID --
  admin name-lookup is project-scoped, same trap as template show). NOT
  quota (4x gp.mid = 8 cores/32768MB vs 10/20/51200 limits; quota rejects
  at the API, never lands ERROR). Physical RAM was the limiter: free
  ~3.4/8.5/7.5 GB across the 3 hypervisors; gp.mid=8192MB, so after 2
  landed no host could fit another. Operator deleted TestCluster;
  devteam re-created as TestCluster2 1+1 (the fitting shape; matches
  stage6/H2). Also noted: cluster was built from a GUI template
  "TestTemplate", not the walkthrough's "test1".
- **FINDING (logged, unruled): tenant quota over-promises testcloud
  capacity** -- devteam hit NoValidHost at under 2/3 of its RAM quota
  (51200MB vs ~19GB schedulable, fragmented). Fine at Roosevelt scale;
  misleading here. Options for a later ruling: trim testcloud tenant
  quotas to schedulable reality, or set the expectation in clientdocs.
- **Decommission audit (operator-requested, read-only):** cloud-wide
  running set = 3x beta-cluster nodes, capi-mgmt-v2, 3 amphorae; zero
  stray/SHUTOFF servers, zero volumes. Domains: 5 platform + devteam
  (live) + beta + foil1; acme fully gone. Recommendations queued:
  (1) lbtest LB leftover (beta): reclaim ~1GB + FIP 10.12.6.97 -- clean
  path is kubectl delete service from beta's kubeconfig (k8s-managed;
  direct Octavia delete would dangle/recreate), main-chat asset;
  (2) beta-cluster teardown = the big reclaim (~17GB, doubles free RAM)
  -- BLOCKED on the cross-stream ruling that its resize-acceptance role
  is done (ledger marks node_count=2 deliberate);
  (3) foil1: dormant shell (no instances/LBs/FIPs/volumes) -- RECOMMEND
  offboard via tenant-offboard.sh (retires a live domain, closes its
  waived Horizon rows; re-onboarding is proven cheap if a foil is needed
  again);
  (4) capi-mgmt-v2 and platform domains: KEEP (CAPI control plane).
- Numbers consumed: none.
REVERT: documentation-only entry (this file + docs/session-ledger.md); git checkout HEAD~ -- docs/v1-redeploy-changelog.md docs/session-ledger.md.

### 2026-07-08 (addendum 41, jumphost stream) -- DOCFIX-123: new end-to-end Jenkins+Kubernetes guide (single-flow HUB) + flannel-driver hole closed; devteam package instantiated

Overnight deliverable (operator away, questions pre-answered before step-away).
Root problem the operator named: the information a client needs to stand up a
Jenkins-to-Kubernetes workflow is spread across 6+ handover files with heavy
duplication and no single flow, so a client must jump between documents and can
fall into workflow holes that cause errors (the flannel driver being the live
example -- devteam's self-made flannel cluster has sat un-converged 6h+ this
session, real confirmation).

- **NEW clientdocs/jenkins-kubernetes-guide.md (DOCFIX-123):** one
  top-to-bottom workflow a client can complete WITHOUT switching documents.
  Covers both patterns the operator ruled in scope: Pattern A (pipeline deploys
  INTO the cluster via a stored kubeconfig credential -- the spine) and Pattern
  B (cluster as an elastic Jenkins build-agent pool). Sections: component map +
  the 3-account rule; prerequisites + clouds.yaml; cluster self-create (the
  ruled path) with the calico/no-flannel + no-Public/Hidden + no-baked-keypair
  rules and the right-size-to-quota-AND-host-capacity (NoValidHost) caveat;
  kubeconfig fetch/understand (client-cert auth, API on the cluster floating IP
  :6443, treat as secret, cert expiry marked VERIFY-LIVE); store as a Jenkins
  Secret-file credential; Pattern A manifest + Jenkinsfile + LB/quota guidance;
  Pattern B cloud/pod-template setup; a stock end-to-end worked example;
  teardown + quota hygiene; troubleshooting (7 verbatim symptoms incl. the
  Missing-auth-url, Public/Hidden 403, NoValidHost, stuck-IN_PROGRESS=flannel,
  keypair-wrong-account, app-cred-cannot-create, kubeconfig-expiry); glossary;
  and a cross-link map so the guide is the HUB and the deep-dive docs are owned
  once. Built from a two-agent read-only inventory (dup/gap map) + conventions
  audit; the Jenkins-to-cluster bridge (kubeconfig-as-credential, dynamic
  agents, a real deploy stage, the NoValidHost sizing rule) was the true hole
  and is now owned here.
- **Flannel hole closed in the other doc devteam uses:** tenant-skill
  references/kubernetes.md "Creating your own cluster template" gains the
  explicit second hard rule (use calico; flannel never converges; a long
  CREATE_IN_PROGRESS is almost always this). Was implicit (calico in the CLI
  block only); now stated.
- **Gates:** pure ASCII+LF; zero internal-term leakage (the guide is now
  scanned -- added to LEAK_FILES in tests/clientdocs-skill/run-tests.sh, a
  harness change); placeholders are standard tokens only
  (TENANT_SHORT_NAME/AUTH_URL/REGION/CA_BUNDLE_FILE/ACCOUNT_CONTACT + THIS
  banner). README Contents + Packaging lists updated. L7 sweep receipt
  re-recorded (30 files). repo-lint 0 fail; clientdocs-skill 8/8;
  clientdocs-scripts 47/47; full gauntlet 38/38.
- **Package:** guide instantiated to ~/devteam-package/jenkins-kubernetes-guide.md
  (devteam values, TEMPLATE banner + title suffix removed, zero tokens
  verified); operator pushes with the rest tonight.
- **HELD for operator (morning):** the broad ease-of-use review across ALL
  handover docs (single-workflow, minimize doc-switching, component-detail) --
  a judgment call the operator wants to drive; the running DOCFIX list
  (handover-pack.md data review + dashboard URL, cred-.txt-file dashboard_url
  field, which is a change to the cloud-mutating onboard script -> gated); the
  offboard Phase-E0 always-403 defect (below). devteam's stuck flannel cluster
  (TestCluster2) awaits a gated delete + calico rebuild.
- Numbers consumed: DOCFIX-123.
REVERT: git checkout HEAD~ -- clientdocs tests/clientdocs-skill/run-tests.sh docs/session-ledger.md docs/v1-redeploy-changelog.md; rm clientdocs/jenkins-kubernetes-guide.md ~/devteam-package/jenkins-kubernetes-guide.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep (re-pin receipt to 29 files); no cloud-side change to revert.

### 2026-07-08 (addendum 42, jumphost stream) -- devteam cluster unblock: stuck flannel cluster gone + devteam-k8s calico template DELIVERED (live op, no numbers)

Operator ruled (on return): delete the wedged cluster + deliver a vetted
template. Executed in logged window ops-devteam-template.

- **TestCluster2 (flannel) CONFIRMED GONE:** by the time the operator returned,
  the cluster list was empty and no orphan nodes/stacks/LBs remained; the two
  gp.mid nodes' RAM was reclaimed (hosts 1-2 back to 8192MB used). The stuck
  flannel cluster either timed out to a cleaned failure or was deleted client/
  operator-side; net state verified clean, so no agent delete was needed.
- **devteam-k8s template DELIVERED (stage5, tenant-onboard.sh devteam 5):**
  created in devteam-prod as devteam-svc app-cred. Verified attributes:
  network_driver=calico, hidden=False (no Public/Hidden), coe=kubernetes,
  image=ubuntu-jammy-kube (ae90b82f...), external=provider-ext,
  fixed_network=devteam-net, master_lb_enabled=True, flavor=capi.node. Closes
  the addendum-39 "no delivered template" gap; devteam now has the vetted
  starting point the clientdocs promise.
- Numbers consumed: none (live op).
REVERT: template only -- as devteam-svc: openstack coe cluster template delete devteam-k8s. Doc side: git checkout HEAD~ -- logs/as-executed-index.md docs/v1-redeploy-changelog.md docs/session-ledger.md.

### 2026-07-08 (addendum 43, jumphost stream) -- DOCFIX-124: clientdocs consolidation M1-M6 + shared glossary (M4); operator ruled ALL moves

Operator ruled the whole consolidation proposal
(docs/clientdocs-workflow-review-20260708.md) in. Implemented under clientdocs
discipline; gauntlet 38/38, L7 receipt 31 files, repo-lint 0 fail.

- **M5 (error-holes) -- self-service-guide.md:** the Kubernetes section now
  carries the two rules that were only in the specialist docs: use the calico
  driver (flannel never converges), and size to physical host capacity AND
  quota (the "No valid host" symptom, not a quota problem, start small + scale).
  These are the workflow holes that were causing client errors.
- **M3 (entry point) -- welcome.md:** the "read in this order" list now
  includes the Jenkins+Kubernetes guide and names the Handover Pack as the
  single value reference; added a "Going straight to a task?" goal-to-document
  map so a client jumps to the one guide that owns their workflow.
- **M1/M2 (de-dup) -- interpreted as single-source-of-truth + pointers, NOT
  hard deletion:** the tenant AI skill (SKILL.md) needs the account model and
  clouds.yaml inline as operating context (it may not have the other docs
  loaded), and ci-integration-guide.md's clouds.yaml block is coupled to the
  clientdocs-scripts P2 test. So handover-pack.md section 2 is marked the
  authoritative account reference; clouds.yaml.template is named the canonical
  auth block; the copies in self-service-guide.md, SKILL.md, and
  ci-integration-guide.md keep an audience-appropriate copy with a one-line
  "authoritative source is X" pointer that resolves drift. (Flagged: this is
  the softer reading of M1/M2; hard deletion is a quick follow-up if preferred.)
- **M6 (fixes) -- Jenkinsfile.example:** script paths corrected from the stale
  clientdocs-starter-kit/ prefix to the delivered scripts/ layout; the
  misleading "Worked sequence" stage renamed "Infrastructure acceptance" with a
  comment clarifying it is an infra smoke build (not an app deploy) and a
  cross-link to the Jenkins guide's deploy pipeline.
- **M4 (glossary) -- NEW clientdocs/glossary.md:** plain-language definitions of
  every component across the pack (tenancy, accounts, networking, compute,
  Kubernetes, access/trust), grouped by theme. Added to LEAK_FILES (leakage-
  scanned), README Contents + Packaging (ships to every client), and
  cross-linked from welcome.md and the Jenkins guide. Standard tokens only.
- **FINDING (operator, flag not fix): the delivered ~/devteam-package/
  handover-pack.md still contains literal {{CUSTODIAN_1}}/{{CUSTODIAN_2}}** --
  the custodian names were never filled at instantiation (consistent with
  devteam's placeholder-custodian internal-test status, but a real handover
  must fill them). Not touched -- operator's value to supply.
- Numbers consumed: DOCFIX-124.
REVERT: git checkout HEAD~ -- clientdocs tests/clientdocs-skill/run-tests.sh docs/session-ledger.md docs/v1-redeploy-changelog.md; rm clientdocs/glossary.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep (re-pin receipt to 30 files).

### 2026-07-08 (addendum 44, jumphost stream) -- DOCFIX-125: day2 {{THIS}}-in-prose reword (survived instantiation) + full devteam package re-instantiation

Caught by staging-first re-instantiation of the devteam package.

- **DOCFIX-125 -- day2-operations.md line 7:** the meta-line explaining the
  {{THIS}} notation used the literal token in DELIVERED prose, so a `{{THIS}}`
  survived into every instantiated copy. Reworded to plain prose ("the specific
  names and endpoints for your tenancy were filled in for you at handover").
  This was the logged clientdocs-sweep candidate; now closed. Skill harness 8/8,
  L7 receipt re-recorded.
- **Full devteam package re-instantiated (~/devteam-package):** all delivered
  files rebuilt from current templates via a staging-first helper that asserts
  zero unfilled tokens before copy. This caught real drift the per-file updates
  had missed -- the package kubernetes.md was missing the addendum-41 flannel
  rule; it is now present. glossary.md added; welcome/self-service/ci-guide/
  SKILL/jenkins-guide all current; clouds.yaml.template filled (its header says
  we fill it at delivery); handover-pack.md updated by delta (canonical-account
  note) to preserve its client-specific values (CIDR/date/delivery). Package
  token-clean EXCEPT the flagged {{CUSTODIAN_1/2}} (operator value, addendum 43).
- Numbers consumed: DOCFIX-125.
REVERT: git checkout HEAD~ -- clientdocs/tenant-skill/references/day2-operations.md clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep. Package is a jumphost artifact (re-run the instantiation helper to rebuild).

### 2026-07-08 (addendum 45, jumphost stream) -- gated script fixes: DOCFIX-126 offboard E0 always-403 + DOCFIX-127 onboard cred-file dashboard_url

Both operator-approved (build offline + harness; live verification gated).

- **DOCFIX-126 -- tenant-offboard.sh Phase E0 always-403:** admin CANNOT list
  another user's application credentials (base policy
  identity:list_application_credentials = admin_or_owner; measured DOCFIX-095).
  The E0 inventory previously counted that 403 as an identity failure and
  FATAL'd at exit 23 BEFORE Phase F (local cred-dir retirement) -- exactly what
  the first live offboard (foil1, addendum 40) hit, stranding ~/tenant-foil1
  for a manual mv. Fix: a new branch recognizes the expected 403 (matches
  identity:list_application_credentials / not-authorized / HTTP 403), reports it
  as "cannot inventory as admin (not owner) -- creds cascade on user delete, OK",
  and does NOT increment ID_FAIL. App creds still delete via keystone cascade on
  user delete, so nothing is left behind. Harness: new case
  e0-appcred-403-not-fatal (MOCK_SCEN=appcred403) asserts exit 0 + the OK line +
  NO "FATAL: identity teardown"; tenant-offboard 20->21, gauntlet 38/38.
- **DOCFIX-127 -- tenant-onboard.sh cred-file dashboard_url:** the generated
  ~/tenant-<client>/*.txt handover cred files carried auth_url but not the web
  dashboard URL (operator's "add the horizon url" item). Added a tagged,
  env-overridable DASHBOARD_URL default (the dashboard is a web UI, NOT an API
  service, so it is not in the keystone catalog and cannot be discovered --
  same literal-with-override pattern as KEYSTONE_VIP; as-built
  https://10.12.4.58/horizon). A dashboard_url= field is now written to the
  domain-admin cred (stage1) and every per-user cred (stage2). Consumers parse
  these by name (awk /^field=/), so the added field is order-independent and
  breaks nothing (tenant-offboard auth_url parse, stage7 canary password parse
  both verified name-based).
  - COVERAGE NOTE (honest): the onboard harness exercises stages 3/4/7 only and
    mocks cred files as INPUTS; the cred-WRITING stages (1/2) have no unit
    coverage (pre-existing boundary). So this field's write is confirmed at the
    gated live onboard, as the cred-writing code always has been. Logged future
    item: add stage1/2 cred-write harness coverage.
- Numbers consumed: DOCFIX-126, DOCFIX-127.
REVERT: git checkout HEAD~ -- scripts/tenant-offboard.sh scripts/tenant-onboard.sh tests/tenant-offboard/run-tests.sh clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep.

### 2026-07-08 (addendum 46, jumphost stream) -- DOCFIX-128: handover package consolidation phase 1 (glossary folded in, hard de-dup, no extra files)

Operator re-ruling: consolidate the handover package WITHOUT extra files, remove
outliers, hard de-dup (delete, not pointer-mark), and do not let the AI skill
block it -- the skill is rewritten from the consolidated docs next (phase 2).
This supersedes the addendum-43 softer M1/M2 (pointers-keep-copies).

- **Glossary folded in, glossary.md REMOVED:** the standalone glossary.md added
  earlier today is now a "## Glossary" appendix at the end of handover-pack.md
  (definitions grouped by theme; the client's own values stay in sections 1-3).
  No extra file. References updated: README Contents/Packaging, welcome.md, the
  Jenkins guide section 11, and LEAK_FILES (glossary now scanned as part of
  handover-pack via the sweep, not a separate harness line).
- **Hard de-dup of the account model:** self-service-guide.md's duplicate
  Account/Use/Do-not TABLE is DELETED, replaced by a one-sentence orientation
  that points to handover-pack section 2 (the canonical rights table). The
  addendum-43 "keep copy + pointer" is gone -- the duplication itself is gone.
- **Distinct task guides KEPT** (operator's chosen consolidation level): intake,
  welcome, self-service, handover-pack, ci-integration, jenkins-kubernetes,
  acceptance, ai-assistant. Net top-level docs 9 -> 8 (glossary removed).
- Gates: L7 receipt 31 -> 30 files, gauntlet 38/38, repo-lint 0 fail.
- **PHASE 2 NEXT (this session): rewrite the tenant AI skill** (SKILL.md +
  references) from the consolidated handover package -- remove the skill's own
  duplication of the account model / clouds.yaml, align to the new structure
  (glossary location, Jenkins guide, driver/capacity rules).
- Numbers consumed: DOCFIX-128.
REVERT: git checkout HEAD~ -- clientdocs tests/clientdocs-skill/run-tests.sh docs/v1-redeploy-changelog.md docs/session-ledger.md; git checkout HEAD~ -- clientdocs/glossary.md (restore deleted file); then bash scripts/repo-lint.sh --record-clientdocs-sweep.

### 2026-07-08 (addendum 47, jumphost stream) -- DOCFIX-129: tenant AI skill aligned to the consolidated handover package (phase 2)

Operator: "once the files are complete you can rewrite the skill based on the
current handover package." Assessment on reading the skill: SKILL.md + the four
references were already well-structured and consistent with the consolidated
docs (the account model is an operating summary that defers to the Handover Pack
as authoritative; the boundary model, auth, and discipline all match). A
from-scratch regeneration would have risked regressing that polished content
(DOCFIX-109) for no gain, so this is a thorough ALIGNMENT/REFRESH that closes the
real gaps -- the skill predated tonight's consolidation and was missing the exact
error signatures the consolidation was about. (If a deeper regeneration is
wanted, it is a follow-up.)

- **references/troubleshooting.md -- three failure signatures added** to the
  "known signatures" list (these were the error-holes the consolidation
  targeted, and the AI skill lacked them): cluster TEMPLATE create fails /
  "Unable to create cluster template" = Public/Hidden flag; cluster stuck
  CREATE_IN_PROGRESS = flannel (rebuild with calico); "No valid host was found"
  = capacity, not quota, size down.
- **references/kubernetes.md** -- cluster-create section now carries the
  right-size-to-capacity (NoValidHost) caveat (was quota-only); the "Clusters
  and CI" section points to the delivered Jenkins + Kubernetes Implementation
  Guide as the single end-to-end pipeline reference.
- **SKILL.md** -- Companion Documents rewritten to list all delivered docs
  including the Jenkins + Kubernetes Implementation Guide and to note the
  Glossary now lives at the end of the Handover Pack; routing gains a pointer to
  the Jenkins guide for the full Jenkins-to-Kubernetes workflow.
- Skill harness 8/8 (T4 placeholder set, T5 routing reachability, leakage, ASCII
  all still hold); gauntlet 38/38; L7 receipt 30 files.
- **Package re-instantiated (~/devteam-package):** glossary.md removed; the
  Glossary appendix folded into the package handover-pack.md (devteam-
  substituted, client-specific values preserved); all skill/reference files
  refreshed; zero unfilled tokens except the flagged {{CUSTODIAN_1/2}}.
- Numbers consumed: DOCFIX-129.
REVERT: git checkout HEAD~ -- clientdocs/tenant-skill clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep; re-run the instantiation helper to rebuild the package.

### 2026-07-08 (addendum 48, jumphost stream) -- DOCFIX-130: jenkins-kubernetes-guide.md post-handover workflow gaps (k8s ServiceAccount how-to + dashboard-user CLI pointer)

Reviewing the guide end to end while supporting devteam's first self-service
cluster surfaced a cluster of gaps a real Jenkins-to-Kubernetes team hits, plus a
structural weakness: the guide was CLI-first with ad-hoc dashboard asides, but
devteam drives the dashboard. All changes are ADDITIVE to
clientdocs/jenkins-kubernetes-guide.md -- no existing text removed, no client value
or {{token}} touched, no new token introduced.

STRUCTURE -- dashboard vs command line made explicit:
- New Section 0 selector "Two ways to drive the cloud": cluster lifecycle
  (create/resize/delete, networks, FIPs, LBs) can be dashboard OR CLI; everything
  Kubernetes (kubeconfig, kubectl/helm, Jenkins, deploy, ingress, storage, k8s
  teardown) is CLI-ONLY -- so a GUI user crosses to the CLI at Section 3 and stays.
  Convention set: differing steps are marked **On the command line** /
  **In the dashboard**; unmarked steps are CLI and apply to both.
- Dashboard equivalents added where lifecycle is GUI-able: Section 2.3 create
  (template + keypair + small size + Public/Hidden unchecked -- the path devteam
  actually used), Section 2.5 resize, Section 8 teardown (release FIP/LB/cluster in
  the dashboard; k8s-object delete stays kubectl-only, Service first).

GAP CLOSURES (the 7 reviewed with the operator):
1. Registry (5.0): stated plainly the platform provides NO container registry
   (VERIFIED -- catalog has only glance + product-streams, no registry service);
   bring your own; public pulls work (cluster pulled its own system images); added
   the private-registry `kubectl create secret docker-registry` + `imagePullSecrets`
   procedure the guide previously only named.
2. Ingress (5.4 new): the guide urged "one ingress instead of N LoadBalancers" with
   no how-to; added the ingress-nginx helm install (one LB) + an Ingress manifest,
   and folded in the no-platform-DNS point (map your own hostnames to the ingress IP).
3. Scaling (2.5 new): `openstack coe cluster resize <cluster> <count>` (VERIFIED
   syntax via --help) + dashboard Resize; same capacity limit as first create noted.
4. Persistent storage (5.5 new): SELF-VERIFYING -- `kubectl get storageclass`; if a
   (default) class exists a PVC binds, if none the cluster has no PV support and to
   contact us. Deliberately does NOT assert a StorageClass (delivered template has
   volume_driver unset -- MEASURED; whether the cinder-CSI default yields a class is
   UNVERIFIED, see operator follow-up).
5. Cert expiry (3): the guide flagged expiry but not how to see it; added a
   `kubectl config view --raw | base64 -d | openssl x509 -noout -enddate` one-liner.
6. No DNS (5.4): folded into ingress -- LoadBalancer/ingress give an IP, not a name.
7. Tooling (1): install pointer for the OpenStack client (coe commands) + kubectl +
   helm, and a kubectl-vs-1.34 version-skew caveat.
- Troubleshooting (9): added ImagePullBackOff/ErrImagePull (registry) and PVC
  Pending (no StorageClass) symptom rows.

- Leak harness caught two self-inflicted leaks on the first pass (the literal terms
  "Horizon" and "magnum"/python-magnumclient in the selector + tooling text) -- both
  reworded to "the dashboard" / "the OpenStack client (coe commands)"; the harness
  did its job. FINAL: leak harness PASS 8/0, gauntlet ALL GREEN (38), repo-lint
  0 fail (1 legacy WARN), L7 receipt re-recorded (30 files).
- No D-NNN governs client-guide content (DOCFIX-class, cf. DOCFIX-123). Read-only
  cloud audit only (catalog, template fields, resize --help); no mutation.
- OPERATOR FOLLOW-UP (logged, not actioned): the delivered cluster template has
  volume_driver unset and no docker volume; confirm whether the built cluster gets a
  default StorageClass from the cinder-CSI default. If NOT, persistent volumes are
  unavailable out of the box -- candidate template improvement (needs a D-NNN /
  future DOCFIX). Verifying live means kubectl into devteam's running cluster
  (tenant isolation -- deferred to operator).
- DELIVERY NOTE: template edit only -- ~/devteam-package NOT re-instantiated; refresh
  it at the next package sweep so devteam's copy carries the new sections.
- Numbers consumed: DOCFIX-130.
REVERT: git checkout HEAD~ -- clientdocs/jenkins-kubernetes-guide.md clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep.

### 2026-07-08 (addendum 49, jumphost stream) -- DOCFIX-131: tenant-offboard.sh Phase A cluster captures stdout-only (latent false-exit-21 bug)

The Phase A cluster captures still merged stderr (`2>&1`) -- the last such sites in
the offboard (Phase C/D were fixed in DOCFIX-113). A benign CLI deprecation warning
on `coe cluster list` would word-split into the delete loop and, worse, keep the
wait loop from ever seeing "empty" -- a FALSE "clusters not gone in budget" FATAL
exit 21 on a tenant whose clusters WERE gone. Client-facing offboard; latent
correctness/safety bug, not cosmetics. Operator-greenlit from the S-class batch.

- **Fix:** new `tinv()` -- the tenant-scoped twin of the DOCFIX-113 `inv()`: runs
  the command under `tenant_env` in the capture subshell, stdout-only, stderr to a
  tempfile SURFACED as a NOTE. The three Phase A `coe cluster list` captures
  (inventory ~line 180, wait loop, final check) route through it. The wait loop
  gains retry-on-list-error and the final check fails closed if the list errors
  (mirrors the Phase C LB-wait pattern) -- an empty stdout on a genuinely failed
  list no longer reads as "gone".
- **Harness:** the mock's `warnstderr` scenario no longer EXCLUDES coe (the
  exclusion existed precisely because this was unfixed); the existing
  stderr-not-argv test now exercises the Phase A path, plus a new explicit
  `coe-stderr-not-argv` assertion (NOTE surfaced + no false exit 21). 22/22 (was
  21/21). Gauntlet 38/38; repo-lint 0 fail; L7 receipt re-recorded (tenant-offboard.sh
  is a pinned tenant-facing file; internal change, no clientdocs staleness).
- Closes the addendum-25 inventory's last Phase-A stderr-merge item; the other two
  (domain-show/token-issue fail-closed cosmetics; Phase E0 display-only pollution)
  remain lower-priority candidates.
- Numbers consumed: DOCFIX-131.
REVERT: git checkout HEAD~ -- scripts/tenant-offboard.sh tests/tenant-offboard/run-tests.sh clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep.

### 2026-07-08 (addendum 50, jumphost stream) -- DOCFIX-132: S7 dead-code removal (validate checks d011-03/04)

Operator-greenlit S-class item S7 (docs/script-quality-findings-20260707.md). Pure
deletion, no behavior change:
- checks/d011-04-octavia-lb.sh: `CF` was assigned from a cred-file probe loop and
  never read (0 uses); removed the 2 dead lines.
- checks/d011-03-vip-tenant.sh: the guard `vr_is_ipv4 "$VHOST" || [ -n "$VHOST" ] ||
  fail` fires only when VHOST is empty, which `[ -n "$VHOST" ]` alone catches -- the
  vr_is_ipv4 leg was subsumed (any valid IPv4 is non-empty). Dropped it; identical.
- Gauntlet green (validate harness).
- Numbers consumed: DOCFIX-132.
REVERT: git checkout HEAD~ -- scripts/checks/d011-03-vip-tenant.sh scripts/checks/d011-04-octavia-lb.sh docs/v1-redeploy-changelog.md docs/session-ledger.md.

### 2026-07-08 (addendum 50, jumphost stream) -- DOCFIX-133: S4 trust-list filter consolidated to scripts/trust_filter.py

Operator-greenlit S-class item S4. Three near-identical inline python-in-bash trust
filters (tenant-offboard sweep-preamble + audit inventory; tenant-acceptance trustee
lookup) -- inline python beyond the house one-liner scale -- replaced by one
fixture-tested scripts/trust_filter.py parameterized for the three shapes:
- `--fields {ID,TRUSTOR,TRUSTEE}` (space-joined), `--trustor-in` (ws/comma uid set,
  default keep-all), `--first`, `--error-token` (default 'trust-parse-error'; '' = silent).
- Preserves each caller's exact contract: offboard fail-closed grep on
  'trust-parse-error' (exit 20); acceptance empty-on-error + first-trustee.
- SDIR added to tenant-offboard.sh + tenant-acceptance.sh (matches the onboard
  net_overlap.py invocation pattern). tenant-onboard was not a trust-filter site.
- New tests/trust_filter/run-tests.sh (9/9: 3 shapes, comma set, hit/miss, parse-error
  token + silent, empty set, bad-fields rc 2). Gauntlet 39 harnesses ALL GREEN (new
  harness auto-discovered); offboard 22/22, acceptance 16/16. L7 receipt re-recorded
  (both tenant scripts are pinned tenant-facing files).
- Numbers consumed: DOCFIX-133.
REVERT: git checkout HEAD~ -- scripts/trust_filter.py tests/trust_filter/run-tests.sh scripts/tenant-offboard.sh scripts/tenant-acceptance.sh clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md.

### 2026-07-08 (addendum 50, jumphost stream) -- DOCFIX-134: S8a FIP pool literals to lib-net.sh (single source of truth)

Operator-greenlit S-class item S8, SCOPED to its low-risk slice (S8a). The D-003
provider FIP allocation pool (10.12.5.0-10.12.7.254) was hardcoded in
phase-04-network-create.sh (env-overridable default) and phase-04-network-verify.sh
(pinned literal). Both already source lib-net.sh, so:
- Added FIP_POOL_START / FIP_POOL_END to lib-net.sh (D-003-tagged; hard-rule #3
  centralized constant keyed by role, under the existing SC2034 carve-out).
- create: FIP_START/FIP_END defaults reference the constants (still env-overridable);
  verify: pins to the constants.
- Gauntlet 39/39 (phase-04 + phase-04-create harnesses green); constants resolve
  when sourced.
- Numbers consumed: DOCFIX-134.
- DEFERRED (bigger than the "S" estimate; operator decision in the session ledger):
  S8b (KEYSTONE_VIP to lib-net.sh across 6 files incl. 3 tenant scripts that do NOT
  currently source lib-net -- sourcing plumbing into client-critical scripts) and
  S8c (provider-bundle-check.py restates plane CIDRs -- needs a generated/parsed
  form since python cannot source bash).
REVERT: git checkout HEAD~ -- scripts/lib-net.sh scripts/phase-04-network-create.sh scripts/phase-04-network-verify.sh docs/v1-redeploy-changelog.md docs/session-ledger.md.

### 2026-07-08 (addendum 51, jumphost stream) -- DOCFIX-135: tenant AI skill registry gap aligned to DOCFIX-130

devteam (via the AI assistant) asked whether there is a registry setup step. The
DOCFIX-130 guide addition (no platform registry; bring-your-own; private-registry
imagePullSecret) had NOT been mirrored into the tenant skill -- grep of
clientdocs/tenant-skill/ returned zero registry content. Same skill-lag class as
DOCFIX-129. Additive, leak-clean, no new token:
- references/kubernetes.md: new "Images: bring your own registry" section (platform
  hosts no registry; public pulls work; private-registry `kubectl create secret
  docker-registry` + imagePullSecrets; secret injected at CI run time; pointer to
  the Jenkins+K8s Implementation Guide section 5.0).
- references/troubleshooting.md: new ImagePullBackOff/ErrImagePull signature under
  section 4 (known failure signatures).
- Skill harness 8/8 (T3 leak, T4 placeholder set unchanged, T6 ASCII); gauntlet
  39/39; repo-lint 0 fail; L7 receipt re-recorded.
- Package re-instantiated: ~/devteam-package/tenant-skill/references/{kubernetes,
  troubleshooting}.md (devteam + live-measured AUTH_URL tokens; registry content
  present; zero leftover tokens).
- Numbers consumed: DOCFIX-135.
REVERT: git checkout HEAD~ -- clientdocs/tenant-skill/references/kubernetes.md clientdocs/tenant-skill/references/troubleshooting.md clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep; re-instantiate the package skill files.

### 2026-07-08 (addendum 52, jumphost stream) -- DOCFIX-136: tenant AI skill full sweep, close remaining DOCFIX-130 skill-lag (6 topics)

Operator-requested full sweep of the skill against the DOCFIX-130 guide (after the
registry gap, DOCFIX-135). Grep-audit found 6 guide topics absent from the skill;
all added concise (skill stays terse; full detail defers to the guide by pointer).
Additive, leak-clean, no new token.
- references/kubernetes.md:
  - Cluster resize command (`openstack coe cluster resize`) -- was prose "scale the
    worker count" with no command.
  - kubeconfig client-cert expiry: time-limited; re-fetch; openssl one-liner to read
    the expiry (skill previously covered only app-cred expiry).
  - dashboard-vs-CLI split: create/resize/delete either way, kubeconfig onward is
    CLI-only (the dashboard cannot drive kubectl) -- for GUI-first tenants.
  - no platform DNS: LoadBalancer/ingress give an IP not a hostname; pointer to the
    guide section 5.4 ingress how-to.
  - NEW "Persistent storage" section: kubectl get storageclass; (default) binds,
    none = no PV support / PVC stays Pending -> contact.
  - ServiceAccount pointer: scoped SA token > full-admin kubeconfig for build agents
    (guide section 6.1).
- references/troubleshooting.md: two new signatures -- PVC Pending (no StorageClass)
  and kubeconfig cert-expired ("worked before, now a cert error").
- Skill harness 8/8 (T3 leak, T4 placeholder set unchanged, T6 ASCII); gauntlet
  39/39; repo-lint 0 fail; L7 receipt re-recorded.
- Package re-instantiated: ~/devteam-package/tenant-skill/references/{kubernetes,
  troubleshooting}.md.
- Skill now covers the full DOCFIX-130 topic set; ingress kept as recommendation +
  guide pointer (full helm how-to stays guide-only, by design).
- Numbers consumed: DOCFIX-136.
REVERT: git checkout HEAD~ -- clientdocs/tenant-skill/references/kubernetes.md clientdocs/tenant-skill/references/troubleshooting.md clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep; re-instantiate the package skill files.

### 2026-07-08 (addendum 53, jumphost stream) -- DOCFIX-137: repo-lint L8 guide<->skill coupling guard (enforceable skill-lag gate)

The tenant AI skill fell behind the Jenkins+K8s guide twice (DOCFIX-135 registry,
DOCFIX-136 six more topics) because guide edits shipped without mirroring into the
skill. Operator asked to make "sweep the skill on guide changes" ENFORCEABLE, not a
note. Added lint check L8, mirroring the L7 sweep-receipt pattern:
- clientdocs/guide-skill-coupling.txt pins the sha256 of
  clientdocs/jenkins-kubernetes-guide.md (INTERNAL file, never shipped -- a .txt, so
  not caught by L7's *.md/*.sh manifest).
- When the guide changes, repo-lint FAILS naming the skill files to review
  (tenant-skill/references/kubernetes.md + troubleshooting.md), until the pin is
  re-recorded: bash scripts/repo-lint.sh --record-guide-skill-coupling (rewrites ONLY
  that pin). Skipped when the guide is absent.
- repo_lint.py: new COUPLING_FLAG + record block + L8 check; docstring L1-L7 -> L1-L8.
  repo-lint.sh: the new flag rides the existing --* passthrough; runner comment updated.
- Harness tests/repo-lint/run-tests.sh +5 (missing pin FAILS, recorded PASSES, guide
  drift FAILS with L7 re-recorded so L8 fires alone, re-record PASSES, no-guide
  SKIPPED) -- 28/28. Gauntlet 39/39; repo-lint 0 fail.
- Calibration: like L7, the gate forces a conscious stop + review + explicit
  re-record; it cannot cheaply PROVE the skill was edited, but it makes skipping the
  review a deliberate act -- which is exactly what was missing.
- Numbers consumed: DOCFIX-137.
REVERT: git checkout HEAD~ -- scripts/repo_lint.py scripts/repo-lint.sh tests/repo-lint/run-tests.sh clientdocs/guide-skill-coupling.txt docs/v1-redeploy-changelog.md docs/session-ledger.md.

### 2026-07-08 (addendum 54, jumphost stream) -- DOCFIX-138: S8b KEYSTONE_VIP default centralized into lib-net.sh

Operator-approved S-class item S8b. The keystone PUBLIC endpoint literal (10.12.4.50)
was the default in 6 scripts across 3 forms; now all reference one lib-net.sh constant
(hard-rule #3; Roosevelt = 1 edit not 6). lib-net verified source-safe before wiring
(constants + read-only helpers, no set/side-effects, source-safe guard).
- lib-net.sh: new KEYSTONE_VIP_DEFAULT="10.12.4.50" (D-020/D-052-tagged, under the
  existing SC2034 carve-out).
- Wired (each stays env-overridable; a lib-net source added where absent):
  - phase-03-admin-openrc.sh: `KEYSTONE_VIP="${KEYSTONE_VIP:-$KEYSTONE_VIP_DEFAULT}"`
    (had SCRIPT_DIR; added source).
  - tenant-onboard.sh: top-level SDIR + source (was function-local only); line 23 ref.
  - tenant-offboard.sh + tenant-acceptance.sh: source added after their existing SDIR
    (DOCFIX-133); offboard KEYSTONE_VIP ref; acceptance foil_env inline AUTH_URL now
    `https://${KEYSTONE_VIP_DEFAULT}:5000/v3`.
  - phase-06-kubeconfig-gate.sh + phase-06-k8s-bootstrap.sh (KEYSTONE_HOSTPORT form):
    local SCRIPT_DIR + source before the block; `${KEYSTONE_VIP_DEFAULT}:5000`.
- Resolve-checked: KEYSTONE_VIP -> 10.12.4.50, KEYSTONE_HOSTPORT -> 10.12.4.50:5000.
  Two header-comment mentions of the literal RETAINED as as-run provenance (doc only,
  not an executable second source).
- Gauntlet 39/39 (onboard/offboard/acceptance/phase-06 all green under the new
  sourcing); repo-lint 0 fail (L7 receipt re-recorded for the 3 tenant scripts --
  internal change, no clientdocs staleness).
- S8 now FULLY done (S8a DOCFIX-134 + S8b here); S8c ruled won't-do.
- Numbers consumed: DOCFIX-138.
REVERT: git checkout HEAD~ -- scripts/lib-net.sh scripts/phase-03-admin-openrc.sh scripts/phase-06-kubeconfig-gate.sh scripts/phase-06-k8s-bootstrap.sh scripts/tenant-onboard.sh scripts/tenant-offboard.sh scripts/tenant-acceptance.sh clientdocs/sweep-receipt.txt docs/v1-redeploy-changelog.md docs/session-ledger.md; then bash scripts/repo-lint.sh --record-clientdocs-sweep.

### 2026-07-09 (jumphost stream) -- ledger machine-derived block re-seed (no number consumed)

Routine regenerate-from-scan of the docs/session-ledger.md machine-derived block (OWNER:
ledger-scan; regenerate, never hand-edit). It lagged the repo: showed DOCFIX next-free 130
(the 2026-07-08 pre-sweep seed) while scripts/ledger-scan.sh reports 139 post-DOCFIX-138 --
a stale next-free pointer that would mis-seed the forthcoming DC-DC fork. Re-seeded from the
2026-07-09 scan: "as of" line -> post-addendum-54/DOCFIX-138; DOCFIX 130 -> 139; provenance
line -> 2026-07-09 scan. D (076) and BUNDLEFIX (012) were already current; the D-068/D-071
and SEC-001/003/004 rows already matched the scan (narrative below the block was current).
No identifier consumed (ledger regen, not a DOCFIX). repo-lint 0 fail.
REVERT: git checkout HEAD~ -- docs/session-ledger.md docs/v1-redeploy-changelog.md

Next-free after this push (per scan, keep this line unwrapped): D-076, DOCFIX-139, BUNDLEFIX-012.
