<!-- CONSOLIDATED SKILL SNAPSHOT for Chat/project upload.
     Source: .claude/skills/openstack-cloud-ops/ (SKILL.md + references/) regenerated at repo HEAD 4dcf898, 2026-07-25.
     WHY: the Chat-side mount is stale (docs/audit/skill-divergence-20260725.md). Chat's skill
     surface is single-file, so the on-demand references are FLATTENED here into one document.
     DERIVED artifact -- regenerate from the skill dir at each stage-close sweep (SKILL.md rule).
     Authoritative copy remains .claude/skills/openstack-cloud-ops/ in the repo. -->

# openstack-cloud-ops -- consolidated skill (Chat upload snapshot, 2026-07-25)

================================================================================
## SKILL.md (main entry -- always in context)
================================================================================

---
name: openstack-cloud-ops
description: "Operate, install, extend, and troubleshoot the Omega Cloud - a commercial multi-tenant Charmed OpenStack (Caracal 2024.1) deployment managed with Juju and MAAS, with Vault TLS, OVN, Ceph, Octavia, and Magnum/CAPI tenant Kubernetes. Use this skill for ANY work touching OpenStack, Juju, MAAS, Magnum, CAPI, Ceph, OVN, Octavia, Keystone, Vault-for-OpenStack, tenant onboarding, or the repository this skill operates against - including writing or reviewing bash/python operational scripts, debugging failed deploys or cluster creates, runbook work, design-decision (D-NNN) discussion, and incident triage. Use it even for seemingly simple OpenStack questions: this deployment has strict operating discipline and known charm traps that make generic answers wrong."
---

# openstack-cloud-ops

Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant
self-administered OpenStack cloud, built as virtual rehearsals (VR0 single-DC
"testcloud"; VR1 two-DC + Office1 headend) for a future bare-metal
multi-datacenter deployment ("Roosevelt"). The CURRENT phase/stage lives in
repo `docs/CURRENT-STATE.md` ONLY (GA-R1) -- never trust a phase claim from
this file or memory. The governing design constraint is
MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables
alongside the running cloud, so transferable answers beat quick fixes.

## Step 0 - locate the source of truth

This repository (GitBucket, git.baldurkeep.com) is authoritative for
everything: bundle, runbooks, scripts, design decisions, as-built values. This
skill is a discipline-and-routing layer OVER that repo, not a substitute for
it. Never hardcode the repo's own name/path in a command -- it has already been
renamed/forked once (see D-110) and will again; discover it fresh each session.

1. Look for a local clone: a repo dir in the working tree, or a `$HOME` entry
   containing `bundle.yaml` + `docs/design-decisions.md`. If found, confirm
   identity with `git remote -v`, then `git log -1` to note HEAD and work from it.
2. No clone and you have shell + network: ask the operator for the exact clone
   URL (org/group and repo name -- do not guess or reuse one from memory; it
   may have moved, per this same repo's own history). The repo may be private;
   if the clone fails, ask the operator to provide access or the relevant files.
3. No clone obtainable (e.g. chat without sandbox network): say so, ask the
   operator to paste the relevant runbook/script, and proceed only on what is
   actually in front of you.

**Divergence rule:** if this skill and repo HEAD disagree, the repo wins -
but FLAG the divergence to the operator rather than silently following either.
The repo is a living draft; this skill's invariants (discipline, hardening)
change slowly, its facts (IPs, versions, phase status) go stale fast.

**Session bookends (GA-R1/GA-R4).** OPEN: read `docs/CURRENT-STATE.md` in full
BEFORE any other doc (it is the only status authority), then `docs/session-ledger.md`
+ `bash scripts/ledger-scan.sh` to recover what is in flight and reconcile it against
repo ground truth (open decisions, open SEC count, next-free numbers). Re-do this
after any compaction. CLOSE: a bounded ledger summary (15-line hard cap, GA-R4 rule
1); the full session body goes to `docs/archive/` in the same close commit; commit
and push -- a session ending without its bookend is treated as NOT durable by the
next session. See `references/operating-discipline.md` (Session continuity).
## Step 1 - detect the environment

- **Live shell to the jumphost / infra** (Claude Code on `vcloud` or
  similar): you may RUN read-only audits directly. Every mutation remains
  individually human-gated - present the command, state what it changes, wait
  for approval. A live shell relaxes the transport, never the discipline.
- **Chat / no infra shell**: operate the gated copy-paste model - prepare
  labeled blocks, the operator runs them and pastes output back. Never assume
  a block ran or succeeded; wait for the pasted evidence.

Read `references/operating-discipline.md` before doing either.

## The three hard operating rules (non-negotiable)

1. **Execute only the current runbook step, exactly as written.** No added
   scope, no adjacent improvements, no live re-architecture mid-step. Findings
   and improvement ideas are LOGGED (changelog / D-NNN proposal), never
   executed live mid-step.
2. **Never use an inferred value.** No IP, ID, name, or scope goes into a
   command unless it was measured this session or carried from confirmed
   as-built. If a value would be inferred: stop and measure it. Never run a
   destructive or session-altering command from memory without confirming it
   is the minimal correct action for the current live state.
3. **Prefer dynamic lookups over hardcoded literals.** Discover VIPs, project
   names, IDs, and version sets at runtime. Where a literal is unavoidable it
   is tagged and centralized (`scripts/lib-net.sh`, `lib-hosts.sh`), keyed by
   stable identity (CIDR, hostname) - never by drifting IDs.

Corollary that governs everything: **verify before mutate**. A read-only audit
precedes every mutation; destructive and secret-handling steps are gated
individually, never batched.

## Routing - where to go for what

| Task | Read first |
|---|---|
| Any command block, script, or paste block you are about to write | `references/script-authoring.md` |
| Deploy / redeploy / teardown (VR0, single-DC testcloud) | repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md` |
| VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission) | repo `docs/CURRENT-STATE.md` FIRST (stage/gate STATUS lives there only), then `docs/dc-dc-deployment-workflow.md` (stage identity, links, gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two. |
| Something is broken (triage, incidents) | `references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md` |
| CAPI / Magnum / mgmt-VM recovery | repo `runbooks/ops-capi-recovery.md` |
| Deliver ANY repo change (script, runbook, doc) | run `bash scripts/repo-lint.sh` + the touched script's `tests/<name>/run-tests.sh` BEFORE handing it over |
| Pre-deploy gate (before add-model) | `bash scripts/preflight.sh` -- THE single entry; do not run the sub-gates piecemeal |
| Is the cloud actually healthy? (post-deploy, post-restart, pre-change baseline, incident) | `bash scripts/cloud-assert.sh` (add `--capture` at deploy completion for the committed BOM) |
| Full-cloud restart after outage/maintenance | repo `runbooks/ops-restart-procedure.md` |
| VR1 site unreachable after a vcloud reboot (`ssh voffice1` / `office1-*` time out) | `bash scripts/site-baseleg.sh check office1` -- `office1-local` is an ISOLATED libvirt net, so the host's base L3 leg (`10.10.0.10/24` + compose route `10.10.1.0/24 via 10.10.0.20`) is dropped every reboot; the `After=libvirtd` unit re-adds it (D-126 amendment / D-128). If MISS and the unit is not yet installed: `sudo bash scripts/site-baseleg.sh apply office1`. This is the vcloud path; the workstation reaches the same over the tailnet (D-107). |
| Starting any consequential live session | `bash scripts/run-logged.sh <label>` first (as-executed log; docs/as-executed-log-convention.md) |
| Credential exposures / security TODOs | repo `docs/security-ledger.md` -- add a row, never only a script comment |
| Resuming work / what is still in flight (esp. after compaction) | repo `docs/CURRENT-STATE.md` first, then `docs/session-ledger.md` + `bash scripts/ledger-scan.sh` to reconcile against repo ground truth |
| Tenant onboarding / tenant self-service | repo `scripts/tenant-onboard.sh` + `runbooks/tenant-onboarding-v2-DRAFT.md` + `appendix-C/D` |
| Network / plane / IPAM questions | `references/environment.md`, repo `scripts/lib-net.sh`, NetBox (the IPAM apex -- but during VR1 the WORKING apex is `office1-netbox`, 10.10.1.10, per DOCFIX-195; `netbox.baldurkeep.com` is a READ-ONLY v1 reference, write-back deferred to end-of-deployment) |
| ANY change request to a built surface | grep repo `docs/design-decisions.md` for the governing D-NNN FIRST - PROPOSED/OPEN means the operator has not ruled: present options, do not implement |
| Why is it built this way? / proposing changes | repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number |
| Versions, channels, pins | repo `docs/CURRENT-STATE.md` section 7 (VR1 substrate -- measured pin authority); `runbooks/appendix-B-asbuilt-version-lock.md` (VR0 charm channels) |
| Environment facts (hosts, repo, planes) | `references/environment.md` |
| OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`) | `references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113) |
| Creating/changing a `libvirt_domain`; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong | `references/platform-traps.md` -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, `acpi=off`, missing `cpu` block = no `svm`, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge) |

## Standard loops (repeatable session shapes)

**Session bootstrap (jumphost):** Claude Code sessions launch from the repo
root, always (GA-R7 rule 6 -- a session launched from a subdirectory gets its
own empty project dir and a divergent auto-memory; the GA-F06 origin).
`git -C "$REPO" pull` (set `$REPO` once per
session -- see `runbooks/README.md` Conventions / `references/environment.md`) ->
`bash scripts/repo-lint.sh` (0 fail expected) -> if touching the live cloud,
`bash scripts/run-logged.sh <label>` to open the logged shell. Repo HEAD and a
clean lint are the preconditions for everything else.

**VR1 operating model (D-128 -- invariant; do not re-derive this each session):**
Claude runs ON the vcloud jumphost -- do NOT relocate the session to a workstation.
Work splits in two, origins FIXED BY DESIGN: **Plane 1** (substrate -- the OUTER
`tofu` root `qemu:///system` + local `virsh`, creating the DC *host* VMs) runs on
vcloud and has no Roosevelt analog; **Plane 2** (MAAS / NetBox / the INNER `tofu`
root `qemu+ssh`-from-Office1 / tenant tests) EXECUTES on `voffice1` via `ssh
voffice1 '...'` so runbooks read "from the Office1 headend" and transfer. Claude's
host only picks the TRANSPORT, not where work runs. Durable Office1 reach = the
D-126 base-leg oneshot (`scripts/site-baseleg.sh check office1` FIRST if `ssh
voffice1` is dead post-reboot) -- NOT tailscale-on-vcloud (D-107 is the operator's
WORKSTATION path / human GUI, left untouched). DCs sit outside the `/22` carve and
keep the D-126 `-J`-through-`vvr1-dc0` shape. **Durable host->site reach is standing
per-SITE tooling, DCs included (so the gap does not reopen per-DC):** `site-baseleg.sh`
is site-keyed. At each DC standup, MEASURE how vcloud reaches that DC's containment/
service net -- if it is an ISOLATED host-local net, add a MEASURED leg row + `install`
the unit as part of standup definition-of-done (NEVER an inferred DC IP -- the harness
fails on an un-commented DC supernet); a routed/NAT net or an already-reachable
`qemu+ssh` path needs none. Register item 20 tracks this. **Rack-side twin invariant
(D-131 sub-1, RULED 2026-07-21): every internet-isolated DC rack gets the repo-carried
`scripts/dc-rack-net.sh install <site>` at standup (reboot-persistent rack bridge legs
+ the node-DNS dnsmasq forwarder aliased on metal-admin, forwarding to region BIND over
the rack's own transit) -- MAAS 3.7's rack-only agent resolver SERVFAILs on isolated
racks, so WITHOUT this, commissioning silently times out (appendix-A carries the
symptom pair).** Two measured MAAS traps with standing consequences: MAAS virsh PODS
are refuted for this repo's DCs (per-machine `power_type=virsh` via
`scripts/maas-node-power.sh` is the ruled shape -- D-103/D-123 amendments 2026-07-20),
and a pod delete CASCADES to its linked machine records -- read the pod's machine list
BEFORE any `vm-host delete`; non-empty = STOP (appendix-A, 2026-07-21 incident).
**Per-DC power credential (SEC-012 dc0, SEC-016 dc1, RULED 2026-07-23 -- standing
DC-standup invariant): each DC gets its OWN dedicated MAAS->libvirt power key, NEVER
a cross-DC reuse.** The REGION's MAAS snap dials the power address, so the split is:
the maas-node-power SCRIPT's mapping virsh uses the DC SERVICE key (via the region's
`~/.ssh/config` Host <rack-transit-ip>), while MAAS's own power ops use the dedicated
key installed in the snap (`/var/snap/maas/current/root/.ssh/` + a per-host `ssh
config` block so each rack uses its own key). FRAGILITY: the snap-side key lives under
per-revision `/var/snap/maas/current/` and may not survive a snap refresh -- re-assert
after any refresh. Roosevelt analog: per-DC IPMI/BMC credentials.

**Change-delivery loop:** grep for prior art (zeroth decision) -> grep
design-decisions for the governing D-NNN -> edit -> `bash scripts/repo-lint.sh`
-> run the touched script's harness (create one if missing -- no script change
ships without its harness) -> deliver with an entry in the SESSION changelog
(ONE changelog per session, GA-R2/D1 -- never per-topic files) carrying a
per-item revert. Under blanket approval, the changelog IS the review surface:
every item states what, why (evidence), and how to revert. Changelogs are
session-scoped scratch: consolidated per-stage at stage close, NEVER citable
as status or decision authority.

**Deploy loop (VR0, single-DC testcloud):** phase-00 runbook (D-061 destroy
path) -> `bash scripts/preflight.sh` PASS -> phase-01..08 gated -> `bash
scripts/cloud-assert.sh --capture` -> commit the asbuilt/ BOM.

**Deploy loop (VR1 DC-DC, the current live mission):** read
`docs/dc-dc-deployment-workflow.md`'s tooling gap register FIRST -- check
whether the stage you're about to run has any open blocking item -> Stage 1
(`dc-dc-phase0-vcloud-prep.md`) gated -> Stage 2..7 gated in order, per
`runbooks/README.md`'s "VR1 DC-DC track" table -> if a stage's `tofu apply`
fails partway, read `runbooks/dc-dc-teardown-rollback.md`'s rollback decision
tree BEFORE reaching for a teardown (fix-forward is usually correct). Run
`bash scripts/opentofu-validate.sh` before any tofu step; whether a given
root/step has been exercised for real is a STATUS question -- check
CURRENT-STATE, never this file's memory of it. **At STAGE
CLOSE** (GA-R6): a stage or verification-type gate CLOSES only on a NAMED executable
check whose captured output the closing commit cites -- prose cannot close a stage,
and there is NO conditional close (a remainder splits into its own gate row or the
stage stays open, E3). Ruling-type gates close per GA-R5 (recorded, pushed ruling).
The close-out set is executable: gauntlet ALL GREEN, repo-lint 0-fail, the stage's
GA-R2 consolidation commit landed, the GA-R7 memory review run, MERGE the stage
branch to `main` (merge commit, NOT squash; operator-gated), then RETIRE the branch
(local + remote). **ALSO sweep THIS skill in the same close:** fold in any new
INVARIANT the stage established and reconcile any divergence from repo HEAD (facts
delegate to the repo and go stale; invariants live here and must be kept current). The
AUTHORITATIVE, versioned skill is THIS dir (`.claude/skills/openstack-cloud-ops/` -- SKILL.md
+ references); keep it current here. THEN REGENERATE the dated single-file Chat-upload
snapshot from it (concatenate SKILL.md + references -> `.claude/skills/openstack-cloud-ops-consolidated-<date>.md`,
ASCII/LF byte-verified) so the Chat/project skill surface can be refreshed -- the snapshot
is a DERIVED artifact, never the source, and the Chat mount going stale is a known failure
mode (it caused a failed clone + a mis-shaped handoff, 2026-07-25; docs/audit/skill-divergence-20260725.md).
The next stage branches off post-merge `main`. `main` is
trunk-of-record only for stages that have merged (precedents: Stage 2 -> PR #1 `e468d2e`
2026-07-15; Stage 3 -> merge commit 2026-07-21, vr1-dc0 scope, dc1 HELD under G12). See
`docs/dc-dc-deployment-workflow.md` "Cross-cutting discipline".

**Incident loop:** capture the verbatim error -> `bash scripts/cloud-assert.sh`
(the service-own-verdict sweep localizes the layer) -> appendix-A by exact
message -> recorded fix, gated -> log the finding (new root causes become
appendix-A/DOCFIX material).

## Record authority and session discipline (GA-R1..R7, ratified 2026-07-18)

Ratified text: repo `docs/audit/ga-rulings.md` -- this is the operating summary.

- **`docs/CURRENT-STATE.md` is the ONLY status authority (GA-R1).** Stage,
  gates, plan counts, version pins, decision-queue state live THERE and nowhere
  else; other surfaces point or are history. Any commit that changes a status it
  carries updates it in the SAME commit (C1; repo-lint L10 enforces). It is
  authoritative over documents, never over measurement -- a fresh capture wins
  and corrects it in the capture's commit (C2).
- **Plan counts and versions are quoted only from captured command output** --
  a dated capture file or a quoted command + output; never from memory or
  another document's prose (GA-R1 rule 2). Legal stage/gate status tokens are
  OPEN / BLOCKED / HELD / CLOSED, used only in CURRENT-STATE (vocabulary ruling,
  Option A: readiness is not a state; event words are history, never states).
- **Rulings are gated like mutations (GA-R5).** ONE decision per operator
  exchange -- batch adoptions are invalid. A ruling exists only when its Status
  line quotes the question AND the operator's exact utterance, dated, committed
  and pushed before dependent work. Ambiguous/template answers rule NOTHING:
  stop and ask. Never adopt from an inferred or reconstructed ruling -- an
  operator ruling is a value under hard rule 2.
- **D-number admission (GA-R3):** a new D-number needs architectural consequence
  beyond the stage, a Roosevelt-delta (the A1 test: a Roosevelt build session
  would grep it before touching a built surface), or supersession. Everything
  else is OPERATIONAL: runbook edit / session-changelog line / as-built row,
  never a D-number. Entries carry [ARCH]/[OPS] tags; doubt resolves DOWN to OPS.
- **Memory governance (GA-R7):** auto-memory records observations and
  conventions ONLY -- never operator policy, priorities, or postures. Any
  memory claiming operator intent is PROPOSED until confirmed verbatim; remove
  on discovery. Durable observations GRADUATE to their repo surface; memory is
  never the only copy. Stage close includes a full memory review (rules 1-4).
- **Reconciliation circuit-breaker.** If you have been fixing records to match
  records for more than one cycle without touching the actual deliverable,
  STOP and report to the operator. Record-churn presenting as progress is this
  project's measured failure mode (GA-F06); escalating is the correct move.

## Posture

- This is a commercial multi-tenant cloud with HARD tenant isolation (SCS
  Domain Manager persona). Treat tenant-visible surfaces and cross-domain
  boundaries as security-relevant in every change.
- The operator community here values debate and industry best practice over
  quick fixes. Push back with sources when you disagree; own mistakes plainly
  and concisely. Fabricated flags, values, or version numbers are the cardinal
  sin - if you have not verified an option name or version, say so and verify.
- Responses stay concise. Decisions get explicit rationale.


================================================================================
## references/operating-discipline.md (loaded on demand in Claude Code; inlined here for Chat)
================================================================================

# Operating discipline

How work is executed on this cloud. These conventions exist because the
runbooks and scripts ARE the product: a fix that works but is undocumented,
unrepeatable, or Roosevelt-incompatible is a failure.

## The gated execution model

Every command block is labeled so a command line is never mistaken for prose,
and so mutation risk is explicit (same convention as the repo runbooks):

- `RUN -- <loc>` - the block CHANGES state; run it at <loc> (jumphost, a unit
  via `juju ssh`, the mgmt VM...).
- `CHECK (read-only) -- <loc>` - safe to re-run.
- `GATE:` - hard stop. Do not proceed unless the stated condition holds.
- `Expect:` - what a passing result looks like. Always state it: the operator
  should never have to guess whether output is good.
- `> CAUTION:` - destructive, secret-handling, or irreversible.

Sequence discipline: read-only audit -> present the mutation -> operator
approves/runs -> verify the result -> next step. One gated mutation at a time;
never batch destructive steps. If output comes back unexpected, STOP and
re-derive from the live state - do not improvise a correction inline.

**Chat (no shell):** you prepare blocks, the operator runs and pastes output.
Treat pasted output as the only evidence; a block you wrote but saw no output
for did not happen.

**Live shell:** you may execute `CHECK (read-only)` blocks yourself. `RUN` and
`> CAUTION:` blocks still get presented and human-approved first - state what
will change and why it is the minimal correct action.

## Irreversible / one-shot / secret steps

- Start every consequential session with `bash scripts/run-logged.sh <label>`
  (opens a script(1)-logged shell to ~/as-executed/; index the session in
  logs/as-executed-index.md -- content NEVER commits, the index always does).
- Retrieve the exact prior working command from the as-executed log or runbook
  VERBATIM. Never improvise vault init, unseal, authorize-charm, CA issuance,
  or anything one-shot. (DOCFIX-006: a mis-redirected `vault operator init`
  loses the unseal keys forever.)
- Secrets never transit argv, clipboard, scrollback, or /tmp. Capture straight
  to a 0600 file under $HOME with `umask 077`; unseal via hidden prompt (L4);
  transfer via base64 pipe into a root-written 0600 file (L-P6-4). Never echo
  a secret to verify it - verify by length/format from the file.
- Never run `maas list` (prints the API key - DOCFIX-016). Never trust a juju
  action's human-formatted output for a captured secret or cert - pull from
  `--format json` (indented YAML block-scalars corrupt PEMs; DOCFIX-021).
- Authorize charms with short-lived child tokens: `juju run` persists action
  params in the operation log (DOCFIX-011).

## Record-keeping: D-NNN / DOCFIX-NNN / BUNDLEFIX-NNN

- `D-NNN` - design decisions (`docs/design-decisions.md`). Append-only:
  superseded entries stay in place, marked, with the superseding entry
  appended. `DOCFIX-NNN` - runbook fixes. `BUNDLEFIX-NNN` - bundle fixes.
- ALWAYS grep the repo for the next free number before assigning one -
  collisions have happened. State "next-free verified" when you assign.
- PARALLEL WORK numbering: when more than one workstream edits the shared numbered
  ledgers (this repo + a parallel Claude Code stream), do NOT each independently grab
  "next-free" -- that collides. One stream owns a given number/block; each DECLARES every
  identifier it consumes in its commit; the other re-greps AFTER the parallel push lands
  and resumes above it. `scripts/ledger-scan.sh` derives next-free from decision HEADERS
  (not prose mentions -- a "next-free D-072" pointer must not inflate the count).
  Keep "Next-free:" pointer lines on ONE line (a word-wrapped continuation escapes the
  line-based exclusion); or just rely on the scan, which is the next-free authority.
- Mid-task findings are logged as proposals, not acted on (hard rule 1).
  A finding that changes a runbook becomes a DOCFIX; one that changes
  architecture becomes a D-NNN with status PROPOSED until the operator rules.
- Status vocabulary: PROPOSED/OPEN -> ADOPTED/DECIDED -> SUPERSEDED (by
  D-MMM). Do not implement PROPOSED items.
- Before acting on any change request, grep design-decisions for the
  governing decision on that surface (and its dependents - e.g. a runbook
  step may rely on the current state). Finding a PROPOSED/OPEN decision
  means presenting its recorded options for a ruling, not picking one.

## Delivery rules (files handed to the operator)

- Multi-file changes ship as repo-relative ZIPs, never loose files (the
  Windows/GitHub Desktop loose-file workflow misplaces them).
- Everything committed is ASCII-only, LF-only. Validate before delivery:
  non-ASCII with `grep -nP '[^\x00-\x7F]'` (or a Python byte read); CR bytes
  with a Python `data.count(b"\x0d")` - a `grep $'\r'` false-positives on
  `$r...` tokens. Non-ASCII in OpenStack config has caused silent daemon
  failures (mod_wsgi UnicodeDecodeError).
- On-box script delivery over `juju ssh` goes as a base64 pipe decoded to a
  file, then the FILE is executed - never a raw heredoc (paste-mangling and
  stdin-consumption both bite; see script-authoring on `read` vs pipes).
- Windows-side steps are PowerShell-native (no bash heredocs, no backslash
  continuations). The operator commits from Windows; the jumphost only pulls.

- Credential exposures and security obligations get a ROW in
  docs/security-ledger.md at discovery time (owner + status) - never only a
  script-comment note; the ledger is reviewed at every phase-00 and handoff.
- Under operator-granted blanket approval, the delivery contract is: implement,
  then hand over ONE cumulative repo-relative ZIP plus a changelog where every
  item states what changed, the evidence for why, and a per-item revert. The
  changelog is the review surface; opinion-weighted calls are flagged as such.

## Session continuity: the in-flight ledger

Long sessions on this cloud get COMPACTED, often several times. Work that lives only in
chat scrollback is lost at compaction. `docs/session-ledger.md` is the durable record of
what is IN FLIGHT (active build, backlog, operator-gated items, open decisions, numbering
state, state facts). Standing practice:
- Read the ledger AND run `scripts/ledger-scan.sh` at session start; reconcile.
- The scan is the DRIFT CHECK: it derives open decisions (last Status line per block, so an
  amendment closes it), OPEN security rows, and next-free numbers straight from the repo.
  The narrative ledger must not claim CLOSED what the scan shows OPEN, nor omit what it
  surfaces. A ledger without this check rots (cf. D-070's unexercised-snapshot critique).
- Update the ledger at every deliverable/commit -- it is a standing deliverable like the
  changelog. Changelog = what changed; ledger = what is still open.

## Boundary mutations: adds-before-removes with a readback gate

When tightening a security boundary (SG rules, firewall, policy) that a live flow depends on,
NEVER remove the permissive rule before the specific replacements are proven present. Create
the narrow rules, READ THEM BACK (fail if any is missing), and only THEN remove the wide rule
by its MEASURED id. There must be no window in which a required flow (e.g. the magnum
conductor -> mgmt apiserver poll) is unauthorized. Verify the tightened boundary by FUNCTIONAL
outcome afterwards, not by assuming the rule text is sufficient.

## Attestation gates for unautomatable safety items

Some acceptance items cannot be scripted -- e.g. a SECOND person performing the Vault unseal
(D-069). Never auto-pass such an item. Gate it on a ledger ROW (docs/security-ledger.md):
report PASS_PENDING_MANUAL while the row is OPEN, PASS only when it is attested CLOSED/rehearsed,
HOLD if the row is missing. Auto-passing an undone safety item is the exact failure D-070 retired.

## Debate and correction norms

- Challenge weak reasoning with sources; the operator prefers industry best
  practice over the quick fix and will push back hard on unverified claims.
  When challenged, verify (docs, source, live probe) rather than defend.
- Own mistakes plainly, in one or two sentences, then fix. No self-flagellation.
- When a prior decision looks wrong, propose superseding it through the D-NNN
  process - do not quietly deviate from it.

## Troubleshooting entry discipline

Before any server-side hypothesis for "X used to work, now it doesn't" on a
web UI: eliminate client state first (incognito window, ~10 seconds), then a
server-side curl, THEN hypothesize. Before any command that acts on tenant
resources: confirm which project/domain scope the shell holds
(`openstack token issue`-level certainty) - `server create` and friends use
ambient scope silently. Full triage method: references/troubleshooting.md.

## Guards are load-bearing: reword the content, never weaken the check

When a repo-lint guard (or any fail-closed check) goes RED on text you
believe is legitimate, the fix is to REWORD the text or add the guard's
own documented opt-out marker -- never to loosen the guard. Precedent: L3
correctly flagged the config.xml tombstones; the tombstones were reworded
and the guard kept. (Rotated from the session ledger's standing-lessons
block, 2026-07-19.)


================================================================================
## references/environment.md (loaded on demand in Claude Code; inlined here for Chat)
================================================================================

# Environment - Omega Cloud (VR0 DC0 testcloud)

Facts here are ANCHORS, not command inputs. Anything marked (verify) must be
re-measured or re-read from the repo/live cloud before use in a command -
hard rule 2 applies. Snapshot date: 2026-07. The repo is fresher than this file.

## The three deployments (this file covers VR0 in detail; VR1 is a pointer)

- **Testcloud (VR0 DC0, now):** four KVM host VMs (openstack0-3) on a single
  hypervisor, managed by MAAS + Juju. Single-DC virtual rehearsal. The rest
  of THIS file is VR0's own ground truth.
- **VR1 DC-DC (in progress, D-100..D-110):** two-DC + Office1-headend
  rehearsal -- VR0's testcloud host is being REPURPOSED into DC1 (not a
  second concurrent cloud). This file does NOT carry VR1's own facts (Office1
  network, DC1/DC2 mesh links, per-DC MAAS/Vault roots, etc.) -- those live
  in `docs/dc-dc-deployment-workflow.md` (master tracker: stage status +
  tooling gap register, read this first for any VR1 work) and
  `docs/dc-dc-buildout-design.md` (the design itself). Runbooks:
  `runbooks/README.md`'s "VR1 DC-DC track" section. IaC: `opentofu/`
  (UNVALIDATED -- see its own README).
- **Roosevelt (future):** bare-metal, multi-DC, commercial production
  (3310 Roosevelt Blvd, Eugene OR). Dedicated node roles (gateway/controller/
  compute split) - unlike the hyperconverged testcloud. Every design choice is
  judged by its transfer to Roosevelt -- VR1 is the rehearsal for THIS, not
  an end in itself.

## Stack (verify against appendix-B for pins)

Charmed OpenStack Caracal 2024.1 - Juju/MAAS/component versions live in
`runbooks/appendix-B-asbuilt-version-lock.md` (VR0) and `docs/CURRENT-STATE.md`
section 7 (VR1 substrate) - Vault TLS (charm-pki
root CA), OVN 24.03, Ceph Squid, Octavia (amphora), Barbican,
mysql-innodb-cluster, RabbitMQ, Magnum + magnum-capi-helm driver + azimuth
(SUBSTRATE below the charms -- pins + traps in `references/platform-traps.md`:
libvirt/KVM via the `dmacvicar/libvirt` OpenTofu provider `0.9.8` (tofu +
component pins: `docs/CURRENT-STATE.md` section 7, the pin authority);
**LXD pinned to the `5.21` LTS track** -- MAAS 3.6/3.7 cannot talk
to LXD >= 6.7, D-114; OPNsense edges driven by REST API, D-113; the vcloud
host is ITSELF a KVM guest, so nesting is a per-level requirement)
capi-helm-charts (kubeadm engine), in-cloud single-homed CAPI mgmt VM
(capi-mgmt-v2, k8s-snap, D-035). NetBox is the IPAM apex: never hand-edit
downstream MAAS or overlays for network values.

## Control points

- **Jumphost:** `vcloud` - all live commands run here. Has juju,
  the openstack CLI (SNAP - cannot read /tmp; use $HOME), jq, kubectl.
- **Repo:** `https://git.baldurkeep.com/OpenStack/openstack-caracal-dc-dc`
  (web) / `.../git/OpenStack/openstack-caracal-dc-dc.git` (clone) -- a
  documented ANCHOR FACT (this repo's name/path has already changed once, see
  D-110; re-verify with `git remote -v` before trusting it, same as any other
  fact in this file). repo-lint: allow-selfref. Operator commits from Windows
  (PowerShell / GitHub Desktop - strips exec bits; `.gitattributes` pins LF);
  the jumphost only pulls.
- **Session `$REPO` convention:** set `$REPO` once per session to your clone's
  root (see `runbooks/README.md` Conventions); every runbook RUN block
  references `$REPO`, never a hardcoded clone path. `repo-lint` L9 enforces
  this for scripts/RUN-blocks/skill files.
- **Juju model:** `openstack`. **MAAS profile:** `admin` (call
  `maas admin ...` directly; NEVER `maas list` - it prints the API key).
- **Management substrate (verify; NEVER touch in teardown):** the MAAS
  machines hosting juju, lxd, and tailscale are hard-excluded from teardown
  scripts. Resolve system_ids live via `scripts/lib-hosts.sh` - system_ids
  are re-minted on every re-enrollment (DOCFIX-040).

## The six network planes (D-052 / D-053; verify against scripts/lib-net.sh)

| Plane | CIDR | Carries | Notes |
|---|---|---|---|
| provider-public | 10.12.4.0/22 | Public API VIPs + tenant FIPs (Pattern A, D-060) | gw .4.1; untagged |
| metal-admin | 10.12.8.0/22 | MAAS PXE, operator/admin endpoint, default binding | gw .8.1; DC-local |
| metal-internal | 10.12.12.0/22 | ALL service-to-service control (internal API, DB, MQ, Vault, peers) | tagged VID 103 via br-internal; no gw |
| data-tenant | 10.12.16.0/22 | Tenant Geneve overlay | no gw |
| storage | 10.12.32.0/22 | Ceph public | no gw |
| replication | 10.12.36.0/22 | Ceph cluster (OSD replication) | no gw |

- API VIPs: triple per clustered charm (provider/admin/internal), matching
  last octet in the .50-.60 band, 11 clustered charms (verify count live).
- Tenant pool: 10.20.0.0/16 (hybrid model D-016 - pool in NetBox, per-project
  /24s Neutron-managed). Avoid collisions with capi-mgmt (10.20.0.0/24) and
  existing tenant /24s - list live before allocating.
- Provider NIC rule (D-057/D-060): the provider uplink must land in OVS
  `br-ex`, never enslaved to a Linux bridge, and `br-ex` carries no L3 config.

## Repo map (what lives where)

- `bundle.yaml` - the canonical bundle; VIPs/units baked in for testcloud.
- `runbooks/phase-00..08-*.md` - the gated deploy sequence, in order, each
  ending in a hard gate. `runbooks/README.md` has the label conventions.
- `runbooks/appendix-A-troubleshooting.md` - symptom->cause->fix index keyed
  by D-NNN/DOCFIX-NNN. First stop for any known-looking failure.
- `runbooks/appendix-B` - version lock. `appendix-C` - identity/RBAC.
  `appendix-D` - Magnum trust model. `ops-capi-recovery.md` - CAPI/Magnum
  post-deploy operations.
- `docs/design-decisions.md` - the D-NNN architectural record (append-only
  discipline; superseded entries stay, marked).
- `scripts/` - phase scripts + `lib-net.sh` / `lib-hosts.sh` (pinned values)
  + tenant onboarding/acceptance. `tests/<script>/` - offline fakebin
  regression harnesses.
- `policies/domain-manager-policy.yaml` + `policies/overrides.zip` - the SCS
  Domain Manager RBAC override (D-051/D-064); the zip ships IN the bundle
  (keystone resources, DOCFIX-071) and provider-bundle-check drift-guards it.
- Operational tooling (2026-07 hardening set): `scripts/preflight.sh` (single
  pre-deploy gate: lint -> bundle invariants -> Charmhub channel assert -> live
  MAAS pre-flight), `scripts/repo-lint.sh`/`repo_lint.py` (static hygiene,
  L1-L6), `scripts/cloud-assert.sh` (behavioral verifier + `--capture` BOM to
  `asbuilt/<date>/`), `scripts/run-logged.sh` (as-executed session logger),
  `scripts/channel_assert.py`. `runbooks/ops-restart-procedure.md` (full-cloud
  restart). `docs/security-ledger.md` (exposure/obligation rows).
  `logs/as-executed-index.md` (committed index; log content stays jumphost-only).
- No KVM snapshot restore path exists (D-070 superseded D-012):
  rebuild-from-runbooks IS the restore path; baselines come from cloud-assert
  `--capture`.
- **VR1 DC-DC additions (2026-07-09/10), not part of VR0's map above:**
  `docs/dc-dc-deployment-workflow.md` (master tracker), `docs/dc-dc-buildout-
  design.md` (design), `docs/design-decisions.md` D-100..D-110,
  `runbooks/dc-dc-phase0..6-*.md` + `dc-dc-teardown-rollback.md`,
  `opentofu/` (libvirt/MAAS/OPNsense IaC, UNVALIDATED -- see its own
  README), `netbox/dc-dc-prefixes-import.py`, `scripts/dc-dc-*.sh` (MTU/Ceph
  budget calculators, Ceph replication tooling).

## Identity / tenancy model (see appendix-C/D and D-051, D-064, D-066)

Domain-per-client. Operator provisions: domain + a domain `manager` (SCS
Domain Manager persona - the plain `admin` role is NOT domain-confinable) +
quotas. The tenant self-services everything inside: projects, users, roles
(only member + load-balancer_member assignable - never admin/manager),
app credentials, networks, templates, clusters. Magnum mints per-cluster
trust app-creds carrying the trustor's roles frozen at mint time (D-039:
trustor needs load-balancer_member or CAPO 403s on Octavia). Cluster create
must run as a password identity, not an app-cred (trust-creation block,
D-066). Every identity command is DOMAIN-QUALIFIED (`--domain`,
`--user-domain`, `--project-domain`) - scope-default resolution silently
lands in the wrong domain and 404s misleadingly.


================================================================================
## references/platform-traps.md (loaded on demand in Claude Code; inlined here for Chat)
================================================================================

# Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense

The substrate BELOW the cloud (the hypervisor, the provisioner, the edge
routers) has its own version boundaries and its own silent-failure modes.
Every item below is either (a) sourced to a vendor/upstream URL, or (b)
MEASURED on this project and sourced to the repo's own D-NNN / changelog.
Nothing here is generic advice: each entry exists because it cost, or would
have cost, a session.

Companion files: `references/opentofu-provider-docs.md` (provider schema +
fetch methodology), `references/troubleshooting.md` (method). Start at the
verbatim-error index at the bottom if you have an error string in hand.

## Version pins that actually matter (and why)

| Thing | Pin | Why the pin exists |
|---|---|---|
| `dmacvicar/libvirt` provider | `0.9.8` | 0.9.0 was a compatibility-BREAKING rewrite; anything remembered from 0.8-era examples is wrong (see below) |
| OpenTofu | live pin: CURRENT-STATE.md section 7 | Stage-1 as-run value is historical (git history) |
| LXD | **`5.21` LTS track** | MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114; see below) |
| MAAS | 3.7.2 (VR0) | the LXD ceiling above is a property of MAAS 3.6/3.7 |
| OPNsense | live pin: CURRENT-STATE.md section 7 (26.7 as of 2026-07-18) | ISC-DHCP left core AT 26.1 -- Kea/dnsmasq only on every later release (see below) |
| Vault charm | `1.8/stable` | 1.16 is an incompatible charm, deliberately NOT an upgrade (appendix-B, D-068) |

Charm channels/revisions are appendix-B's job, not this file's. This file
covers only the substrate the charms sit on.

---

## 1. dmacvicar/libvirt provider (pinned 0.9.8)

### 1a. The 0.8 -> 0.9 break invalidates most examples you will find

v0.9.0 (2025-11-08) is a deliberate, compatibility-breaking rewrite on the
plugin framework, with the design principle "HCL maps almost 1:1 to libvirt
XML"; the legacy provider stayed on the `v0.8` branch. v0.9.1 (2025-11-30)
then code-generated the schema from libvirt's XML schema, renaming attributes
again (`unit` -> `memory_unit`, `os.arch` -> `os.type_arch`, `os.machine` ->
`os.type_machine`, `os.kernel_args` -> `os.cmdline`, camelCase -> snake_case
throughout).
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.0
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1

**Consequence:** blog posts, StackOverflow answers, and model memory almost
all describe 0.8. Nested structures are attribute-style objects
(`os = { ... }`, `devices = { ... }`), not HCL blocks. Confirm against
`docs/resources/*.md` + `examples/*.tf` at the pinned tag, or
`tofu providers schema -json`.

### 1b. Bare `memory` is KiB, not MiB -- 1024x too little RAM (DOCFIX-188)

The 0.9.1 migration guide states the rule explicitly: "Value/unit pairs are
explicit -- whenever libvirt exposes a value with a unit attribute the
provider now has two attributes (`memory` + `memory_unit`, `capacity` +
`capacity_unit`, etc.). **Leaving the unit unset lets libvirt use its
default.**"
(https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1)

And libvirt's default is KiB: "The units for this value are determined by the
optional attribute `unit`, which defaults to 'KiB'"
(https://libvirt.org/formatdomain.html#memory-allocation).

So `memory = 2048` with no `memory_unit` is **2 MiB**, not 2 GiB. In 0.8,
`memory` meant MiB -- that is exactly how this shipped here.

    memory      = var.memory_mib
    memory_unit = "MiB"      # REQUIRED. Never omit.

Signatures (all measured, 2026-07-12): QEMU cmdline `-m size=2048k`;
`virsh dominfo` -> `Max memory: 2048 KiB`; a FreeBSD guest echoes
`/boot.config` (262 bytes of serial output) and then triple-faults handing off
to `/boot/loader`. `tofu validate` CANNOT catch it (the attribute is
optional) -- `scripts/opentofu-validate.sh` check **S1** does.
Evidence: `docs/archive/changelogs/changelog-20260712-libvirt-memory-unit-rootcause.md`,
`docs/archive/incident-20260712-opnsense-edge-boot-triplefault.md`.

The provider's own current domain examples now carry `memory_unit = "MiB"`
(`docs/resources/domain.md`) -- a fresh reader of the docs would get this
right; a reader of any 0.8-era example would not.

### 1c. No `features` block => `acpi=off` => FreeBSD panics (DOCFIX-190)

The whole `features` attribute is optional, so omitting it is schema-valid --
and libvirt then renders the machine with `acpi=off` (measured on the QEMU
cmdline: `-machine pc-i440fx-noble,...,acpi=off`).

    features = {
      acpi = true
      apic = {}          # acpi is a Boolean; apic is a nested object
    }

(schema: `docs/resources/domain.md`, `features.acpi` (Boolean),
`features.apic` (Attributes).)

Three distinct consequences, only one of which is loud:
- FreeBSD/OPNsense **panics** (it finds the local APIC via ACPI's MADT and
  ships no `atpic` fallback): `panic: running without device atpic requires a
  local APIC` -> `db>` prompt -> domain shows `running` while burning 100% of
  one core forever.
- Linux guests boot but degrade: no clean ACPI shutdown/reboot signalling.
- **MAAS drives power off/on via ACPI**, so a MAAS-managed node VM without
  ACPI can only ever be hard-stopped. That would have been a nasty Stage-3
  debug.
Evidence: `docs/archive/changelogs/changelog-20260712-libvirt-acpi-kernel-panic.md`.
Guard: `scripts/opentofu-validate.sh` check **S2**.

### 1d. No `cpu` block => generic CPU => NO `svm` => nested KVM impossible

Found 2026-07-13. With no `cpu` attribute, libvirt hands the guest a generic
emulated CPU model with no virtualization flag. Measured: host is an **AMD
EPYC 9965**, the default-CPU guest was handed an **`Opteron_G3`**. Nothing
errors -- the guest just has no `/dev/kvm` and no `svm` in `/proc/cpuinfo`.

Ubuntu's own nested-virt guide states the guest-side requirement: set the
guest `cpu mode` to `host-model` or `host-passthrough` for the guest to see
`svm`/`vmx`
(https://ubuntu.com/server/docs/how-to/virtualisation/enable-nested-virtualisation/).

    cpu = {
      mode = "host-passthrough"
      # per-caller: pass svm through, or disable it as router hardening
      features = var.expose_nested_virt ? [] : [
        { name = "svm", policy = "disable" }
      ]
    }

**The host side is a SEPARATE requirement and bites at every level.** The
vcloud host is ITSELF a KVM guest (`systemd-detect-virt` -> `kvm`), so nesting
must be enabled at each level, not just ours:
- check: `cat /sys/module/kvm_amd/parameters/nested` -> `Y` or `1`
  (Intel: `/sys/module/kvm_intel/parameters/nested`)
- persist: `options kvm-amd nested=1` in a file under `/etc/modprobe.d/`
  (same source as above)

Why this matters here (D-114): LXD **virtual machines** are qemu/KVM guests,
so MAAS composing service VMs inside a site containment VM needs working
nested KVM at that depth; and `nova-compute` on the DC nodes needs it one
level deeper still. `modules/node-vm` still has the missing-`cpu`-block defect
-- logged, NOT fixed, because DC1 is gated behind Office1
(`docs/archive/changelogs/changelog-20260713-d114-voffice1-nested-virt.md`).

### 1e. "will be updated in-place" DOES NOT mean "the guest stays up"

A plan line reading `libvirt_domain.vm will be updated in-place` **bounced a
live guest** (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and
no DHCP on the Office1 edge). "In-place" is a statement about the Terraform
RESOURCE, not about the domain.

Upstream confirms the mechanism: the v0.9.4 release notes describe domain
update as "updating a domain (e.g. changing memory) **would undefine and
re-define it**" (the 0.9.4 fix was only to preserve NVRAM/TPM files across
that undefine), and v0.9.6 added "configurable update shutdown behavior so
domain updates can request guest shutdown and wait with a configurable timeout
before forcing a stop."
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4
- https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.6

**Rule: treat ANY `tofu apply` whose plan touches a `libvirt_domain` as an
OUTAGE of that guest. Schedule and gate it as one.** Read the plan's resource
list before applying; `0 to change` on the domains is the only assurance that
a running guest is not about to bounce
(`docs/session-ledger.md`, standing lesson 2).

Corollary (DOCFIX-187/DOCFIX-188): some domain attributes are CREATE-time to
libvirt (max boot memory, `machine`, `cpu`). The provider will happily plan an
in-place update that libvirt then ignores. If a changed value does not show up
in `virsh dominfo`, `destroy` + `undefine` the domain and re-apply (volumes
survive an `undefine`).

Second corollary (measured 2026-07-20/21): the undefine/redefine also
**REGENERATES every UNPINNED NIC MAC**. The plan reads `0 to add, N to change,
0 to destroy`, the apply succeeds -- and every touched domain now has brand-new
MACs. Anything that keys on those MACs is silently stranded: in the 2026-07-21
commissioning incident all 9 MAAS-enlisted nodes became UNKNOWN at PXE (no
"Performing PXE boot" event, silent 30-min timeouts), because a serial-console
retrofit apply had drifted every boot-interface MAC. **Rule: any domain whose
MACs are load-bearing (MAAS-enlisted nodes above all) carries config-pinned
MACs** -- `mac = { address = ... }` per interface; see `modules/node-vm`'s
`interface_macs` + `tests/node-vm`. VR-only trap: Roosevelt metal has fixed
MACs.

### 1f. Two more upstream behaviours worth knowing (from the release notes)

- `capacity_unit = "GiB"` on a volume caused **"Provider produced inconsistent
  result after apply"** on every apply before 0.9.4 (libvirt normalizes to
  bytes). Fixed in 0.9.4 -- if you see that error string, check the provider
  version first.
- `tofu destroy` of a **`dir` pool** used to DELETE the backing directory
  (`StoragePoolDelete`). Since 0.9.4 the directory is preserved by default,
  with an explicit `destroy.delete = true` to opt back in. Relevant to
  `modules/dc-storage-pool`.
  (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4)
- **UNVERIFIED:** the 0.9.4 notes name a `destroy.shutdown.timeout` option
  while `docs/resources/domain.md` documents `destroy = { graceful, timeout }`,
  and I could not find the 0.9.6 update-shutdown attribute documented at all.
  Before relying on either, read the real schema:
  `tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'`.

### 1g. `forward=bridge` networks + `br_netfilter`: a global FORWARD-drop silently kills bridged frames (D-125)

A `libvirt_network` with `forward = { mode = "bridge" }` (dmacvicar schema:
`bridge = { name = "<host-bridge>" }`, no `ips`, no NAT) attaches guest taps to a
**pre-existing** host bridge -- libvirt does NOT create the bridge, so it must
already exist (declare it in the host's netplan, not the tofu) or the network
fails to start. This is the Model B "bridge-in" WAN path (D-125): the inner
OPNsense WAN rides `br-vr1-dc0-wan` out through `vvr1-dc0`'s IP-less uplink NIC to
the single vcloud ISP NAT.

Two silent-failure traps ride along:

- **`br_netfilter` makes bridged frames traverse the L3 `FORWARD` chain.** So a
  GLOBAL forward-drop -- `iptables -P FORWARD DROP`, or an nftables forward hook
  with `policy drop` and no interface match -- silently kills the WAN bridge even
  though it is "just L2". Keep any DC-LOCAL forward-drop (SEC-010) **scoped to the
  transit interface** (`oifname/iifname "<transit>"`), never global. Note
  `net.ipv4.ip_forward=0` is L3-only and does NOT affect pure bridging -- do not
  reach for it expecting to gate a bridge, and do not assume `ip_forward=1` is
  required just because an older NAT-based design needed it.
- **Foreign-MAC egress across a nested bridge is unprovable until apply.** When the
  bridged guest is itself inside another guest (OPNsense inside `vvr1-dc0`), the
  outer NAT's `virbr` sees a foreign source MAC arriving on the containment VM's
  port; an anti-spoof/nwfilter drop there blackholes egress. `tofu validate` is
  clean regardless. TEST IN ISOLATION FIRST: attach a throwaway guest to the
  bridge, confirm it gets an address and pings out, BEFORE the real chain rides
  it. (D-125's fallback for this failure is double-NAT: masquerade at `vvr1-dc0`
  instead of relying on nested L2.)

## 2. AppArmor blocks qemu on any non-default pool path

libvirt's stock `abstractions/libvirt-qemu` grants qemu the DEFAULT pool path
(`/var/lib/libvirt/images`) only. A pool anywhere else is blocked by AppArmor
**even with perfect POSIX permissions**, and the failure names nothing:

- libvirt/qemu says only: `Could not open '<path>': Permission denied`
- the actual denial is in the kernel log:
  `apparmor="DENIED" operation="open" profile="libvirt-<domain-uuid>" name="<path>"`
- the domain DEFINES fine and fails at START -- so `tofu apply` can "succeed"
  into a dead guest.

Upstream (still open, provider issue #920, with the same workaround):
https://github.com/dmacvicar/terraform-provider-libvirt/issues/920

Fix, in the vendor-sanctioned override include (survives package upgrade --
do NOT edit the shipped abstraction):

    # /etc/apparmor.d/local/abstractions/libvirt-qemu
    /var/lib/libvirt/vr1/** rwk,

This is now installed by `bash scripts/prereqs/install-apparmor-libvirt.sh`
(pool parent overridable via `VR1_POOL_PARENT`), wired into
`scripts/prereqs/install-all.sh` and reported by `check-prereqs.sh`. It cost a
session on 2026-07-12 (DOCFIX-186) because it existed only as hand-applied host
state; `docs/archive/changelogs/changelog-20260713-apparmor-libvirt-prereq.md`.

## 3. MAAS + LXD

### 3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114)

Canonical's own announcement: LXD 6.7 "consolidates some API endpoints" that
MAAS's pinned **pylxd 2.3.5** cannot speak to (fixed in **pylxd >= 2.3.9**,
not yet in a MAAS release). Verbatim guidance: "We recommend using LXD <=6.6
or the 5.21 LTS release until further notice."
https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749

VR0 runs LXD **5.21.4** and is therefore safe by accident of timing; D-114
makes it a decision. **LXD 5.21 LTS is supported to June 2029**
(https://canonical.com/blog/lxd_5-21-0_lts).

**The live hazard is the snap auto-refresh, not the install.** LXD's docs:
"By default, installed snaps update automatically when new releases are
published to the channel they're tracking"; the `latest` track "typically
points to the latest feature release", is "a continuously rolling release
track", and is "not recommended for general use"; a bare `snap install lxd`
uses the most recent LTS track, "which is currently 5.21".
https://canonical.com/lxd/docs/latest/reference/releases-snap/
=> install/refresh explicitly on `5.21/stable` and verify the tracked channel
(`snap list lxd`, `snap info lxd`) before blaming MAAS for an LXD it can no
longer talk to.

### 3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off

"Bridges created by LXD are managed, which means that in addition to creating
the bridge interface itself, LXD also sets up a local `dnsmasq` process to
provide DHCP, IPv6 route announcements and DNS services to the network"
(`ipv4.dhcp` defaults to **true**).
https://canonical.com/lxd/docs/latest/reference/network_bridge/

MAAS's own LXD VM-host procedure therefore has you disable exactly that,
because MAAS must be the DHCP/PXE authority for the machines it composes:

    lxc network set lxdbr0 dns.mode=none
    lxc network set lxdbr0 ipv4.dhcp=false
    lxc network set lxdbr0 ipv6.dhcp=false

https://canonical.com/maas/docs/how-to-manage-machines

This is the concrete form of D-114's DHCP warning. On `office1-local`,
**OPNsense/Kea is authoritative** (10.10.0.0/24, pool .100-.199). Two DHCP
servers on one L2 is an intermittent, genuinely unpleasant failure. Never
bridge `lxdbr0` onto the site LAN without disabling its DHCP first.

### 3c. Two different LXDs in this cloud -- do not conflate them (D-114)

Conflating these caused a real, recorded error:
- **Juju-created LXD CONTAINERS** on the OpenStack nodes (`lxd:N` placements
  in `bundle.yaml`, 21 of them). MAAS never sees these and never needs to.
- **A MAAS-registered LXD KVM host** (the VR0 `lxd` machine) into which MAAS
  **COMPOSES VMs** (`maas $PROFILE vm-host compose ...`) -- VR0's `tailscale`
  machine is one. These ARE MAAS machines: enlisted, commissioned, deployed,
  powered, released.

D-114 extends the second pattern per site; the first is explicitly out of
scope. Also note: MAAS only sees VMs in the LXD **project** the VM host was
registered with -- VMs in other projects are invisible to it and unaffected
(MAAS LP#1923251, https://bugs.launchpad.net/maas/+bug/1923251).

**Composed-VM networking -- MEASURE, do not assume.** A MAAS-bundled copy of
the MAAS docs states: "When composing a virtual machine with LXD, MAAS uses
either the 'maas' LXD profile, or (if that doesn't exist) the 'default' LXD
profile. The profile is used to determine which bridge to use."
(served by a MAAS instance's own docs:
https://maas.cloud.cbh.kth.se/MAAS/docs/cli/how-to-use-lxd.html -- **I could
NOT locate this sentence on canonical.com/maas.io today; the docs site has been
reorganized and the old URLs 404. Treat the profile->bridge mechanism as
PLAUSIBLE-BUT-UNCONFIRMED against a primary source.**) Since a composed machine
that PXEs on the wrong L2 is the failure this predicts, do the cheap
measurement instead of trusting either source: `lxc profile show maas` /
`lxc profile show default` on the LXD host, and confirm the NIC's parent bridge
before composing.

Because LXD **VMs** (not containers) are qemu/KVM guests, this pattern needs
nested KVM at whatever depth the LXD host sits -- see 1d.

## 4. OPNsense edge (traps entered at 26.1; they persist on the live 26.7)

### 4a. `config.xml` is GUI-owned. Do not hand-author it. (D-112, D-113)

The config-xml path is **DELETED** (template, renderer, ISO builder, and the
module's `config_seed` volume + cdrom disk -- D-113 amendment, 2026-07-13).
Configure the edge over the **REST API** (`scripts/opnsense-api.sh`); mint an
API key with `scripts/opnsense-bootstrap-apikey.sh`. Three separate defects
(no sshd, no console, an inert `<dhcpd>` block) all traced to hand-writing
that file, and a full-config push DROPS ~667 migration-populated elements --
including the box's only two firewall pass rules -- surviving only on an
UNDOCUMENTED self-heal at boot. If you think the API cannot express something,
that is a finding to raise against D-113, not a reason to write XML.

Two traps that follow:
- **A config ISO can NEVER be read on a nano image** (D-112, root-caused in
  upstream `opnsense/core:src/sbin/opnsense-importer`: the importer probes for
  a read-only root and `bootstrap_and_exit 0`s on a pre-installed image before
  scanning any media). Any instruction to build one is a trap.
- Factory default: LAN `192.168.1.1/24`, **SSH disabled**, **all inbound
  blocked on WAN**. An unanswered ping to the WAN address is NOT a fault.

### 4b. ISC dhcpd is GONE from core in 26.1 -- an `<dhcpd>` config is inert

26.1 release notes: "ISC-DHCP moves to a plugin. It will be automatically
installed during upgrades. **It is not installed on new installations because
it is not being used**, but you can still install and keep using it." Also:
"Dnsmasq is now the default for DHCPv4 and DHCPv6 as well as RA out of the
box."
https://docs.opnsense.org/releases/CE_26.1.html

So on a fresh 26.1-or-later image: an ISC `<dhcpd>` block configures nothing (that was
DOCFIX-193), and the GUI default backend is **dnsmasq**, not Kea. This edge
deliberately runs **Kea** (10.10.0.0/24, pool .100-.199, first real lease
served to `voffice1` 2026-07-13).

### 4c. REST API: auth, endpoints, and the reconfigure step everyone forgets

- Auth is **HTTP basic with an API key/secret pair**, not the root password:
  "API access is part of the local user authentication system, but uses
  key/secret pairs to separate account information from machine to machine
  communication." URL pattern: `https://<host>/api/<module>/<controller>/<command>`.
  https://docs.opnsense.org/development/how-tos/api.html
- Kea DHCPv4 (exact paths):
  `/api/kea/dhcpv4/get`, `/set`, `/add_subnet`, `/set_subnet/$uuid`,
  `/del_subnet/$uuid`, `/add_reservation`, `/set_reservation/$uuid`,
  `/get_reservation`, `/add_option`; service control:
  `/api/kea/service/reconfigure|restart|start|stop|status`.
  **A `set*` call only edits config -- it is NOT live until you POST
  `/api/kea/service/reconfigure`.** A silently-unapplied change looks exactly
  like a change that did not take.
  https://docs.opnsense.org/development/api/core/kea.html
- **Firewall rules have a self-rollback safety net -- use it.** The filter
  controller exposes `savepoint`, `apply/$rollback_revision`,
  `cancel_rollback/$rollback_revision`, `revert/$revision`: "When calling
  `savepoint()` a new config revision will be created and the timestamp will be
  returned for later use. **If the `cancelRollback(savepoint)` is not called
  within 60 seconds, the firewall will rollback to the previous state**."
  For any remote rule change on a router you reach THROUGH that router, a
  savepoint means a mistake self-reverts instead of locking you out.
  https://docs.opnsense.org/development/api/core/firewall.html
- Do NOT re-implement vendor internals to bootstrap. OPNsense stores API
  secrets as `crypt(secret,'$6$')`; we deliberately do not mint keys offline --
  a format drift would fail SILENTLY (keys that never authenticate). Call the
  vendor's own model instead (`scripts/opnsense-bootstrap-apikey.sh`).
  (D-113 amendment.)

### 4d. root's shell is tcsh -- `$(...)` dies QUIETLY over SSH

Measured on the Office1 edge: root's shell is **tcsh**. tcsh has no `$(...)`
command substitution (it uses backquotes); a `$(...)` inside a quoted remote
command is rejected with:

    Illegal variable name.

It already cost a defect: a pre-install snapshot silently no-op'd and the
install then ran with no rollback point behind it
(`docs/archive/changelogs/changelog-20260713-office1-dhcp-apply.md`). Because the step was
non-fatal, nothing stopped.

**Always feed remote commands to `sh -s`:**

    ssh -i "$OPNSENSE_SSH_KEY" root@<edge> 'sh -s' <<'SH'
    ...POSIX shell here...
    SH

(tcsh's substitution syntax: https://man.freebsd.org/cgi/man.cgi?query=tcsh)

---

## Verbatim error -> cause index (grep this first)

| Exact string you see | Cause | Where |
|---|---|---|
| guest triple-faults right after echoing `/boot.config`; `virsh dominfo` -> `Max memory: 2048 KiB`; QEMU `-m size=2048k` | `libvirt_domain.memory` with no `memory_unit` -- KiB, not MiB | 1b |
| `panic: running without device atpic requires a local APIC` (then `db>`, 100% of one core, domain "running") | no `features` block -> `acpi=off` | 1c |
| guest has no `/dev/kvm`; `/proc/cpuinfo` has no `svm`; CPU model reads `Opteron_G3` on an EPYC host | no `cpu` block -> generic emulated CPU | 1d |
| `Could not open '<path>': Permission denied` (domain defines, then fails to START; nothing names AppArmor) -- check `dmesg` for `apparmor="DENIED" operation="open" profile="libvirt-<uuid>"` | non-default libvirt pool path not granted in AppArmor | 2 |
| `Provider produced inconsistent result after apply` on a volume with `capacity_unit` | provider < 0.9.4 | 1f |
| plan says `will be updated in-place`, guest uptime resets anyway | provider undefines + redefines the domain | 1e |
| MAAS commissioning times out at 30 min, NO PXE event for a known machine | an in-place apply regenerated unpinned NIC MACs; MAAS records stale | 1e |
| `Illegal variable name.` from an `ssh root@<edge> '...'` | root's shell is tcsh; `$(...)` unsupported | 4d |
| OPNsense boots on FACTORY DEFAULTS with a correct config ISO attached; console shows `>>> Invoking import script 'importer'` and nothing more | the Configuration Importer can never fire on a nano image | 4a |
| a Kea/firewall API `set` returns success but the box's behaviour does not change | no `service/reconfigure` (or no firewall `apply`) call | 4c |
| MAAS cannot talk to an LXD VM host after an LXD update | LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS | 3a |
| a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2 | `lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it) | 3b, 3c |

---

## NetBox (upstream, `netbox.baldurkeep.com`)

### `HTTP 403 Forbidden` on every API call -- but the SAME token works from `curl`

**Symptom.** `urllib`/`requests`/`pynetbox` get `403 Forbidden` from the upstream NetBox on every
endpoint, including ones a valid token plainly reads. `curl` with the identical token and URL
returns `200`. It reads exactly like a bad or revoked token; it is not.

**Cause.** Something in front of upstream NetBox (WAF / reverse proxy) filters on **User-Agent**.
The default Python UA (`Python-urllib/3.12`) is blocked; `curl/8.5.0` is allowed. Measured
2026-07-13: same token, same URL, UA the only variable -- `None -> 403`, `curl/8.5.0 -> 200`.

**Fix.** Send an accepted `User-Agent` header. This applies to **`pynetbox`** too (it sends
`python-requests/...`), so the repo's NetBox tooling must set it:

```python
nb = pynetbox.api(url, token=tok)
nb.http_session.headers["User-Agent"] = "curl/8.5.0"
```

**Do not** diagnose this as a token/permission problem and go re-mint credentials -- that is the
trap. Confirm with `curl` first; if curl works and Python does not, it is the UA.

**Note:** this is an UPSTREAM-only trap so far. The Office1 sandbox NetBox (4.6.4, no proxy in
front) accepts the default Python UA. Upstream is 4.5.8 and also uses **v1** (40-char, bare) tokens,
while the sandbox is 4.6 and uses **v2** (`nbt_<key>.<plaintext>`). Any tool pointed at BOTH must
handle both auth shapes and the UA -- do not assume they are the same NetBox.


================================================================================
## references/script-authoring.md (loaded on demand in Claude Code; inlined here for Chat)
================================================================================

# Script authoring - house style and hardening

Every script and paste block in this project follows these rules. They are
not style preferences: each one encodes a failure that actually happened.
Read this in full before writing ANY bash or python for this cloud.

## Zeroth decision: does it already exist?

Before writing ANY operational script or check block, search the repo:
`grep -rli <topic> scripts/ runbooks/` - the deploy/verify surface is heavily
scripted and duplicating an existing script creates drift (e.g. the haproxy
backend sweep already exists as `scripts/phase-03-core-verify.sh` 3.1b).
Route to, extend, or fix the existing artifact; write new only when nothing
covers the need, and say which search came up empty.

## First decision: what kind of block is this?

The error-handling regime depends on execution mode. Choosing wrong is itself
a bug:

1. **Executed script** (`bash script.sh`, own process): `set -uo pipefail`
   at minimum; `set -e` acceptable and usually right, with the capture
   caveats below. Exit codes are the interface.
2. **Interactive paste block** (operator pastes into their shell): NEVER a
   bare `exit` (it kills their shell) and no `set -e` (it can kill their
   shell's options or abort mid-paste). Wrap the whole block in a subshell
   `( { ...; } )` so a stray exit is contained; signal failure by printing
   `FAIL: ...` lines the operator can read.
3. **Verify/count-gate block** (greps that legitimately return zero matches):
   run WITHOUT `set -e`, and end every count-grep with `|| true` - a zero
   count is a valid answer, not an error (L1). `bash -n` cannot catch this;
   it is behavior, not syntax.

## The header contract (executed scripts)

Every script opens with a comment block stating: path + argument synopsis;
what it does (one paragraph, referencing the D-NNN/DOCFIX it implements);
**whether it mutates anything** ("Mutates NOTHING" / what it changes and the
gate protecting it); usage line; exit-code contract; and "ASCII + LF".
House exit codes: `0` PROCEED, `1` HOLD (a gate failed), `2` precondition
missing (tool absent, wrong model, helper not found). Then:

    set -euo pipefail            # or set -uo pipefail; see regime above
    shopt -s inherit_errexit 2>/dev/null || true
    IFS=$'\n\t'

Source shared constants instead of restating them:
`. "$SCRIPT_DIR/lib-net.sh"` (planes, VIP bands, helpers) and `lib-hosts.sh`
(hostnames, octets, system_id resolution). If a value you need is not in a
lib, consider adding it there rather than inlining a literal.

## Hardening rules (each one is a scar)

**SIGPIPE races break guards in BOTH directions.** `cmd | grep -q X` under
pipefail: on match, grep exits, the producer takes SIGPIPE (141), the pipeline
reports failure despite the match. In an `... || die` verify this FALSE-DIES on
success; in an `... && die` guard it FAILS OPEN -- the 2026-07 sweep found a
duplicate-CIDR guard that let collisions through exactly this way. Treat every
`| grep -q` on a live pipe as a defect regardless of which way the test points.

**Pipefail + SIGPIPE race.** `cmd | grep -q X` under pipefail falsely fails:
`grep -q` closes the pipe on first match, SIGPIPE (141) kills the producer
(`juju ssh` especially). Capture, then test:

    OUT=$(cmd 2>&1 || true)
    grep -q "pattern" <<<"$OUT"

**`set -e` kills id-captures silently.** `ID=$(openstack ... || die ...)` -
if the subshell exits non-zero before your handler fires, `set -e` aborts the
assignment line with no message. Append `|| true` to each capture, then
validate the captured value explicitly (see whole-output validation below).

**Whole-output validation, never extract-then-check.** Do not pipe raw output
through `awk`/`grep` to extract a field and then test the fragment - a
partial failure yields a plausible-looking fragment. Capture the WHOLE output,
validate its shape (e.g. `is_id(){ [[ "$1" =~ ^[0-9a-f]{32}$ ]]; }` for a
keystone id), and only then use it.

**Centralize `</dev/null` in a wrapper, not per call.** The house pattern is a
2-line helper (`rc()`, `rcap()`, `J()`) that appends `</dev/null 2>&1` once;
every call site then stays clean and un-forgettable. Heredoc-payload ssh
(`ssh ... bash -s <<'EOF'`) is the ONE exemption -- stdin IS the delivery there.

**Inner stdin consumption.** Any `ssh`/`sudo`/`juju ssh` inside a heredoc,
pipe, or loop eats the remaining stdin and truncates the block. Append
`</dev/null` to EVERY inner invocation (`</dev/tty` only when it genuinely
must prompt).

**`read` and `... | bash` are mutually exclusive.** A script piped to bash
has the pipe as stdin, so `read` returns empty at EOF - this once silently
created a passwordless user. For paste-safety AND working prompts: base64-
decode to a file, then run the file (it inherits the terminal stdin).

**juju invocation shape.** `juju ssh -m <model>` - the `-m` flag goes BEFORE
the target. `juju run <unit> <action>` output: use `--format json` for
anything captured; confirm long actions via `juju show-operation <N>`, not
the streamed log (a wait-timeout does not mean the hook failed).

**Snap confinement.** The openstack CLI snap cannot read `/tmp` - stage files
under `$HOME`. Same for `juju attach-resource` payloads.

**Client output ordering.** `openstack -f value -c X -c Y` returns columns in
ALPHABETICAL order, not flag order. When order matters: `-f json | jq`, or
single-column queries. After any jq returning null, run `jq 'keys'` - key
casing is command-specific (Title-Case in lists, hyphenated in quota show).

**Environment isolation.** Any block that switches identity runs in a
subshell that first unsets all OS_* vars:
`( for v in $(env | awk -F= '/^OS_/{print $1}'); do unset "$v"; done; export ... )`
Thread `OS_CACERT` explicitly into isolated subshells - it gets stripped.
Confirm scope before acting: which project/domain the token holds, not which
you intended.

**Stable keys, not drifting IDs.** Look up MAAS subnets by CIDR, machines by
hostname, CAPI CRs by LISTING then operating on the exact returned name
(the OpenStackCluster suffix is random per create - a wrong-name patch
silently no-ops). system_ids are re-minted on re-enrollment.

**Verify the launched cmdline, not the config text.** For OpenStack debs run
via LSB-init-wrapped systemd: a flag "present in a file" proves nothing.
Gate on behavior - the init script's `show-args`, `ps -ww -C <daemon> -o args`
on the live process, and probe under the daemon's RESTRICTED PATH
(`env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin sh -c 'command -v helm'`) - an
interactive shell's PATH masks daemon-PATH failures.

**sed is not a verifier.** A non-matching `sed -i` exits 0 having changed
nothing. Assert the post-edit content; never trust sed's exit code as proof
of the edit.

**ASCII + LF, validated.** Non-ASCII: `grep -nP '[^\x00-\x7F]'` or a Python
byte read. CR check: Python `data.count(b"\x0d")` - `grep $'\r'` false-
positives on `$r...` tokens. Non-ASCII in conf.d has silently killed daemons.

**Python helpers live in .py files** tested against fixtures - no inline
python-in-bash beyond one-liners. Do not assume `jq` exists off-jumphost;
gate on it (`need_jq`) or use python3.

**Secrets in scripts:** whitelist-write to 0600 files under a 0700 dir
(`umask 077` first), never echo, never argv, measure secret length from the
file rather than asserting an expected length (this deployment's app-cred
secrets are 86 chars, not the commonly assumed 43 - never hardcode either).

**Structured-output captures take stdout ONLY; never merge stderr.** `x=$(openstack ... -f json 2>&1)`
is a defect: any stderr line (a `--long` deprecation warning, an SDK notice) lands at char 0 and
kills the JSON parse -- this fail-closed a live SG audit in 2026-07. Capture stdout alone; send
stderr to a tempfile; surface it on parse failure, never discard it (`2>/dev/null` trades the bug
for silent error-masking). The `vr_json` helper in `scripts/lib-validate.sh` encodes this. Also
drop deprecated flags that only add stderr noise (`--long` is unneeded; fields are present without it).

**Fixture-test throwaway probe one-liners like real code.** A measurement `ss`/`tcpdump`/`awk`
pipeline is code, and it has the same parse traps: `ss` DROPS the State column under a
`state established` filter (peer is then `$NF`, not `$5`); `cut -d:` splits IPv4-mapped IPv6
(`[::ffff:10.0.0.1]:443`) at the wrong colon; `grep -oE 'src [0-9.]+'` greedily eats the trailing
`.` into the next field; charmed magnum logs to `/var/log/magnum/magnum-conductor.log`, NOT
journald. One measurement chain in 2026-07 shipped four such bugs before each was caught. Run the
parse against a fixture line (mapped-v6, plain-v4, the real column layout) BEFORE handing the
operator a probe -- the same discipline as a committed script.

## Testing: nothing ships on `bash -n`

`bash -n` validates parse, not behavior - and most of the failures above are
behavioral. Every nontrivial script gets an offline regression harness at
`tests/<script-name>/run-tests.sh` in the established pattern:

- `fakebin/` contains fake `juju` / `openstack` / `maas` / `kubectl` /
  `ssh` executables that replay fixture output and log the calls they
  receive; real coreutils stay real.
- The harness sets `PATH="$BIN:$PATH"`, a scratch `HOME`, and env-injected
  fixture paths, runs the target, and asserts BOTH the exit code and an
  output regex per case: `run <want_rc> <regex> <label>`.
- Cover: the happy path, each gate's failure branch (asserting HOLD not
  crash), each precondition (exit 2), and any DOCFIX behavior the script
  claims (e.g. "verify-first never creates a rule when one exists").
- Read an existing harness (e.g. `tests/phase-07-conductor-graft/`) before
  writing a new one; match its shape.

For MUTATING scripts, the harness fakebin is STATEFUL: the fake `maas`/`juju`
advances a phase file on each mutation call so post-mutation verification reads
post-mutation reality (see tests/phase-00-teardown-d061 -- canary survival,
decompose-detection, substrate-collision aborts). A `--no-prompt` flag on a
destructive script exists FOR its harness, nothing else.

**Harnesses are HERMETIC: fakes shadow, deletion un-shadows.** Never simulate
an unavailable tool by DELETING its fake -- on a real host that un-shadows the
real binary and the test falls through to live infrastructure (a jumphost
gauntlet run reached real Charmhub this way). Simulate failure with a FAILING
fake (prints a realistic error, exits nonzero); keep the fail-loud-on-unmatched
arm in every fake so an uncovered call can never silently hit a real CLI.

**A script migration commit MUST carry its harness.** The D-060 revert updated
the scripts and left their harnesses testing the retired D-058 world -- red-at-
HEAD tests that trained everyone to ignore red. If you change a script's
behavior or vocabulary, the same commit updates fixtures and expectations.

**Static contract:** `bash scripts/repo-lint.sh` must exit 0 before anything is
delivered (L1 ASCII/LF, L2 stale tokens, L3 ghost refs, L4 deprecated refs,
L5 numbering, L6 bare invocations, L7 clientdocs sweep receipt). Changing any
tenant-facing surface (tenant-*.sh, the onboarding contract, tenant runbooks,
the policy zip, clientdocs/ itself) trips lint L7 until clientdocs/ has been
reviewed for staleness and the receipt is re-recorded:
`bash scripts/repo-lint.sh --record-clientdocs-sweep` (the sweep IS the review;
recording without reviewing defeats the rule). A guard script that must NAME stale tokens
opts out per-file with a `repo-lint: allow-stale-tokens` marker in its header.
Committed binaries (e.g. policies/overrides.zip) need a `*.zip binary`
.gitattributes rule or the LF normalization corrupts them in transit.
Introducing a tool dependency (jq, python module) means adding its presence
gate in the same change.

Read-only verify scripts DETECT and report; remediation stays a gated human
step even when the fix is obvious - the script's job is evidence, the
operator's job is the mutation decision.

**Mock worlds must be TRUTHFUL to the real CLI/contract, and test expectations must match the
SPECIFIED contract.** A mock that emits a 409 must actually have the resource pre-exist; a mock of
an openstack list command must emit the real deprecation warning ON STDERR (or the stderr-separation
rule above never gets exercised); a harness assertion that contradicts the documented exit contract
(e.g. expecting a lone SKIP to lower the overall verdict when the contract says SKIP never lowers it)
is a TEST bug, not a code bug -- fix the test. Three 2026-07 "failures" were untruthful mocks or
wrong assertions, each caught by running the harness before delivery.


================================================================================
## references/troubleshooting.md (loaded on demand in Claude Code; inlined here for Chat)
================================================================================

# Troubleshooting - method, reflexes, and routing

The repo's `runbooks/appendix-A-troubleshooting.md` is the operational
symptom->cause->fix index, keyed by D-NNN/DOCFIX-NNN (it now includes the
mysql-innodb-cluster recovery signatures and a point-of-use identifier index).
ALWAYS check it before diagnosing from first principles - most failures on
this cloud have been seen before and have a validated fix.
`ops-capi-recovery.md` covers the CAPI/Magnum stack;
`runbooks/ops-restart-procedure.md` covers full-cloud recovery.
This file gives you the method and the reflexes that must fire BEFORE and
WHILE you read those.

**Substrate failures (libvirt/KVM guest won't boot, AppArmor, MAAS+LXD,
OPNsense edge):** these are NOT in appendix-A. Go to
`references/platform-traps.md` -- it ends in a verbatim-error -> cause index
keyed to exact strings (`panic: running without device atpic requires a local
APIC`, `Illegal variable name.`, a bare `Permission denied` that never names
AppArmor, `-m size=2048k`, ...). Match the string BEFORE hypothesizing.

**VR1 DC-DC track:** appendix-A above is VR0-specific -- nothing in the VR1
DC-DC track (`docs/dc-dc-deployment-workflow.md`) has been executed yet, so
there is no equivalent incident index for it (an OpenTofu/MAAS-`vm_host`/
OPNsense-edge failure will not be in appendix-A). Apply the SAME triage
method below regardless, but do not expect a pre-catalogued fix for a VR1-
specific symptom -- log a genuinely new finding as a DOCFIX/D-NNN candidate
per this repo's own discipline rather than assuming it must already be
documented somewhere.

## Triage method

1. **Reproduce and capture verbatim.** The exact error text routes you in
   appendix-A. Paraphrased symptoms match nothing.
2. **Eliminate cheap layers first.** Web UI misbehaving: incognito/client
   state (~10s) before any server hypothesis. CLI misbehaving: confirm the
   token scope (project/domain) before blaming the service - wrong ambient
   scope produces misleading 404s/403s, especially with domain-per-client
   identity (an un-domain-qualified lookup resolves in the WRONG domain).
3. **Establish what is actually true, live.** `bash scripts/cloud-assert.sh`
   first - it runs every service-own-verdict gate in one read-only sweep and
   localizes the failing layer. Then the targeted read-only audit of that
   surface before touching anything. juju status, the service's own state
   (haproxy admin socket, mysql cluster-status, vault status, ovn-appctl),
   the relevant logs. Hard rule 2: measurements, not memory.
4. **Match against appendix-A**, then design-decisions if it smells
   architectural. If matched: follow the recorded fix exactly (verbatim-
   reference rule for anything one-shot).
5. **If unmatched:** smallest-possible hypothesis, read-only test to confirm
   or kill it, and log the finding. New root causes become appendix-A /
   DOCFIX material - capturing them is part of the fix.
6. **Least-destructive first, individually gated.** Reload before restart,
   restart before rebuild, rebuild before redeploy. Never batch.

## Reflexes (internalize these; they override intuition)

- **juju-green is necessary, not sufficient.** `active/idle` is BLIND to:
  a DOWN haproxy backend (D-045 - hid a dead nova-api for 3 days), a missing
  magnum trustee domain (D-046 - magnum reports ready regardless), an
  unparsed-but-attached policy override, and per-backend/service state
  generally. Gate on the service's OWN verdict (admin sockets, functional
  probes), never on juju status alone.
- **"Reports OK while broken" generalizes.** Charm-side validation is often
  parse-level only (the keystone policy zip validates YAML, not semantics).
  Acceptance is BEHAVIORAL: prove the capability works, not that the config
  is present.
- **Distinguish down from thrashing.** A host that looks "down" to juju/OVN
  (ovsdb inactivity-probe storms) may be swap-thrashing, not down (D-040).
  Check `who -b` / `uptime` / `journalctl -k | grep -i oom` before treating
  it as an outage.
- **Read the reason field before the status field.** Magnum health UNHEALTHY
  has three signatures (D-042 amendment): reason EMPTY = conductor cannot
  reach the mgmt API (VM down?); all-Ready except infrastructure "not found"
  = the cosmetic driver-contract miss; a reason citing LB failure = real,
  check Octavia. Same status, three different responses.
- **Single-node mgmt VM does not self-heal (D-035/D-041).** Workload nodes
  wedged with the `uninitialized` taint, magnum reconcile dead, addons
  Pending -> check `capi-mgmt-v2` is ACTIVE before anything else. Manual
  start is POLICY (down is a signal), not a defect.
- **Known blast-radius traps:** `juju destroy-model` decomposes MAAS
  pod-composed machines - machine retention is `juju remove-machine
  --keep-instance` (D-061); mysql-innodb-cluster never bootstraps at
  num_units 1 - deploy at 3 (D-062); `reboot-cluster-from-complete-outage`
  is destructive against an already-healthy cluster - check cluster status
  FIRST; vault restarts sealed by design - sealed-after-reboot is expected,
  not a fault; the magnum `domain-setup` action must be re-run after every
  redeploy (D-046).
- **Green-in-the-shell, broken-in-the-daemon.** Interactive shells have a
  different PATH, env, and stdin than LSB-init daemons. Verify under the
  daemon's conditions (restricted PATH, live-process args) - see
  script-authoring.
- **The absence of an expected resource is a finding, not a gap to fill.**
  If something that should exist does not (a domain, a rule, an image
  property), find out WHY it is absent before recreating it - it may have
  been removed deliberately (check design-decisions) or its absence may be
  the actual root cause several layers up.

**Prove event-driven flows by FUNCTIONAL OUTCOME, not packet/snapshot forensics.** For a
bursty, per-operation client flow (e.g. the magnum conductor's helm/watch to the mgmt apiserver),
an instantaneous `ss` snapshot or a short `tcpdump` window will usually catch NOTHING -- the
connection is idle at rest and short-lived under load. That absence is expected, not a fault.
The right proof is a forced real operation that SUCCEEDS end-to-end (a cluster resize reaching
UPDATE_COMPLETE proves the conductor reached the mgmt apiserver through the boundary), plus a
TCP handshake from the real source. Reach for established-state or a forced functional op before
concluding a flow is broken; a service's own successful state transition outranks a captured packet.

## Escalation and blockers

- Missing vault unseal keys, lost one-shot secrets, or anything requiring
  material only the operator holds: STOP and escalate. Do not improvise
  around a recovery blocker (a vault re-init wipes the TLS plane).
- If a fix would deviate from a runbook or contradict a D-NNN: log it as a
  proposal and get a ruling. Mid-incident is exactly when discipline pays.
- Two failed fix attempts on the same hypothesis = the hypothesis is wrong.
  Step back to the audit stage; widen what you are measuring.


================================================================================
## references/opentofu-provider-docs.md (loaded on demand in Claude Code; inlined here for Chat)
================================================================================

# OpenTofu / provider documentation -- sources, versions, and fetch method

Read this before writing or extending anything under `repo opentofu/` (the
VR1 DC-DC infrastructure-as-code, D-103). Built up during the 2026-07-09
OpenTofu delivery; see `repo opentofu/README.md` for the full, per-module
account (schema notes, an "Audit pass" section, and every finding's exact
sourcing) -- this file is the index into that plus the fetch methodology,
so a future session does not have to rediscover it by trial and error.

## The core lesson: doc-summarization is lossy, real example code is not

Every provider doc page fetched this session went through a summarizing
model, which reliably got FIELD NAMES right but repeatedly lost or
mis-stated CALL SYNTAX (block vs. attribute style, exact nesting). The
`opentofu/` scaffold shipped with a real syntax bug from trusting doc
summaries alone (fixed once real example `.tf` files were fetched and
read directly -- see `repo opentofu/README.md`'s syntax-bug note). **For any
new provider/resource: find real example `.tf`/`.go` source before trusting
a doc-summary's syntax claims.** Doc-summaries are fine for "what fields
exist"; they are not reliable for "how do I write this."

## Fetch gotchas

- The Terraform/OpenTofu Registry's own doc pages (`registry.terraform.io/
  providers/.../docs`) are client-side-rendered -- a plain fetch returns only
  the page shell, no content. Go straight to the provider's GitHub repo
  instead.
- GitHub's raw file URLs need the CORRECT default branch (varies: `main` for
  some repos, `master` for others -- check `api.github.com/repos/<org>/<repo>`
  for the `default_branch` field first, don't guess).
- A general web search's own synthesized "summary" of a doc page can itself
  be wrong (encountered once this session: a search summary confidently
  named the wrong branch and file path). Treat search-result summaries as
  leads to verify, not answers -- always follow through to the primary
  source.
- When a fetch refuses to "reproduce verbatim" (copyright caution), ask for
  a specific, narrow thing instead -- "list bare field names only, no
  prose" reliably works where "reproduce this page" does not.
- `api.github.com/repos/<org>/<repo>/contents/<path>` (a JSON directory
  listing) is a reliable way to discover the REAL current file layout when
  a guessed raw-file path 404s -- doc reorganizations happen (one provider
  moved from `website/docs/r/*.markdown` to `docs/resources/*.md` between
  the versions training data and current reality disagree on).

## Providers used in `repo opentofu/`

### `dmacvicar/libvirt` (pinned `0.9.8`, confirmed 2026-07-09)

**Before writing any `libvirt_domain`, read `references/platform-traps.md`
section 1.** Four schema-level traps there have each cost real time and none
of them is a validate-time error: bare `memory` is KiB not MiB; no `features`
block means `acpi=off`; no `cpu` block means no `svm` (nested KVM silently
impossible); and "will be updated in-place" still BOUNCES the guest.

- Registry: https://registry.terraform.io/providers/dmacvicar/libvirt/latest
  (JS-rendered -- use the GitHub repo for actual content)
- GitHub: https://github.com/dmacvicar/terraform-provider-libvirt (default
  branch `main`; there is no `CHANGELOG.md` -- the release NOTES on the
  releases page are the changelog, and they are unusually informative:
  `.../releases/tag/v0.9.0` is the breaking-rewrite rationale, `v0.9.1` is the
  0.9.0->0.9.1 attribute-rename migration guide)
- **Version boundary that invalidates most examples on the internet:** 0.9.0
  (2025-11-08) is a compatibility-BREAKING rewrite (plugin framework, "HCL maps
  almost 1:1 to libvirt XML"); the legacy provider lives on the `v0.8` branch.
  0.9.1 then code-generated the schema and renamed attributes again. Anything
  0.8-era -- including model memory -- is wrong for this pin.
- Docs: `docs/resources/*.md`, `docs/data-sources/*.md` (NOT the older
  `website/docs/r/*.markdown` layout some search results/training data
  reference -- that's stale)
- **Real example `.tf` files (the higher-value source): `examples/*.tf`** --
  `domain_with_network.tf`, `alpine_cloudinit.tf` were the two that resolved
  every syntax question this session. Fetch these FIRST for any new
  resource, before the doc pages.
- Key fact: this provider code-generates its schema as a 1:1 mirror of
  libvirt's own domain/network/pool XML (confirmed via a general search of
  provider discussion, not the docs pages themselves) -- when a Terraform
  schema question stalls, libvirt's own native XML docs
  (https://libvirt.org/formatdomain.html, https://www.libvirt.org/pci-addresses.html)
  are a legitimate, stable cross-reference for what a nested attribute is
  likely to be named/shaped, since the mapping is direct.

### `canonical/maas` (pinned `2.7.2`, confirmed 2026-07-09)
- Registry: https://registry.terraform.io/providers/canonical/maas/latest
  (same JS-rendering issue -- use GitHub)
- GitHub: https://github.com/canonical/terraform-provider-maas (default
  branch `master` -- NOT `main`, confirmed by checking `default_branch`
  directly rather than assuming)
- Docs: `docs/resources/*.md`, `docs/index.md` (provider block config)
- Moved from an older `maas/maas` registry namespace -- use `canonical/maas`,
  not `maas/maas`, for anything current.
- Key lesson (see `repo opentofu/README.md`'s "MAAS registration + tc netem
  research" section for the full account): `maas_vm_host` (register a virsh
  chassis) and `maas_vm_host_machine` (MAAS composes/creates a new VM
  itself) are NOT interchangeable -- read the schema (which fields are
  INPUTS) before assuming a resource matches a design intent just because
  its name sounds related.

### OpenTofu core (language features, not a provider)
- `local-exec` provisioner: https://opentofu.org/docs/language/resources/provisioners/local-exec/
- `terraform_data` (built-in, replaces the older `null_resource` for
  provisioner-only use): https://opentofu.org/docs/language/resources/tf-data/
  (note the URL slug is `tf-data`, not `terraform_data` or `terraform-data`)
- Both pages rendered cleanly on a direct fetch (static docs, no JS-rendering
  issue) -- these were the most reliable fetches of the whole session.

### OPNsense (not a Terraform provider -- the guest OS itself)

**SUPERSEDED RESEARCH WARNING (2026-07-12/13).** `repo opentofu/README.md`'s
"OPNsense deployment research" section and the 2026-07-09 changelogs describe a
**config-ISO / Configuration-Importer** delivery path. That path is DEAD and its
code is DELETED:
- **D-112:** the Configuration Importer can NEVER fire on a pre-installed nano
  image (upstream `opnsense-importer` probes for a read-only root and exits
  before scanning media). The ISO was inert; nothing was ever going to read it.
- **D-113 (ADOPTED 2026-07-13, option a2):** OPNsense STAYS, but config moves to
  the **REST API**. `opentofu/templates/opnsense-config.xml.tmpl`,
  `scripts/opnsense-render-config.sh` and `scripts/opnsense-build-config-iso.sh`
  are DELETED, along with the module's `config_seed` volume + cdrom disk.
  **A rendered-`config.xml` push would now CLOBBER live API-managed DHCP.**

Current path: `scripts/opnsense-api.sh` (REST client) +
`scripts/opnsense-bootstrap-apikey.sh` (mints a key via OPNsense's own model
over SSH). Traps, exact endpoints, the mandatory `service/reconfigure` step, the
60-second firewall auto-rollback, and the tcsh trap: `references/platform-traps.md`
section 4.

- Official docs: https://docs.opnsense.org/manual/install.html;
  REST API: https://docs.opnsense.org/development/how-tos/api.html;
  26.1 release notes (ISC-DHCP left core; dnsmasq is the new default):
  https://docs.opnsense.org/releases/CE_26.1.html
- Still-valid finding from the 2026-07-09 research: cloud-init is unreliable on
  FreeBSD, so `modules/cloudinit-vm` genuinely does not apply to OPNsense
  (https://forum.opnsense.org/index.php?topic=42517.0), and the nano image (not
  the installer image) is the one to use.
- Historical only (the ISO mechanism these describe cannot fire on nano):
  https://github.com/opnsense/core/issues/5733, .../issues/10017.

## Where the full findings live (don't duplicate here, it will drift)

**Caveat first (2026-07-13):** `repo opentofu/README.md` predates D-112/D-113
and its "OPNsense deployment research" section still describes the DELETED
config-ISO path -- see the superseded-research warning above before acting on
anything it says about OPNsense. Its libvirt schema notes also predate the four
domain traps in `references/platform-traps.md` section 1. Everything else in it
still stands.

`repo opentofu/README.md` is the authoritative, per-module account --
schema notes, what's confirmed vs. assumed per attribute, and a dated
"Audit pass" section recording a later re-review (a real documentation
error was found and fixed there: an early claim that libvirt interface
list-order directly sets OPNsense's LAN/WAN role conflated two separate
mechanisms -- read that section before trusting anything about NIC
ordering/roles). Treat this file as the index/methodology; treat that one
as the ground truth for what each module actually does and why.