openstack-cloud-ops -- consolidated skill (Chat upload snapshot, 2026-07-25)

================================================================================

SKILL.md (main entry -- always in context)

================================================================================

name: openstack-cloud-ops

description: "Operate, install, extend, and troubleshoot the Omega Cloud - a commercial multi-tenant Charmed OpenStack (Caracal 2024.1) deployment managed with Juju and MAAS, with Vault TLS, OVN, Ceph, Octavia, and Magnum/CAPI tenant Kubernetes. Use this skill for ANY work touching OpenStack, Juju, MAAS, Magnum, CAPI, Ceph, OVN, Octavia, Keystone, Vault-for-OpenStack, tenant onboarding, or the repository this skill operates against - including writing or reviewing bash/python operational scripts, debugging failed deploys or cluster creates, runbook work, design-decision (D-NNN) discussion, and incident triage. Use it even for seemingly simple OpenStack questions: this deployment has strict operating discipline and known charm traps that make generic answers wrong."

openstack-cloud-ops

Operating skill for the Omega Cloud: a commercial, multi-tenant, tenant self-administered OpenStack cloud, built as virtual rehearsals (VR0 single-DC "testcloud"; VR1 two-DC + Office1 headend) for a future bare-metal multi-datacenter deployment ("Roosevelt"). The CURRENT phase/stage lives in repo docs/CURRENT-STATE.md ONLY (GA-R1) -- never trust a phase claim from this file or memory. The governing design constraint is MINIMIZE DELTA TO ROOSEVELT: the runbooks and scripts are primary deliverables alongside the running cloud, so transferable answers beat quick fixes.

Step 0 - locate the source of truth

This repository (GitBucket, git.baldurkeep.com) is authoritative for everything: bundle, runbooks, scripts, design decisions, as-built values. This skill is a discipline-and-routing layer OVER that repo, not a substitute for it. Never hardcode the repo's own name/path in a command -- it has already been renamed/forked once (see D-110) and will again; discover it fresh each session.

Look for a local clone: a repo dir in the working tree, or a $HOME entry containing bundle.yaml + docs/design-decisions.md. If found, confirm identity with git remote -v, then git log -1 to note HEAD and work from it.
No clone and you have shell + network: ask the operator for the exact clone URL (org/group and repo name -- do not guess or reuse one from memory; it may have moved, per this same repo's own history). The repo may be private; if the clone fails, ask the operator to provide access or the relevant files.
No clone obtainable (e.g. chat without sandbox network): say so, ask the operator to paste the relevant runbook/script, and proceed only on what is actually in front of you.

Divergence rule: if this skill and repo HEAD disagree, the repo wins - but FLAG the divergence to the operator rather than silently following either. The repo is a living draft; this skill's invariants (discipline, hardening) change slowly, its facts (IPs, versions, phase status) go stale fast.

Session bookends (GA-R1/GA-R4). OPEN: read docs/CURRENT-STATE.md in full BEFORE any other doc (it is the only status authority), then docs/session-ledger.md

bash scripts/ledger-scan.sh to recover what is in flight and reconcile it against repo ground truth (open decisions, open SEC count, next-free numbers). Re-do this after any compaction. CLOSE: a bounded ledger summary (15-line hard cap, GA-R4 rule 1); the full session body goes to docs/archive/ in the same close commit; commit and push -- a session ending without its bookend is treated as NOT durable by the next session. See references/operating-discipline.md (Session continuity).

Step 1 - detect the environment
Live shell to the jumphost / infra (Claude Code on vcloud or similar): you may RUN read-only audits directly. Every mutation remains individually human-gated - present the command, state what it changes, wait for approval. A live shell relaxes the transport, never the discipline.
Chat / no infra shell: operate the gated copy-paste model - prepare labeled blocks, the operator runs them and pastes output back. Never assume a block ran or succeeded; wait for the pasted evidence.

Read references/operating-discipline.md before doing either.

The three hard operating rules (non-negotiable)

Execute only the current runbook step, exactly as written. No added scope, no adjacent improvements, no live re-architecture mid-step. Findings and improvement ideas are LOGGED (changelog / D-NNN proposal), never executed live mid-step.
Never use an inferred value. No IP, ID, name, or scope goes into a command unless it was measured this session or carried from confirmed as-built. If a value would be inferred: stop and measure it. Never run a destructive or session-altering command from memory without confirming it is the minimal correct action for the current live state.
Prefer dynamic lookups over hardcoded literals. Discover VIPs, project names, IDs, and version sets at runtime. Where a literal is unavoidable it is tagged and centralized (scripts/lib-net.sh, lib-hosts.sh), keyed by stable identity (CIDR, hostname) - never by drifting IDs.

Corollary that governs everything: verify before mutate. A read-only audit precedes every mutation; destructive and secret-handling steps are gated individually, never batched.

Routing - where to go for what

Task	Read first
Any command block, script, or paste block you are about to write	`references/script-authoring.md`
Deploy / redeploy / teardown (VR0, single-DC testcloud)	repo `runbooks/README.md`, then the phase-NN runbook; conventions in `references/operating-discipline.md`
VR1 DC-DC buildout (two-DC + Office1 headend -- the current live mission)	repo `docs/CURRENT-STATE.md` FIRST (stage/gate STATUS lives there only), then `docs/dc-dc-deployment-workflow.md` (stage identity, links, gap register), then `runbooks/README.md`'s "VR1 DC-DC track" section for the `dc-dc-phase0..6` runbooks + `dc-dc-teardown-rollback.md`. SEPARATE track from VR0's phase-NN above -- do not conflate stage numbers between the two.
Something is broken (triage, incidents)	`references/troubleshooting.md`, then repo `runbooks/appendix-A-troubleshooting.md`
CAPI / Magnum / mgmt-VM recovery	repo `runbooks/ops-capi-recovery.md`
Deliver ANY repo change (script, runbook, doc)	run `bash scripts/repo-lint.sh` + the touched script's `tests/<name>/run-tests.sh` BEFORE handing it over
Pre-deploy gate (before add-model)	`bash scripts/preflight.sh` -- THE single entry; do not run the sub-gates piecemeal
Is the cloud actually healthy? (post-deploy, post-restart, pre-change baseline, incident)	`bash scripts/cloud-assert.sh` (add `--capture` at deploy completion for the committed BOM)
Full-cloud restart after outage/maintenance	repo `runbooks/ops-restart-procedure.md`
VR1 site unreachable after a vcloud reboot (`ssh voffice1` / `office1-*` time out)	`bash scripts/site-baseleg.sh check office1` -- `office1-local` is an ISOLATED libvirt net, so the host's base L3 leg (`10.10.0.10/24` + compose route `10.10.1.0/24 via 10.10.0.20`) is dropped every reboot; the `After=libvirtd` unit re-adds it (D-126 amendment / D-128). If MISS and the unit is not yet installed: `sudo bash scripts/site-baseleg.sh apply office1`. This is the vcloud path; the workstation reaches the same over the tailnet (D-107).
Starting any consequential live session	`bash scripts/run-logged.sh <label>` first (as-executed log; docs/as-executed-log-convention.md)
Credential exposures / security TODOs	repo `docs/security-ledger.md` -- add a row, never only a script comment
Resuming work / what is still in flight (esp. after compaction)	repo `docs/CURRENT-STATE.md` first, then `docs/session-ledger.md` + `bash scripts/ledger-scan.sh` to reconcile against repo ground truth
Tenant onboarding / tenant self-service	repo `scripts/tenant-onboard.sh` + `runbooks/tenant-onboarding-v2-DRAFT.md` + `appendix-C/D`
Network / plane / IPAM questions	`references/environment.md`, repo `scripts/lib-net.sh`, NetBox (the IPAM apex -- but during VR1 the WORKING apex is `office1-netbox`, 10.10.1.10, per DOCFIX-195; `netbox.baldurkeep.com` is a READ-ONLY v1 reference, write-back deferred to end-of-deployment)
ANY change request to a built surface	grep repo `docs/design-decisions.md` for the governing D-NNN FIRST - PROPOSED/OPEN means the operator has not ruled: present options, do not implement
Why is it built this way? / proposing changes	repo `docs/design-decisions.md` (D-NNN); grep before assigning a new number
Versions, channels, pins	repo `docs/CURRENT-STATE.md` section 7 (VR1 substrate -- measured pin authority); `runbooks/appendix-B-asbuilt-version-lock.md` (VR0 charm channels)
Environment facts (hosts, repo, planes)	`references/environment.md`
OpenTofu / libvirt / MAAS / OPNsense IaC work (repo `opentofu/`)	`references/opentofu-provider-docs.md` FIRST (sources + fetch method), then repo `opentofu/README.md` (per-module ground truth, schema notes, audit findings -- but its OPNsense section is SUPERSEDED by D-112/D-113)
Creating/changing a `libvirt_domain`; anything about nested KVM, a libvirt storage pool, LXD versions, an OPNsense edge, or a VM that boots wrong	`references/platform-traps.md` -- per-tool traps + version pins + a verbatim-error index (KiB-vs-MiB memory, `acpi=off`, missing `cpu` block = no `svm`, "in-place" bounces the guest, AppArmor on non-default pools, MAAS-vs-LXD 6.7, lxdbr0's dnsmasq, tcsh on the edge)

Standard loops (repeatable session shapes)

Session bootstrap (jumphost): Claude Code sessions launch from the repo root, always (GA-R7 rule 6 -- a session launched from a subdirectory gets its own empty project dir and a divergent auto-memory; the GA-F06 origin). git -C "$REPO" pull (set $REPO once per session -- see runbooks/README.md Conventions / references/environment.md) -> bash scripts/repo-lint.sh (0 fail expected) -> if touching the live cloud, bash scripts/run-logged.sh <label> to open the logged shell. Repo HEAD and a clean lint are the preconditions for everything else.

VR1 operating model (D-128 -- invariant; do not re-derive this each session): Claude runs ON the vcloud jumphost -- do NOT relocate the session to a workstation. Work splits in two, origins FIXED BY DESIGN: Plane 1 (substrate -- the OUTER tofu root qemu:///system + local virsh, creating the DC host VMs) runs on vcloud and has no Roosevelt analog; Plane 2 (MAAS / NetBox / the INNER tofu root qemu+ssh-from-Office1 / tenant tests) EXECUTES on voffice1 via ssh voffice1 '...' so runbooks read "from the Office1 headend" and transfer. Claude's host only picks the TRANSPORT, not where work runs. Durable Office1 reach = the D-126 base-leg oneshot (scripts/site-baseleg.sh check office1 FIRST if ssh voffice1 is dead post-reboot) -- NOT tailscale-on-vcloud (D-107 is the operator's WORKSTATION path / human GUI, left untouched). DCs sit outside the /22 carve and keep the D-126 -J-through-vvr1-dc0 shape. Durable host->site reach is standing per-SITE tooling, DCs included (so the gap does not reopen per-DC): site-baseleg.sh is site-keyed. At each DC standup, MEASURE how vcloud reaches that DC's containment/ service net -- if it is an ISOLATED host-local net, add a MEASURED leg row + install the unit as part of standup definition-of-done (NEVER an inferred DC IP -- the harness fails on an un-commented DC supernet); a routed/NAT net or an already-reachable qemu+ssh path needs none. Register item 20 tracks this. **Rack-side twin invariant (D-131 sub-1, RULED 2026-07-21): every internet-isolated DC rack gets the repo-carried scripts/dc-rack-net.sh install <site> at standup (reboot-persistent rack bridge legs

the node-DNS dnsmasq forwarder aliased on metal-admin, forwarding to region BIND over the rack's own transit) -- MAAS 3.7's rack-only agent resolver SERVFAILs on isolated racks, so WITHOUT this, commissioning silently times out (appendix-A carries the symptom pair). Two measured MAAS traps with standing consequences: MAAS virsh PODS are refuted for this repo's DCs (per-machine power_type=virsh via scripts/maas-node-power.sh is the ruled shape -- D-103/D-123 amendments 2026-07-20), and a pod delete CASCADES to its linked machine records -- read the pod's machine list BEFORE any vm-host delete; non-empty = STOP (appendix-A, 2026-07-21 incident). Per-DC power credential (SEC-012 dc0, SEC-016 dc1, RULED 2026-07-23 -- standing DC-standup invariant): each DC gets its OWN dedicated MAAS->libvirt power key, NEVER a cross-DC reuse.** The REGION's MAAS snap dials the power address, so the split is: the maas-node-power SCRIPT's mapping virsh uses the DC SERVICE key (via the region's ~/.ssh/config Host ), while MAAS's own power ops use the dedicated key installed in the snap (/var/snap/maas/current/root/.ssh/ + a per-host ssh config block so each rack uses its own key). FRAGILITY: the snap-side key lives under per-revision /var/snap/maas/current/ and may not survive a snap refresh -- re-assert after any refresh. Roosevelt analog: per-DC IPMI/BMC credentials.

Change-delivery loop: grep for prior art (zeroth decision) -> grep design-decisions for the governing D-NNN -> edit -> bash scripts/repo-lint.sh -> run the touched script's harness (create one if missing -- no script change ships without its harness) -> deliver with an entry in the SESSION changelog (ONE changelog per session, GA-R2/D1 -- never per-topic files) carrying a per-item revert. Under blanket approval, the changelog IS the review surface: every item states what, why (evidence), and how to revert. Changelogs are session-scoped scratch: consolidated per-stage at stage close, NEVER citable as status or decision authority.

Deploy loop (VR0, single-DC testcloud): phase-00 runbook (D-061 destroy path) -> bash scripts/preflight.sh PASS -> phase-01..08 gated -> bash scripts/cloud-assert.sh --capture -> commit the asbuilt/ BOM.

Deploy loop (VR1 DC-DC, the current live mission): read docs/dc-dc-deployment-workflow.md's tooling gap register FIRST -- check whether the stage you're about to run has any open blocking item -> Stage 1 (dc-dc-phase0-vcloud-prep.md) gated -> Stage 2..7 gated in order, per runbooks/README.md's "VR1 DC-DC track" table -> if a stage's tofu apply fails partway, read runbooks/dc-dc-teardown-rollback.md's rollback decision tree BEFORE reaching for a teardown (fix-forward is usually correct). Run bash scripts/opentofu-validate.sh before any tofu step; whether a given root/step has been exercised for real is a STATUS question -- check CURRENT-STATE, never this file's memory of it. At STAGE CLOSE (GA-R6): a stage or verification-type gate CLOSES only on a NAMED executable check whose captured output the closing commit cites -- prose cannot close a stage, and there is NO conditional close (a remainder splits into its own gate row or the stage stays open, E3). Ruling-type gates close per GA-R5 (recorded, pushed ruling). The close-out set is executable: gauntlet ALL GREEN, repo-lint 0-fail, the stage's GA-R2 consolidation commit landed, the GA-R7 memory review run, MERGE the stage branch to main (merge commit, NOT squash; operator-gated), then RETIRE the branch (local + remote). ALSO sweep THIS skill in the same close: fold in any new INVARIANT the stage established and reconcile any divergence from repo HEAD (facts delegate to the repo and go stale; invariants live here and must be kept current). The next stage branches off post-merge main. main is trunk-of-record only for stages that have merged (precedents: Stage 2 -> PR #1 e468d2e 2026-07-15; Stage 3 -> merge commit 2026-07-21, vr1-dc0 scope, dc1 HELD under G12). See docs/dc-dc-deployment-workflow.md "Cross-cutting discipline".

Incident loop: capture the verbatim error -> bash scripts/cloud-assert.sh (the service-own-verdict sweep localizes the layer) -> appendix-A by exact message -> recorded fix, gated -> log the finding (new root causes become appendix-A/DOCFIX material).

Record authority and session discipline (GA-R1..R7, ratified 2026-07-18)

Ratified text: repo docs/audit/ga-rulings.md -- this is the operating summary.

docs/CURRENT-STATE.md is the ONLY status authority (GA-R1). Stage, gates, plan counts, version pins, decision-queue state live THERE and nowhere else; other surfaces point or are history. Any commit that changes a status it carries updates it in the SAME commit (C1; repo-lint L10 enforces). It is authoritative over documents, never over measurement -- a fresh capture wins and corrects it in the capture's commit (C2).
Plan counts and versions are quoted only from captured command output -- a dated capture file or a quoted command + output; never from memory or another document's prose (GA-R1 rule 2). Legal stage/gate status tokens are OPEN / BLOCKED / HELD / CLOSED, used only in CURRENT-STATE (vocabulary ruling, Option A: readiness is not a state; event words are history, never states).
Rulings are gated like mutations (GA-R5). ONE decision per operator exchange -- batch adoptions are invalid. A ruling exists only when its Status line quotes the question AND the operator's exact utterance, dated, committed and pushed before dependent work. Ambiguous/template answers rule NOTHING: stop and ask. Never adopt from an inferred or reconstructed ruling -- an operator ruling is a value under hard rule 2.
D-number admission (GA-R3): a new D-number needs architectural consequence beyond the stage, a Roosevelt-delta (the A1 test: a Roosevelt build session would grep it before touching a built surface), or supersession. Everything else is OPERATIONAL: runbook edit / session-changelog line / as-built row, never a D-number. Entries carry [ARCH]/[OPS] tags; doubt resolves DOWN to OPS.
Memory governance (GA-R7): auto-memory records observations and conventions ONLY -- never operator policy, priorities, or postures. Any memory claiming operator intent is PROPOSED until confirmed verbatim; remove on discovery. Durable observations GRADUATE to their repo surface; memory is never the only copy. Stage close includes a full memory review (rules 1-4).
Reconciliation circuit-breaker. If you have been fixing records to match records for more than one cycle without touching the actual deliverable, STOP and report to the operator. Record-churn presenting as progress is this project's measured failure mode (GA-F06); escalating is the correct move.

Posture

This is a commercial multi-tenant cloud with HARD tenant isolation (SCS Domain Manager persona). Treat tenant-visible surfaces and cross-domain boundaries as security-relevant in every change.
The operator community here values debate and industry best practice over quick fixes. Push back with sources when you disagree; own mistakes plainly and concisely. Fabricated flags, values, or version numbers are the cardinal sin - if you have not verified an option name or version, say so and verify.
Responses stay concise. Decisions get explicit rationale.

================================================================================

references/operating-discipline.md (loaded on demand in Claude Code; inlined here for Chat)

================================================================================

Operating discipline

How work is executed on this cloud. These conventions exist because the runbooks and scripts ARE the product: a fix that works but is undocumented, unrepeatable, or Roosevelt-incompatible is a failure.

The gated execution model

Every command block is labeled so a command line is never mistaken for prose, and so mutation risk is explicit (same convention as the repo runbooks):

RUN -- <loc> - the block CHANGES state; run it at (jumphost, a unit via juju ssh, the mgmt VM...).
CHECK (read-only) -- <loc> - safe to re-run.
GATE: - hard stop. Do not proceed unless the stated condition holds.
Expect: - what a passing result looks like. Always state it: the operator should never have to guess whether output is good.
> CAUTION: - destructive, secret-handling, or irreversible.

Sequence discipline: read-only audit -> present the mutation -> operator approves/runs -> verify the result -> next step. One gated mutation at a time; never batch destructive steps. If output comes back unexpected, STOP and re-derive from the live state - do not improvise a correction inline.

Chat (no shell): you prepare blocks, the operator runs and pastes output. Treat pasted output as the only evidence; a block you wrote but saw no output for did not happen.

Live shell: you may execute CHECK (read-only) blocks yourself. RUN and > CAUTION: blocks still get presented and human-approved first - state what will change and why it is the minimal correct action.

Irreversible / one-shot / secret steps

Start every consequential session with bash scripts/run-logged.sh <label> (opens a script(1)-logged shell to ~/as-executed/; index the session in logs/as-executed-index.md -- content NEVER commits, the index always does).
Retrieve the exact prior working command from the as-executed log or runbook VERBATIM. Never improvise vault init, unseal, authorize-charm, CA issuance, or anything one-shot. (DOCFIX-006: a mis-redirected vault operator init loses the unseal keys forever.)
Secrets never transit argv, clipboard, scrollback, or /tmp. Capture straight to a 0600 file under $HOME with umask 077; unseal via hidden prompt (L4); transfer via base64 pipe into a root-written 0600 file (L-P6-4). Never echo a secret to verify it - verify by length/format from the file.
Never run maas list (prints the API key - DOCFIX-016). Never trust a juju action's human-formatted output for a captured secret or cert - pull from --format json (indented YAML block-scalars corrupt PEMs; DOCFIX-021).
Authorize charms with short-lived child tokens: juju run persists action params in the operation log (DOCFIX-011).

Record-keeping: D-NNN / DOCFIX-NNN / BUNDLEFIX-NNN

D-NNN - design decisions (docs/design-decisions.md). Append-only: superseded entries stay in place, marked, with the superseding entry appended. DOCFIX-NNN - runbook fixes. BUNDLEFIX-NNN - bundle fixes.
ALWAYS grep the repo for the next free number before assigning one - collisions have happened. State "next-free verified" when you assign.
PARALLEL WORK numbering: when more than one workstream edits the shared numbered ledgers (this repo + a parallel Claude Code stream), do NOT each independently grab "next-free" -- that collides. One stream owns a given number/block; each DECLARES every identifier it consumes in its commit; the other re-greps AFTER the parallel push lands and resumes above it. scripts/ledger-scan.sh derives next-free from decision HEADERS (not prose mentions -- a "next-free D-072" pointer must not inflate the count). Keep "Next-free:" pointer lines on ONE line (a word-wrapped continuation escapes the line-based exclusion); or just rely on the scan, which is the next-free authority.
Mid-task findings are logged as proposals, not acted on (hard rule 1). A finding that changes a runbook becomes a DOCFIX; one that changes architecture becomes a D-NNN with status PROPOSED until the operator rules.
Status vocabulary: PROPOSED/OPEN -> ADOPTED/DECIDED -> SUPERSEDED (by D-MMM). Do not implement PROPOSED items.
Before acting on any change request, grep design-decisions for the governing decision on that surface (and its dependents - e.g. a runbook step may rely on the current state). Finding a PROPOSED/OPEN decision means presenting its recorded options for a ruling, not picking one.

Delivery rules (files handed to the operator)

Multi-file changes ship as repo-relative ZIPs, never loose files (the Windows/GitHub Desktop loose-file workflow misplaces them).
Everything committed is ASCII-only, LF-only. Validate before delivery: non-ASCII with grep -nP '[^\x00-\x7F]' (or a Python byte read); CR bytes with a Python data.count(b"\x0d") - a grep $'\r' false-positives on $r... tokens. Non-ASCII in OpenStack config has caused silent daemon failures (mod_wsgi UnicodeDecodeError).
On-box script delivery over juju ssh goes as a base64 pipe decoded to a file, then the FILE is executed - never a raw heredoc (paste-mangling and stdin-consumption both bite; see script-authoring on read vs pipes).
Windows-side steps are PowerShell-native (no bash heredocs, no backslash continuations). The operator commits from Windows; the jumphost only pulls.
Credential exposures and security obligations get a ROW in docs/security-ledger.md at discovery time (owner + status) - never only a script-comment note; the ledger is reviewed at every phase-00 and handoff.
Under operator-granted blanket approval, the delivery contract is: implement, then hand over ONE cumulative repo-relative ZIP plus a changelog where every item states what changed, the evidence for why, and a per-item revert. The changelog is the review surface; opinion-weighted calls are flagged as such.

Session continuity: the in-flight ledger

Long sessions on this cloud get COMPACTED, often several times. Work that lives only in chat scrollback is lost at compaction. docs/session-ledger.md is the durable record of what is IN FLIGHT (active build, backlog, operator-gated items, open decisions, numbering state, state facts). Standing practice:

Read the ledger AND run scripts/ledger-scan.sh at session start; reconcile.
The scan is the DRIFT CHECK: it derives open decisions (last Status line per block, so an amendment closes it), OPEN security rows, and next-free numbers straight from the repo. The narrative ledger must not claim CLOSED what the scan shows OPEN, nor omit what it surfaces. A ledger without this check rots (cf. D-070's unexercised-snapshot critique).
Update the ledger at every deliverable/commit -- it is a standing deliverable like the changelog. Changelog = what changed; ledger = what is still open.

Boundary mutations: adds-before-removes with a readback gate

When tightening a security boundary (SG rules, firewall, policy) that a live flow depends on, NEVER remove the permissive rule before the specific replacements are proven present. Create the narrow rules, READ THEM BACK (fail if any is missing), and only THEN remove the wide rule by its MEASURED id. There must be no window in which a required flow (e.g. the magnum conductor -> mgmt apiserver poll) is unauthorized. Verify the tightened boundary by FUNCTIONAL outcome afterwards, not by assuming the rule text is sufficient.

Attestation gates for unautomatable safety items

Some acceptance items cannot be scripted -- e.g. a SECOND person performing the Vault unseal (D-069). Never auto-pass such an item. Gate it on a ledger ROW (docs/security-ledger.md): report PASS_PENDING_MANUAL while the row is OPEN, PASS only when it is attested CLOSED/rehearsed, HOLD if the row is missing. Auto-passing an undone safety item is the exact failure D-070 retired.

Debate and correction norms

Challenge weak reasoning with sources; the operator prefers industry best practice over the quick fix and will push back hard on unverified claims. When challenged, verify (docs, source, live probe) rather than defend.
Own mistakes plainly, in one or two sentences, then fix. No self-flagellation.
When a prior decision looks wrong, propose superseding it through the D-NNN process - do not quietly deviate from it.

Troubleshooting entry discipline

Before any server-side hypothesis for "X used to work, now it doesn't" on a web UI: eliminate client state first (incognito window, ~10 seconds), then a server-side curl, THEN hypothesize. Before any command that acts on tenant resources: confirm which project/domain scope the shell holds (openstack token issue-level certainty) - server create and friends use ambient scope silently. Full triage method: references/troubleshooting.md.

Guards are load-bearing: reword the content, never weaken the check

When a repo-lint guard (or any fail-closed check) goes RED on text you believe is legitimate, the fix is to REWORD the text or add the guard's own documented opt-out marker -- never to loosen the guard. Precedent: L3 correctly flagged the config.xml tombstones; the tombstones were reworded and the guard kept. (Rotated from the session ledger's standing-lessons block, 2026-07-19.)

================================================================================

references/environment.md (loaded on demand in Claude Code; inlined here for Chat)

================================================================================

Environment - Omega Cloud (VR0 DC0 testcloud)

Facts here are ANCHORS, not command inputs. Anything marked (verify) must be re-measured or re-read from the repo/live cloud before use in a command - hard rule 2 applies. Snapshot date: 2026-07. The repo is fresher than this file.

The three deployments (this file covers VR0 in detail; VR1 is a pointer)

Testcloud (VR0 DC0, now): four KVM host VMs (openstack0-3) on a single hypervisor, managed by MAAS + Juju. Single-DC virtual rehearsal. The rest of THIS file is VR0's own ground truth.
VR1 DC-DC (in progress, D-100..D-110): two-DC + Office1-headend rehearsal -- VR0's testcloud host is being REPURPOSED into DC1 (not a second concurrent cloud). This file does NOT carry VR1's own facts (Office1 network, DC1/DC2 mesh links, per-DC MAAS/Vault roots, etc.) -- those live in docs/dc-dc-deployment-workflow.md (master tracker: stage status + tooling gap register, read this first for any VR1 work) and docs/dc-dc-buildout-design.md (the design itself). Runbooks: runbooks/README.md's "VR1 DC-DC track" section. IaC: opentofu/ (UNVALIDATED -- see its own README).
Roosevelt (future): bare-metal, multi-DC, commercial production (3310 Roosevelt Blvd, Eugene OR). Dedicated node roles (gateway/controller/ compute split) - unlike the hyperconverged testcloud. Every design choice is judged by its transfer to Roosevelt -- VR1 is the rehearsal for THIS, not an end in itself.

Stack (verify against appendix-B for pins)

Charmed OpenStack Caracal 2024.1 - Juju/MAAS/component versions live in runbooks/appendix-B-asbuilt-version-lock.md (VR0) and docs/CURRENT-STATE.md section 7 (VR1 substrate) - Vault TLS (charm-pki root CA), OVN 24.03, Ceph Squid, Octavia (amphora), Barbican, mysql-innodb-cluster, RabbitMQ, Magnum + magnum-capi-helm driver + azimuth (SUBSTRATE below the charms -- pins + traps in references/platform-traps.md: libvirt/KVM via the dmacvicar/libvirt OpenTofu provider 0.9.8 (tofu + component pins: docs/CURRENT-STATE.md section 7, the pin authority); LXD pinned to the 5.21 LTS track -- MAAS 3.6/3.7 cannot talk to LXD >= 6.7, D-114; OPNsense edges driven by REST API, D-113; the vcloud host is ITSELF a KVM guest, so nesting is a per-level requirement) capi-helm-charts (kubeadm engine), in-cloud single-homed CAPI mgmt VM (capi-mgmt-v2, k8s-snap, D-035). NetBox is the IPAM apex: never hand-edit downstream MAAS or overlays for network values.

Control points

Jumphost: vcloud - all live commands run here. Has juju, the openstack CLI (SNAP - cannot read /tmp; use $HOME), jq, kubectl.
Repo: https://git.baldurkeep.com/OpenStack/openstack-caracal-dc-dc (web) / .../git/OpenStack/openstack-caracal-dc-dc.git (clone) -- a documented ANCHOR FACT (this repo's name/path has already changed once, see D-110; re-verify with git remote -v before trusting it, same as any other fact in this file). repo-lint: allow-selfref. Operator commits from Windows (PowerShell / GitHub Desktop - strips exec bits; .gitattributes pins LF); the jumphost only pulls.
Session $REPO convention: set $REPO once per session to your clone's root (see runbooks/README.md Conventions); every runbook RUN block references $REPO, never a hardcoded clone path. repo-lint L9 enforces this for scripts/RUN-blocks/skill files.
Juju model: openstack. MAAS profile: admin (call maas admin ... directly; NEVER maas list - it prints the API key).
Management substrate (verify; NEVER touch in teardown): the MAAS machines hosting juju, lxd, and tailscale are hard-excluded from teardown scripts. Resolve system_ids live via scripts/lib-hosts.sh - system_ids are re-minted on every re-enrollment (DOCFIX-040).

The six network planes (D-052 / D-053; verify against scripts/lib-net.sh)

Plane	CIDR	Carries	Notes
provider-public	10.12.4.0/22	Public API VIPs + tenant FIPs (Pattern A, D-060)	gw .4.1; untagged
metal-admin	10.12.8.0/22	MAAS PXE, operator/admin endpoint, default binding	gw .8.1; DC-local
metal-internal	10.12.12.0/22	ALL service-to-service control (internal API, DB, MQ, Vault, peers)	tagged VID 103 via br-internal; no gw
data-tenant	10.12.16.0/22	Tenant Geneve overlay	no gw
storage	10.12.32.0/22	Ceph public	no gw
replication	10.12.36.0/22	Ceph cluster (OSD replication)	no gw

API VIPs: triple per clustered charm (provider/admin/internal), matching last octet in the .50-.60 band, 11 clustered charms (verify count live).
Tenant pool: 10.20.0.0/16 (hybrid model D-016 - pool in NetBox, per-project /24s Neutron-managed). Avoid collisions with capi-mgmt (10.20.0.0/24) and existing tenant /24s - list live before allocating.
Provider NIC rule (D-057/D-060): the provider uplink must land in OVS br-ex, never enslaved to a Linux bridge, and br-ex carries no L3 config.

Repo map (what lives where)

bundle.yaml - the canonical bundle; VIPs/units baked in for testcloud.
runbooks/phase-00..08-*.md - the gated deploy sequence, in order, each ending in a hard gate. runbooks/README.md has the label conventions.
runbooks/appendix-A-troubleshooting.md - symptom->cause->fix index keyed by D-NNN/DOCFIX-NNN. First stop for any known-looking failure.
runbooks/appendix-B - version lock. appendix-C - identity/RBAC. appendix-D - Magnum trust model. ops-capi-recovery.md - CAPI/Magnum post-deploy operations.
docs/design-decisions.md - the D-NNN architectural record (append-only discipline; superseded entries stay, marked).
scripts/ - phase scripts + lib-net.sh / lib-hosts.sh (pinned values)
- tenant onboarding/acceptance. tests/<script>/ - offline fakebin regression harnesses.
policies/domain-manager-policy.yaml + policies/overrides.zip - the SCS Domain Manager RBAC override (D-051/D-064); the zip ships IN the bundle (keystone resources, DOCFIX-071) and provider-bundle-check drift-guards it.
Operational tooling (2026-07 hardening set): scripts/preflight.sh (single pre-deploy gate: lint -> bundle invariants -> Charmhub channel assert -> live MAAS pre-flight), scripts/repo-lint.sh/repo_lint.py (static hygiene, L1-L6), scripts/cloud-assert.sh (behavioral verifier + --capture BOM to asbuilt/<date>/), scripts/run-logged.sh (as-executed session logger), scripts/channel_assert.py. runbooks/ops-restart-procedure.md (full-cloud restart). docs/security-ledger.md (exposure/obligation rows). logs/as-executed-index.md (committed index; log content stays jumphost-only).
No KVM snapshot restore path exists (D-070 superseded D-012): rebuild-from-runbooks IS the restore path; baselines come from cloud-assert --capture.
VR1 DC-DC additions (2026-07-09/10), not part of VR0's map above: docs/dc-dc-deployment-workflow.md (master tracker), docs/dc-dc-buildout- design.md (design), docs/design-decisions.md D-100..D-110, runbooks/dc-dc-phase0..6-*.md + dc-dc-teardown-rollback.md, opentofu/ (libvirt/MAAS/OPNsense IaC, UNVALIDATED -- see its own README), netbox/dc-dc-prefixes-import.py, scripts/dc-dc-*.sh (MTU/Ceph budget calculators, Ceph replication tooling).

Identity / tenancy model (see appendix-C/D and D-051, D-064, D-066)

Domain-per-client. Operator provisions: domain + a domain manager (SCS Domain Manager persona - the plain admin role is NOT domain-confinable) + quotas. The tenant self-services everything inside: projects, users, roles (only member + load-balancer_member assignable - never admin/manager), app credentials, networks, templates, clusters. Magnum mints per-cluster trust app-creds carrying the trustor's roles frozen at mint time (D-039: trustor needs load-balancer_member or CAPO 403s on Octavia). Cluster create must run as a password identity, not an app-cred (trust-creation block, D-066). Every identity command is DOMAIN-QUALIFIED (--domain, --user-domain, --project-domain) - scope-default resolution silently lands in the wrong domain and 404s misleadingly.

================================================================================

references/platform-traps.md (loaded on demand in Claude Code; inlined here for Chat)

================================================================================

Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense

The substrate BELOW the cloud (the hypervisor, the provisioner, the edge routers) has its own version boundaries and its own silent-failure modes. Every item below is either (a) sourced to a vendor/upstream URL, or (b) MEASURED on this project and sourced to the repo's own D-NNN / changelog. Nothing here is generic advice: each entry exists because it cost, or would have cost, a session.

Companion files: references/opentofu-provider-docs.md (provider schema + fetch methodology), references/troubleshooting.md (method). Start at the verbatim-error index at the bottom if you have an error string in hand.

Version pins that actually matter (and why)

Thing	Pin	Why the pin exists
`dmacvicar/libvirt` provider	`0.9.8`	0.9.0 was a compatibility-BREAKING rewrite; anything remembered from 0.8-era examples is wrong (see below)
OpenTofu	live pin: CURRENT-STATE.md section 7	Stage-1 as-run value is historical (git history)
LXD	`5.21` LTS track	MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114; see below)
MAAS	3.7.2 (VR0)	the LXD ceiling above is a property of MAAS 3.6/3.7
OPNsense	live pin: CURRENT-STATE.md section 7 (26.7 as of 2026-07-18)	ISC-DHCP left core AT 26.1 -- Kea/dnsmasq only on every later release (see below)
Vault charm	`1.8/stable`	1.16 is an incompatible charm, deliberately NOT an upgrade (appendix-B, D-068)

Charm channels/revisions are appendix-B's job, not this file's. This file covers only the substrate the charms sit on.

1. dmacvicar/libvirt provider (pinned 0.9.8)

1a. The 0.8 -> 0.9 break invalidates most examples you will find

v0.9.0 (2025-11-08) is a deliberate, compatibility-breaking rewrite on the plugin framework, with the design principle "HCL maps almost 1:1 to libvirt XML"; the legacy provider stayed on the v0.8 branch. v0.9.1 (2025-11-30) then code-generated the schema from libvirt's XML schema, renaming attributes again (unit -> memory_unit, os.arch -> os.type_arch, os.machine -> os.type_machine, os.kernel_args -> os.cmdline, camelCase -> snake_case throughout).

Consequence: blog posts, StackOverflow answers, and model memory almost all describe 0.8. Nested structures are attribute-style objects (os = { ... }, devices = { ... }), not HCL blocks. Confirm against docs/resources/*.md + examples/*.tf at the pinned tag, or tofu providers schema -json.

1b. Bare `memory` is KiB, not MiB -- 1024x too little RAM (DOCFIX-188)

The 0.9.1 migration guide states the rule explicitly: "Value/unit pairs are explicit -- whenever libvirt exposes a value with a unit attribute the provider now has two attributes (memory + memory_unit, capacity + capacity_unit, etc.). Leaving the unit unset lets libvirt use its default." (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.1)

And libvirt's default is KiB: "The units for this value are determined by the optional attribute unit, which defaults to 'KiB'" (https://libvirt.org/formatdomain.html#memory-allocation).

So memory = 2048 with no memory_unit is 2 MiB, not 2 GiB. In 0.8, memory meant MiB -- that is exactly how this shipped here.

memory      = var.memory_mib
memory_unit = "MiB"      # REQUIRED. Never omit.

Signatures (all measured, 2026-07-12): QEMU cmdline -m size=2048k; virsh dominfo -> Max memory: 2048 KiB; a FreeBSD guest echoes /boot.config (262 bytes of serial output) and then triple-faults handing off to /boot/loader. tofu validate CANNOT catch it (the attribute is optional) -- scripts/opentofu-validate.sh check S1 does. Evidence: docs/archive/changelogs/changelog-20260712-libvirt-memory-unit-rootcause.md, docs/archive/incident-20260712-opnsense-edge-boot-triplefault.md.

The provider's own current domain examples now carry memory_unit = "MiB" (docs/resources/domain.md) -- a fresh reader of the docs would get this right; a reader of any 0.8-era example would not.

1c. No `features` block => `acpi=off` => FreeBSD panics (DOCFIX-190)

The whole features attribute is optional, so omitting it is schema-valid -- and libvirt then renders the machine with acpi=off (measured on the QEMU cmdline: -machine pc-i440fx-noble,...,acpi=off).

features = {
  acpi = true
  apic = {}          # acpi is a Boolean; apic is a nested object
}

(schema: docs/resources/domain.md, features.acpi (Boolean), features.apic (Attributes).)

Three distinct consequences, only one of which is loud:

FreeBSD/OPNsense panics (it finds the local APIC via ACPI's MADT and ships no atpic fallback): panic: running without device atpic requires a local APIC -> db> prompt -> domain shows running while burning 100% of one core forever.
Linux guests boot but degrade: no clean ACPI shutdown/reboot signalling.
MAAS drives power off/on via ACPI, so a MAAS-managed node VM without ACPI can only ever be hard-stopped. That would have been a nasty Stage-3 debug. Evidence: docs/archive/changelogs/changelog-20260712-libvirt-acpi-kernel-panic.md. Guard: scripts/opentofu-validate.sh check S2.

1d. No `cpu` block => generic CPU => NO `svm` => nested KVM impossible

Found 2026-07-13. With no cpu attribute, libvirt hands the guest a generic emulated CPU model with no virtualization flag. Measured: host is an AMD EPYC 9965, the default-CPU guest was handed an Opteron_G3. Nothing errors -- the guest just has no /dev/kvm and no svm in /proc/cpuinfo.

Ubuntu's own nested-virt guide states the guest-side requirement: set the guest cpu mode to host-model or host-passthrough for the guest to see svm/vmx (https://ubuntu.com/server/docs/how-to/virtualisation/enable-nested-virtualisation/).

cpu = {
  mode = "host-passthrough"
  # per-caller: pass svm through, or disable it as router hardening
  features = var.expose_nested_virt ? [] : [
    { name = "svm", policy = "disable" }
  ]
}

The host side is a SEPARATE requirement and bites at every level. The vcloud host is ITSELF a KVM guest (systemd-detect-virt -> kvm), so nesting must be enabled at each level, not just ours:

check: cat /sys/module/kvm_amd/parameters/nested -> Y or 1 (Intel: /sys/module/kvm_intel/parameters/nested)
persist: options kvm-amd nested=1 in a file under /etc/modprobe.d/ (same source as above)

Why this matters here (D-114): LXD virtual machines are qemu/KVM guests, so MAAS composing service VMs inside a site containment VM needs working nested KVM at that depth; and nova-compute on the DC nodes needs it one level deeper still. modules/node-vm still has the missing-cpu-block defect -- logged, NOT fixed, because DC1 is gated behind Office1 (docs/archive/changelogs/changelog-20260713-d114-voffice1-nested-virt.md).

1e. "will be updated in-place" DOES NOT mean "the guest stays up"

A plan line reading libvirt_domain.vm will be updated in-place bounced a live guest (measured 2026-07-13: uptime 8m36s -> 6s; ~30s with no routing and no DHCP on the Office1 edge). "In-place" is a statement about the Terraform RESOURCE, not about the domain.

Upstream confirms the mechanism: the v0.9.4 release notes describe domain update as "updating a domain (e.g. changing memory) would undefine and re-define it" (the 0.9.4 fix was only to preserve NVRAM/TPM files across that undefine), and v0.9.6 added "configurable update shutdown behavior so domain updates can request guest shutdown and wait with a configurable timeout before forcing a stop."

Rule: treat ANY tofu apply whose plan touches a libvirt_domain as an OUTAGE of that guest. Schedule and gate it as one. Read the plan's resource list before applying; 0 to change on the domains is the only assurance that a running guest is not about to bounce (docs/session-ledger.md, standing lesson 2).

Corollary (DOCFIX-187/DOCFIX-188): some domain attributes are CREATE-time to libvirt (max boot memory, machine, cpu). The provider will happily plan an in-place update that libvirt then ignores. If a changed value does not show up in virsh dominfo, destroy + undefine the domain and re-apply (volumes survive an undefine).

Second corollary (measured 2026-07-20/21): the undefine/redefine also REGENERATES every UNPINNED NIC MAC. The plan reads 0 to add, N to change, 0 to destroy, the apply succeeds -- and every touched domain now has brand-new MACs. Anything that keys on those MACs is silently stranded: in the 2026-07-21 commissioning incident all 9 MAAS-enlisted nodes became UNKNOWN at PXE (no "Performing PXE boot" event, silent 30-min timeouts), because a serial-console retrofit apply had drifted every boot-interface MAC. Rule: any domain whose MACs are load-bearing (MAAS-enlisted nodes above all) carries config-pinned MACs -- mac = { address = ... } per interface; see modules/node-vm's interface_macs + tests/node-vm. VR-only trap: Roosevelt metal has fixed MACs.

1f. Two more upstream behaviours worth knowing (from the release notes)

capacity_unit = "GiB" on a volume caused "Provider produced inconsistent result after apply" on every apply before 0.9.4 (libvirt normalizes to bytes). Fixed in 0.9.4 -- if you see that error string, check the provider version first.
tofu destroy of a dir pool used to DELETE the backing directory (StoragePoolDelete). Since 0.9.4 the directory is preserved by default, with an explicit destroy.delete = true to opt back in. Relevant to modules/dc-storage-pool. (https://github.com/dmacvicar/terraform-provider-libvirt/releases/tag/v0.9.4)
UNVERIFIED: the 0.9.4 notes name a destroy.shutdown.timeout option while docs/resources/domain.md documents destroy = { graceful, timeout }, and I could not find the 0.9.6 update-shutdown attribute documented at all. Before relying on either, read the real schema: tofu providers schema -json | jq '.provider_schemas[].resource_schemas.libvirt_domain'.

1g. `forward=bridge` networks + `br_netfilter`: a global FORWARD-drop silently kills bridged frames (D-125)

A libvirt_network with forward = { mode = "bridge" } (dmacvicar schema: bridge = { name = "<host-bridge>" }, no ips, no NAT) attaches guest taps to a pre-existing host bridge -- libvirt does NOT create the bridge, so it must already exist (declare it in the host's netplan, not the tofu) or the network fails to start. This is the Model B "bridge-in" WAN path (D-125): the inner OPNsense WAN rides br-vr1-dc0-wan out through vvr1-dc0's IP-less uplink NIC to the single vcloud ISP NAT.

Two silent-failure traps ride along:

br_netfilter makes bridged frames traverse the L3 FORWARD chain. So a GLOBAL forward-drop -- iptables -P FORWARD DROP, or an nftables forward hook with policy drop and no interface match -- silently kills the WAN bridge even though it is "just L2". Keep any DC-LOCAL forward-drop (SEC-010) scoped to the transit interface (oifname/iifname "<transit>"), never global. Note net.ipv4.ip_forward=0 is L3-only and does NOT affect pure bridging -- do not reach for it expecting to gate a bridge, and do not assume ip_forward=1 is required just because an older NAT-based design needed it.
Foreign-MAC egress across a nested bridge is unprovable until apply. When the bridged guest is itself inside another guest (OPNsense inside vvr1-dc0), the outer NAT's virbr sees a foreign source MAC arriving on the containment VM's port; an anti-spoof/nwfilter drop there blackholes egress. tofu validate is clean regardless. TEST IN ISOLATION FIRST: attach a throwaway guest to the bridge, confirm it gets an address and pings out, BEFORE the real chain rides it. (D-125's fallback for this failure is double-NAT: masquerade at vvr1-dc0 instead of relying on nested L2.)

2. AppArmor blocks qemu on any non-default pool path

libvirt's stock abstractions/libvirt-qemu grants qemu the DEFAULT pool path (/var/lib/libvirt/images) only. A pool anywhere else is blocked by AppArmor even with perfect POSIX permissions, and the failure names nothing:

libvirt/qemu says only: Could not open '<path>': Permission denied
the actual denial is in the kernel log: apparmor="DENIED" operation="open" profile="libvirt-<domain-uuid>" name="<path>"
the domain DEFINES fine and fails at START -- so tofu apply can "succeed" into a dead guest.

Upstream (still open, provider issue #920, with the same workaround): https://github.com/dmacvicar/terraform-provider-libvirt/issues/920

Fix, in the vendor-sanctioned override include (survives package upgrade -- do NOT edit the shipped abstraction):

# /etc/apparmor.d/local/abstractions/libvirt-qemu
/var/lib/libvirt/vr1/** rwk,

This is now installed by bash scripts/prereqs/install-apparmor-libvirt.sh (pool parent overridable via VR1_POOL_PARENT), wired into scripts/prereqs/install-all.sh and reported by check-prereqs.sh. It cost a session on 2026-07-12 (DOCFIX-186) because it existed only as hand-applied host state; docs/archive/changelogs/changelog-20260713-apparmor-libvirt-prereq.md.

3. MAAS + LXD

3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114)

Canonical's own announcement: LXD 6.7 "consolidates some API endpoints" that MAAS's pinned pylxd 2.3.5 cannot speak to (fixed in pylxd >= 2.3.9, not yet in a MAAS release). Verbatim guidance: "We recommend using LXD <=6.6 or the 5.21 LTS release until further notice." https://discourse.maas.io/t/maas-incompatibility-with-lxd-6-7/15749

VR0 runs LXD 5.21.4 and is therefore safe by accident of timing; D-114 makes it a decision. LXD 5.21 LTS is supported to June 2029 (https://canonical.com/blog/lxd_5-21-0_lts).

The live hazard is the snap auto-refresh, not the install. LXD's docs: "By default, installed snaps update automatically when new releases are published to the channel they're tracking"; the latest track "typically points to the latest feature release", is "a continuously rolling release track", and is "not recommended for general use"; a bare snap install lxd uses the most recent LTS track, "which is currently 5.21". https://canonical.com/lxd/docs/latest/reference/releases-snap/ => install/refresh explicitly on 5.21/stable and verify the tracked channel (snap list lxd, snap info lxd) before blaming MAAS for an LXD it can no longer talk to.

3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off

"Bridges created by LXD are managed, which means that in addition to creating the bridge interface itself, LXD also sets up a local dnsmasq process to provide DHCP, IPv6 route announcements and DNS services to the network" (ipv4.dhcp defaults to true). https://canonical.com/lxd/docs/latest/reference/network_bridge/

MAAS's own LXD VM-host procedure therefore has you disable exactly that, because MAAS must be the DHCP/PXE authority for the machines it composes:

lxc network set lxdbr0 dns.mode=none
lxc network set lxdbr0 ipv4.dhcp=false
lxc network set lxdbr0 ipv6.dhcp=false

https://canonical.com/maas/docs/how-to-manage-machines

This is the concrete form of D-114's DHCP warning. On office1-local, OPNsense/Kea is authoritative (10.10.0.0/24, pool .100-.199). Two DHCP servers on one L2 is an intermittent, genuinely unpleasant failure. Never bridge lxdbr0 onto the site LAN without disabling its DHCP first.

3c. Two different LXDs in this cloud -- do not conflate them (D-114)

Conflating these caused a real, recorded error:

Juju-created LXD CONTAINERS on the OpenStack nodes (lxd:N placements in bundle.yaml, 21 of them). MAAS never sees these and never needs to.
A MAAS-registered LXD KVM host (the VR0 lxd machine) into which MAAS COMPOSES VMs (maas $PROFILE vm-host compose ...) -- VR0's tailscale machine is one. These ARE MAAS machines: enlisted, commissioned, deployed, powered, released.

D-114 extends the second pattern per site; the first is explicitly out of scope. Also note: MAAS only sees VMs in the LXD project the VM host was registered with -- VMs in other projects are invisible to it and unaffected (MAAS LP#1923251, https://bugs.launchpad.net/maas/+bug/1923251).

Composed-VM networking -- MEASURE, do not assume. A MAAS-bundled copy of the MAAS docs states: "When composing a virtual machine with LXD, MAAS uses either the 'maas' LXD profile, or (if that doesn't exist) the 'default' LXD profile. The profile is used to determine which bridge to use." (served by a MAAS instance's own docs: https://maas.cloud.cbh.kth.se/MAAS/docs/cli/how-to-use-lxd.html -- I could NOT locate this sentence on canonical.com/maas.io today; the docs site has been reorganized and the old URLs 404. Treat the profile->bridge mechanism as PLAUSIBLE-BUT-UNCONFIRMED against a primary source.) Since a composed machine that PXEs on the wrong L2 is the failure this predicts, do the cheap measurement instead of trusting either source: lxc profile show maas / lxc profile show default on the LXD host, and confirm the NIC's parent bridge before composing.

Because LXD VMs (not containers) are qemu/KVM guests, this pattern needs nested KVM at whatever depth the LXD host sits -- see 1d.

4. OPNsense edge (traps entered at 26.1; they persist on the live 26.7)

4a. `config.xml` is GUI-owned. Do not hand-author it. (D-112, D-113)

The config-xml path is DELETED (template, renderer, ISO builder, and the module's config_seed volume + cdrom disk -- D-113 amendment, 2026-07-13). Configure the edge over the REST API (scripts/opnsense-api.sh); mint an API key with scripts/opnsense-bootstrap-apikey.sh. Three separate defects (no sshd, no console, an inert <dhcpd> block) all traced to hand-writing that file, and a full-config push DROPS ~667 migration-populated elements -- including the box's only two firewall pass rules -- surviving only on an UNDOCUMENTED self-heal at boot. If you think the API cannot express something, that is a finding to raise against D-113, not a reason to write XML.

Two traps that follow:

A config ISO can NEVER be read on a nano image (D-112, root-caused in upstream opnsense/core:src/sbin/opnsense-importer: the importer probes for a read-only root and bootstrap_and_exit 0s on a pre-installed image before scanning any media). Any instruction to build one is a trap.
Factory default: LAN 192.168.1.1/24, SSH disabled, all inbound blocked on WAN. An unanswered ping to the WAN address is NOT a fault.

4b. ISC dhcpd is GONE from core in 26.1 -- an `<dhcpd>` config is inert

26.1 release notes: "ISC-DHCP moves to a plugin. It will be automatically installed during upgrades. It is not installed on new installations because it is not being used, but you can still install and keep using it." Also: "Dnsmasq is now the default for DHCPv4 and DHCPv6 as well as RA out of the box." https://docs.opnsense.org/releases/CE_26.1.html

So on a fresh 26.1-or-later image: an ISC <dhcpd> block configures nothing (that was DOCFIX-193), and the GUI default backend is dnsmasq, not Kea. This edge deliberately runs Kea (10.10.0.0/24, pool .100-.199, first real lease served to voffice1 2026-07-13).

4c. REST API: auth, endpoints, and the reconfigure step everyone forgets

Auth is HTTP basic with an API key/secret pair, not the root password: "API access is part of the local user authentication system, but uses key/secret pairs to separate account information from machine to machine communication." URL pattern: https://<host>/api/<module>/<controller>/<command>. https://docs.opnsense.org/development/how-tos/api.html
Kea DHCPv4 (exact paths): /api/kea/dhcpv4/get, /set, /add_subnet, /set_subnet/$uuid, /del_subnet/$uuid, /add_reservation, /set_reservation/$uuid, /get_reservation, /add_option; service control: /api/kea/service/reconfigure|restart|start|stop|status. A set* call only edits config -- it is NOT live until you POST /api/kea/service/reconfigure. A silently-unapplied change looks exactly like a change that did not take. https://docs.opnsense.org/development/api/core/kea.html
Firewall rules have a self-rollback safety net -- use it. The filter controller exposes savepoint, apply/$rollback_revision, cancel_rollback/$rollback_revision, revert/$revision: "When calling savepoint() a new config revision will be created and the timestamp will be returned for later use. If the cancelRollback(savepoint) is not called within 60 seconds, the firewall will rollback to the previous state." For any remote rule change on a router you reach THROUGH that router, a savepoint means a mistake self-reverts instead of locking you out. https://docs.opnsense.org/development/api/core/firewall.html
Do NOT re-implement vendor internals to bootstrap. OPNsense stores API secrets as crypt(secret,'$6$'); we deliberately do not mint keys offline -- a format drift would fail SILENTLY (keys that never authenticate). Call the vendor's own model instead (scripts/opnsense-bootstrap-apikey.sh). (D-113 amendment.)

4d. root's shell is tcsh -- `$(...)` dies QUIETLY over SSH

Measured on the Office1 edge: root's shell is tcsh. tcsh has no $(...) command substitution (it uses backquotes); a $(...) inside a quoted remote command is rejected with:

Illegal variable name.

It already cost a defect: a pre-install snapshot silently no-op'd and the install then ran with no rollback point behind it (docs/archive/changelogs/changelog-20260713-office1-dhcp-apply.md). Because the step was non-fatal, nothing stopped.

Always feed remote commands to sh -s:

ssh -i "$OPNSENSE_SSH_KEY" root@<edge> 'sh -s' <<'SH'
...POSIX shell here...
SH

(tcsh's substitution syntax: https://man.freebsd.org/cgi/man.cgi?query=tcsh)

Verbatim error -> cause index (grep this first)

Exact string you see	Cause	Where
guest triple-faults right after echoing `/boot.config`; `virsh dominfo` -> `Max memory: 2048 KiB`; QEMU `-m size=2048k`	`libvirt_domain.memory` with no `memory_unit` -- KiB, not MiB	1b
`panic: running without device atpic requires a local APIC` (then `db>`, 100% of one core, domain "running")	no `features` block -> `acpi=off`	1c
guest has no `/dev/kvm`; `/proc/cpuinfo` has no `svm`; CPU model reads `Opteron_G3` on an EPYC host	no `cpu` block -> generic emulated CPU	1d
`Could not open '<path>': Permission denied` (domain defines, then fails to START; nothing names AppArmor) -- check `dmesg` for `apparmor="DENIED" operation="open" profile="libvirt-<uuid>"`	non-default libvirt pool path not granted in AppArmor	2
`Provider produced inconsistent result after apply` on a volume with `capacity_unit`	provider < 0.9.4	1f
plan says `will be updated in-place`, guest uptime resets anyway	provider undefines + redefines the domain	1e
MAAS commissioning times out at 30 min, NO PXE event for a known machine	an in-place apply regenerated unpinned NIC MACs; MAAS records stale	1e
`Illegal variable name.` from an `ssh root@<edge> '...'`	root's shell is tcsh; `$(...)` unsupported	4d
OPNsense boots on FACTORY DEFAULTS with a correct config ISO attached; console shows `>>> Invoking import script 'importer'` and nothing more	the Configuration Importer can never fire on a nano image	4a
a Kea/firewall API `set` returns success but the box's behaviour does not change	no `service/reconfigure` (or no firewall `apply`) call	4c
MAAS cannot talk to an LXD VM host after an LXD update	LXD auto-refreshed past 6.6; MAAS 3.6/3.7 needs LXD <= 6.6 or 5.21 LTS	3a
a MAAS-composed VM gets a lease MAAS did not issue, or PXEs on the wrong L2	`lxdbr0` runs its own dnsmasq (verified); the composing LXD profile's bridge is the other candidate (unconfirmed -- measure it)	3b, 3c

NetBox (upstream, `netbox.baldurkeep.com`)

`HTTP 403 Forbidden` on every API call -- but the SAME token works from `curl`

Symptom. urllib/requests/pynetbox get 403 Forbidden from the upstream NetBox on every endpoint, including ones a valid token plainly reads. curl with the identical token and URL returns 200. It reads exactly like a bad or revoked token; it is not.

Cause. Something in front of upstream NetBox (WAF / reverse proxy) filters on User-Agent. The default Python UA (Python-urllib/3.12) is blocked; curl/8.5.0 is allowed. Measured 2026-07-13: same token, same URL, UA the only variable -- None -> 403, curl/8.5.0 -> 200.

Fix. Send an accepted User-Agent header. This applies to pynetbox too (it sends python-requests/...), so the repo's NetBox tooling must set it:

nb = pynetbox.api(url, token=tok)
nb.http_session.headers["User-Agent"] = "curl/8.5.0"

Do not diagnose this as a token/permission problem and go re-mint credentials -- that is the trap. Confirm with curl first; if curl works and Python does not, it is the UA.

Note: this is an UPSTREAM-only trap so far. The Office1 sandbox NetBox (4.6.4, no proxy in front) accepts the default Python UA. Upstream is 4.5.8 and also uses v1 (40-char, bare) tokens, while the sandbox is 4.6 and uses v2 (nbt_<key>.<plaintext>). Any tool pointed at BOTH must handle both auth shapes and the UA -- do not assume they are the same NetBox.

================================================================================

references/script-authoring.md (loaded on demand in Claude Code; inlined here for Chat)

================================================================================

Script authoring - house style and hardening

Every script and paste block in this project follows these rules. They are not style preferences: each one encodes a failure that actually happened. Read this in full before writing ANY bash or python for this cloud.

Zeroth decision: does it already exist?

Before writing ANY operational script or check block, search the repo: grep -rli <topic> scripts/ runbooks/ - the deploy/verify surface is heavily scripted and duplicating an existing script creates drift (e.g. the haproxy backend sweep already exists as scripts/phase-03-core-verify.sh 3.1b). Route to, extend, or fix the existing artifact; write new only when nothing covers the need, and say which search came up empty.

First decision: what kind of block is this?

The error-handling regime depends on execution mode. Choosing wrong is itself a bug:

Executed script (bash script.sh, own process): set -uo pipefail at minimum; set -e acceptable and usually right, with the capture caveats below. Exit codes are the interface.
Interactive paste block (operator pastes into their shell): NEVER a bare exit (it kills their shell) and no set -e (it can kill their shell's options or abort mid-paste). Wrap the whole block in a subshell ( { ...; } ) so a stray exit is contained; signal failure by printing FAIL: ... lines the operator can read.
Verify/count-gate block (greps that legitimately return zero matches): run WITHOUT set -e, and end every count-grep with || true - a zero count is a valid answer, not an error (L1). bash -n cannot catch this; it is behavior, not syntax.

The header contract (executed scripts)

Every script opens with a comment block stating: path + argument synopsis; what it does (one paragraph, referencing the D-NNN/DOCFIX it implements); whether it mutates anything ("Mutates NOTHING" / what it changes and the gate protecting it); usage line; exit-code contract; and "ASCII + LF". House exit codes: 0 PROCEED, 1 HOLD (a gate failed), 2 precondition missing (tool absent, wrong model, helper not found). Then:

set -euo pipefail            # or set -uo pipefail; see regime above
shopt -s inherit_errexit 2>/dev/null || true
IFS=$'\n\t'

Source shared constants instead of restating them: . "$SCRIPT_DIR/lib-net.sh" (planes, VIP bands, helpers) and lib-hosts.sh (hostnames, octets, system_id resolution). If a value you need is not in a lib, consider adding it there rather than inlining a literal.

Hardening rules (each one is a scar)

SIGPIPE races break guards in BOTH directions. cmd | grep -q X under pipefail: on match, grep exits, the producer takes SIGPIPE (141), the pipeline reports failure despite the match. In an ... || die verify this FALSE-DIES on success; in an ... && die guard it FAILS OPEN -- the 2026-07 sweep found a duplicate-CIDR guard that let collisions through exactly this way. Treat every | grep -q on a live pipe as a defect regardless of which way the test points.

Pipefail + SIGPIPE race. cmd | grep -q X under pipefail falsely fails: grep -q closes the pipe on first match, SIGPIPE (141) kills the producer (juju ssh especially). Capture, then test:

OUT=$(cmd 2>&1 || true)
grep -q "pattern" <<<"$OUT"

set -e kills id-captures silently. ID=$(openstack ... || die ...) - if the subshell exits non-zero before your handler fires, set -e aborts the assignment line with no message. Append || true to each capture, then validate the captured value explicitly (see whole-output validation below).

Whole-output validation, never extract-then-check. Do not pipe raw output through awk/grep to extract a field and then test the fragment - a partial failure yields a plausible-looking fragment. Capture the WHOLE output, validate its shape (e.g. is_id(){ [[ "$1" =~ ^[0-9a-f]{32}$ ]]; } for a keystone id), and only then use it.

Centralize </dev/null in a wrapper, not per call. The house pattern is a 2-line helper (rc(), rcap(), J()) that appends </dev/null 2>&1 once; every call site then stays clean and un-forgettable. Heredoc-payload ssh (ssh ... bash -s <<'EOF') is the ONE exemption -- stdin IS the delivery there.

Inner stdin consumption. Any ssh/sudo/juju ssh inside a heredoc, pipe, or loop eats the remaining stdin and truncates the block. Append </dev/null to EVERY inner invocation (</dev/tty only when it genuinely must prompt).

read and ... | bash are mutually exclusive. A script piped to bash has the pipe as stdin, so read returns empty at EOF - this once silently created a passwordless user. For paste-safety AND working prompts: base64- decode to a file, then run the file (it inherits the terminal stdin).

juju invocation shape. juju ssh -m <model> - the -m flag goes BEFORE the target. juju run <unit> <action> output: use --format json for anything captured; confirm long actions via juju show-operation <N>, not the streamed log (a wait-timeout does not mean the hook failed).

Snap confinement. The openstack CLI snap cannot read /tmp - stage files under $HOME. Same for juju attach-resource payloads.

Client output ordering. openstack -f value -c X -c Y returns columns in ALPHABETICAL order, not flag order. When order matters: -f json | jq, or single-column queries. After any jq returning null, run jq 'keys' - key casing is command-specific (Title-Case in lists, hyphenated in quota show).

Environment isolation. Any block that switches identity runs in a subshell that first unsets all OS* vars: `( for v in $(env | awk -F= '/^OS/{print $1}'); do unset "$v"; done; export ... )ThreadOS_CACERT` explicitly into isolated subshells - it gets stripped. Confirm scope before acting: which project/domain the token holds, not which you intended.

Stable keys, not drifting IDs. Look up MAAS subnets by CIDR, machines by hostname, CAPI CRs by LISTING then operating on the exact returned name (the OpenStackCluster suffix is random per create - a wrong-name patch silently no-ops). system_ids are re-minted on re-enrollment.

Verify the launched cmdline, not the config text. For OpenStack debs run via LSB-init-wrapped systemd: a flag "present in a file" proves nothing. Gate on behavior - the init script's show-args, ps -ww -C <daemon> -o args on the live process, and probe under the daemon's RESTRICTED PATH (env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin sh -c 'command -v helm') - an interactive shell's PATH masks daemon-PATH failures.

sed is not a verifier. A non-matching sed -i exits 0 having changed nothing. Assert the post-edit content; never trust sed's exit code as proof of the edit.

ASCII + LF, validated. Non-ASCII: grep -nP '[^\x00-\x7F]' or a Python byte read. CR check: Python data.count(b"\x0d") - grep $'\r' false- positives on $r... tokens. Non-ASCII in conf.d has silently killed daemons.

Python helpers live in .py files tested against fixtures - no inline python-in-bash beyond one-liners. Do not assume jq exists off-jumphost; gate on it (need_jq) or use python3.

Secrets in scripts: whitelist-write to 0600 files under a 0700 dir (umask 077 first), never echo, never argv, measure secret length from the file rather than asserting an expected length (this deployment's app-cred secrets are 86 chars, not the commonly assumed 43 - never hardcode either).

Structured-output captures take stdout ONLY; never merge stderr. x=$(openstack ... -f json 2>&1) is a defect: any stderr line (a --long deprecation warning, an SDK notice) lands at char 0 and kills the JSON parse -- this fail-closed a live SG audit in 2026-07. Capture stdout alone; send stderr to a tempfile; surface it on parse failure, never discard it (2>/dev/null trades the bug for silent error-masking). The vr_json helper in scripts/lib-validate.sh encodes this. Also drop deprecated flags that only add stderr noise (--long is unneeded; fields are present without it).

Fixture-test throwaway probe one-liners like real code. A measurement ss/tcpdump/awk pipeline is code, and it has the same parse traps: ss DROPS the State column under a state established filter (peer is then $NF, not $5); cut -d: splits IPv4-mapped IPv6 ([::ffff:10.0.0.1]:443) at the wrong colon; grep -oE 'src [0-9.]+' greedily eats the trailing . into the next field; charmed magnum logs to /var/log/magnum/magnum-conductor.log, NOT journald. One measurement chain in 2026-07 shipped four such bugs before each was caught. Run the parse against a fixture line (mapped-v6, plain-v4, the real column layout) BEFORE handing the operator a probe -- the same discipline as a committed script.

Testing: nothing ships on `bash -n`

bash -n validates parse, not behavior - and most of the failures above are behavioral. Every nontrivial script gets an offline regression harness at tests/<script-name>/run-tests.sh in the established pattern:

fakebin/ contains fake juju / openstack / maas / kubectl / ssh executables that replay fixture output and log the calls they receive; real coreutils stay real.
The harness sets PATH="$BIN:$PATH", a scratch HOME, and env-injected fixture paths, runs the target, and asserts BOTH the exit code and an output regex per case: run <want_rc> <regex> <label>.
Cover: the happy path, each gate's failure branch (asserting HOLD not crash), each precondition (exit 2), and any DOCFIX behavior the script claims (e.g. "verify-first never creates a rule when one exists").
Read an existing harness (e.g. tests/phase-07-conductor-graft/) before writing a new one; match its shape.

For MUTATING scripts, the harness fakebin is STATEFUL: the fake maas/juju advances a phase file on each mutation call so post-mutation verification reads post-mutation reality (see tests/phase-00-teardown-d061 -- canary survival, decompose-detection, substrate-collision aborts). A --no-prompt flag on a destructive script exists FOR its harness, nothing else.

Harnesses are HERMETIC: fakes shadow, deletion un-shadows. Never simulate an unavailable tool by DELETING its fake -- on a real host that un-shadows the real binary and the test falls through to live infrastructure (a jumphost gauntlet run reached real Charmhub this way). Simulate failure with a FAILING fake (prints a realistic error, exits nonzero); keep the fail-loud-on-unmatched arm in every fake so an uncovered call can never silently hit a real CLI.

A script migration commit MUST carry its harness. The D-060 revert updated the scripts and left their harnesses testing the retired D-058 world -- red-at- HEAD tests that trained everyone to ignore red. If you change a script's behavior or vocabulary, the same commit updates fixtures and expectations.

Static contract: bash scripts/repo-lint.sh must exit 0 before anything is delivered (L1 ASCII/LF, L2 stale tokens, L3 ghost refs, L4 deprecated refs, L5 numbering, L6 bare invocations, L7 clientdocs sweep receipt). Changing any tenant-facing surface (tenant-.sh, the onboarding contract, tenant runbooks, the policy zip, clientdocs/ itself) trips lint L7 until clientdocs/ has been reviewed for staleness and the receipt is re-recorded: bash scripts/repo-lint.sh --record-clientdocs-sweep (the sweep IS the review; recording without reviewing defeats the rule). A guard script that must NAME stale tokens opts out per-file with a repo-lint: allow-stale-tokens marker in its header. Committed binaries (e.g. policies/overrides.zip) need a `.zip binary` .gitattributes rule or the LF normalization corrupts them in transit. Introducing a tool dependency (jq, python module) means adding its presence gate in the same change.

Read-only verify scripts DETECT and report; remediation stays a gated human step even when the fix is obvious - the script's job is evidence, the operator's job is the mutation decision.

Mock worlds must be TRUTHFUL to the real CLI/contract, and test expectations must match the SPECIFIED contract. A mock that emits a 409 must actually have the resource pre-exist; a mock of an openstack list command must emit the real deprecation warning ON STDERR (or the stderr-separation rule above never gets exercised); a harness assertion that contradicts the documented exit contract (e.g. expecting a lone SKIP to lower the overall verdict when the contract says SKIP never lowers it) is a TEST bug, not a code bug -- fix the test. Three 2026-07 "failures" were untruthful mocks or wrong assertions, each caught by running the harness before delivery.

================================================================================

references/troubleshooting.md (loaded on demand in Claude Code; inlined here for Chat)

================================================================================

Troubleshooting - method, reflexes, and routing

The repo's runbooks/appendix-A-troubleshooting.md is the operational symptom->cause->fix index, keyed by D-NNN/DOCFIX-NNN (it now includes the mysql-innodb-cluster recovery signatures and a point-of-use identifier index). ALWAYS check it before diagnosing from first principles - most failures on this cloud have been seen before and have a validated fix. ops-capi-recovery.md covers the CAPI/Magnum stack; runbooks/ops-restart-procedure.md covers full-cloud recovery. This file gives you the method and the reflexes that must fire BEFORE and WHILE you read those.

Substrate failures (libvirt/KVM guest won't boot, AppArmor, MAAS+LXD, OPNsense edge): these are NOT in appendix-A. Go to references/platform-traps.md -- it ends in a verbatim-error -> cause index keyed to exact strings (panic: running without device atpic requires a local APIC, Illegal variable name., a bare Permission denied that never names AppArmor, -m size=2048k, ...). Match the string BEFORE hypothesizing.

VR1 DC-DC track: appendix-A above is VR0-specific -- nothing in the VR1 DC-DC track (docs/dc-dc-deployment-workflow.md) has been executed yet, so there is no equivalent incident index for it (an OpenTofu/MAAS-vm_host/ OPNsense-edge failure will not be in appendix-A). Apply the SAME triage method below regardless, but do not expect a pre-catalogued fix for a VR1- specific symptom -- log a genuinely new finding as a DOCFIX/D-NNN candidate per this repo's own discipline rather than assuming it must already be documented somewhere.

Triage method

Reproduce and capture verbatim. The exact error text routes you in appendix-A. Paraphrased symptoms match nothing.
Eliminate cheap layers first. Web UI misbehaving: incognito/client state (~10s) before any server hypothesis. CLI misbehaving: confirm the token scope (project/domain) before blaming the service - wrong ambient scope produces misleading 404s/403s, especially with domain-per-client identity (an un-domain-qualified lookup resolves in the WRONG domain).
Establish what is actually true, live. bash scripts/cloud-assert.sh first - it runs every service-own-verdict gate in one read-only sweep and localizes the failing layer. Then the targeted read-only audit of that surface before touching anything. juju status, the service's own state (haproxy admin socket, mysql cluster-status, vault status, ovn-appctl), the relevant logs. Hard rule 2: measurements, not memory.
Match against appendix-A, then design-decisions if it smells architectural. If matched: follow the recorded fix exactly (verbatim- reference rule for anything one-shot).
If unmatched: smallest-possible hypothesis, read-only test to confirm or kill it, and log the finding. New root causes become appendix-A / DOCFIX material - capturing them is part of the fix.
Least-destructive first, individually gated. Reload before restart, restart before rebuild, rebuild before redeploy. Never batch.

Reflexes (internalize these; they override intuition)

juju-green is necessary, not sufficient. active/idle is BLIND to: a DOWN haproxy backend (D-045 - hid a dead nova-api for 3 days), a missing magnum trustee domain (D-046 - magnum reports ready regardless), an unparsed-but-attached policy override, and per-backend/service state generally. Gate on the service's OWN verdict (admin sockets, functional probes), never on juju status alone.
"Reports OK while broken" generalizes. Charm-side validation is often parse-level only (the keystone policy zip validates YAML, not semantics). Acceptance is BEHAVIORAL: prove the capability works, not that the config is present.
Distinguish down from thrashing. A host that looks "down" to juju/OVN (ovsdb inactivity-probe storms) may be swap-thrashing, not down (D-040). Check who -b / uptime / journalctl -k | grep -i oom before treating it as an outage.
Read the reason field before the status field. Magnum health UNHEALTHY has three signatures (D-042 amendment): reason EMPTY = conductor cannot reach the mgmt API (VM down?); all-Ready except infrastructure "not found" = the cosmetic driver-contract miss; a reason citing LB failure = real, check Octavia. Same status, three different responses.
Single-node mgmt VM does not self-heal (D-035/D-041). Workload nodes wedged with the uninitialized taint, magnum reconcile dead, addons Pending -> check capi-mgmt-v2 is ACTIVE before anything else. Manual start is POLICY (down is a signal), not a defect.
Known blast-radius traps: juju destroy-model decomposes MAAS pod-composed machines - machine retention is juju remove-machine --keep-instance (D-061); mysql-innodb-cluster never bootstraps at num_units 1 - deploy at 3 (D-062); reboot-cluster-from-complete-outage is destructive against an already-healthy cluster - check cluster status FIRST; vault restarts sealed by design - sealed-after-reboot is expected, not a fault; the magnum domain-setup action must be re-run after every redeploy (D-046).
Green-in-the-shell, broken-in-the-daemon. Interactive shells have a different PATH, env, and stdin than LSB-init daemons. Verify under the daemon's conditions (restricted PATH, live-process args) - see script-authoring.
The absence of an expected resource is a finding, not a gap to fill. If something that should exist does not (a domain, a rule, an image property), find out WHY it is absent before recreating it - it may have been removed deliberately (check design-decisions) or its absence may be the actual root cause several layers up.

Prove event-driven flows by FUNCTIONAL OUTCOME, not packet/snapshot forensics. For a bursty, per-operation client flow (e.g. the magnum conductor's helm/watch to the mgmt apiserver), an instantaneous ss snapshot or a short tcpdump window will usually catch NOTHING -- the connection is idle at rest and short-lived under load. That absence is expected, not a fault. The right proof is a forced real operation that SUCCEEDS end-to-end (a cluster resize reaching UPDATE_COMPLETE proves the conductor reached the mgmt apiserver through the boundary), plus a TCP handshake from the real source. Reach for established-state or a forced functional op before concluding a flow is broken; a service's own successful state transition outranks a captured packet.

Escalation and blockers

Missing vault unseal keys, lost one-shot secrets, or anything requiring material only the operator holds: STOP and escalate. Do not improvise around a recovery blocker (a vault re-init wipes the TLS plane).
If a fix would deviate from a runbook or contradict a D-NNN: log it as a proposal and get a ruling. Mid-incident is exactly when discipline pays.
Two failed fix attempts on the same hypothesis = the hypothesis is wrong. Step back to the audit stage; widen what you are measuring.

================================================================================

references/opentofu-provider-docs.md (loaded on demand in Claude Code; inlined here for Chat)

================================================================================

OpenTofu / provider documentation -- sources, versions, and fetch method

Read this before writing or extending anything under repo opentofu/ (the VR1 DC-DC infrastructure-as-code, D-103). Built up during the 2026-07-09 OpenTofu delivery; see repo opentofu/README.md for the full, per-module account (schema notes, an "Audit pass" section, and every finding's exact sourcing) -- this file is the index into that plus the fetch methodology, so a future session does not have to rediscover it by trial and error.

The core lesson: doc-summarization is lossy, real example code is not

Every provider doc page fetched this session went through a summarizing model, which reliably got FIELD NAMES right but repeatedly lost or mis-stated CALL SYNTAX (block vs. attribute style, exact nesting). The opentofu/ scaffold shipped with a real syntax bug from trusting doc summaries alone (fixed once real example .tf files were fetched and read directly -- see repo opentofu/README.md's syntax-bug note). For any new provider/resource: find real example .tf/.go source before trusting a doc-summary's syntax claims. Doc-summaries are fine for "what fields exist"; they are not reliable for "how do I write this."

Fetch gotchas

The Terraform/OpenTofu Registry's own doc pages (registry.terraform.io/ providers/.../docs) are client-side-rendered -- a plain fetch returns only the page shell, no content. Go straight to the provider's GitHub repo instead.
GitHub's raw file URLs need the CORRECT default branch (varies: main for some repos, master for others -- check api.github.com/repos/<org>/<repo> for the default_branch field first, don't guess).
A general web search's own synthesized "summary" of a doc page can itself be wrong (encountered once this session: a search summary confidently named the wrong branch and file path). Treat search-result summaries as leads to verify, not answers -- always follow through to the primary source.
When a fetch refuses to "reproduce verbatim" (copyright caution), ask for a specific, narrow thing instead -- "list bare field names only, no prose" reliably works where "reproduce this page" does not.
api.github.com/repos/<org>/<repo>/contents/<path> (a JSON directory listing) is a reliable way to discover the REAL current file layout when a guessed raw-file path 404s -- doc reorganizations happen (one provider moved from website/docs/r/*.markdown to docs/resources/*.md between the versions training data and current reality disagree on).

Providers used in `repo opentofu/`

`dmacvicar/libvirt` (pinned `0.9.8`, confirmed 2026-07-09)

Before writing any libvirt_domain, read references/platform-traps.md section 1. Four schema-level traps there have each cost real time and none of them is a validate-time error: bare memory is KiB not MiB; no features block means acpi=off; no cpu block means no svm (nested KVM silently impossible); and "will be updated in-place" still BOUNCES the guest.

Registry: https://registry.terraform.io/providers/dmacvicar/libvirt/latest (JS-rendered -- use the GitHub repo for actual content)
GitHub: https://github.com/dmacvicar/terraform-provider-libvirt (default branch main; there is no CHANGELOG.md -- the release NOTES on the releases page are the changelog, and they are unusually informative: .../releases/tag/v0.9.0 is the breaking-rewrite rationale, v0.9.1 is the 0.9.0->0.9.1 attribute-rename migration guide)
Version boundary that invalidates most examples on the internet: 0.9.0 (2025-11-08) is a compatibility-BREAKING rewrite (plugin framework, "HCL maps almost 1:1 to libvirt XML"); the legacy provider lives on the v0.8 branch. 0.9.1 then code-generated the schema and renamed attributes again. Anything 0.8-era -- including model memory -- is wrong for this pin.
Docs: docs/resources/*.md, docs/data-sources/*.md (NOT the older website/docs/r/*.markdown layout some search results/training data reference -- that's stale)
Real example .tf files (the higher-value source): examples/*.tf -- domain_with_network.tf, alpine_cloudinit.tf were the two that resolved every syntax question this session. Fetch these FIRST for any new resource, before the doc pages.
Key fact: this provider code-generates its schema as a 1:1 mirror of libvirt's own domain/network/pool XML (confirmed via a general search of provider discussion, not the docs pages themselves) -- when a Terraform schema question stalls, libvirt's own native XML docs (https://libvirt.org/formatdomain.html, https://www.libvirt.org/pci-addresses.html) are a legitimate, stable cross-reference for what a nested attribute is likely to be named/shaped, since the mapping is direct.

`canonical/maas` (pinned `2.7.2`, confirmed 2026-07-09)

Registry: https://registry.terraform.io/providers/canonical/maas/latest (same JS-rendering issue -- use GitHub)
GitHub: https://github.com/canonical/terraform-provider-maas (default branch master -- NOT main, confirmed by checking default_branch directly rather than assuming)
Docs: docs/resources/*.md, docs/index.md (provider block config)
Moved from an older maas/maas registry namespace -- use canonical/maas, not maas/maas, for anything current.
Key lesson (see repo opentofu/README.md's "MAAS registration + tc netem research" section for the full account): maas_vm_host (register a virsh chassis) and maas_vm_host_machine (MAAS composes/creates a new VM itself) are NOT interchangeable -- read the schema (which fields are INPUTS) before assuming a resource matches a design intent just because its name sounds related.

OpenTofu core (language features, not a provider)

local-exec provisioner: https://opentofu.org/docs/language/resources/provisioners/local-exec/
terraform_data (built-in, replaces the older null_resource for provisioner-only use): https://opentofu.org/docs/language/resources/tf-data/ (note the URL slug is tf-data, not terraform_data or terraform-data)
Both pages rendered cleanly on a direct fetch (static docs, no JS-rendering issue) -- these were the most reliable fetches of the whole session.

OPNsense (not a Terraform provider -- the guest OS itself)

SUPERSEDED RESEARCH WARNING (2026-07-12/13). repo opentofu/README.md's "OPNsense deployment research" section and the 2026-07-09 changelogs describe a config-ISO / Configuration-Importer delivery path. That path is DEAD and its code is DELETED:

D-112: the Configuration Importer can NEVER fire on a pre-installed nano image (upstream opnsense-importer probes for a read-only root and exits before scanning media). The ISO was inert; nothing was ever going to read it.
D-113 (ADOPTED 2026-07-13, option a2): OPNsense STAYS, but config moves to the REST API. opentofu/templates/opnsense-config.xml.tmpl, scripts/opnsense-render-config.sh and scripts/opnsense-build-config-iso.sh are DELETED, along with the module's config_seed volume + cdrom disk. A rendered-config.xml push would now CLOBBER live API-managed DHCP.

Current path: scripts/opnsense-api.sh (REST client) + scripts/opnsense-bootstrap-apikey.sh (mints a key via OPNsense's own model over SSH). Traps, exact endpoints, the mandatory service/reconfigure step, the 60-second firewall auto-rollback, and the tcsh trap: references/platform-traps.md section 4.

Official docs: https://docs.opnsense.org/manual/install.html; REST API: https://docs.opnsense.org/development/how-tos/api.html; 26.1 release notes (ISC-DHCP left core; dnsmasq is the new default): https://docs.opnsense.org/releases/CE_26.1.html
Still-valid finding from the 2026-07-09 research: cloud-init is unreliable on FreeBSD, so modules/cloudinit-vm genuinely does not apply to OPNsense (https://forum.opnsense.org/index.php?topic=42517.0), and the nano image (not the installer image) is the one to use.
Historical only (the ISO mechanism these describe cannot fire on nano): https://github.com/opnsense/core/issues/5733, .../issues/10017.

Where the full findings live (don't duplicate here, it will drift)

Caveat first (2026-07-13): repo opentofu/README.md predates D-112/D-113 and its "OPNsense deployment research" section still describes the DELETED config-ISO path -- see the superseded-research warning above before acting on anything it says about OPNsense. Its libvirt schema notes also predate the four domain traps in references/platform-traps.md section 1. Everything else in it still stands.

repo opentofu/README.md is the authoritative, per-module account -- schema notes, what's confirmed vs. assumed per attribute, and a dated "Audit pass" section recording a later re-review (a real documentation error was found and fixed there: an early claim that libvirt interface list-order directly sets OPNsense's LAN/WAN role conflated two separate mechanisms -- read that section before trusting anything about NIC ordering/roles). Treat this file as the index/methodology; treat that one as the ground truth for what each module actually does and why.

openstack-cloud-ops -- consolidated skill (Chat upload snapshot, 2026-07-25)

SKILL.md (main entry -- always in context)

openstack-cloud-ops

Step 0 - locate the source of truth

Step 1 - detect the environment

The three hard operating rules (non-negotiable)

Routing - where to go for what

Standard loops (repeatable session shapes)

Record authority and session discipline (GA-R1..R7, ratified 2026-07-18)

Posture

references/operating-discipline.md (loaded on demand in Claude Code; inlined here for Chat)

Operating discipline

The gated execution model

Irreversible / one-shot / secret steps

Record-keeping: D-NNN / DOCFIX-NNN / BUNDLEFIX-NNN

Delivery rules (files handed to the operator)

Session continuity: the in-flight ledger

Boundary mutations: adds-before-removes with a readback gate

Attestation gates for unautomatable safety items

Debate and correction norms

Troubleshooting entry discipline

Guards are load-bearing: reword the content, never weaken the check

references/environment.md (loaded on demand in Claude Code; inlined here for Chat)

Environment - Omega Cloud (VR0 DC0 testcloud)

The three deployments (this file covers VR0 in detail; VR1 is a pointer)

Stack (verify against appendix-B for pins)

Control points

The six network planes (D-052 / D-053; verify against scripts/lib-net.sh)

Repo map (what lives where)

Identity / tenancy model (see appendix-C/D and D-051, D-064, D-066)

references/platform-traps.md (loaded on demand in Claude Code; inlined here for Chat)

Platform traps and version pins -- libvirt/KVM, MAAS+LXD, OPNsense

Version pins that actually matter (and why)

1. dmacvicar/libvirt provider (pinned 0.9.8)

1a. The 0.8 -> 0.9 break invalidates most examples you will find

1b. Bare memory is KiB, not MiB -- 1024x too little RAM (DOCFIX-188)

1c. No features block => acpi=off => FreeBSD panics (DOCFIX-190)

1d. No cpu block => generic CPU => NO svm => nested KVM impossible

1e. "will be updated in-place" DOES NOT mean "the guest stays up"

1f. Two more upstream behaviours worth knowing (from the release notes)

1g. forward=bridge networks + br_netfilter: a global FORWARD-drop silently kills bridged frames (D-125)

2. AppArmor blocks qemu on any non-default pool path

3. MAAS + LXD

3a. MAAS 3.6/3.7 is INCOMPATIBLE with LXD >= 6.7 (D-114)

3b. LXD's own bridge runs a dnsmasq DHCP server -- MAAS says turn it off

3c. Two different LXDs in this cloud -- do not conflate them (D-114)

4. OPNsense edge (traps entered at 26.1; they persist on the live 26.7)

4a. config.xml is GUI-owned. Do not hand-author it. (D-112, D-113)

4b. ISC dhcpd is GONE from core in 26.1 -- an <dhcpd> config is inert

4c. REST API: auth, endpoints, and the reconfigure step everyone forgets

4d. root's shell is tcsh -- $(...) dies QUIETLY over SSH

Verbatim error -> cause index (grep this first)

NetBox (upstream, netbox.baldurkeep.com)

HTTP 403 Forbidden on every API call -- but the SAME token works from curl

references/script-authoring.md (loaded on demand in Claude Code; inlined here for Chat)

Script authoring - house style and hardening

Zeroth decision: does it already exist?

First decision: what kind of block is this?

The header contract (executed scripts)

Hardening rules (each one is a scar)

Testing: nothing ships on bash -n

references/troubleshooting.md (loaded on demand in Claude Code; inlined here for Chat)

Troubleshooting - method, reflexes, and routing

Triage method

Reflexes (internalize these; they override intuition)

Escalation and blockers

references/opentofu-provider-docs.md (loaded on demand in Claude Code; inlined here for Chat)

OpenTofu / provider documentation -- sources, versions, and fetch method

The core lesson: doc-summarization is lossy, real example code is not

Fetch gotchas

Providers used in repo opentofu/

dmacvicar/libvirt (pinned 0.9.8, confirmed 2026-07-09)

canonical/maas (pinned 2.7.2, confirmed 2026-07-09)

OpenTofu core (language features, not a provider)

OPNsense (not a Terraform provider -- the guest OS itself)

Where the full findings live (don't duplicate here, it will drift)

1b. Bare `memory` is KiB, not MiB -- 1024x too little RAM (DOCFIX-188)

1c. No `features` block => `acpi=off` => FreeBSD panics (DOCFIX-190)

1d. No `cpu` block => generic CPU => NO `svm` => nested KVM impossible

1g. `forward=bridge` networks + `br_netfilter`: a global FORWARD-drop silently kills bridged frames (D-125)

4a. `config.xml` is GUI-owned. Do not hand-author it. (D-112, D-113)

4b. ISC dhcpd is GONE from core in 26.1 -- an `<dhcpd>` config is inert

4d. root's shell is tcsh -- `$(...)` dies QUIETLY over SSH

NetBox (upstream, `netbox.baldurkeep.com`)

`HTTP 403 Forbidden` on every API call -- but the SAME token works from `curl`

Testing: nothing ships on `bash -n`

Providers used in `repo opentofu/`

`dmacvicar/libvirt` (pinned `0.9.8`, confirmed 2026-07-09)

`canonical/maas` (pinned `2.7.2`, confirmed 2026-07-09)