| 2026-08-03 |

my reap poll reported REAPED while the controller was down -- absence-test defect
...
The background wait-loop polled 'juju models | grep -q name: admin/vr1-dc0' and
broke when the grep MISSED, reporting 'REAPED after ~2730s'. The model was still
there. With the API refused, juju models prints NOTHING, the grep matches
nothing, and the absence-test read empty output as 'gone'. Verified after: port
17070 still REFUSED, model still present.
This is verbatim the rule this repo already states -- 'could not look' is never
'nothing there' -- failing in a checker I wrote in the same session I wrote that
sentence into this very runbook. Same command also used 'cmd | head -3; rc=0',
which captures head's status, not the command's.
Path M gains a second instrument warning: poll on a POSITIVE signal -- require
juju models to actually return model lines, then ask whether the target is among
them; treat empty output or a non-zero juju exit as REFUSE, never as gone.
No mutating action taken. Controller remains down; awaiting the operator's
decision on the rebuild.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

model destroyed OK, then --force orphaned it and took the controller DOWN
...
destroy-model returned 'Model destroyed.' EXIT 0 with a clean progressive drain
(36/56 -> 26 -> 20 -> 16 -> 9 -> 0) and a correct MAAS release: census 10, count
UNCHANGED, nine nodes Ready/owner=None, controller VM still Deployed. Proxies
PASS. M.4 model-defaults corrected and read back.
Then add-model refused: model already exists, stuck at life: dead. Root cause
from the controller log -- the destroy left the model doc alive while its STATUS
doc was gone. undertaker crash-loops on 'cannot set status: model not found';
modelcache crash-loops on 'status doc <uuid>:e not found' (732 iterations in ~12
min). The API server depends on modelcache, so 17070 is connection refused and
THE CONTROLLER IS DOWN -- while the VM pings and systemctl reads active. A
jujud restart did not fix it. LP #1737487 class.
OWNED, two of my decisions are implicated: I carried --force --no-wait over from
the 07-31 stall where agents were STOPPED and force was genuinely required; this
run's agents were ALIVE and draining, so force was almost certainly unnecessary
and it is the documented cause of this inconsistency. And when the model would
not reap I re-issued destroy against a model already dead, which moved it back
to dying and re-armed the loop.
Path M amended: try without --force when agents are alive; reserve --force
--no-wait for the measured stall; never re-issue destroy on a dead model; if the
controller is already down on this symptom, rebuild the controller rather than
attempt state-DB surgery.
Not degrading: nine nodes Ready, MAAS healthy, both proxies PASS, model-defaults
now controller-level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Path M: the juju MODEL teardown + release path, from the runs we already logged
...
Operator: "You have that information from previous teardowns for the juju
release path ... find the previously logged information in the repo. Log the
commands and steps for future reference ... this has bit in the past and will be
a reusable and needed set for the future."
I had said the release path was the one thing I could not prove in advance. It
was already proven and logged, in CURRENT-STATE under MODEL TEARDOWN 2026-07-31.
Two corrections to the command I proposed:
1. --no-wait BELONGS IN IT and I had excluded it, reasoning from juju help that
it was reckless. Measured 07-31: the plain destroy-model STALLED TERMINALLY --
'attempt 30 ... model not empty, found 26 machines, 37 applications', flat ~19
minutes, application set byte-identical, because ALL 26 agents were stopped so
no teardown hook could execute. --force --no-wait cleared it (18->5->2, then
Model destroyed.). The help text talks you out of the flag that works.
2. The release path is proven, not unknown: all nine role nodes came back
Ready/owner=None, zero stranded, no maas machine release needed or run.
Expected post-state is 9 Ready + 1 Deployed -- the controller VM stays
Deployed in the controller model, so expecting 10 raises a false alarm.
Also recovered, and live for this rebuild: destroy-model TAKES THE MODEL CONFIG
WITH IT. On 07-31 that silently removed apt-mirror and the spaces work and
nothing in the repo would have caught it. Measured today: this controller's
model-defaults carry apt-mirror=http://10.12.8.4/ubuntu and nothing else -- wrong
after the convergence -- while the three settings the deploy needs were set at
MODEL level and will be destroyed. Defaults apply to NEW models only, so they
must be fixed BEFORE add-model.
And the VR0 pod warning does not transfer: VR0's virsh-POD MAAS decomposes
pod-composed machines on destroy-model; VR1 uses per-machine power_type=virsh,
not pods, so there is nothing to decompose.
The teardown runbook documented only SUBSTRATE teardown -- the model layer had no
procedure, which is why this kept being re-derived. Now Path M, M.0-M.6.
Final command: juju destroy-model vr1-dc0 --force --no-wait --no-prompt
BLOCKED by the permission layer and NOT run; no workaround attempted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |

Items 3.9 + 3.10: the teardown false clear existed in three places, not one
...
3.9 was the readiness audit's most dangerous item -- the teardown runbook is what an operator
reaches for DURING a failed Stage 5, under time pressure.
The false clear was in THREE places, not the one the audit named. Besides Step 2's vm-host
read -- which can never return, since this repo uses per-machine power_type=virsh and
instantiates no maas_vm_host module -- the same wrong premise sat in the "READ BEFORE ANY DC
TEARDOWN" header telling the operator to remove the maas-vm-host record, and in the
Relationship-to-D-061 claim that no VR1 DC had reached Stage 4. The header instance is the
consequential one: it is read FIRST, so it bypassed any fix confined to Step 2.
Step 2 is now a two-lens MACHINE census run from the headend (maas is measurably absent on
vcloud): lens 1 enumerates what exists and ends in a countable RECORDS REQUIRING ATTRIBUTION,
lens 2 corroborates against lib-hosts pinned boot MACs and exits 1 on any hit. Demonstrated
three ways against a fixture -- records present -> exit 1, zero -> exit 0, empty roster ->
REFUSE -- so it is a gate rather than a formality. The 2026-07-21 pod-cascade precedent is
retained as the reason associations are read before any destroy.
Step 3's six phantom module.dc1_* targets are retired; all 8 targets now resolve, verified
independently against ^module "X" across all three roots. The real insight: scoping a DC is a
ROOT choice, not a -target choice, since each substrate root holds exactly one site.
Step 4's VERIFY moved to qemu+ssh from the headend behind a virsh version REFUSAL, because an
unreachable URI, a stopped VM and a bad key otherwise return the same empty result. The
decision tree gains "no branch reaches a destroy without Step 2 passing" -- it never mentioned
the MAAS gate -- and the virsh destroy vs tofu destroy verb distinction. Also fixed: Step 1
backed up the wrong state file, "Two paths" pointed twice at a nonexistent Step 6, and $REPO
silently meant two different clones.
3.10: item 17 CLOSED with measured evidence, and its own stated fix corrected (it closed by
D-125 bridge-in, not the replication the entry claimed). Item 19 disambiguated 19a/19b rather
than renumbered, because both are cited by number from outside the file. Item 20 MEASURED, not
asserted -- verdict no leg required, with the rule mismatch written in rather than resolved
silently, plus an expiry condition; the voffice1-side reboot durability recorded as UNMEASURED
with the commands that would resolve it.
repo-lint 0 fail / 615 files, both files ASCII+LF.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-19 |

Batch 4 C6: docs disposition -- 24 history docs archived, 16 retained, refs rewritten
...
DISPOSITION MANIFEST (archive, one-line justifications): clientdocs-
workflow-review (v1 tenant-docs era) / D-057-DECIDED-append +
D-057-REVIEW-ITEMS + README-D057-PACK + D-058-renumber (executed v1
decision packs) / D-068-openbao-assessment-DRAFT (supplementary; the
cited analysis retained) / DOCFIX-064-phase08-changelist + docfix-draft-
20260702 (executed/draft) / handoff-20260703 + handoff-20260705
(superseded handoffs) / incident-20260712-triplefault (trap routed to
platform-traps 1b/1c long ago) / netbox-vip-queue + phase-00-maas-
standup-notes + script-quality-findings + session-findings-2026-07-02 +
tenant-cidr-overlap-PLAN + v1-pre-deploy-fixes (v1-era executed/dated) /
repo-lint-nextfree-bug-FINDING (absorbed DOCFIX-105/107; repo_lint.py
pointer updated) / upstream-bug-draft-dashboard-tls (draft, dated) /
model-a-fallback-plan (D-123 Model B adopted; fallback preserved in
archive) / stage3-adversarial-review-20260716 (R3-F register,
dispositioned in 2.10) / dc-dc-replication-DR-seed (absorbed per
workflow) / dc-dc-ipv6-charm-research + dc-dc-netem-and-ula-gua-proposal
(stage 4+ research, refs updated). RETAINED 16 top-level, justified in
the session changelog. Live-surface refs to all moved files rewritten to
archive paths (18 files). G3 cell + ledger + changelog updated same
commit (the L10 fire on a Status-line path rewrite is satisfied by the
CURRENT-STATE touch -- the check worked as designed).
Revert: git revert this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi
|
| 2026-07-16 |

docs(teardown-rollback): Model B reconciliation banner (two-root DC teardown)
...
runbooks/dc-dc-teardown-rollback.md was written 2026-07-10 for the SINGLE-ROOT
shape (OpenTofu creates DC node-VM domains directly on vcloud; maas-vm-host
registers the vcloud host). Under D-123 Model B that is operationally wrong for
the DCs and a mis-directed rollback is a hazard, so added a top-of-file
reconciliation banner (body not yet rewritten). Claims grep-verified:
- TWO roots / TWO state files: OUTER (opentofu/, vcloud) owns vvr1-dc0 + transit
+ the D-125 vcloud ISP uplink; INNER (opentofu/vr1-dc0-substrate/) owns the 9
node VMs + 6 planes + vr1-dc0-wan + edge + inner pool ON vvr1-dc0's libvirt.
- maas-vm-host now registers vvr1-dc0's INNER virsh -> Office1 region.
- Whole-DC site-down is ONE object: virsh destroy vvr1-dc0 (D-122/D-123).
- Teardown ORDER: inner root first (or virsh destroy vvr1-dc0), then outer;
D-061 "clean up MAAS record before destroying libvirt" still applies.
- Model B -> A revert is docs/model-a-fallback-plan.md, not this runbook.
Office1/mesh/pool steps unaffected. Doc-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|
| 2026-07-10 |

DOCFIX-175/176: OpenTofu state file security + VR1 teardown/rollback runbook
...
Found while reviewing what's missing from the VR1 buildout ahead of
tomorrow's execution.
DOCFIX-175: terraform.tfstate stores sensitive=true variables (like
maas_api_key) in plaintext regardless of the flag, which only redacts
CLI/plan output (verified against HashiCorp's own docs). Added .gitignore
coverage, a "State file handling" section in opentofu/README.md, an inline
caveat on maas_api_key's description, and a "SECURE THE STATE FILE" callout
at Stage 1 Step 10 -- the actual first real tofu apply (an early draft
wrongly assumed Stage 3; corrected after reading the runbooks directly).
DOCFIX-176: new runbooks/dc-dc-teardown-rollback.md for the OpenTofu/
libvirt/MAAS-vm_host layer, which D-061's existing teardown scripts don't
cover (different layer, juju/MAAS-machine). Two paths (scoped -target
teardown or full VR1 destroy), a rollback decision tree favoring
fix-forward over destroy, and explicit mesh-link shared-infrastructure
handling.
Also folded in: a fix to docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md
after re-running ledger-scan.sh caught that its own narration (and the
session-ledger mirror) had reintroduced the exact false-positive class
DOCFIX-174 had just fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|