Stage 5 dc0: remediate ceph-mon/2 + ceph-radosgw/0 "allocating" (apt CLOSE-WAIT); durable capture ...

Live-cloud, operator-gated (2 exchanges). Two units stuck `allocating` were root-caused
NOT to a down proxy (both dc0 proxies PASS; security.ubuntu.com 200/0.67s) but to
cloud-init `apt-get update` parked in CLOSE-WAIT to 10.12.8.4:3142 for ~12.5h (apt has no
client read timeout) since the 08-04 redeploy. Rebuilt each from the rack (D-138):
remove-unit --no-prompt -> remove-machine --force --no-prompt (REQUIRED: remove-unit does
NOT cascade to a never-provisioned dead-agent container) -> add-unit --to <bundle
placement>. Both fresh containers' cloud-init finished (~211s); mons bootstrapped 3/3, 4
OSDs active, storage cascade cleared. A 2nd apt-cacher-ng mode (radosgw-hacluster 404 on a
rotated point-release -> exit 100) SELF-HEALED via juju hook retry. Measured after-state
17:34:08Z: census 62 active; remaining non-active all deferred-by-design + gss +
barbican-vault settling.

Durable capture (operator-directed "a then b"):
- docs/audit/stage5-dc0-ceph-remediation-20260805.txt: transcribed capture (NOT script(1));
  the before-state (CLOSE-WAIT sockets, ~45000s etimes, both containers) is unrecoverable
  and lives only here.
- docs/CURRENT-STATE.md: dated status block; measured 62 active (NOT the unmeasured 40/47,
  C2); records the --force fact and the phase-03 Step-3.1 gate defect (expects 1, VR1 has 4
  deferred + gss) as durable finding.
- runbooks/appendix-A-troubleshooting.md: NEW entry for the symptom pair (both modes +
  gated remove/--force/re-add remediation). Drafted; operator-reviewed.
- docs/changelog-20260805-stage5-dc0-ceph-remediation.md: session changelog (separate
  same-day file from the skill-DOCFIX changelog, disclosed in header); F2/F3/F4 owed,
  unnumbered.

repo-lint 0 fail / 1 legacy warn; ledger-scan DOCFIX next-free still 210 (no token leaked);
all touched files ASCII-clean. Status lives in docs/CURRENT-STATE.md ONLY.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728

Ruling 3 commit 1: bundle.yaml is VIP-free; dc0's VIPs are now a RENDERED overlay ...

The 11 inline vip: lines are gone from bundle.yaml and live in
overlays/vr1-dc0-vips.yaml, GENERATED by scripts/render-dc-overlays.py from
render/values/vr1-dc0-vips.yaml -- the renderer's first real output.
placement's then-empty options: block was dropped rather than left parsing
as null. Ruling 3's dual-stack ADD is commit 2 and is NOT in here.

PROVEN NEUTRAL by the ruling's own named check: provider-bundle-check on the
pre-extraction bundle and on the merged pair produce line-for-line identical
output, same known designate failure included. A structural diff separately
confirmed only vip moved -- applications 56 -> 56, relations 108 -> 108,
every other key byte-identical.

The 11 inline comments were carried, not dropped: extracted verbatim by
parser rather than retyped, via new optional per-app comment support in the
renderer. An app with no comment emits none, so dc1 still reproduces
byte-for-byte at 1634 bytes. The B1/B5 tokens stay DEFINED in bundle.yaml's
header and the overlay header names that, so the references do not dangle.

The blast-radius repair is the bulk of this, and every item was found BEFORE
the edit by the read-only agent sweep:

- preflight.sh P2 validated the bare base (11 phantom failures post-move) ->
  merged input.
- pre-flight-checks.sh CHECK 1 read the base as raw text and would have seen
  ZERO VIPs. This is BLOCKER-1, fixed the ruled way and not by bumping the
  count. Measured trap: the file sets IFS=$'\n\t', so a space-joined string
  reached grep as ONE filename, grep exited 2, pipefail propagated and the
  gate died silently mid-check. Fixed with a bash array.
- phase-01's RUN block gates juju deploy on 11/11/0 and would have read
  0/0/0 and ABORTED THE DEPLOY. Both guards now span base + overlay, and a
  second latent bug was fixed in the same edit: grep -c over two files prints
  one count PER FILE, so the bare $( ) captured a two-line string and every
  numeric test would have failed anyway. Now awk-summed; verified 11/11/0.
- phase-01's Octavia SAN derivation raised KeyError, swallowed by || true
  into an empty VIP. Now reads the overlay, deriving 10.12.4.57.
- phase-00 teardown, dc-dc-phase4's "NO VIP edits" guidance, appendix-A L3,
  and render-baseline's now-unverifiable bundle hash all repaired.

Harness 30 -> 31. The fixture base is now bundle + dc0 overlay, since that
pair is the deploy input; T16 renamed to match. NEW T31 asserts the bare base
fails for every clustered principal, so "deployed without its overlay" is a
tested state rather than an assumption.

Gauntlet ALL GREEN (85) on vcloud; repo-lint 0 fail / 610 files scanned.
Still red and expected: CHECK 1 at OK=11 want 13, and designate -- R11's
.61/.62 remain ruled-but-not-built, which is commit 2's job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

set-interface-v4: reload the pf filter on --commit (RULED); appendix-A rewritten ...

RULING (GA-R5) 2026-07-27. Question as presented: (a) add `configctl filter reload` to the
script's --commit path; (b) leave the script and rely on DC standup DoD item 8; (c) put it behind
an opt-in flag. Operator answer, exact utterance: "a". OPS under GA-R3 -- a script fix completing
an already-documented operation, governed by D-113; no new D-number.

DEFECT (appendix-A, measured on the fresh dc1 edge 2026-07-23): --commit ran `configctl interface
reconfigure`, which applies the address to the kernel but does NOT regenerate the pf ruleset, so
OPNsense never creates its automatic outbound-NAT rules for the (re-)addressed LAN. Forwarded LAN
packets left the WAN un-NAT'd and replies never returned -- the edge passed NO LAN traffic while
looking perfectly configured. Self-masking: any later plugin install or config apply reloads the
filter as a side effect, so dc0's 07-20 qga work hid it there until dc1 exposed it.

Option (b) rejected on evidence, not taste: DoD item 8 already existed and did NOT fire at dc1,
and prose-only controls have a measured failure record here (D-137's own enforcement sat as prose
at phase-3:498 and missed BOTH DC standups).

PLACEMENT IS THE SUBSTANCE OF THE FIX, and my first attempt got it wrong -- caught before
shipping. Putting the reload on the next line of the `interface reconfigure` heredoc would have
been a fix that fixes nothing exactly when it matters: re-addressing the interface you arrived on
drops the session DURING the reconfigure, so a following command in that stream never executes --
and that drop case IS this incident's own scenario (dc1's LAN moved 192.168.1.1 -> 10.12.64.1
over the LAN itself). As shipped the reload runs over a FRESH connection to PROBE_HOST.

Also added a `pfctl -s nat` read-back with the `proto carp` boilerplate filtered out, since
boilerplate-only is precisely appendix-A's broken-state signature. REPORTED, not gated: an edge's
first addressing may legitimately have no gateway and so no NAT to generate, and hard-failing
would block a correct run. The hard gate stays address-on-the-kernel.

Harness 59/59 (was 53), cases 15-15f. Case 15f caught a real inconsistency -- appendix-A still
said "not yet ruled". Case 15e caught a bug in my own first draft, which scanned to EOF and so
failed the script for having the very hard gate the case exists to preserve.

appendix-A rewritten: the FIX stays for an edge already in this state, plus a PREVENTED-IN-THE-TOOL
block recording the placement detail and why prose prevention was rejected.

Gauntlet ALL GREEN (81), repo-lint 0-fail. No live edge touched -- both DC edges are already
addressed, so this affects only the NEXT edge addressing.

With this, every substantive Stage 4 in-stage item is closed or split to gate row G17; only the
GA-R6 close-out set remains.

Revert: git revert this commit's script + harness + appendix-A changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf

Delivery batch: lib-hosts VR1 arms, D-133 guards, appendix-A pf entry, DoD 8-9 (gauntlet 77 GREEN) ...

lib-hosts vr1-dc0/vr1-dc1 populated from the measured identity table
(tofu-name keys, pinned boot MACs, D-134 octets, per-DC power URIs +
tags) + host_sysid_by_bootmac (the VR1 resolver) + one-selection guard;
dc-selector rewritten, 48 PASS. reenroll-hosts + carve-host-interfaces
gain explicit vr1-* refusals (VR0 flows superseded by D-133 -- populated
arms must not enable them). lib-net vr1-dc0 VID caveat. Appendix-A:
edge pf-never-regenerated symptom (masking trap named). Phase-2 DoD:
items 8 (filter reload + LAN egress probe) + 9 (dc-mirror), item 4
range corrected to D-134 .201-.254. CURRENT-STATE same-commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw

Recovery COMPLETE: 9/9 Ready again (shapes exact, power=virsh); retire-fully fully executed; SEC-013 narrowed ...

- Operator-ruled recovery: PXE re-enlist (~2 min) -> maas-node-power.sh
  dry+commit (9/9 verified) -> re-commission -> ALL 9 READY ~3 min
  (docs/audit/incident-20260721-recovery-verify.txt; new MAAS hostnames)
- voffice1 cleanup: vr1-dc0-maas dir + tfstate + SEC-013 key file removed,
  absence verified; SEC-013 row narrowed to CLI-profile-only
- CURRENT-STATE incident paragraph RESOLVED same commit (GA-R1 C1/C2)
- NEW appendix-A entry: pod delete cascades to linked machine records --
  read the pod's machine list BEFORE vm-host delete; non-empty = STOP

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XRtm5sDQgUsZyTJk3k7dz

Incident docs: 2 appendix-A entries, platform-traps MAC-regen corollary, LP draft ...

Closes the remaining documentation queue from the 2026-07-21
commissioning incident. Appendix-A gains the MAC-drift and
agent-resolver symptoms (verbatim, with checks + recorded fixes);
platform-traps 1e gains the unpinned-MAC regeneration corollary + index
row; LP draft ready for the operator to file (sanitization warning
included). Still queued: stale pod cleanup at stage close.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01STHXiHfoxHqq8fGRVvb66G

appendix-A: new incident class -- MAAS says dhcp_on=True but no dhcpd runs (Temporal wedged) ...

Records the 2026-07-20 incident by verbatim symptom: config vs running state
disagree, MAAS's own service_set reports healthy, and the region journal
shows Temporal 'Not enough hosts to serve the request'. Check is pgrep -c
dhcpd on EVERY controller; fix is snap restart maas on the region, verified
behaviorally. Includes the scope trap that made it invisible for days: an
already-built site keeps working because deployed VMs hold their addresses,
so it only surfaces when new nodes try to PXE.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiUu1oqt76tWvV4vEC3NAr

Batch 4 C1: GA-R4 rule-6 ledger rotation -- 1187 -> 131 lines (GA-F13) ...

Bodies rotated VERBATIM to docs/archive/session-ledger-rotated-20260719.md
(Live state 2026-07-13, OPEN WORK VR1+VR0, NetBox write-path bugs, PINNED
rulings, standing lessons, state facts, project-completion, NetBox import
narrative). Routed-before-rotating (durable homes): juju format=line 3.6
trap -> appendix-A; guards-are-load-bearing -> operating-discipline; VR0
operational facts + admin/controller -> maas-as-built-reference.md
(lessons 1-6 verified already covered by platform-traps/L8). Live ledger
keeps header + machine-derived fence + two compact still-live blocks
(NetBox write-path bugs pointer, project-completion) + the bounded session
summaries. 131 < 300: the F1 cap is now ENFORCEABLE. Fences OK; ledger-
scan counters unchanged; repo-lint 0 fail.

Revert: git revert this commit (restores the 1187-line ledger).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi

GA-F10/H1 + GA-R1/R6: token migration, State-line demotions, E1 type tags (2.5+2.6) ...

Mapping record: docs/audit/h1-token-map-20260719.md -- all distinct
tokens dispositioned, none unmapped. Live-surface migration: workflow
doc (26 flags -> 0; header drops the tracker model, 8 State lines demote
to identity + CURRENT-STATE pointers), runbooks (18 -> 0; close-out
instructions RETARGETED to record status in CURRENT-STATE, not the
workflow tracker), readiness doc (5 -> 0), CURRENT-STATE itself (6 -> 0:
G9 STOPPED -> BLOCKED with named blocker; stale "GA-R1 NOT yet ratified"
header + pre-D-130 divergence rows corrected per C2). Gate rows gain
GA-R6/E1 type tags ([V]/[R]) with legend. CLAUDE.md expectation line
demoted per S5b. History classes (ledger, changelogs, design-decisions)
deliberately untouched -- their disposition is Batch 4/GA-R3 structural.
Extractor re-run: zero flagged occurrences on live non-history surfaces.
Batch 2 items 2.5 + 2.6.

Revert: git revert this commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZbUeRE8weySLSGY1odrCi

addendum 39: devteam cluster-template 403 diagnosed (Public/Hidden flag -> clustertemplate:publish admin_api, BY DESIGN, no cloud change) -- DOCFIX-121 clientdocs template-creation guidance + DOCFIX-122 appendix-A symptom entry; L7 receipt re-recorded; ~/devteam-package re-rendered; stage5-for-devteam option logged (unruled) ...

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

addendum 37: D-075 ADOPTED+EXECUTED -- dashboard VIP root redirect(/->302 /horizon)+no-autoindex via conf-enabled override (RedirectMatch backend-port leak caught in verify, replaced with per-dir rewrite); DOCFIX-118 phase-03 PER-REBUILD block + appendix-A "Index of /" entry ...

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

D-072 dashboard TLS repair (executed) + DOCFIX-089 fail-closed check + BUNDLEFIX-011 ...

Identifiers consumed: D-072, DOCFIX-089, BUNDLEFIX-011.

Since-deploy dashboard VIP TLS defect fixed: cluster binding -> metal-admin
(live bind + durable bundle edit); https 200 CA-verified on both reachable
VIPs, all gates green, post-fix BOM committed. phase-03 3.3 rewritten
fail-closed; appendix-A symptom entry added; Launchpad bug draft ready for
operator submission.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FCGjrRXtbPRu5qEThw3CDz

Patches

Script and file updates

MAAS script changes

Post deploy changes

Updates and patches

repo: sanitation sweep -- canonical phase-NN runbook + appendices; design-decisions D-001..D-042; canonical bundle; superseded files moved to remove/ for review