| 2026-08-06 |
Stage 5 dc0: memcached scaled to 3 LIVE (operator-approved) + consumer verify
...
Live scale-up executed on the dc0 rack (operator "Both approved"):
juju add-unit memcached -n 2 --to lxd:1,lxd:2 -m vr1-dc0 (exit 0).
- CONVERGED 3/3 active/idle, one per control node (memcached/0 control-01, /1 control-02,
/2 control-03; all "Unit is ready and clustered", 11211/tcp). Clean install, no F5 apt-wedge.
Capture docs/audit/stage5-dc0-memcached-scaleup-20260806.txt.
- Consumer verify (D-121 verify-at-deploy): nova-cloud-controller sees all 3 servers
(memcache_servers = .115,.166,.165:11211). designate coordination backend_url shows ONE
(memcached/0) -- designate is workload-blocked pre-Stage-7, so re-check at designate
activation; recorded, not a scale-up defect.
- bundle.yaml re-staged to the dc0 rack (42845edb == repo HEAD).
- CURRENT-STATE reconciled: memcached config=3 AND live=3.
Record-only commit (the live mutation itself was the add-unit above). repo-lint 0 fail.
Revert of the LIVE state (if ever needed): juju remove-unit memcached/1 memcached/2 -m vr1-dc0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Stage 5 dc0: memcached 1->3 config-of-record [BUNDLEFIX-055 + DOCFIX-212]
...
Operator ruling 2026-08-06, exact utterance "3 units (restore intent)" -- restoring the
2026-07-31 memcached->3 direction that BUNDLEFIX-053 never folded and the retired
dc-ha-scaleup.yaml had carried unrealized. Found while reconciling a stale comment:
- Full merge-diff (measured) of the archived overlay vs bundle.yaml showed the ONLY divergence
was memcached (overlay num_units:3 / base 1) -- BUNDLEFIX-053 folded the HA chain but not this.
Corrects two earlier misstatements (this session): "memcached=3 in bundle.yaml" (it was 1) and
the retirement's "wholly redundant / all-keys no-op" claim (it diverged on memcached).
- BUNDLEFIX-055: bundle.yaml memcached num_units 1->3, to:[lxd:0]->[lxd:0,1,2] + the
3-independent-caches rationale (no hacluster/VIP; clients hash across the full server list).
This completes the fold, making the archived overlay genuinely, fully redundant.
provider-bundle-check 57/0 (memcached is orthogonal to the arity/VIP gates).
- DOCFIX-212: design-decisions D-121 "Left single (NOT scaled)" list -- memcached removed,
recorded as =3.
- CURRENT-STATE: records the CONFIG=3 / LIVE=1 divergence -- a live add-unit (operator wants it
deployed live) or a teardown+redeploy reconciles it. The live scale-up is a separate gated step.
Gauntlet ALL GREEN (99 harnesses); repo-lint 0 fail (1 pre-existing legacy L1 warn).
Owed: re-stage the functionally-changed bundle.yaml to the dc0 rack (gated); live add-unit to 3.
Revert: bundle.yaml memcached back to num_units:1 + to:[lxd:0]; revert the DOCFIX-212 D-121 line
and the CURRENT-STATE gap note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Stage 5 dc0: RETIRE + archive dc-ha-scaleup.yaml (operator-ruled; R6 superseded) [DOCFIX-211]
...
Operator ruling 2026-08-06, exact utterance "Retire the redundancy and archive." The
overlay was wholly redundant -- BUNDLEFIX-053 folded its entire content into bundle.yaml,
so applying it was an all-keys no-op deep-merge (Task-#1 post-wave review, measured).
- Archived: git mv overlays/dc-ha-scaleup.yaml -> docs/archive/dc-ha-scaleup-RETIRED-20260806.yaml
+ a do-not-deploy banner. docs/archive/ chosen explicitly (no prior overlay-archive precedent).
- design-decisions.md: GA-R5 SUPERSESSION note on D-121's R6 (original 2026-07-27 record retained
as history); fixed :503 and :4563 "overlay encodes v-a" refs -> bundle.yaml/BUNDLEFIX-053.
- Harness (tests/provider-bundle-check): T17 REMOVED (== T18 once vault-hacluster is in base);
T17b/T32/T33/T34 RE-POINTED onto good.yaml (base HA chain). T33/T34 now use the deterministic
mutate() helper, not sed-on-overlay (no silent no-op fixture; a wrong key path KeyErrors). Each
FAIL case proven to fire against the real checker message. 58->57 cases; provider-bundle-check 57/0.
- Runbook dc-dc-phase4 Step-4 note + Step-12.3(a) gate: the two-phase "base at cluster_count:1 then
apply the overlay to reach 3" model RETIRED -- base deploys 3-unit HA directly, so cluster_count:3
is the expected post-Step-4 value.
- Rendered vips overlays: edited the render SOURCE (render/values/vr1-dc{0,1}-vips.yaml, D-136) and
re-rendered (not hand-edited); overlay diff = the two comment lines only; render-drift 4/0.
Hand-maintained vr1-dc{0,1}-machines.yaml deploy-command comments + provider-bundle-check.py +
cloud-assert.sh comments + dc-dc-deployment-workflow.md item 22 reconciled.
- CURRENT-STATE: no edit needed -- active status already omits the HA overlay from the deploy input;
remaining mentions are dated historical narrative (retained as history, GA-R1).
- Also finalizes Item 3/5 changelog notes (F9 approved+staged; Task #1 recommendation ruled).
Gauntlet ALL GREEN (99 harnesses); repo-lint 0 fail (1 pre-existing legacy L1 warn).
Staging: dc0 rack vr1-dc0-vips.yaml now stale-by-one-comment vs the re-render (functionally
identical); re-syncs + sha-verifies at the next deploy per D-138 (logged, no rack write for a comment).
Revert: git mv the overlay back (strip banner) + git revert this commit + re-render vips from
reverted values.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Stage 5 dc0 F-A/BUNDLEFIX-054: correct bundle.yaml HA-chain header + session changelog
...
- bundle.yaml:23 HA-chain header was stale after BUNDLEFIX-053 (vault-hacluster),
the D-020 amendment (vault metal-only VIP) and R11 (designate VIP). Corrected FROM
the measured merged dc0 deploy input: 12 -> 13 hacluster subordinates/:ha relations;
ALL 13 carry a VIP (12 the full provider+metal-admin+metal-internal triple, vault a
METAL-ONLY pair per D-020); retired the stale "designate has no HAProxy VIP" note
(designate gained triple VIP .62 under R11); clarified rabbitmq is the 14th HA app but
native-clustered (not in the 13). COMMENT-ONLY -- no deploy semantics change.
- Session changelog docs/changelog-20260806-stage5-dc0-f4-postwave.md (GA-R2):
F4 (14/14 measurement-backed + vault ha_enabled), F8 (ceph-radosgw resolved),
F9 (staging drift measured, re-stage gated), F-A, and the Task #1 finding that
dc-ha-scaleup.yaml is now redundant with bundle.yaml (recommend retiring; operator
ruling owed since R6/D-121 reference it -- logged, not executed).
Gauntlet ALL GREEN (99); repo-lint 0 fail (1 pre-existing legacy L1 warn).
Revert: git revert this commit (restores the prior 6-line header; changelog is additive).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Stage 5 dc0 F4: 14/14 HA now measurement-backed + vault ha_enabled MEASURED
...
Live read-only capture from the dc0 rack (juju status -m vr1-dc0, D-138 path):
docs/audit/stage5-dc0-juju-status-14of14-20260806.txt.
- 14/14 D-121-enumerated HA apps at scale=3 (per changelog-20260805-d121-ha-scaleup.md:24).
HONEST SPLIT (GA-R6 E3, not rounded): 12 active/idle; 2 blocked-at-scale=3 on KNOWN
non-HA items -- octavia (configure-resources) + designate (nameservers). Model-wide
152 active/idle, 7 blocked, 1 unknown (gss, normal). Converts the CURRENT-STATE
14/14 line from OPERATOR-ATTESTED to MEASUREMENT-BACKED (C2: measurement wins).
- F4 vault ha_enabled MEASURED: vault status HA Enabled=FALSE x3, storage=mysql,
unsealed, shared cluster id. CAUSE (measured, corrects a mid-session mis-blame of the
mysql backend): charm renders storage "mysql" with no ha_enabled and exposes no such
option (only vip + dns-ha-access-record) -- HA is charm/VIP model, not vault-native.
CONFIRMS D-121 (v-a) as-built; vault-native/Raft HA stays owned by D-068. No new decision.
- F8 ceph-radosgw RESOLVED: 3 units active/idle "Unit is ready" (prior stale-status converged).
- F9 staging drift MEASURED (re-stage gated, owed at next deploy): dc0 rack bundle.yaml +
vr1-dc0-vips.yaml STALE vs HEAD; dc1 rack has no ~/repo-stage (HELD).
Revert: git revert this commit (record-only; no live-cloud change was made -- all reads).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Session close 2026-08-06 (GA-R4 bookend): Task #2 D-020 vault-metal-only DONE + sweep
...
Bookend for the Task #2 session (deliverable commits 07e2d9d, a08783c already pushed).
- docs/session-ledger.md: bounded <=15-line SESSION CLOSE 2026-08-06 block (ledger now
299 lines, one from the 300 cap -- next close needs rotation).
- docs/audit/queued-findings-20260806-task2-vault-metal-only.txt: the sweep. FIRST SURFACE
items: bundle.yaml:23 stale "12 charms" (F-A); rack ~/repo-stage overlays now stale (F-B);
Task #2 guidance unexecutable as written (F-C); render-drift red at part-2 close but
undocumented (F-D). Owed: F4/F8/F9/Task-1. Owned: W-1/W-2/W-3.
Gauntlet ALL GREEN (99); repo-lint 0 fail. Memory: instrument-currency instance sixteen
(appended locally, not repo-tracked). Status ONLY in docs/CURRENT-STATE.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
pre-flight-checks CHECK 1: reset awk app-name tracker on applications: (hardening)
...
Defensive follow-up to 07e2d9d. The vault metal-only branch keys on the app name
tracked from 2-space header lines across bundle.yaml then the overlay. bundle.yaml is
VIP-free (ruling 3) so no stale name can currently leak, but reset name="" on each
applications: block as cheap insurance if bundle.yaml ever regains a vip -- otherwise a
stale bundle key could grade the first overlay vip line under the wrong app.
pre-flight-checks harness 32/0; repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Task #2: commit D-020 vault-metal-only amendment; teach renderer + gates; harnesses green
...
The RULED-but-uncommitted D-020 amendment (vault METAL-ONLY, 2026-08-05) is now
committed and enforced end-to-end. Discovered mid-task that the overlays are RENDERED
from render/values/*.yaml (D-136) and hand-editing them is forbidden by the render-drift
gate -- which was ALREADY red at the part-2 close (undocumented) from that session's dc0
overlay hand-edit. Resolved by teaching the renderer, not by hand-editing (advisor-reviewed;
the amendment is ruled, so this is OPS implementation).
Vault VIP shape is now the metal PAIR (metal-admin + metal-internal, no provider-public,
no v6) across all four consumers, each with a failing-direction fixture:
- provider-bundle-check.py: per-DC/family-aware vault exception; STILL band- and
octet-uniqueness-checked (advisor caught the first draft's early `continue` disarming
octet_owner for .61 -- proven rc=0 draft / rc=1 fixed). T54/T55/T56.
- render-dc-overlays.py: name-keyed metal-only render branch (drops provider + v6);
overlays re-rendered (diff vs HEAD = exactly the one vault line each). T15b.
- pre-flight-checks.sh CHECK 1: awk name-tracking + vault metal-pair branch. T28b.
- render/values/*.yaml: vault comment (part of rendered bytes).
D-121 status 12/14 -> 14/14 in CURRENT-STATE, with the missing juju-status capture
DECLARED as an owed gap (GA-R1 rule 2) rather than papered over.
Verify: gauntlet ALL GREEN (99); repo-lint 0 fail; provider-bundle-check 58/0,
render-dc-overlays 24/0, render-drift 4/0, pre-flight-checks 32/0.
Body: docs/changelog-20260805-task2-vault-metal-only-commit.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
| 2026-08-05 |
Session close 2026-08-05 (part 2): D-121 HA 14/14 + vault metal-only; GA-R4 bookend
...
Bounded ledger summary + rotation (07-31/08-01 -> archive/session-ledger-rotated-20260805.md,
315->288) + sweep queued-findings-20260805-d121-ha-vault.txt (9 first-surface). D-121 executed
live -- all 14 control-plane apps to 3-unit HA; D-020 amended vault->metal-only (ratified, HELD
uncommitted pending the provider-bundle-check harness reconcile, Task #2). Status ONLY in
CURRENT-STATE.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
CURRENT-STATE: D-121 HA scale-up now 12/14 (nova-cc, rabbitmq, barbican DONE)
...
Measured 2026-08-05: nova-cloud-controller + rabbitmq-server (native erlang 3-node cluster)
+ barbican scaled to 3-unit HA. 12 of 14 HA apps at 3-unit HA. Only keystone (cloud-wide
auth VIP blip) and vault (operator unseal) remain -- both operator-gated. vault HA also
resolves the barbican-vault secrets-storage dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
BUNDLEFIX-053: bundle.yaml to full 3-unit HA (D-121); reverse BUNDLEFIX-002 vault de-HA
...
Reverses the D-009/BUNDLEFIX-002/003 decorative-single-unit posture to real 3-unit HA per
operator ruling 2026-08-05 ("superseded the HA removal from all items ... stand up all
remaining HA apps ... HA blocks sign off"; "testing as if Roosevelt"). Executes adopted
D-121 incl. its (v-a) vault sub-ruling.
bundle.yaml:
- num_units 1->3 + to:[lxd:0,1,2] on 14 HA apps (keystone glance nova-cc placement
neutron-api cinder ceph-radosgw dashboard octavia barbican magnum designate rabbitmq vault)
- hacluster cluster_count 1->3 x12
- vault-hacluster (cluster_count:3) + [vault:ha, vault-hacluster:ha] uncommented -- the
BUNDLEFIX-002 reversal (D-121 (v-a): vault MySQL-backed HA)
- rabbitmq min-cluster-size:3 (D-009 amendment)
- governing comments reconciled (D-009/BUNDLEFIX-002/003 -> D-121 built)
overlays unchanged (VIPs already set).
Every change verified against charm/juju docs or the deployed charm (not assumed):
hacluster cluster_count default 3; rabbitmq min-cluster-size; vault ha endpoint=metal-internal
like keystone; vault MySQL-backend HA per HashiCorp docs (ha_enabled+lock_table). Validated:
provider-bundle-check.py --dc vr1-dc0 PASS (13 hacluster subs cluster_count==num_units; 13
principals carry a VIP; 109 relations well-formed); repo-lint 0 fail.
Live: Wave 1 (8 apps) scaled to 3-unit HA (see CURRENT-STATE + changelog). Waves 2-4 and the
post-wave bundle/overlay reconciliation (pinned task) remain. Body:
docs/changelog-20260805-d121-ha-scaleup.md. Status in docs/CURRENT-STATE.md ONLY.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
CURRENT-STATE: barbican-vault/0 verify result (C2) -- genuine incomplete, missing vault_url
...
Item (b) verify supersedes the "settling" note in 71c5b97: barbican-vault/0 is NOT settling
and NOT deferred. vault provided per-unit role_id + token but the secrets-storage databag
lacks vault_url (app-data empty) -> charm logs "Requesting access to vault (None)". Not
deploy-blocking (barbican/0 active on software backend). Measured 2026-08-05 17:55Z.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Stage 5 dc0: remediate ceph-mon/2 + ceph-radosgw/0 "allocating" (apt CLOSE-WAIT); durable capture
...
Live-cloud, operator-gated (2 exchanges). Two units stuck `allocating` were root-caused
NOT to a down proxy (both dc0 proxies PASS; security.ubuntu.com 200/0.67s) but to
cloud-init `apt-get update` parked in CLOSE-WAIT to 10.12.8.4:3142 for ~12.5h (apt has no
client read timeout) since the 08-04 redeploy. Rebuilt each from the rack (D-138):
remove-unit --no-prompt -> remove-machine --force --no-prompt (REQUIRED: remove-unit does
NOT cascade to a never-provisioned dead-agent container) -> add-unit --to <bundle
placement>. Both fresh containers' cloud-init finished (~211s); mons bootstrapped 3/3, 4
OSDs active, storage cascade cleared. A 2nd apt-cacher-ng mode (radosgw-hacluster 404 on a
rotated point-release -> exit 100) SELF-HEALED via juju hook retry. Measured after-state
17:34:08Z: census 62 active; remaining non-active all deferred-by-design + gss +
barbican-vault settling.
Durable capture (operator-directed "a then b"):
- docs/audit/stage5-dc0-ceph-remediation-20260805.txt: transcribed capture (NOT script(1));
the before-state (CLOSE-WAIT sockets, ~45000s etimes, both containers) is unrecoverable
and lives only here.
- docs/CURRENT-STATE.md: dated status block; measured 62 active (NOT the unmeasured 40/47,
C2); records the --force fact and the phase-03 Step-3.1 gate defect (expects 1, VR1 has 4
deferred + gss) as durable finding.
- runbooks/appendix-A-troubleshooting.md: NEW entry for the symptom pair (both modes +
gated remove/--force/re-add remediation). Drafted; operator-reviewed.
- docs/changelog-20260805-stage5-dc0-ceph-remediation.md: session changelog (separate
same-day file from the skill-DOCFIX changelog, disclosed in header); F2/F3/F4 owed,
unnumbered.
repo-lint 0 fail / 1 legacy warn; ledger-scan DOCFIX next-free still 210 (no token leaked);
all touched files ASCII-clean. Status lives in docs/CURRENT-STATE.md ONLY.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
DOCFIX-209: correct SKILL.md session-CLOSE bookend + re-seed ledger machine-block
...
SKILL.md's always-loaded close-bookend line conflated two ratified mechanisms:
GA-R4 rule 2 (ledger rotation) and GA-R2/D1 (changelog = session body-of-record).
"the full session body goes to docs/archive/ in the same close commit" is wrong on
both counts -- the per-session body is the changelog (docs/changelog-<date>-<label>.md,
cited by a Body: line in the SESSION CLOSE block), and docs/archive/ holds rotated
ledgers (session-ledger-rotated-*.md) + per-stage consolidated records
(archive/changelogs/, archive/stage-records/), never per-session bodies. Corrected in
SKILL.md and the byte-identical line in the derived consolidated-20260727 snapshot.
operating-discipline.md (Session continuity) was already correct -- no edit.
Disposition operator-ruled 2026-08-05 "Proceed as DOCFIX-209": both same-day rulings,
read correctly, back the practice and contradict the skill line, so no ratified text
changes. Body: docs/changelog-20260805-skill-close-convention-docfix.md.
Also re-seeded session-ledger.md machine-derived block from ledger-scan (prior seed
2026-08-02, stale): 4 open decisions (D-142 added), 29 open SEC (SEC-033 added),
next-free D-143 / DOCFIX-210 / BUNDLEFIX-053. No status-bearing surface touched
(a skill routing line is not a status claim -- no GA-R1 L10 coupling). Stage 5 dc0
untouched. repo-lint 0 fail / 1 legacy warn; skill files ASCII-clean; ledger fences OK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Close-fix 2026-08-05: add Body changelog, verify root CA (openssl), reconcile D-142 scan-visibility + durability
...
Advisor-caught close gaps against the repo's own convention:
- Body: NEW docs/changelog-20260805-vault-init-ovn-resolved.md (prior closes cite a
changelog Body: line; the 08-05 close block had Sweep: but no Body:). Ledger + CURRENT-STATE
now cite it.
- Root CA: decoded the ACTUAL pasted PEM with openssl on the rack (not a self-decode).
Confirms notBefore Aug 5 02:05:57 2026 / notAfter Aug 2 01:06:27 2036 GMT; adds sha256
75:DF:33:97:...:35:A1. as-exit as-built + CURRENT-STATE updated to measured fact.
- D-142 Status now leads "PROPOSED / OPEN" so ledger-scan surfaces it (was invisible; scan
keys the open list off the Status token) -> reconciles the close block's "4 open decisions".
- Durability line completed: voffice1 PULLED to sync (was 3321c57, 4 behind); dc0 rack
~/repo-stage unaffected (docs-only; preflight sha verified); gauntlet not owed (docs-only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
SESSION CLOSE 2026-08-05 (GA-R4 bookend): vault init DONE + ovn-central RESOLVED; D-142 saved
...
Bounded 15-line ledger summary + close sweep (3 first-surface). Stage 5 remains
OPEN -- session bookend, not a stage close.
- vault init complete (operator-run one-shot, dc0 rack, -m vr1-dc0); root CA
generated; vault active/idle.
- ovn-central/3,4,5 all active -- OVN NB/SB cluster formed (the redeploy's purpose).
- D-142 vault-init QoL sweep SAVED (approved-in-principle, impl deferred; R2 open).
- Close sweep: ceph-mon/2 + ceph-radosgw/0 apt-wedge cascade (triage next);
/background unavailable over Remote Control; R2 transport gap.
Durability: 0 uncommitted / 0 unpushed; repo-lint 0 fail / 1 legacy warn.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
Stage 5 dc0: vault init DONE + ovn-central RESOLVED; D-142 vault-init QoL saved
...
Live (operator-run one-shot, recorded no-secrets): phase-02-vault-bringup Steps
2.1-2.3 on the dc0 rack (-m vr1-dc0), vault active/idle, root CA generated
(valid 2026-08-05 -> 2036-08-02). ovn-central/3,4,5 ALL active -- OVN NB/SB
cluster formed; the multi-session cert saga is closed by the two fixes landed
this cycle (dc-node-etchosts Step 1.2b CN delivery + D-052 '' -> metal-admin).
Engineering saved (PROPOSED, not executed -- operator: run current commands now,
test QoL next opportunity):
- D-142 PROPOSED: vault-init workflow QoL sweep (APPROVED-IN-PRINCIPLE, IMPL
DEFERRED; R2 off-host transport UNRESOLVED). Distinct from D-068 (substrate) /
D-011.6 (manual-unseal bar).
- docs/audit/vault-init-qol-proposal-20260805.md: R1-R5 + full hidden-prompt
safety analysis + tee-write residual + pre-init writability probe + R3 harness
constraints + operator's verbatim safety constraint.
- runbook-fold-register F13: -m openstack -> -m vr1-dc0 / run-location = DC rack
(D-138) on phase-02-vault-bringup (scope stretch stated).
CURRENT-STATE + as-exec updated (L10 coupling). Security hygiene: child token in
operator paste was ttl=10m/expired -> benign, not stored. Remaining (separate):
ceph-mon/2 + ceph-radosgw/0 allocating (apt-wedge) cascade.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
D-052 RE-AMENDMENT: ovn-central '' default metal-internal -> metal-admin + network binding reference
...
- ROOT: the '' default is the MANAGEMENT binding (primary NIC + juju agent<->controller),
universal '' = metal-admin across all 56 apps. The 08-03 amendment ('' -> metal-internal,
never deployed until 08-04) left ovn-central single-legged on the isolated metal-internal
plane -> no route to controller (10.12.8.5:17070 unreachable) -> agent-binary download failed
-> ovn-central never started. Its cert rationale was already withdrawn; cert CN is fixed
independently by the /etc/hosts postruncmd (dc-node-etchosts.sh, Step 1.2b).
- bundle.yaml: ovn-central '' -> metal-admin; functional endpoints (certificates, ovsdb*,
coordinator) STAY metal-internal. provider-bundle-check 55/55, repo-lint 0 fail.
- D-052 RE-AMENDMENT 2026-08-05 recorded (GA-R5, operator utterance quoted). Scope: ovn-central
only; every other binding re-verified correct vs dataflow (ruled exceptions: D-072 dashboard,
D-106 designate:dnsaas).
- NEW docs/network-space-binding-reference.md: 6-plane roles + CIDRs + RHOSP analogs, the ''
management-binding rule, full 56-app placement matrix, dataflow confirmation, ruled exceptions.
- Stage 5 remains OPEN. NEXT: re-stage bundle to rack, re-home ovn-central on the live model,
then vault init (operator-only) -> ovn cert -> converge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
| 2026-08-04 |
Stage 5 dc0 redeploy: Path M teardown (clean) + ovn-central cert DELIVERY bug fixed
...
- Path M teardown of vr1-dc0 complete + clean: graceful drain (36->15) + juju
resolved --no-retry (error-unit wedge, 15->3) + --force --no-wait (agents-stopped
tail, 3->0). NO orphan (controller healthy), M.5 cascade clean (10 machines
UNCHANGED, 9 Ready + 1 Deployed). Both --force and resolved operator-ruled (GA-R5).
- dc-node-etchosts.sh: render runcmd: -> postruncmd: -- juju model-config forbids a
top-level runcmd in cloudinit-userdata. The 08-04 fix proved the CONCEPT live but
never the DELIVERY (harness graded cloud-init YAML, which accepts runcmd, not juju
acceptance). Harness switched to postruncmd + NEW T10 (juju-accepted key, rejects
bare runcmd), MUTATION-PROVEN, 10/10. Step 1.2b now live-passing; re-staged to rack.
- M.6 rebuild to the deploy step: add-model (cred vr1-dc0-cred), spaces PASS 0 fatal,
egress 8/8 PASS, dry-run PASS (9 machines, per-DC tags). Stage 5 remains OPEN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fg98z7QyzwYUs8fsWCn728
|
SESSION CLOSE 2026-08-04 (GA-R4 bookend): ovn-central cert PROVEN + WIRED; SEC-033 filed
...
Bookend for the session that root-caused ovn-central's cert failure (isolated
metal-internal plane -> empty CN), proved the /etc/hosts fix live, and wired it
via dc-node-etchosts.sh + phase-01 Step 1.2b. Stage 5 remains OPEN.
- docs/session-ledger.md: bounded 2026-08-04 summary; oldest block (07-31 node
carve) rotated to docs/archive/session-ledger-rotated-20260804.md; 295 lines.
- docs/security-ledger.md: SEC-033 (tls-certificates relation databag exposes
vault's global-client private key to any juju model reader; interface-level).
- docs/audit/queued-findings-20260804-ovn-cert-fix.txt: close sweep, 4 FIRST
SURFACE (SEC-033-now-filed, live-model test drift, rack repo-stage prereq,
juju-routing research note).
- docs/CURRENT-STATE.md: SESSION CLOSE 2026-08-04 pointer; NEXT = step 3 redeploy.
Gauntlet ALL GREEN (99), repo-lint 0 fail. voffice1 synced to HEAD.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
ovn-central cert: root-caused, fix PROVEN live, and wired for redeploy
...
Root cause (measured, corrects the committed rdns_mode/binding framing):
charm-ovn-central derives its TLS common_name from get_hostname(its
metal-internal address); that plane is deliberately isolated (D-052) with no
reachable resolver, so the reverse lookup returns None -> empty CN -> vault
issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR
exist; only the reverse is unreachable from the isolated plane.
Binding approach REFUTED live (3 configs): the charm uses the metal-internal
address regardless of which endpoint is rebound, and juju default-route
selection is not tied to the default binding. So the app STAYS on
metal-internal (D-052-correct for its OVSDB/certificates data type).
Fix PROVEN end-to-end (controlled single-unit test): an /etc/hosts reverse
entry -> CN populated -> vault issued ovn-central_0.server.cert ->
/etc/ovn/{cert_host,key_host,ovn-central.crt} written; control units without
it stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN.
Wired (verify-at-provision on the redeploy):
- NEW scripts/dc-node-etchosts.sh + tests/dc-node-etchosts (harness 9/9):
renders a per-DC cloudinit-userdata adding each node's metal-internal
address -> hostname to /etc/hosts at provision; CIDR derived from lib-net
(dc0 10.12.12.0/22, dc1 10.12.72.0/22). Rendered runcmd executed live and
correctly scoped to metal-internal.
- runbooks/phase-01-bundle-deploy.md Step 1.2b: gated pre-deploy step
(after add-model, before deploy), VR1-only.
- A NEW mechanism borrowing D-008's shape, not D-008 itself.
Records: reeval RESOLUTION + CURRENT-STATE RESOLVED+WIRED + changelog; the old
rdns_mode remediation is VOID. Gauntlet ALL GREEN (99), repo-lint 0 fail.
Live tests were reversible; model left at its captured before-state.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
CORRECT ovn-central root cause: NOT rdns_mode -- metal-internal is isolated (no reachable resolver)
...
Autonomous step-1 measurement refuted the committed rdns_mode diagnosis:
- rdns_mode=2 on ALL planes; the PTR EXISTS at the region BIND (10.12.8.6
answers dig -x 10.12.12.122). "enable rdns_mode" is a VOID no-op.
- metal-internal is deliberately isolated (link-scoped routes only; ping -I
eth1 10.12.8.6 -> NO ROUTE). The region controller has no metal-internal
interface, so no resolver is reachable on that plane.
- systemd-resolved scopes the reverse of a container's OWN metal-internal
address to eth1 -> no reachable resolver -> empty cert CN. Discriminating
test (advisor-directed): both links set to the reachable 10.12.8.6 STILL
failed "No route to host" -> NOT a dns_servers change either.
The fix is therefore a DECISION (decouple ovn-central's cert CN from
metal-internal reverse-DNS), a precondition for any redeploy (same MAAS,
verify at provision time). VOID banners + CORRECTION blocks added to the
reeval, remediation-plan, sweep, and CURRENT-STATE. Refutation of LP #2044324
and the vault-issuance rule are unchanged and stand.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-08-03 |
CURRENT-STATE: correct ovn-central root cause (reeval) + record Stage-5 sweep (GA-R1 C1)
...
Supersede the LP #2044324 framing with the measured root cause (metal-internal
reverse-DNS gap -> empty CN -> no server cert), point to the reeval + remediation
+ sweep audit docs, and record sweep findings F1/F2/F3. Satisfies the L10 status-
in-same-commit rule the prior audit-docs commit (eb63339) tripped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
audit: ovn-central cert reeval + remediation plan + Stage-5 misses sweep
...
Re-evaluation of the prior (wrong) ovn-central "awaiting server certificate
data" diagnosis, from fresh live measurement + 3 source-level agents + an
adversarial review. Corrected root cause: ovn-central derives its cert CN from
a REVERSE lookup of its metal-internal address; metal-internal is the only
plane of six without reverse-DNS PTRs, so get_hostname() -> None -> empty
common_name -> vault issues no server cert -> ovn-central blocks. LP #2044324
is NO MATCH; no upstream fix / channel bump helps.
- ovn-central-cert-reeval-20260803.md -- root cause, evidence, what the
prior diagnosis got wrong, ranked remedies, why-no-PTR.
- ovn-central-cert-remediation-plan-20260803.md -- gated rdns_mode + re-fire
sequence with corrected (non-false-green) acceptance criteria.
- stage5-sweep-misses-20260803.md -- F1 metal-internal rdns outlier +
standup tooling never sets rdns_mode; F2 four hacluster units blocked on
stale pre-D-141 IPv6 VIP CIB resources; F3 octavia error is downstream of
neutron VIP TLS. Meta: incomplete-transition tails.
All findings MEASURED read-only this session; every remediation step is
operator-gated and NOT executed. SEC-033 (relation databag exposes vault
global-client key) noted for separate filing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
reconcile provider-bundle-check harness to D-141 (sweep F1 CLOSED)
...
The 2026-08-03 deploy session reverted the dc0 VIP overlay to IPv4-only
under D-141 (3e691cd) but shipped it without the companion harness update,
leaving run-tests-all RED at 1/98 (provider-bundle-check, 4 dual-family
cases T19/T21/T25/T45 asserting a shape the deploy input no longer has).
Reconcile by re-pointing, never deleting (the checker's dual-family path is
still live code and D-141 rule-3 promotes v6 later):
- new synthetic dual.yaml fixture on the MEASURED all-GUA legs of the
pre-revert deploy input (3e691cd^), present in the current apex -- NOT the
stale DUAL6 ULA constant D-139 deprecated
- T19 assertion REPLACED with the v4-only invariant (0 dual-family)
- T21/T25/T45 re-pointed to dual.yaml (dual-family PASS + apex-refusal
controls preserved)
- corrected the now-vacuous v4only.yaml comment
scripts/provider-bundle-check.py is UNCHANGED -- it is family-agnostic (a
vip is a v4 triple OR a dual-family sextet); this is a harness reconcile only.
Verified: provider-bundle-check 55/55 ALL PASS (count unchanged 55->55, pure
re-point); new T19 failing-direction proven; full gauntlet GAUNTLET: ALL
GREEN (98 harnesses) (was 1/98 FAIL); repo-lint 0 fail / 1 legacy warn.
CURRENT-STATE updated same commit (GA-R1 C1). Three follow-ons LOGGED to the
close sweep (FN1 stale DUAL6 ULA; FN2 generated overlay header; FN3 dual.yaml
builder can silently no-op at promotion).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
SESSION CLOSE 2026-08-03 (GA-R4 bookend): dc0 bundle deployed, controller rebuilt, vault up; ovn-central cert deferred
...
Stage 5 remains OPEN; session bookend, not a stage close.
Deploy mid-Stage-5: bundle deployed + mostly converged (9 machines started, mysql
ONLINE, vault init+unseal+root-CA, ~25 units active, 0 error). DOCFIX-208 fixed
the machines-overlay omission; D-135 amendment (b) converged dc0 onto the apt
proxy; D-141 v4 VIP revert cleared keystone Invalid vips; controller rebuilt via
new Path C after a --force destroy orphaned it.
ovn-central x3 DEGRADED and DEFERRED: LP #2044324, cert request missing
common_name -> no server cert -> OVN cluster not formed. Three remedies exhausted.
GATE RED AT CLOSE (recorded): gauntlet 1/98 FAIL provider-bundle-check -- the
D-141 v4 revert left the deploy overlay v4-only while the harness asserts
dual-family; logged, harness owes a reconcile. repo-lint 0 fail.
Bookend: ledger summary (292 lines after rotating 07-30 pt5), sweep
queued-findings-20260803 (4 first surface), CURRENT-STATE close block, savegame
Step 1b (pull inner clones at close, operator-directed). voffice1 pulled to sync.
Owned: twice asserted a wrong ovn-central cert root cause; flagged a ruled
binding exception (D-072) nearly reverted without grepping the D-NNN; shipped the
v4 revert without its harness update.
Next: escalate LP #2044324; reconcile provider-bundle-check to D-141; phase-03
core verify. Status ONLY in CURRENT-STATE.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
DEFER ovn-central cert to a new session: bounce failed; precise defect = missing common_name
...
Bounce done, did not work. Three remedies exhausted, all deterministic (not the
inconsistent race): vault reissue-certificates; juju bind default->metal-internal;
full remove-relation + integrate (fresh id certificates:142). All leave
ovn-central with ca+client.cert only, no server cert.
Precise defect for the LP escalation: ovn-central/0 publishes sans:[10.12.12.122],
private-address, ingress-address, unit_name -- and NO common_name. The
tls-certificates interface needs a common_name to sign a SERVER cert; without it
vault issues only the client cert + CA. Missing CN traces to the Skipping
internal/admin/public 'no local address found' (LP #2044324).
Resume point recorded: escalate LP #2044324; decide accept-degraded vs the
UNVERIFIED os-*-network avenue. Live state carried forward: ovn-central default is
metal-internal (stays, architecturally correct); certificates:142; v4 VIPs; vault
init+unseal+root-CA; mysql ONLINE.
Process-defect note owned (operator-flagged): twice asserted an ovn-central root
cause the evidence did not support this session; next session treats my
hypotheses as unverified until measured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
correct framing: ovn-central metal-internal binding is architecturally right and STAYS; cert bug separate
...
Operator: ovn-central IS a metal-internal service by the plane's intended purpose.
D-052 explicitly categorizes OVN NB/SB DB (ovsdb*) under metal-internal;
ovn-central has no public/admin API endpoint and all functional endpoints already
bind metal-internal, so the '' default belongs there too. The metal-admin default
was an API-charm convention mis-inherited by a non-API service.
What was wrong was ONLY my premise that the rebind would fix the cert bug -- it
does not. The binding (correct categorization) and the cert failure (charm
LP #2044324, server-cert request/response) are SEPARATE. The D-052 amendment
stands on architectural grounds; the binding is retained.
CURRENT-STATE and the D-052 amendment corrected accordingly. Next: bounce the
certificates relation to force a fresh exchange against the correct single-plane
baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
OWNED: the ovn-central binding fix did NOT resolve the cert issue; hypothesis was wrong
...
juju bind ovn-central metal-internal applied (EXIT 0, bundle+live). Post-bind the
cert handler re-ran and the SAME skip warnings persist (internal/admin/public, no
local address found); vault still issues only ca+client.cert, no server cert;
ovn-central still awaiting server certificate data.
So moving the default binding did NOT make internal/admin/public resolve -- those
endpoint spaces are not tied to the default binding (they need os-*-network config
or dedicated bindings ovn-central lacks). The default-on-metal-admin was NOT the
cause; the real fault is the server-cert request/response (core of LP #2044324),
which the binding doesn't touch. The D-052 amendment ruling was taken on a wrong
premise I supplied -- owned.
Keep-or-revert the binding is an open operator call (harmless either way,
ovn-central was already broken). Actual cert remedy unresolved: bounce units /
investigate server-cert request format vs vault charm / escalate LP / accept
degraded. Awaiting direction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
apply ovn-central default -> metal-internal in bundle; WITHDRAW dashboard fix (D-072)
...
Applied the ruled ovn-central fix to bundle.yaml: '' default metal-admin ->
metal-internal, with an explanatory comment citing LP #2044324 and the D-052
amendment. provider-bundle-check PASS, repo-lint 0 fail.
WITHDRAWN: the openstack-dashboard:cluster fix. On reading bundle.yaml:667 before
applying, found it is D-072 / BUNDLEFIX-011 -- a RULED, as-executed fix: horizon
renders haproxy's 443 backend on the cluster-binding address but only creates
apache SSL vhosts for the default+public addresses, so cluster on metal-internal
= dashboard VIP HTTPS dead (L4 check masks it). metal-admin is deliberate.
Reverting it would reintroduce the exact bug D-072 repaired.
OWNED: my sweep flagged it because it compared binding VALUES against generic
D-052 plane purposes without grepping the governing D-NNN -- which CLAUDE.md
explicitly requires before touching a built surface. The in-bundle comment was
right above the line. Caught by reading the bundle before applying, not by the
sweep. Sweep audit, D-052 amendment, and CURRENT-STATE all corrected: 1 real
deviation (ovn-central), 5 non-issues.
Dashboard binding UNCHANGED. Next: re-stage bundle + live juju bind ovn-central.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
