Fork: 0
OpenStack / openstack-caracal-dc-dc
Transfer to URL with SHA Find file
Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260724-caveman-disable.md
@JANeumatrix JANeumatrix 1 day ago 14 KB Juju client setup on voffice1: vr1-maas cloud + both DC credentials
Raw Blame History

Changelog -- 2026-07-24 -- caveman plugin: disable on live host + governance + guardrails

Session scope: acting on a 4-seat council review of the third-party caveman Claude Code plugin (installed + off-by-default-guarded 2026-07-23, commit 338c919). Operator accepted the review recommendations (exact utterance: "I'll accept your recommendations"). Harness-config + repo-governance only -- NO cloud mutation, NO runbook change. CURRENT-STATE deliberately untouched (adding a SEC row is not an L10 trigger; the G14 open-SEC count reconciles at Stage 4 close -- see item 2). Council finding that drove the disable: enabled:true runs third-party node on every session start + every prompt REGARDLESS of CAVEMAN_DEFAULT_MODE=off (the env guard gates arming/terse-output, not hook execution), and the plugin's own honest docs put net token savings at break-even-to-negative on this host's gated-approval workload -- so a standing per-turn execution surface on a credential-bearing jumphost was not worth carrying.

Item 1 -- caveman DISABLED on the live host (enabled:false, both scopes)

  • ~/.claude/settings.json (user scope, authoritative for vcloud): enabledPlugins."caveman@caveman" flipped true -> false. Stops the SessionStart + UserPromptSubmit hooks from wiring at the next session start.
  • /home/jessea123/openstack-caracal-dc-dc/.claude/settings.json (committed repo scope): added enabledPlugins {"caveman@caveman": false}. Project scope outranks user scope (Local > Project > User), so this is a fail-closed, travels-with-the-clone off-switch across the operator's workstations -- mirrors the dual placement used for the env guard.
  • CAVEMAN_DEFAULT_MODE=off retained in both files as a residual layer; the extraKnownMarketplaces.caveman entry RETAINED (plugin stays installed) so a deliberate re-enable for hardened-workflow deployment testing remains cheap.
  • Verified: both settings.json parse (jq); user + repo enabledPlugins."caveman@caveman" = false; no ~/.claude/.caveman-active flag (not armed). repo-lint 0 fail / 1 legacy warn (unchanged).
  • Revert: set both enabledPlugins."caveman@caveman" back to true (repo copy via git revert/edit; user copy by hand). Note: re-enabling re-incurs the SEC-017 re-verify duty.

Item 2 -- SEC-017 opened (supply-chain re-verify obligation)

  • Added row SEC-017 to docs/security-ledger.md (next-free by grep; highest prior = SEC-016). Records the standing third-party per-turn code-execution surface, its point-in-time benign verdict at SHA 0d95a81, the not-declaratively-pinnable caveat, the disable disposition, and the STANDING DUTY to re-verify the hooks on any re-enable or plugin update.
  • Register call = OPS (no D-number): fails GA-R3's Roosevelt-delta / A1 grep test (no bare-metal analog of a jumphost output-style plugin); precedent D-129 (OPS plugin-install stays OPS). The SEC row is the durable home the security-ledger mandate requires ("a row at discovery, never only a note") -- the prior changelog (338c919) is GA-R2 non-citable/archive-bound, so the posture had no durable home before this.
  • Count: open SEC rows 12 -> 13. This is NOT an L10 status-couple trigger (L10 fires on docs/audit/*, design-decisions Status lines, and session-ledger gate rows -- not security-ledger.md), so CURRENT-STATE is untouched; the G14 count reconciles at Stage 4 close per its existing deferral pattern.
  • Self-measured drift baseline (sha256, 2026-07-24; executing cache copy == git clone at 0d95a81, only a .in_use runtime marker differing):
    • src/hooks/caveman-activate.js bc0af3ba327657630e5f6c60be619e6b5394bf03edc29b1384a15f09a1b348bf
    • src/hooks/caveman-mode-tracker.js 3e34f004d4a9e65609f8c95b47dfe5f0e3a6f2d16b88c7f8af730187ee2fb246
    • src/hooks/caveman-config.js f47fe2e6440578eeb20408bc3131b63fb8e1ae4f6ea26dbdcca33e7901e5ee8c
    • src/hooks/caveman-stats.js 433a345a279c2d54be51864ef77c0a7b19d0f8604b9a441ce0a1eded31f113e8
    • src/hooks/cavecrew-model-overrides.js 9f2601e8551653609f0b9d700c9bd75bffa43747d2db18ffe83fb73de5dc3607
    • .claude-plugin/plugin.json b76999bd244fa59c2b8805b1c2b573458e2d065b58e5904fc8e9e545e3dac7a6
  • Revert: delete the SEC-017 row from docs/security-ledger.md (git revert <sha> for the commit).

Item 3 -- guardrails: repo-lint L11 + fail-closed compress prohibition

Council-reviewed guardrail set. (a) a repo-lint hash-baseline of the plugin files was REJECTED (layering violation -- the plugin lives outside the repo under ~/.claude, non-portable across workstations, and duplicates the plugin's own checksums.sha256; drift-on-update belongs in the SEC-017 re-verify duty, not repo-lint). Built (b) and (c):

  • (b) repo-lint L11 -- caveman-compress residue detector. scripts/repo_lint.py gains L11: FAIL on any *.original.md in the tree (the backup signature of a lossy in-place /caveman-compress rewrite). Excludes .git, .claude worktrees, .terraform, __pycache__ (reuses the all_text() skip set); opt-out marker repo-lint: allow-original-md per the L9 convention. Docstring catalog + repo-lint.sh header updated L1-L10 -> L1-L11. Harness tests/repo-lint/run-tests.sh gains T42 (residue FAILS), T43 (opt-out suppresses), T44 (worktree excluded): 45/45 PASS (was 42). It is a POST-HOC detector (the original is already overwritten when the backup exists) -- it complements, not replaces, (c).
  • (c) fail-closed compress prohibition. Mechanism measured read-only: /caveman-compress <file> is a SKILL that mutates by running python3 -m scripts <file> from the skill dir (in-place overwrite + <name>.original.md backup). Two layers added: (1) .claude/settings.json deny rules Bash(*python3 -m scripts*) + Bash(*caveman-compress*) -- verified no legit repo usage collides (the strings appear only inside repo_lint.py's L11 text, never in an executed argv); (2) a CLAUDE.md "Tooling guardrails" section prohibiting the compressor against ANY repo-tracked prose file, scoped to the whole record corpus (not a 4-file hand-list, which would rot). Note the pre-existing Skill(*) ask rule already gates the skill invocation itself; the deny is the fail-closed backstop that does not depend on that routing.
  • Net enforcement stack: plugin DISABLED (item 1) is primary; Skill(*) ask + the compress deny gate invocation; L11 catches residue; CLAUDE.md is the human-readable rationale.
  • Verification: jq valid; all touched files ASCII/LF; repo-lint 0 fail / 1 legacy warn; bash scripts/run-tests-all.sh -> GAUNTLET: ALL GREEN (77 harnesses).
  • Revert: git revert <sha> -- removes L11 (+ its 3 harness cases), the two deny rules, and the CLAUDE.md section together.

(Session continued into the VR1 DC-mirror work -- distinct topic, same session, so per GA-R2 it lands in this one changelog.)

Item 4 -- DC1 debmirror PAUSED; bandwidth measured NOT the bottleneck

  • Operator asked to pause one sync so a single mirror completes before tomorrow, on the premise the two syncs share the vcloud uplink. PAUSED dc1 (live): systemctl stop dc1-mirror-sync.service dc1-mirror-sync.timer on the dc1 rack; debmirror killed (0 procs), held at 329.9 GiB (resumable; status left stale "RUNNING"). Timer stopped so it cannot auto-restart and re-contend.
  • MEASURED result: pausing dc1 did NOT speed dc0 up -- solo dc0 averaged ~37 GiB/h vs ~35 GiB/h while contended. So the vcloud uplink is NOT the binding constraint (it is per-connection/per-file, dominated by universe's small-package tail). The pause neither helped nor hurt dc0; dc0 is on track (~70% at 2026-07-24T15:35Z, ETA ~this evening UTC). This finding fed the DC1 decision (item 5): rather than idle dc1, switch it to a caching proxy.
  • Revert: resume dc1's full sync -- on the dc1 rack: systemctl daemon-reload && systemctl start dc1-mirror-sync.service dc1-mirror-sync.timer (debmirror resumes from 329.9 GiB, skipping downloaded files).

Item 5 -- DC1 interim apt caching proxy: D-135 amendment + dc-cache-proxy.sh

  • D-135 AMENDED 2026-07-24 (GA-R5, operator utterance quoted in design-decisions.md): DC1 gets an INTERIM apt caching proxy to unblock its build now; DC0 stays the full-mirror deployment under test; DC1 proxy REPLACED by a full mirror afterward. Both artifact-delivery models exercised this rehearsal (Roosevelt input, D-132/D-133). The paused dc1 debmirror (~330 GiB) is KEPT dormant as the fallback, not deleted.
  • Shipped scripts/dc-cache-proxy.sh (site-keyed check/install; apt-cacher-ng in proxy mode at the D-134 utility .4:3142, consumed via juju model-config apt-http-proxy=http://<LISTEN>:3142; caches archive + security + UCA with no per-repo remap). REUSES the <site>-mirror-net.service net layer (does not re-create it; fails loud if absent, pointing at dc-mirror.sh). Feasibility recon confirmed read-only first: apt-cacher-ng installable (3.7.4-1ubuntu5.24.04.1), all three upstreams 200 via the edge, port 3142 free.
  • Harness tests/dc-cache-proxy/run-tests.sh (15 static cases: syntax, both sites' utility .4 + ruled edge gw, stable-identity keying, port pin, proxy-mode consumption, archive+UCA smoke coverage, net-layer reuse, MEASURED-tag discipline, arg/site validation) -- 15/15 PASS.
  • CURRENT-STATE Stage-4 narrative updated (DC0/DC1 artifact-path split). repo-lint 0 fail / 1 legacy warn; bash scripts/run-tests-all.sh -> GAUNTLET: ALL GREEN (78 harnesses) (was 77; +tests/dc-cache-proxy).
  • Live install on the DC1 rack (apt-get install apt-cacher-ng + dc-cache-proxy.sh install dc1) DONE + verified 2026-07-24 (gated): apt-cacher-ng 3.7.4-1ubuntu5.24.04.1 active on .4:3142, drop-in matches, net layer present, check PASS 8/8 -- proxy serves archive (200) AND UCA caracal (200) (UCA cached with no remap, as expected in proxy mode). NOTE: the DC1 rack reports a pending kernel reboot (running 6.8.0-134 vs installed -136) -- pre-existing, deferred to a maintenance window (a reboot would disrupt the paused mirror).
  • Revert: git revert <sha> (removes the script, harness, D-135 amendment paragraph, and CURRENT-STATE split); if installed live, on the dc1 rack: systemctl disable --now apt-cacher-ng; rm -f /etc/apt-cacher-ng/acng.conf.d/dc1-cache-proxy.conf.

Item 6 -- phase-4 runbook: per-DC MAAS service-credential step (Step 2.0)

  • Operator-directed: the MAAS API key Juju needs is created by NO prior stage -- a standing DEPLOYMENT-TIME task, so it belongs in the runbook. Added Step 2.0 to runbooks/dc-dc-phase4-juju-bundle-per-dc.md (before the bootstrap MUTATION): create a DEDICATED per-DC MAAS admin service user (sudo maas createadmin) + issue its own API key (sudo maas apikey --generate) -- per-DC credential isolation (SEC-012/-016; Roosevelt: per-DC cloud creds), never a cross-DC reuse.
  • Creds location corrected (operator-directed): the key stages in the JUMPHOST per-site creds folder (~/vr1-<dc>-creds/maas-api-key.txt on vcloud, SEC-009 convention) -- NOT a new location on the region (voffice1) as the first walkthrough had it. Generated FROM the jumphost so it transits ssh stdout with no persistent copy on the region; never printed (SEC-006); consumed by Juju via path/stdin, never argv. Each issued key = a per-DC SEC rotation row.
  • Revert: git revert <sha> (removes Step 2.0). No live/cloud state touched.

Item 7 -- Per-DC MAAS service credentials created (SEC-018/-019) [CLOUD MUTATION, driven]

  • Executed phase-4 Step 2.0 for both DCs (operator: "You drive it"), logged window ~/as-executed/2026-07-24-dc-maas-creds.log. Passwordless sudo on voffice1 confirmed; maas createadmin --password supported (read-only precheck).
  • Created dedicated MAAS superusers juju-vr1-dc0 and juju-vr1-dc1 (random passwords generated ON the region, UNSTORED -- Juju auths via the API key), and staged each user's API key into the vcloud per-site folder: ~/vr1-dc0-creds/maas-api-key.txt and ~/vr1-dc1-creds/maas-api-key.txt (0600). Keys generated FROM the jumphost (no persistent region copy), verified by FORMAT ONLY (both parts=3, 70 chars), never printed. Result fail=0.
  • SEC-018 (dc0 key) + SEC-019 (dc1 key) opened -- per-DC rotation obligations (open SEC 13 -> 15; reconciled at Stage 4 close per the G14 pattern). as-executed index entry added (log itself stays uncommitted, secret-adjacent).
  • Revert: on voffice1 sudo maas apikey --username=juju-vr1-dc0 --delete=<key> (+ dc1) and sudo maas deleteuser juju-vr1-dc0 (+ dc1); shred -u the two vcloud key files; git revert <sha> for the SEC/index/changelog rows.

Item 8 -- dc1 VIP overlay drafted (overlays/vr1-dc1-vips.yaml)

  • Operator OK'd the overlay approach (vs editing bundle.yaml in place). Drafted overlays/vr1-dc1-vips.yaml: re-homes the 11 API-service VIPs from bundle.yaml's vr1-dc0 bands into vr1-dc1's MEASURED bands (lib-net vr1-dc1 PLANE_NAME): provider-public 10.12.4 -> 10.12.64, metal-admin 10.12.8 -> 10.12.68, metal-internal 10.12.12 -> 10.12.72; host octets .50-.60 unchanged. Base bundle.yaml stays the vr1-dc0 source of truth; overlay applied only for the dc1 deploy (juju deploy ./bundle.yaml --overlay ./overlays/vr1-dc1-vips.yaml). YAML valid (11 apps), repo-lint 0 fail.
  • Revert: git revert <sha> (removes the overlay). No live/cloud state.

Item 9 -- Juju client setup on voffice1 (cloud + credentials) [client config, not a cloud mutation]

  • Juju client location = voffice1 (operator-ruled after reconsideration): the Office1 headend has a Roosevelt analog and D-128 runs MAAS-facing work there so runbooks read "from the headend"; vcloud (substrate host) has no Roosevelt analog, so driving the cloud from it would not transfer. Juju 3.6.25 (3/stable, matches VR0 + the bundle's Juju 3.6 target) installed on voffice1; removed from vcloud.
  • juju add-cloud --client vr1-maas -- the ONE Office1 MAAS region (type=maas, endpoint http://10.10.0.20:5240/MAAS). juju add-credential both DCs: vr1-dc0-cred + vr1-dc1-cred from the staged keys, piped vcloud->voffice1 into a home-dir yaml (snap can't read /tmp), never printed, shred -u'd. Juju credential store on voffice1 now holds runtime copies (noted on SEC-018/-019; canonical files stay on vcloud). Credentials validate at bootstrap (juju does not pre-validate).
  • Two snap gotchas measured + worked around: the juju snap has a private /tmp (stage bundle/overlay/cred files under $HOME); voffice1 ssh sessions have per-session /tmp. Both relevant to the deploy steps.
  • Revert: ssh voffice1 'juju remove-credential vr1-maas vr1-dc0-cred; juju remove-credential vr1-maas vr1-dc1-cred; juju remove-cloud --client vr1-maas'; git revert <sha> for the SEC/changelog rows.