diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 012d5ab..89846f1 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2186,6 +2186,19 @@ borrowing D-008's SHAPE, not D-008 itself. The DELIVERY is verify-at-provision on the redeploy; the rendered content + cert-issuance chain are proven. Full record: the reeval doc's RESOLUTION 2026-08-04 section. + **>>> SESSION CLOSE 2026-08-04 (GA-R4 bookend). Stage 5 remains OPEN; a session bookend, NOT a + stage close. <<<** ovn-central cert root-caused, fix PROVEN live, and WIRED for the redeploy + (above). 5 commits `bd7a1d0..9b49b84`, gauntlet ALL GREEN (99), repo-lint 0 fail. **SEC now 29 -- + SEC-033 FILED** (tls-certificates relation databag exposes vault's global-client private key to + any juju model reader; interface-level, mitigate via juju RBAC). No new arch numbers (all OPS). + DURABILITY: vcloud 0/0; voffice1 PULLED to `9b49b84`; **dc0 rack `~/repo-stage` lacks the new + `dc-node-etchosts.sh` -- a redeploy prereq (sweep F3).** Live tests were reversible; the model is + at its captured before-state (residue: certificates:145, ovn-central/0 holds its proof cert -- + sweep F2). CLOSE SWEEP: `docs/audit/queued-findings-20260804-ovn-cert-fix.txt` (4 FIRST SURFACE: + SEC-033-now-filed, live-model test drift, rack repo-stage prereq, the juju-routing research + note). **NEXT: STEP 3 -- the clean redeploy** (Path M model teardown -> preflight -> phase-01 + with Step 1.2b -> phase-02 vault init -> converge). Body: + `docs/changelog-20260804-ovn-central-cert-fix.md`. **>>> SESSION CLOSE 2026-08-03 (GA-R4 bookend). Stage 5 remains OPEN; this is a session bookend, NOT a stage close. <<<** DEPLOY IS MID-STAGE-5: bundle deployed + mostly converged (9 machines started, mysql ONLINE, vault init+unseal+root-CA, ~25 units active, 0 error), diff --git a/docs/archive/session-ledger-rotated-20260804.md b/docs/archive/session-ledger-rotated-20260804.md new file mode 100644 index 0000000..20b886a --- /dev/null +++ b/docs/archive/session-ledger-rotated-20260804.md @@ -0,0 +1,17 @@ +# Rotated from docs/session-ledger.md on 2026-08-04 (GA-R4 rule 3 / F1 -- cap restored at this close) + +## SESSION CLOSE 2026-07-31 -- dc0 node carve COMPLETE, controller BOOTSTRAPPED, deploy blocked on prefer-ipv6 (bounded, GA-R4) + +- Branch `dc-dc-stage5-preconditions`, **11 commits** pushed (`a859352..80510f4`). NO stage opened/closed. Scan: 3 decisions, **SEC 25** (SEC-028, -029 opened), D 139 / **DOCFIX 207** / BUNDLEFIX 053. +- **`scripts/dc-node-carve.sh` SHIPPED** (harness 48/48) -- the last no-tool gap. **dc0 v4 carve COMPLETE 10/10, named gate 134/0**, and the diff against the pre-migration capture is **EMPTY**: every node's six legs match Office1 address for address. v6 carve **54/54**; the controller's own v6 restored separately (the tool walks only the nine tagged role nodes). +- **>>> JUJU CONTROLLER LIVE: `vr1-dc0-controller`, active/idle. <<<** FOURTH attempt, first success; the three failures were each a real defect one layer deeper, closed by D-138, the under-carve fix, and D-132 q1 respectively. Agent binaries resolved on attempt 1. +- **Model `vr1-dc0` created, spaces gate PASS (6/6), `apt-mirror` set** to the DC mirror -- key VERIFIED against the live client, not guessed; URL asserted on `dists/jammy/Release` content. +- **2 RULINGS (GA-R5, quoted):** *"Mint juju-vr1-dc0 on the new region (Recommended)"*; *"Copy dc0's PKI overlay to the dc0 rack (Recommended)"*. Both registered BEFORE use -- **SEC-028** (juju service credential; `vm-secret-locations` gained its first `rack` rows) and **SEC-029** (PKI overlay residency). Key proven to AUTHENTICATE before bootstrap, never printed, sha256-verified at every hop. +- **DOCFIX-206: Step 2.0's credential gate was NOT region-scoped, and I hit it** -- credential listed, folder present, user existed, all in OFFICE1. The SKIP branch would have led to an auth failure that reads as a network fault. +- **>>> BUNDLE DEPLOY ATTEMPT 1 FAILED; NOTHING IS HALF-APPLIED (model measured EMPTY). <<<** `unknown option "prefer-ipv6"` on barbican. **The dry-run PASSED and the deploy did not** -- `--dry-run` does not validate config option NAMES. Measured plan is **56 apps / 108 relations**, not the 50/97 preflight still quotes. +- **Root cause measured against CHARMHUB's own schema: 7 of 13 VIP charms declare `prefer-ipv6`, 6 do not**, and barbican never had it at ANY channel -- so R2's uniform application was never valid. **RULED: *"Research what those 6 charms do with v6 VIPs first (Recommended)"*** -- overlay NOT edited, R2 NOT amended. **STAGE 5 IS BLOCKED ON THAT RESEARCH.** +- **FINDING, logged not fixed: `preflight.sh` is REGION-BLIND** -- 19 false negatives for dc0 ("not enrolled in MAAS") because it defaults to the Office1 profile. `MAAS_PROFILE=vr1-dc0-region` drops them to zero. A blanket refusal would be WRONG: dc1's nodes still live in Office1. +- **Mutation testing earned its cost again:** a first pass proved less than it looked (deleting an assertion's MESSAGE only proves it exists), and the sharper pass found the `br-ex` static compare **could not fail**. Added a fixture, re-proved the kill. +- **OWNED -- three instrument errors, all one shape:** a capture parsed at the wrong field, a `systemctl` poll racing an async start, and a charmhub query with a wrong field name returning a uniform "no config" across 13 charms. The last two were caught by the SHAPE of the answer being implausible, not by discipline; that detector is now in auto-memory. +- Gauntlet **ALL GREEN (93)**; repo-lint 0 fail; creds-matrix 65/65. Permission allowlist +12 read-only rules (all pinned to check/verify/assert/plan subcommands). +- **NEXT:** the prefer-ipv6 research, then re-deploy. Steps 1-3.5 and the controller are DONE and need no repeat. Body: `docs/changelog-20260730-dc0-node-carve.md` (11 items). Status ONLY in CURRENT-STATE.md. diff --git a/docs/audit/queued-findings-20260804-ovn-cert-fix.txt b/docs/audit/queued-findings-20260804-ovn-cert-fix.txt new file mode 100644 index 0000000..e2f6a94 --- /dev/null +++ b/docs/audit/queued-findings-20260804-ovn-cert-fix.txt @@ -0,0 +1,101 @@ +============================================================================== +SAVEGAME SWEEP -- 2026-08-04 session (ovn-central cert reeval -> root cause -> +fix PROVEN live -> wired for redeploy; plus the 08-03 provider-bundle-check +reconcile that opened this working session). +Method (ruled 2026-07-31): read the whole session, enumerate every finding / +decision / measurement / mistake, grep each against repo surfaces. +Hit = ALREADY ON SURFACE. No hit = FIRST SURFACE, would be lost on /clear. +Status claims live in docs/CURRENT-STATE.md ONLY. +============================================================================== + +-------------------------------------------------------------------------------------- +FIRST SURFACE -- existed ONLY in the transcript. Listed first, by consequence. +-------------------------------------------------------------------------------------- + +F1. >>> SEC-033 was NOT filed on the security-ledger register. <<< The relation-databag + exposure (juju show-unit on ANY tls-certificates consumer prints vault's GLOBAL-CLIENT + private key in cleartext to any model reader) was written into CURRENT-STATE + the reeval + doc PROSE, but the register of record is docs/security-ledger.md and it has no SEC-033 row + (ledger-scan: SEC still 28, highest SEC-032). This savegame ADDS the SEC-033 row (graduating + the observation to its surface -- GA-R7). grep 'SEC-033' docs/security-ledger.md before this + session: 0 hits -> FIRST SURFACE on the register. + +F2. >>> LIVE dc0 MODEL DRIFTED FROM CURRENT-STATE via this session's reversible tests. <<< + CURRENT-STATE describes the before-state (certificates:142, all 3 ovn-central units broken). + MEASURED now: the certificates relation is certificates:145 (bounced 3x during the fix proof), + and ovn-central/0 carries an ISSUED vault server cert + /etc/ovn/{cert_host,key_host, + ovn-central.crt} (from the single-unit proof; it persists) while /1,/2 are broken. Bindings + were reverted to all-metal-internal (= before-state). This is teardown-bound (the redeploy + wipes it) but recorded so the next session does not read "/0 has a cert, relation is :145" as + a new fault. Measurement wins (GA-R1 C2); no doc correction owed since it is a transient test + residue on a teardown-bound model, not a config-of-record claim. + +F3. >>> REDEPLOY PREREQ: the dc0 rack ~/repo-stage does NOT carry the new + scripts/dc-node-etchosts.sh. <<< Step 1.2b runs it FROM the rack (D-138), so before the + redeploy the rack stage must be updated (scp + sha256-verify, per the D-138 staged-copy rule) + OR the step run from wherever the juju client + repo are co-located. Not staged as of close. + +F4. Developer-doc research finding worth keeping (partially on surface): juju bindings are + explicitly NOT an L3 routing mechanism (Juju charm-guide) and default-route selection is + non-deterministic ("first space", LP#1781856) -- this is WHY moving ovn-central's default + binding did not restore a route and the CN stayed empty. Captured in the reeval doc's + CORRECTION/RESOLUTION and CURRENT-STATE; recorded here so the reasoning is not lost if those + are later condensed. + +-------------------------------------------------------------------------------------- +ALREADY ON SURFACE (grep-confirmed) -- recorded for completeness. +-------------------------------------------------------------------------------------- +- provider-bundle-check reconciled to D-141 (v4-only) + FN1/FN2/FN3: commit 5c9da99; + docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt (AMENDED block); changelog-20260803. +- ovn-central root cause (isolated-plane metal-internal, CN=get_hostname(metal-internal addr)), + LP #2044324 NO MATCH, vault signs any non-empty CN, OVN no CN-content rule: reeval doc + + CURRENT-STATE + changelog-20260804. +- rdns_mode fix VOID / dns_servers fix VOID / binding approach refuted (3 configs): reeval doc + CORRECTION+RESOLUTION, remediation-plan VOID banner, sweep-misses F1 correction, CURRENT-STATE. +- Fix PROVEN (controlled single-unit test: cert issues only with the /etc/hosts entry): reeval + RESOLUTION + changelog-20260804 Item 2. +- NEW mechanism scripts/dc-node-etchosts.sh + tests/dc-node-etchosts (harness 9/9) + runbook + Step 1.2b; a NEW mechanism borrowing D-008's shape, not D-008: committed 9b49b84. +- F2 (hacluster stale IPv6 VIP CIB, 4 units; designate live vip still v6; v6 naming-collision + defect) + F3 (octavia downstream of ovn-central; neutron-api false-green): stage5-sweep-misses + (committed); both resolve via the clean redeploy. +- app STAYS on metal-internal (D-052 data-type placement): reeval RESOLUTION + CURRENT-STATE. + +-------------------------------------------------------------------------------------- +MANDATORY FIVE (structurally invisible) +-------------------------------------------------------------------------------------- +1. Gitignored settings.local.json: allow 313 / ask 11 / deny 0 -- UNCHANGED this session (was + 313 at 08-03 close). Broad ssh allows present as before (Bash(ssh voffice1 *), the -J voffice1 + 172.31.0.2/.6 dc-rack forms). No new allow rule added this session. Same broad-ssh posture the + 08-03 sweep flagged; still gitignored (only recovery copy). +2. Dangling refs: new commits cite scripts/dc-node-etchosts.sh, tests/dc-node-etchosts, + runbooks/phase-01-bundle-deploy.md Step 1.2b, the audit docs -- all resolve (repo-lint 0 fail). +3. Ruling fidelity: NO new GA-R5 rulings this session (all OPS: scripts/runbook/audit). No new + D-/DOCFIX-/BUNDLEFIX numbers assigned (next-free unchanged D-142/DOCFIX-209/BUNDLEFIX-053). +4. As-executed log: not opened (run-logged.sh interactive-only). This session's mutations were + reversible live tests, all reverted; the durable deliverables are committed. Gap declared here. +5. Contradiction detector: F2 above (live model vs CURRENT-STATE before-state) is the residue; + no config-of-record contradiction (deploy input verified v4-only clean; the rdns_mode/dns_servers + corrections already landed in the docs this session). + +-------------------------------------------------------------------------------------- +OWNED (mistakes this session) +-------------------------------------------------------------------------------------- +- Root cause called WRONG THREE TIMES before it was right: (i) "metal-internal rdns_mode + disabled" -- refuted, rdns_mode=2 on all planes; (ii) "dns_servers advertises unroutable + 10.12.4.6" -- refuted, both links set to the reachable resolver still failed; (iii) landed on + the real cause (isolated plane, CN from metal-internal addr). Each corrected by MEASURING before + committing further -- the discipline worked, but the first two shipped into committed records and + needed VOID/CORRECTION passes. +- Explained away a null resolvectl result ("resolved binds to eth1") as a rationalization instead + of measuring; advisor caught it; the discriminating test then established the real constraint. +- TWO false-negative binding tests: concluded the binding fix failed without bouncing the relation + to re-fire the request (Test 1), then mis-attributed which address the CN uses; only the direct + get_hostname measurement + a fresh bounce settled it. Same false-negative shape twice. +- Over-claimed "metal-admin-only DNS" (metal-internal is the outlier that matters, not a posture) + and "the default binding is not the lever" (it is ON the causal path) -- both corrected in-record. +- Guessed dc1's metal-internal CIDR (10.12.68) writing the harness; the harness caught it + (real value 10.12.72.0/22, from lib-net). Fixed; a case that could not have passed on the guess. +- Nearly built the cloudinit wiring on the get_hostname-layer proof WITHOUT confirming a server + cert actually issues (the exact false-negative shape); advisor made the cert-issuance test + blocking; it passed and closed the chain. diff --git a/docs/security-ledger.md b/docs/security-ledger.md index 121aca7..e9bcce4 100644 --- a/docs/security-ledger.md +++ b/docs/security-ledger.md @@ -83,3 +83,4 @@ | SEC-030 | 2026-07-31 | **Local permission rules bypass the committed ask-gating -- ACCEPTED, KNOWN posture, recorded so it is explicit rather than implicit.** `.claude/settings.json` (committed team policy) deliberately puts `juju deploy`, `maas admin machine delete` and `maas admin * update/create/release` in **ask**, and the controller-destruction command in **deny**. `.claude/settings.local.json` (GITIGNORED, 286 allow rules accumulated across sessions, largely from interactive don't-ask-again approvals) contains `Bash(ssh *)`, `Bash(ssh voffice1 *)`, `Bash(ssh vr1-dc0-rack *)`, `Bash(ssh vr1-dc0-maas *)` and `Bash(python3 *)`. Because an `ssh ''` invocation matches on the OUTER ssh command, a destructive `maas admin machine delete` issued over ssh never reaches the ask rule; `python3 *` permits arbitrary execution outright. **CLAUDE.md hard rule 3 states the ask rules enforce per-mutation approval and are not to be worked around, so this is a real divergence between stated and enforced posture.** | 2026-07-31 permission carry-over review; continuation of `queued-findings-20260730-dc0-region-migration` F1 | operator ("Leave them as they are") | **OPEN -- ACCEPTED BY RULING, NOT AN OVERSIGHT.** (1) The operator was shown the five rules verbatim and the gating they bypass, and ruled to keep them: the mutation gate in practice is the operator's conversational approval of each presented command, not the rule engine. That has held all session. (2) **The compensating control is the PRESENTATION discipline** -- every mutation is still stated, justified as minimal, and approved before running. If that discipline lapses, nothing else catches it. (3) **NOT promoted to committed team policy** (ruling: "Promote a curated safe subset"): 36 read-only rules were promoted to `settings.json`; these five stay local-only so the bypass is NOT baked into team policy or carried to Roosevelt. (4) **`Read(//home/jessea123/**)` was deliberately EXCLUDED from promotion** -- it grants read of `~/vr1-*-creds/`, `~/vault-init/`, `~/as-executed/` and `~/tenant-*`, every one of which CLAUDE.md forbids reading into context. Promoting it would have put a secrets-read permission into team policy. (5) Four local wildcards were NARROWED before promotion because they permit mutation: `ip route *`, `ip neigh *`, `bridge fdb *`, `tc qdisc *` all allow add/del on the host running every DC node; only the `show`/`get` verbs were promoted. (6) A rebuild still loses the ~250 unpromoted local rules -- that half of F1 remains open by choice. | | SEC-031 | 2026-08-02 | **The vr1-dc1 OPNsense edge is currently an OPEN ROUTER serving its management GUI to the simulated ISP.** The 2026-08-01 hard power cut (in-place tofu resize bouncing the containment VM) destroyed the edge's FreeBSD base-system user database -- `/etc/master.passwd` (`pwd_mkdb: line #1 too long`) and `/etc/group` (`pw: group 'wheel' does not exist`) -- so `Configuring firewall.....failed.` on every boot since and NO pf ruleset is in force. MEASURED against a control, which is what makes this a finding rather than a worry: from vcloud, the healthy office1 edge (same 26.7 image, same build path) answers WAN ICMP with 100% loss and its GUI with `000`; the dc1 edge answers WAN ICMP at **0% loss** and its GUI **200 in 0.014s**. It also forwards without translating -- simultaneous tcpdump on both taps shows the same packet leaving the WAN leg with source still `10.12.64.2`. Bounded by lab topology: the "ISP" is the `vr1-dc1-uplink` libvirt NAT net on vcloud, not the internet, and D-125 egress isolation is unaffected. Discovered by the read-only dc1 edge assessment, `docs/audit/dc1-edge-assessment-20260802.txt`. | docs/audit/dc1-edge-assessment-20260802.txt | operator | OPEN -- closes when the edge's pf ruleset is restored (the repair regenerates the user DB from config.xml via the proven D-112(c) `local_user_set()` path, or donates it from the pristine `opnsense-26.7-nano.qcow2` in the same pool). Config.xml itself is INTACT -- fsck names every inode it deletes and `/conf/config.xml` is not among them. NOT rebuild-blocking; the exposure ends with the repair. | | SEC-032 | 2026-08-02 | **dc0 edge credentials have been RESIDENT ON THE RACK, undisclosed, since 2026-07-20 -- the dc0 analog of SEC-015, which dc1 opened and remediated the same phase.** Found while staging the dc0 edge rebuild: `ls` (never a read) of `vr1-dc0-rack:~/vr1-dc0-creds/` shows `vr1-dc0-edge_ed25519` (the PRIVATE half, 432 B, 0600) and `opnsense-api.txt` (173 B, 0600), both dated 2026-07-20 -- i.e. left in place at the original edge build. dc1's build shredded both equivalents in the same phase and disclosed the transient as SEC-015; dc0's never did, and 13 days passed with nothing noticing. **THIS ALSO ANSWERS SEC-021(a), which asked for "a decision between RE-MINT and locating an off-jumphost copy": the off-jumphost copy is this one.** `creds-manifests/vm-secret-locations:123` already declares `rack vr1-dc0 ssh vr1-dc0-rack:~/vr1-dc0-creds/*` as an audited location, so this was visible to a `--remote` sweep and simply had not been reconciled -- the gap is reconciliation, not detection. **The 2026-07-20 `opnsense-api.txt` is now DEAD MATERIAL:** the rebuild destroyed the config.xml that held its hash, so it authenticates to nothing. It was deliberately NOT deleted (hard rule 1 -- a finding is logged, not actioned mid-step, and it is the physical evidence behind SEC-021(a)). The edge PRIVATE KEY was likewise left in place: `opnsense-bootstrap-apikey.sh` and `opnsense-set-interface-v4.sh` both SSH the edge DIRECT with no ProxyJump, and the edge LAN is reachable only from the rack, so rack residency is a structural requirement of the current tooling, not an oversight to be quietly cleaned up. **REMEDIATED IN PASSING:** SEC-021(a)'s access gap is closed -- a freshly minted key is now consolidated at `~/vr1-dc0-creds/opnsense-api.txt` on the jumphost (0600, 173 B, shape-verified 1x`key=`/1x`secret=` at 80 chars each, sha256-compared to the source, secret never printed). `creds-matrix.py` E1 for `dc0-edge-api` is consequently CLEAN where it previously read EXPECTED-BUT-ABSENT. | measured read-only during the dc0 edge rebuild, `docs/audit/dc0-edge-rebuild-20260802.txt`; SEC-015 (the dc1 precedent); SEC-021(a); `creds-manifests/vm-secret-locations:123` | operator | **OPEN -- three items, none actioned here.** (1) Rule on the dead 2026-07-20 `opnsense-api.txt` on the rack: shred (the dc1 precedent) or retain as SEC-021 evidence. (2) The `S2 vr1-dc0 EXPECTED-BUT-ABSENT: 'opnsense-api.txt'` finding is STILL RED and is now red for a DIFFERENT reason than when the operator accepted it on 2026-07-30 -- then the credential did not exist; now it exists and `creds-manifests/vr1-dc0.manifest` simply does not declare it. The manifest is DERIVED (`creds-matrix.py --render`), so this is a one-line regeneration, deliberately left to the operator rather than rendered inside a rebuild task. `creds-matrix.tsv:76`'s note key `n-dc0-edge-api-absent` and its `creds-matrix-notes.md:123` text are now factually stale for the same reason -- REPLACE the wording with the new invariant, never delete the row to go green (the note itself says so). (3) Decide whether rack-resident edge keys are the accepted steady state for BOTH DCs (making SEC-015's remediation the outlier) or whether the tooling should gain a ProxyJump path so the key never leaves the jumphost. Custody detail off-repo per D-069. | +| SEC-033 | 2026-08-04 | **The tls-certificates relation databag exposes vault's GLOBAL-CLIENT private key in cleartext to any juju model reader.** `juju show-unit ` (measured identical on ovn-central/0, mysql-innodb-cluster/0, keystone/0) prints vault's `ca` + `client.cert` + `client.key` from the relation databag -- the last being the SHARED global-client PRIVATE key. Inherent to the interface-tls-certificates GLOBAL-CLIENT model (vault distributes one shared client key to every consumer over the relation); NOT newly introduced and NOT deploy-specific -- the same holds on any Charmed-OpenStack + vault deploy. Impact: the global client key is only as protected as juju model access; a read-only juju user can extract it. Discovered while reading the certificates relation databag during the ovn-central cert re-evaluation. | measured read-only 2026-08-04, `docs/audit/ovn-central-cert-reeval-20260803.md`; interface-tls-certificates global-client model | operator | OPEN -- interface/upstream-level, not a deploy defect. Mitigate via juju model RBAC (bound who holds admin/read on the model). Roosevelt analog: identical on every DC's cloud. Accept-or-mitigate is an operator ruling; nothing to action in the deploy itself. | diff --git a/docs/session-ledger.md b/docs/session-ledger.md index 20c539c..3462314 100644 --- a/docs/session-ledger.md +++ b/docs/session-ledger.md @@ -185,21 +185,11 @@ Ready, four named gates green) moved VERBATIM to `docs/archive/session-ledger-rotated-20260803.md`. The live ledger would have breached the 300-line cap with this close's summary. -## SESSION CLOSE 2026-07-31 -- dc0 node carve COMPLETE, controller BOOTSTRAPPED, deploy blocked on prefer-ipv6 (bounded, GA-R4) +## ROTATED 2026-08-04 (GA-R4 rule 3 / F1 -- cap restored at this close) -- Branch `dc-dc-stage5-preconditions`, **11 commits** pushed (`a859352..80510f4`). NO stage opened/closed. Scan: 3 decisions, **SEC 25** (SEC-028, -029 opened), D 139 / **DOCFIX 207** / BUNDLEFIX 053. -- **`scripts/dc-node-carve.sh` SHIPPED** (harness 48/48) -- the last no-tool gap. **dc0 v4 carve COMPLETE 10/10, named gate 134/0**, and the diff against the pre-migration capture is **EMPTY**: every node's six legs match Office1 address for address. v6 carve **54/54**; the controller's own v6 restored separately (the tool walks only the nine tagged role nodes). -- **>>> JUJU CONTROLLER LIVE: `vr1-dc0-controller`, active/idle. <<<** FOURTH attempt, first success; the three failures were each a real defect one layer deeper, closed by D-138, the under-carve fix, and D-132 q1 respectively. Agent binaries resolved on attempt 1. -- **Model `vr1-dc0` created, spaces gate PASS (6/6), `apt-mirror` set** to the DC mirror -- key VERIFIED against the live client, not guessed; URL asserted on `dists/jammy/Release` content. -- **2 RULINGS (GA-R5, quoted):** *"Mint juju-vr1-dc0 on the new region (Recommended)"*; *"Copy dc0's PKI overlay to the dc0 rack (Recommended)"*. Both registered BEFORE use -- **SEC-028** (juju service credential; `vm-secret-locations` gained its first `rack` rows) and **SEC-029** (PKI overlay residency). Key proven to AUTHENTICATE before bootstrap, never printed, sha256-verified at every hop. -- **DOCFIX-206: Step 2.0's credential gate was NOT region-scoped, and I hit it** -- credential listed, folder present, user existed, all in OFFICE1. The SKIP branch would have led to an auth failure that reads as a network fault. -- **>>> BUNDLE DEPLOY ATTEMPT 1 FAILED; NOTHING IS HALF-APPLIED (model measured EMPTY). <<<** `unknown option "prefer-ipv6"` on barbican. **The dry-run PASSED and the deploy did not** -- `--dry-run` does not validate config option NAMES. Measured plan is **56 apps / 108 relations**, not the 50/97 preflight still quotes. -- **Root cause measured against CHARMHUB's own schema: 7 of 13 VIP charms declare `prefer-ipv6`, 6 do not**, and barbican never had it at ANY channel -- so R2's uniform application was never valid. **RULED: *"Research what those 6 charms do with v6 VIPs first (Recommended)"*** -- overlay NOT edited, R2 NOT amended. **STAGE 5 IS BLOCKED ON THAT RESEARCH.** -- **FINDING, logged not fixed: `preflight.sh` is REGION-BLIND** -- 19 false negatives for dc0 ("not enrolled in MAAS") because it defaults to the Office1 profile. `MAAS_PROFILE=vr1-dc0-region` drops them to zero. A blanket refusal would be WRONG: dc1's nodes still live in Office1. -- **Mutation testing earned its cost again:** a first pass proved less than it looked (deleting an assertion's MESSAGE only proves it exists), and the sharper pass found the `br-ex` static compare **could not fail**. Added a fixture, re-proved the kill. -- **OWNED -- three instrument errors, all one shape:** a capture parsed at the wrong field, a `systemctl` poll racing an async start, and a charmhub query with a wrong field name returning a uniform "no config" across 13 charms. The last two were caught by the SHAPE of the answer being implausible, not by discipline; that detector is now in auto-memory. -- Gauntlet **ALL GREEN (93)**; repo-lint 0 fail; creds-matrix 65/65. Permission allowlist +12 read-only rules (all pinned to check/verify/assert/plan subcommands). -- **NEXT:** the prefer-ipv6 research, then re-deploy. Steps 1-3.5 and the controller are DONE and need no repeat. Body: `docs/changelog-20260730-dc0-node-carve.md` (11 items). Status ONLY in CURRENT-STATE.md. +The oldest closed-session summary (2026-07-31 -- dc0 node carve COMPLETE, controller +bootstrapped) moved VERBATIM to `docs/archive/session-ledger-rotated-20260804.md`. The live +ledger stood at 292 lines and this close's summary would have breached the 300-line cap. ## ROTATED 2026-07-31 (GA-R4 rule 3 / F1 -- cap restored at this close) @@ -290,3 +280,16 @@ - **OWNED:** twice asserted a wrong ovn-central cert root cause (the binding); flagged a RULED binding exception (D-072 dashboard) I'd have reverted without grepping the D-NNN (would have killed Horizon HTTPS); shipped the v4 revert without its harness update. - **DURABILITY:** vcloud 0/0; voffice1 was **36 behind, PULLED to sync** (new savegame **Step 1b**, operator-directed); dc0 rack `~/repo-stage` matches HEAD. - **NEXT:** escalate LP #2044324 + decide accept-degraded vs the unverified `os-*-network` avenue; reconcile provider-bundle-check to D-141; then continue Stage-5 (phase-03 core verify). Sweep: `docs/audit/queued-findings-20260803-stage5-deploy-ovn.txt` (**4 FIRST SURFACE**). Status ONLY in CURRENT-STATE.md. + +## SESSION CLOSE 2026-08-04 -- ovn-central cert ROOT-CAUSED, fix PROVEN live, WIRED for redeploy (bounded, GA-R4) + +- Branch `dc-dc-stage5-preconditions`, **5 commits** pushed (`bd7a1d0..9b49b84`; the day opened on the provider-bundle-check reconcile to D-141). NO stage opened/closed. Scan: 3 decisions, **SEC 29** (SEC-033 opened), D 142 / DOCFIX 209 / BUNDLEFIX 053 (no new arch numbers -- all OPS). +- **>>> ovn-central "awaiting server certificate data" ROOT-CAUSED (measured). <<<** charm-ovn-central derives its TLS common_name from `get_hostname(its metal-internal address)`; metal-internal is the DELIBERATELY-ISOLATED D-052 plane with no reachable resolver, so the reverse lookup returns None -> empty CN -> vault issues no server cert -> OVN cluster never forms. rdns_mode=2 and the PTR exist; only the reverse is unreachable from the isolated plane. +- **The prior diagnosis was WRONG on every count, corrected in-record:** LP #2044324 NO MATCH; rdns_mode fix VOID (all planes already 2); dns_servers fix VOID; binding approach REFUTED live (3 configs -- the charm uses the metal-internal address regardless of binding). +- **App STAYS on metal-internal** (D-052-correct for its OVSDB/certificates data type -- operator call). The fix does NOT relocate it. +- **Fix PROVEN end-to-end (controlled single-unit LIVE test):** an /etc/hosts reverse entry -> CN populated -> vault issued `ovn-central_0.server.cert` -> the charm wrote its /etc/ovn cert files; the two control units without the entry stayed broken. OVN imposes no CN-content rule; vault signs any non-empty CN. +- **WIRED for the redeploy:** NEW `scripts/dc-node-etchosts.sh` + `tests/dc-node-etchosts` (9/9) render a per-DC `cloudinit-userdata` adding each node's metal-internal address -> hostname to /etc/hosts at provision (CIDR from lib-net; the harness caught a guessed dc1 value). Applied as gated **Step 1.2b** of `phase-01-bundle-deploy.md`. A NEW mechanism borrowing D-008's shape, NOT D-008. +- **F2/F3 resolve in the clean redeploy:** the hacluster stale-IPv6-CIB block (4 units) is gone from the v4-only config-of-record; octavia's error is downstream of ovn-central. Deploy input verified v4-only. +- Gauntlet **ALL GREEN (99)**, repo-lint 0 fail. voffice1 PULLED to sync (`9b49b84`); dc0 rack `~/repo-stage` lacks the new script (redeploy prereq -- sweep F3). Live tests were reversible; model at its captured before-state (residue: certificates:145, /0 holds its proof cert -- sweep F2). +- **OWNED:** called the root cause wrong THREE times before right; explained away a null resolvectl result (advisor caught it); two false-negative binding tests (didn't re-fire the relation); nearly wired the fix without confirming a server cert issues (advisor made that test blocking). +- **NEXT:** we are at **STEP 3** -- Path M model teardown -> preflight -> phase-01 (with Step 1.2b) -> phase-02 vault init -> converge. Sweep: `docs/audit/queued-findings-20260804-ovn-cert-fix.txt` (4 FIRST SURFACE). Body: `docs/changelog-20260804-ovn-central-cert-fix.md`. Status ONLY in CURRENT-STATE.md.