| 2026-07-31 |

Three post-deploy defects; memcached scaled; ceph-rbd-mirror PINNED (item 22)
...
D1 jammy-backports: RULED + FIXED. Verified on CONTENT (Release fields + a real
.deb fetch), mirror 951G -> 952G. Units moved 22 error -> 17 with 15 apps past
install (was 1), so the fix works and auto-retry is running.
D2 the UPSTREAM UCA is unreachable from nodes -- LOGGED NOT FIXED. Measured FROM
A NODE, not the rack: upstream 000, mirrored 200 with real content. SIX apps set
it across TWO option names (openstack-origin on barbican/magnum/octavia, source
on ceph-mon/ceph-osd/ceph-radosgw); every other charm defaults to
openstack-origin: caracal, same upstream pocket. The UCA signing key is already
on the node, so a raw deb line verifies with no |key suffix. Genuine D-135
experiment result: the full-mirror DC must rewrite every non-Ubuntu source; the
proxy DC needs none. dc1 will not hit this.
D3 prefer-ipv6: true is FATAL on the seven charms that declare it, and is now
the blocker. This is the exact risk flagged earlier today as "never observed
live" and deliberately left OUT of the D-101 ruling note. keystone/0:
"Interface 'eth0' does not have a scope global non-temporary ipv6 address" --
get_relation_ip() returns early with get_ipv6_addr() when the option is true.
Measured on the container: eth0 holds only fe80::/64. The NODES are dual-stacked;
the LXD CONTAINERS are not. All seven true-valued apps are erroring. NOT TAKEN --
R2/D-101 are ruled surfaces and this needs its own GA-R5 exchange. One question
is stated and NOT answered: whether a v6 VIP on a container with no global v6 is
routable.
memcached scaled 1 -> 3 in dc-ha-scaleup (operator-directed); its exclusion
comment re-pointed rather than left stale. ceph-rbd-mirror PINNED as gap
register item 22 with four options and a recommendation -- (d) add the missing
liveness detection now, (b) scale to 2 active/standby at Roosevelt -- and the
overlay now forbids a silent scale without a D-108 amendment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|

Option D half 1: explicit base on all 56 apps, gated by new invariant 12
...
PROVEN AGAINST THE REAL MODEL, NOT INFERRED. The earlier one-app test only
showed the error MOVING to the next app. The fully-based bundle was staged as a
throwaway on the rack and --dry-run against the LIVE 23-application model --
the same command that errored on barbican 265-vs-261 -- exits 0.
Subordinates are in scope, measured: mysql-router is 1154 at 22.04 and 1178 at
24.04. hacluster sharing rev 166 across both bases is not a property to depend
on.
TWO SILENT UNDER-MATCHES, both caught by cross-checking the edit against the
parsed file rather than by review:
(1) a first pass keyed on the block form inserted 44 of 56 -- the twelve
-hacluster apps are single-line FLOW MAPPINGS. The inserter asserts its
app list equals what yaml.safe_load sees, and refuses otherwise.
(2) overlays/dc-ha-scaleup.yaml DEFINES vault-hacluster, an application that
exists nowhere in bundle.yaml, and it had no base. That overlay is a LATER
deploy step, so this would have re-opened the trap AFTER the deploy
succeeded. Found by the new gate on its FIRST run, in a file the fix was
not looking at -- the same later-step-overlay class as the ceph-mon miss
earlier today.
INVARIANT 12 (NEW): every application carries an explicit base equal to the
bundle's OWN default-base -- keyed on that value, not a literal. A bundle with
no default-base REFUSES rather than passing vacuously. Without this, deleting
the 56 lines is a byte-identical PASS and the trap returns at the next partial
failure, at any DC.
Five cases, each mutation-proven individually (harness 50 -> 55/55): neutering
nobase kills T49/T50/T52, wrongbase kills T51, the refuse branch kills T53.
Gauntlet ALL GREEN (93); repo-lint 0 fail.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-29 |

3.8 fixed per-DC -- and the fix exposed two repo records naming the wrong NIC
...
bridge-interface-mappings is out of bundle.yaml and per-DC in overlays/vr1-dc0-machines.yaml
(new) and vr1-dc1-machines.yaml. The machines overlay was chosen over the vips overlay for a
measured reason: both *-vips.yaml are marked GENERATED / do-not-hand-edit, so a key added
there is clobbered on the next render, and preflight.sh already assembles ${DC}-machines.yaml.
Creating the dc0 file also closes NEW-3 from the other side -- preflight's [ -f ] guard was
silently skipping a file the deploy needs.
THE WORSE DEFECT THE FIX EXPOSED. dc0's compute provider-public MACs are 52:54:00:8c:2a:8c /
52:54:00:50:48:88 -- position [1] in each node's macs list. Audit register row U5 and the
D-124 correction both name the [0] values, which are the PXE/boot MACs, and D-124 further
tells the reader to source them "from lib-hosts.sh" -- verified independently: lib-hosts.sh
declares only HOST_BOOT_MAC and cannot supply a provider MAC at all. Following either record
would have bridged br-ex onto the boot NIC: not a missing mapping that fails loudly, but a
wrong one that comes up and misbehaves. Confirmed three ways -- substrate main.tf:109-116 list
order, the macpin plan capture, and live MAAS showing br-ex over enp2s0. The correct values
and the [0]-vs-[1] trap are now recorded in the overlay, next to the data.
NEW-1: 10.12.12.0/22 was named as the geneve/DATA plane; that is metal-internal, data-tenant
is 10.12.16.0/22. Deliberately not swept repo-wide -- the 10.12.12.5x VIP legs are correct
metal-internal values.
NEW-2 phantom vr0-dc0-testcloud.yaml removed. NEW-4/6 stale octavia-pki name and the phantom
${DC}-hostnames.yaml removed (juju errors on a missing overlay). NEW-7 scale-up removed from
the VIP deploy command per R6, ${DC1_MODEL} -> ${DC_MODEL}.
R5/D-106 BUILT: phase-6 Step 5 rewritten configure-not-deploy, the unsatisfiable diff gate
replaced by a presence-and-state assertion, and the "no designate block exists" claim
corrected against bundle.yaml. os-public-hostname stays unset, per the ruling's refused
option (c).
Verified independently rather than from reported output: provider-bundle-check PASSes for
both DCs in preflight P2's own overlay order. repo-lint 0 fail / 617 files.
Logged not fixed: provider-bundle-check treats an absent bridge-interface-mappings as a silent
skip and never checks the MAC prefix against --dc, so it could not fail on either variant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvCyrwvYTTcDYnRErfMsNf
|
| 2026-07-24 |

vr1-dc1 bundle render (9-node role-separated) + expansion review record
...
Render (committee Fork-2 hand-render), validated by the extended checker:
- bundle.yaml: machines "8"-"11" (tags=openstack) -> "0"-"8" role+DC tags (block
style); every `to:` re-homed (quorum trio -> control 0/1/2; nova-compute 3->2 on
compute 3/4; ceph-osd -> storage 5-8; scaled apps + utilities -> control);
ovn-chassis -> the 2 dc1 compute provider MACs (52:54:01:d1:04:02/05:02).
- overlays/vr1-dc1-machines.yaml (NEW): per-DC retag (machines-overlay merge is
VERIFY-LIVE at `juju deploy --dry-run`).
- overlays/dc-ha-scaleup.yaml: tokens rendered to logical control ids 0/1/2
(DC-neutral); header updated; stale ceph-osd-3 comment -> 4.
- provider-bundle-check.py: ovn-chassis MAC check skips the VR0 Pattern-A set on a
role-separated bundle (the validator CAUGHT a real YAML flow-comma render bug).
Validated: base + FULL dc1 deploy input (3 overlays, --dc vr1-dc1) PASS; harness
15/15; gauntlet ALL GREEN (78); repo-lint 0-fail.
Expansion review (3 read-only reviewers) recorded in
docs/audit/stage5-expansion-review-20260724.md: relations/bindings/subordinates
HOLD under role-sep (uniform 6-NIC, SEC-011); no channel/pin change; open
resolve-before-deploy items (vault VIP, v4-vs-dual-stack phasing) + deploy-gate
script gaps + VERIFY-LIVE gates. NetBox->deploy coupling routed to Chat as a new
PROPOSED D-number (Fork-2 escalation). No cloud state mutated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VUjEb7onHpdqFHUnio6iVw
|
| 2026-07-16 |

Stage 3 (vr1-dc0) prep batch: D-121..D-124 rulings + substrate/rack wiring, HA overlay, NetBox importers
...
Prepares the VR1 Stage-3 DC substrate so the live steps reduce to review-plan-and-apply.
Authoring + offline validation only -- NO cloud mutation; every tofu apply, NetBox --commit,
and rack install stays operator-gated. Four decisions ruled this batch, all recorded:
- D-121 (ADOPTED): make the decorative single-unit control plane REAL. 14 services scale 1->3
(D-009's mechanical scale-up pulled into VR1); node layout = Option C (3 control + 2 compute +
3 storage/DC, sizing set) after a whole-host resource validation (256 vCPU/1024 GiB/10 TiB);
vault sub-ruling = v-a (MySQL-backed, charm-verify gated). Placement-balance rule for Stage 5.
- D-122 (ADOPTED): VR1 site shape -- nested-per-site containment, dark fiber East-West only,
dedicated per-site L3 ISP uplink, 2-NIC fabric-routed DC edge (LAN=provider-public, WAN=uplink),
per-site MAAS. Edge NIC count baremetal-matched (nodes 6, edge 2).
- D-123 (ADOPTED Model A): DC nodes vcloud-level; site-down = destroy the vr1-dc0-* group; MAAS
region on Office1 + a rack controller per DC (vvr1-dc0).
- D-124 (ADOPTED Scheme A): the Office1-region<->DC-rack management overlay -- a transit link on
the office1<->dc0 mesh leg; rack sizing 4/8192/80.
Code (all validated, NOT applied):
- OpenTofu: new modules/site-wan (NAT /24 uplink, provider-schema-confirmed); main.tf Stage-3
section -- vr1_dc0_wan, the 2-NIC edge, 8 Option C node VMs (for_each), the vvr1-dc0 rack
(two legs, static IPs from new vr1_dc0_rack_* tfvars -- NetBox-assigned, not invented), netem
held, mesh-link network_name output; stale vr1_dc1_planes comment fixed. Step-9 maas-vm-host
deliberately deferred (DOCFIX-179: its maas provider would force every plan to demand creds).
tofu validate Success; opentofu-validate 11/11.
- NetBox: dc-edge-wan-import.py (58/58) + dc-rack-mgmt-import.py (96/96) -- register the DC-edge
/24s + the rack transit/IP, dry-by-default, whole-plan preflight, operator-supplied literals.
- overlays/dc-ha-scaleup.yaml: the D-121 14-service HA scale-up (Stage-5), placement tokens +
rabbitmq min-cluster-size + hacluster cluster_count + re-declared vault-hacluster.
- scripts/site-headend-install.sh: rack-only mode (enrolls a DC rack to Office1's region).
- runbooks/dc-dc-phase2-tofu-dc-substrate.md: fix-forward (D-119 naming, Steps 4-5 REST-API,
D-121/122 folded in, stale gate corrected).
docs/session-ledger.md + dc-dc-deployment-workflow.md reconciled; three changelogs added.
Gauntlet ALL GREEN; repo-lint 0-fail (1 documented legacy WARN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ck6xh3jWQi5b3Su8Dx1LEH
|