OpenStack/openstack-caracal-dc-dc

Fork: 0

OpenStack / openstack-caracal-dc-dc

History for openstack-caracal-dc-dc / docs / finding-20260718-voffice1-cloudinit-seed-replace.md

2026-07-18	43b29d7 Browse files » Findings: post-reboot outer plan would BOUNCE Office1 headend + edge -- DC0 apply STOPPED ... Read-only orientation + pre-apply re-verification. NOTHING applied, no module touched, no state written. Records findings only, before any fix work. WHAT WAS FOUND Outer `tofu plan` post-reboot = 7 add / 2 change / 7 destroy, NOT the committed 5 add / 2 in-place / 6 destroy in docs/dc0-deploy-readiness.md. The entire delta is `voffice1` (the live headend), not DC0 scope. 1. DOMINANT RISK -- the 2 "in-place" changes bounce both guests. VERIFIED FROM PRIMARY SOURCE (dmacvicar/libvirt v0.9.8 internal/provider/domain_resource.go): Update's only no-bounce path (L1090) requires `domainConfigUnchanged && runningChanged && autostartUnchanged`. An autostart-only diff has runningChanged=FALSE, so it falls through to stopDomainIfRunning -> DomainUndefine -> DomainDefineXML -> DomainSetAutostart. The guest is STOPPED. Applies to voffice1 AND office1-opnsense. 2. `-refresh-only` CANNOT fix it. Read is self-gating (L938): `if isImport \|\| (!state.Autostart.IsNull() && ...)`. State's Autostart is NULL, so autostart is never read back. Matches empirical `grep -c autostart` = 0 on both domains. (isImport DOES read it -- `tofu import` is a candidate bounce-free reconcile path.) 3. Seed-volume replacement is STRUCTURAL, not drift. modules/cloudinit-vm chains libvirt_cloudinit_disk.seed (stages the ISO in /tmp) -> libvirt_volume.seed (create.content.url = that /tmp path). The 2026-07-17 host reboot cleared /tmp, so the disk re-creates with a NEW random filename, so the url changes, so the volume force-replaces -- while attached to the RUNNING domain as sda. The real ISO is intact. Recurs after every host reboot. Warrants D-130 (next-free confirmed 2026-07-18). ALSO RECONCILED - Bookend item (a) CLOSED (measured): vcloud host patched to 6.8.0-136 and rebooted 2026-07-17; apt clean; no reboot-required. D-127 autostart chain PROVEN by a real unattended reboot -- both VMs + all 3 SSH targets recovered with zero manual intervention; base-leg ok, NetBox 302, edge 0% loss. Item (b) STAYS OPEN (edge guest still on 14.3-RELEASE-p16; a host reboot does not patch it). - Stage-3 "HELD NetBox addressing" blocker in dc-dc-deployment-workflow.md is STALE. Measured live against the apex: 172.31.0.0/30 (Transit, scope VR1 DC0), the 172.31.0.0/24 container, and 10.12.8.2/22 all PRESENT. Needs DOCFIX-196. - Version drift: OpenTofu on the jumphost is v1.12.4; README.md:41 says v1.12.3. Coincidental to the above -- the cause is /tmp, not the version bump. TWO OF MY OWN CLAIMS CORRECTED IN-FLIGHT (both were inferences, per hard rule 2) - Cited platform-traps.md 1e as the warrant for the autostart bounce. 1e was measured on a MEMORY change and does NOT transfer to an autostart-only diff. The conclusion held but the reasoning was wrong; replaced with the source evidence above. 1e's operational rule ("0 to change is the only assurance") stands. - Claimed the /tmp trap hits "all 9 DC0 node VMs and every containment VM". WRONG, retracted. libvirt_cloudinit_disk appears in exactly ONE module (cloudinit-vm), instantiated TWICE (main.tf:181 voffice1, main.tf:361 vvr1_dc0). The 9 nodes use modules/node-vm = blank-disk PXE, NO cloud-init; edges use REST config (D-113(a2)). Real exposure = voffice1 + vvr1-dc0 (+vvr1-dc1) = 3. PRIOR-SESSION ROOT CAUSE docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md asserts "the 2 in-place collapse to 0 once the bookend sets those two autostart live" and "no forces-replacement on any existing". Both are FALSE. The collapse prediction was wrong WHEN WRITTEN, not made wrong by the reboot -- it inferred that `virsh autostart` would be picked up by refresh, without reading the provider's Read path. Correcting those assertions is queued as Class 1 doc-integrity work. OPERATOR RULING (2026-07-18): do NOT proceed to DC0 leaving these behind. Fix scripts/data/settings to 100% correct first; the post-teardown redeploy must be error-free from these deliverables. repo-lint: 0 fail (1 legacy WARN). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YArUfQQe6VasfTvzzb9D6P JANeumatrix committed 1 day ago

2026-07-18

43b29d7
Browse files »

Findings: post-reboot outer plan would BOUNCE Office1 headend + edge -- DC0 apply STOPPED ...

Read-only orientation + pre-apply re-verification. NOTHING applied, no module
touched, no state written. Records findings only, before any fix work.

WHAT WAS FOUND

Outer `tofu plan` post-reboot = 7 add / 2 change / 7 destroy, NOT the committed
5 add / 2 in-place / 6 destroy in docs/dc0-deploy-readiness.md. The entire delta
is `voffice1` (the live headend), not DC0 scope.

1. DOMINANT RISK -- the 2 "in-place" changes bounce both guests. VERIFIED FROM
   PRIMARY SOURCE (dmacvicar/libvirt v0.9.8 internal/provider/domain_resource.go):
   Update's only no-bounce path (L1090) requires
   `domainConfigUnchanged && runningChanged && autostartUnchanged`. An
   autostart-only diff has runningChanged=FALSE, so it falls through to
   stopDomainIfRunning -> DomainUndefine -> DomainDefineXML -> DomainSetAutostart.
   The guest is STOPPED. Applies to voffice1 AND office1-opnsense.

2. `-refresh-only` CANNOT fix it. Read is self-gating (L938):
   `if isImport || (!state.Autostart.IsNull() && ...)`. State's Autostart is NULL,
   so autostart is never read back. Matches empirical `grep -c autostart` = 0 on
   both domains. (isImport DOES read it -- `tofu import` is a candidate
   bounce-free reconcile path.)

3. Seed-volume replacement is STRUCTURAL, not drift. modules/cloudinit-vm chains
   libvirt_cloudinit_disk.seed (stages the ISO in /tmp) -> libvirt_volume.seed
   (create.content.url = that /tmp path). The 2026-07-17 host reboot cleared /tmp,
   so the disk re-creates with a NEW random filename, so the url changes, so the
   volume force-replaces -- while attached to the RUNNING domain as sda. The real
   ISO is intact. Recurs after every host reboot. Warrants D-130 (next-free
   confirmed 2026-07-18).

ALSO RECONCILED

- Bookend item (a) CLOSED (measured): vcloud host patched to 6.8.0-136 and rebooted
  2026-07-17; apt clean; no reboot-required. D-127 autostart chain PROVEN by a real
  unattended reboot -- both VMs + all 3 SSH targets recovered with zero manual
  intervention; base-leg ok, NetBox 302, edge 0% loss. Item (b) STAYS OPEN (edge
  guest still on 14.3-RELEASE-p16; a host reboot does not patch it).
- Stage-3 "HELD NetBox addressing" blocker in dc-dc-deployment-workflow.md is STALE.
  Measured live against the apex: 172.31.0.0/30 (Transit, scope VR1 DC0), the
  172.31.0.0/24 container, and 10.12.8.2/22 all PRESENT. Needs DOCFIX-196.
- Version drift: OpenTofu on the jumphost is v1.12.4; README.md:41 says v1.12.3.
  Coincidental to the above -- the cause is /tmp, not the version bump.

TWO OF MY OWN CLAIMS CORRECTED IN-FLIGHT (both were inferences, per hard rule 2)

- Cited platform-traps.md 1e as the warrant for the autostart bounce. 1e was
  measured on a MEMORY change and does NOT transfer to an autostart-only diff. The
  conclusion held but the reasoning was wrong; replaced with the source evidence
  above. 1e's operational rule ("0 to change is the only assurance") stands.
- Claimed the /tmp trap hits "all 9 DC0 node VMs and every containment VM". WRONG,
  retracted. libvirt_cloudinit_disk appears in exactly ONE module (cloudinit-vm),
  instantiated TWICE (main.tf:181 voffice1, main.tf:361 vvr1_dc0). The 9 nodes use
  modules/node-vm = blank-disk PXE, NO cloud-init; edges use REST config (D-113(a2)).
  Real exposure = voffice1 + vvr1-dc0 (+vvr1-dc1) = 3.

PRIOR-SESSION ROOT CAUSE

docs/changelog-20260717-d127-iac-autostart-and-guest-updates.md asserts "the 2
in-place collapse to 0 once the bookend sets those two autostart live" and "no
forces-replacement on any existing". Both are FALSE. The collapse prediction was
wrong WHEN WRITTEN, not made wrong by the reboot -- it inferred that `virsh
autostart` would be picked up by refresh, without reading the provider's Read path.
Correcting those assertions is queued as Class 1 doc-integrity work.

OPERATOR RULING (2026-07-18): do NOT proceed to DC0 leaving these behind. Fix
scripts/data/settings to 100% correct first; the post-teardown redeploy must be
error-free from these deliverables.

repo-lint: 0 fail (1 legacy WARN).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YArUfQQe6VasfTvzzb9D6P

JANeumatrix committed 1 day ago