Newer
Older
openstack-caracal-dc-dc / docs / changelog-20260713-docfix194-opentofu-module-validation.md

DOCFIX-194 (2026-07-13) -- the OpenTofu gate was green over two broken modules

scripts/opentofu-validate.sh printed PASS on 2026-07-13 while two modules were flatly broken -- neither could have applied, and one could not even init. Both are modules that Stage 3 and Stage 4 are built on. This fixes both defects and closes the blind spot that hid them.

Root cause of the miss: root-only validation

tofu validate against the ROOT module only parses modules the root actually instantiates. A module that is not called -- or is called only from a commented-out block -- is never parsed at all. Root's main.tf calls dc-planes, dc-storage-pool, mesh-link, office1-network and opnsense-edge. It does NOT call:

base-image  cloudinit-vm  maas-vm-host  netem-link  node-vm

which is precisely the set Stage 2 (cloudinit-vm, base-image) and Stage 3 (node-vm, maas-vm-host, netem-link) depend on. They had never been parsed by any tool. This was only findable because a tofu binary now exists on the jumphost (OpenTofu v1.12.3) -- the tree was authored in sessions with no binary to self-check, exactly as opentofu/README.md warned.

Defect 1 -- modules/node-vm: libvirt_volume had a format attribute that does not exist

format = {          # WRONG -- rejected: "An argument named format is not expected here"
  type = "qcow2"
}

Verified against the provider's own schema (tofu providers schema -json, dmacvicar/libvirt v0.9.8): libvirt_volume has no top-level format. It nests under target:

target = {          # CORRECT
  format = {
    type = "qcow2"
  }
}

modules/base-image had the correct nesting all along -- which is why it passed and node-vm did not. node-vm builds EVERY DC node (PXE-boot blanks). Stage 3/4 would have failed on first use. The module's own header even said "VERIFY with tofu providers schema -json before the first real apply" -- nobody could, until now.

Defect 2 -- modules/netem-link: a destroy provisioner referencing var.* (could not init)

provisioner "local-exec" {
  when    = destroy
  command = "ssh ${var.vcloud_host_ssh_target} 'sudo tc qdisc del dev ${var.bridge_name} root' || true"
}

OpenTofu rejects this at init time:

Error: Invalid reference from destroy provisioner
Destroy-time provisioners and their connection configurations may only reference
attributes of the related resource, via 'self', 'count.index', or 'each.key'.

So the module could not even initialize, let alone apply. Fixed with the canonical pattern -- stash the values in input, read them back as self.input.*:

input = {
  ssh_target  = var.vcloud_host_ssh_target
  bridge_name = var.bridge_name
}
provisioner "local-exec" {
  when    = destroy
  command = "ssh ${self.input.ssh_target} 'sudo tc qdisc del dev ${self.input.bridge_name} root' || true"
}

netem-link is the DR/latency mechanism for Stage 3 and the Stage 6 failover drill.

The durable fix -- S3: validate EVERY module standalone

The two bugs are one-line fixes. The real defect was the gate. opentofu-validate.sh now runs a new S3 stage that validates every directory under opentofu/modules/ standalone, in a TEMP COPY (so the gate never writes .terraform/ or a module-level .terraform.lock.hcl into the repo -- only the ROOT lock file is tracked, deliberately).

A gate that reports green over unparsed code is worse than no gate: it manufactures confidence. That is what happened here, and it is what S3 prevents.

Verification

tests/opentofu-validate/run-tests.sh: 9 PASS (T2 skipped -- can't exercise the missing-binary guard on a box that has the binary). Three new cases:

  • T8 (the load-bearing one): a fixture whose ROOT IS VALID and does NOT call a BROKEN module. Root-only validation reports PASS on it. S3 must FAIL. If T8 ever goes green, the blind spot is back.
  • T9: the same fixture shape with a CORRECT module passes -- proving T8 fails on the defect, not on the fixture's shape.
  • T10: all 10 real modules validate standalone.

Fixtures use terraform_data (a builtin), so they need no provider download -- the cases stay fast and need no registry access.

Gate against the real tree: S1 PASS, S2 PASS, fmt clean, root validate clean, all 10 modules PASS standalone.

Revert

git revert <this commit>

Restores the two broken modules and the root-only gate. Nothing was instantiated; no live system is affected by either the bug or the fix.