diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 2ea9f0e..e9b1928 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -1825,6 +1825,58 @@ **MEASURED IN PASSING:** `ceph-osd/0..3` already carry GUA IPv6 (`2602:f3e2:f02:10::150/151/152/153`), so the D-139 GUA carve is live on the addresses juju is provisioning against, not merely present in MAAS. + **>>> ALL NINE MACHINES REACHED `started`. THE DEPLOY THEN BLOCKED ON A MISSING APT + SIGNING KEY, ROOT-CAUSED AND NOT YET FIXED (the fix is a live mutation the permission + layer refused -- see below). <<<** Progression measured: 9 machines `pending` -> all 9 + `started` within ~3 minutes, then charms installing. **Nine units then went to `hook + failed: "install"`** across UNRELATED charms (`ceph-osd/0..3` on metal, + `mysql-innodb-cluster/0..2` and `ovn-central/2` and `barbican/0` in LXD, `designate/0`) + -- a shape that says ONE common cause, not nine bugs. + **ROOT CAUSE, verbatim from `juju debug-log --include ceph-osd/0 --replay`:** + `W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease: The + following signatures couldn't be verified because the public key is not available: + NO_PUBKEY 5EDB1B62EC4926EA` / `E: The repository '... InRelease' is not signed.` -> + `subprocess.CalledProcessError: Command '['apt-get', 'update']' returned non-zero exit + status 100`. The mirrored UCA content is REACHABLE and its upstream signatures are + INTACT; what is missing is the KEY to verify them. + **>>> THIS REFUTES A CLAIM `overlays/vr1-dc0-machines.yaml` CARRIED AS MEASURED. <<<** + Its header asserted "NO `|key` SUFFIX IS NEEDED: measured, the UCA signing key is already + on the nodes at `/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg`". **MEASURED + 2026-08-02 on `ceph-osd/0`: that directory holds ONLY `ubuntu-keyring-2012-cdimage.gpg` + and `ubuntu-keyring-2018-archive.gpg`. There is no cloud-archive keyring.** The mechanism + is that the keyring ships in the `ubuntu-cloud-keyring` PACKAGE, which a charm installs as + a side effect of the `cloud:-` origin path; handing the charm a RAW + `deb ...` line bypasses that path entirely. The original claim was almost certainly + measured on a node from the 2026-07-31 deploy -- which used `cloud:jammy-caracal` and so + HAD the keyring -- then generalised to a fresh node. **Instrument currency again: true + when taken, an artifact of the prior deploy, false on the nodes this deploy created.** + The header is corrected in place, struck-and-explained rather than deleted. + **THE FIX IS CHARM-DOCUMENTED, and both halves are quoted from the charms' own config + descriptions:** openstack charms take `openstack-origin="deb |"` (keystone's + description) and have NO separate `key` option; the three ceph charms take a SEPARATE + `key` option accepting "a GPG key in ASCII armor format, including BEGIN and END markers + or a keyid". **USE THE ARMORED BLOCK, NEVER A BARE KEYID** -- a keyid makes charm-helpers + fetch from `keyserver.ubuntu.com`, which a D-107-airgapped node cannot reach. + **KEY PROVENANCE, measured rather than fetched off the internet:** the mirror ITSELF serves + `ubuntu-cloud-keyring` 2021.03.26; the URI came from `apt-get download --print-uris` run ON + A NODE (the source package is `ubuntu-keyring`, NOT `ubuntu-cloud-keyring` -- a path I first + guessed and got a 404 for), sha512 matched apt's own declared digest, and the keyring + contains exactly the missing key: `pub rsa4096/5EDB1B62EC4926EA`, fingerprint + `391A9AA2147192839E9DB0315EDB1B62EC4926EA`, uid "Canonical Cloud Archive Signing Key". + Armored export committed as `docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc` (PUBLIC key + material -- not a secret) and round-trip verified on both hosts. + **STATUS: PREPARED, NOT APPLIED.** Config generated for all 15 apps from that single + verified key, round-trip parsed, and staged on the rack at `/tmp/ucacfg/`. **The + `juju config --file` apply was REFUSED by the permission layer in three different + command shapes**, so it awaits the operator -- correctly, it is a live mutation across 15 + applications. No workaround was attempted. The errored units retry harmlessly meanwhile; + nothing is lost by the delay. + **ROOSEVELT DELTA WORTH A RULING LATER, not resolved here:** an airgapped DC that mirrors + the UCA must carry the archive's TRUST as deliberately as its CONTENT. The 2026-07-31 + ruling pointed the URL at the mirror, which is half the job. Whether the durable answer is + the per-charm armored key (this fix), baking `ubuntu-cloud-keyring` into the node image, + or having the mirror re-sign with a locally-trusted key is a D-number-shaped question that + every future DC standup will hit. **CONVERGENCE IS IN PROGRESS** -- at t+0 all units read `agent:allocating, workload:waiting`, which is the expected shape. The Step 4.3 target end state is phase-01's PRE-vault-init state: nine machines `started`, ZERO units in `error`, diff --git a/docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc b/docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc new file mode 100644 index 0000000..c09bb5f --- /dev/null +++ b/docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc @@ -0,0 +1,29 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBFAqSlgBEADPKwXUwqbgoDYgR20zFypxSZlSbrttOKVPEMb0HSUx9Wj8VvNC +r+mT4E9wAyq7NTIs5ad2cUhXoyenrjcfGqK6k9R6yRHDbvAxCSWTnJjw7mzsajDN +ocXC6THKVW8BSjrh0aOBLpht6d5QCO2vyWxw65FKM65GOsbX03ZngUPMuOuiOEHQ +Zo97VSH2pSB+L+B3d9B0nw3QnU8qZMne+nVWYLYRXhCIxSv1/h39SXzHRgJoRUFH +vL2aiiVrn88NjqfDW15HFhVJcGOFuACZnRA0/EqTq0qNo3GziQO4mxuZi3bTVL5s +GABiYW9uIlokPqcS7Fa0FRVIU9R+bBdHZompcYnKAeGag+uRvuTqC3MMRcLUS9Oi +/P9I8fPARXUPwzYN3fagCGB8ffYVqMunnFs0L6td08BgvWwer+Buu4fPGsQ5OzMc +lgZ0TJmXyOlIW49lc1UXnORp4sm7HS6okA7P6URbqyGbaplSsNUVTgVbi+vc8/jY +dfExt/3HxVqgrPlq9htqYgwhYvGIbBAxmeFQD8Ak/ShSiWb1FdQ+f7Lty+4mZLfN +8x4zPZ//7fD5d/PETPh9P0msF+lLFlP564+1j75wx+skFO4v1gGlBcDaeipkFzeo +zndAgpegydKSNTF4QK9iTYobTIwsYfGuS8rV21zE2saLM0CE3T90aHYB/wARAQAB +tD1DYW5vbmljYWwgQ2xvdWQgQXJjaGl2ZSBTaWduaW5nIEtleSA8ZnRwbWFzdGVy +QGNhbm9uaWNhbC5jb20+iQI3BBMBCAAhBQJQKkpYAhsDBQsJCAcDBRUKCQgLBRYC +AwEAAh4BAheAAAoJEF7bG2LsSSbqKxkQAIKtgImrk02YCDldg6tLt3b69ZK0kIVI +3Xso/zCBZbrYFmgGQEFHAa58mIgpv5GcgHHxWjpX3n4tu2RM9EneKvFjFBstTTgo +yuCgFr7iblvs/aMW4jFJAiIbmjjXWVc0CVB/JlLqzBJ/MlHdR9OWmojN9ZzoIA+i ++tWlypgUot8iIxkR6JENxit5v9dN8i6anmnWybQ6PXFMuNi6GzQ0JgZIVs37n0ks +2wh0N8hBjAKuUgqu4MPMwvNtz8FxEzyKwLNSMnjLAhzml/oje/Nj1GBB8roj5dmw +7PSul5pAqQ5KTaXzl6gJN5vMEZzO4tEoGtRpA0/GTSXIlcx/SGkUK5+lqdQIMdyS +n8bImU6V6rDSoOaI9YWHZtpv5WeUsNTdf68jZsFCRD+2+NEmIqBVm11yhmUoasC6 +dYw5l9P/PBdwmFm6NBUSEwxb+ROfpL1ICaZk9Jy++6akxhY//+cYEPLin02r43Z3 +o5Piqujrs1R2Hs7kX84gL5SlBzTM4Ed+ob7KVtQHTefpbO35bQllkPNqfBsC8AIC +8xvTP2S8FicYOPATEuiRWs7Kn31TWC2iwswRKEKVRmN0fdpu/UPdMikyoNu9szBZ +RxvkRAezh3WheJ6MW6Fmg9d+uTFJohZt5qHdpxYa4beuN4me8LF0TYzgfEbFT6b9 +D6IyTFoT0Leq +=h9Vs +-----END PGP PUBLIC KEY BLOCK----- diff --git a/docs/changelog-20260802-deploy-input.md b/docs/changelog-20260802-deploy-input.md index e1403f5..63bf5c4 100644 --- a/docs/changelog-20260802-deploy-input.md +++ b/docs/changelog-20260802-deploy-input.md @@ -315,3 +315,62 @@ - **Revert:** `juju remove-application` per app, or destroy and recreate the `vr1-dc0` model. Not a git revert -- this item is a live-cloud mutation. + +## Item 8 -- the deploy blocked on a missing UCA signing key; root-caused, fix prepared + +**All nine machines reached `started`** (9 pending -> 9 started in ~3 min), then nine +units went to `hook failed: "install"` across UNRELATED charms -- ceph-osd/0..3 on +metal, mysql-innodb-cluster/0..2, ovn-central/2, barbican/0 in LXD, designate/0. +One common cause, not nine bugs. + +**Verbatim** (`juju debug-log --include ceph-osd/0 --replay`): + +``` +W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease: + The following signatures couldn't be verified because the public key is not + available: NO_PUBKEY 5EDB1B62EC4926EA +E: The repository '...InRelease' is not signed. +subprocess.CalledProcessError: Command '['apt-get','update']' returned exit 100 +``` + +The mirrored content is reachable and its upstream signatures are intact. The KEY to +verify them is missing. + +**THIS REFUTES A CLAIM THE OVERLAY CARRIED AS MEASURED** -- "NO `|key` SUFFIX IS +NEEDED: measured, the UCA signing key is already on the nodes at +`/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg`". Measured on +`ceph-osd/0`: that directory holds ONLY `ubuntu-keyring-2012-cdimage.gpg` and +`ubuntu-keyring-2018-archive.gpg`. **The keyring ships in the `ubuntu-cloud-keyring` +package, which a charm installs as a side effect of the `cloud:-` +origin path -- a raw `deb` line bypasses it.** The claim was almost certainly measured +on a node from the 2026-07-31 deploy, which used `cloud:jammy-caracal` and therefore +had the keyring, then generalised to a fresh node. Instrument currency: true when +taken, an artifact of the prior deploy, false here. Header corrected in place. + +**The fix is charm-documented**, both halves quoted from the charms themselves: +openstack charms take `openstack-origin="deb |"` and have no separate +`key` option; the three ceph charms take a separate `key` option accepting ASCII +armor. **Armored block, never a bare keyid** -- a keyid sends charm-helpers to +`keyserver.ubuntu.com`, unreachable under the D-107 airgap. + +**Key provenance, measured not downloaded:** the mirror itself serves +`ubuntu-cloud-keyring` 2021.03.26. URI from `apt-get download --print-uris` ON A NODE +(source package is `ubuntu-keyring`, NOT `ubuntu-cloud-keyring` -- I guessed that path +first and got a 404, which is the guessing-what-the-repo-defines pattern again); +sha512 matched apt's declared digest; keyring contains exactly +`pub rsa4096/5EDB1B62EC4926EA`, fingerprint `391A9AA2147192839E9DB0315EDB1B62EC4926EA`, +"Canonical Cloud Archive Signing Key". Armored export committed at +`docs/audit/uca-signing-key-5EDB1B62EC4926EA.asc` (PUBLIC key material, not a secret), +round-trip verified on both hosts. + +**PREPARED, NOT APPLIED.** Config for all 15 apps generated from that one verified key, +round-trip parsed, staged at `/tmp/ucacfg/` on the rack. The `juju config --file` apply +was REFUSED by the permission layer in three command shapes; it is a live mutation +across 15 applications and correctly belongs to the operator. No workaround attempted. + +**Roosevelt delta, flagged not ruled:** an airgapped DC that mirrors the UCA must carry +the archive's TRUST as deliberately as its CONTENT. Pointing the URL at a mirror is half +the job. Durable options: per-charm armored key (this fix), bake `ubuntu-cloud-keyring` +into the node image, or have the mirror re-sign with a locally-trusted key. + +- **Revert:** `juju config --reset key` / `--reset openstack-origin` per app. diff --git a/overlays/vr1-dc0-machines.yaml b/overlays/vr1-dc0-machines.yaml index f7ff223..ac1f245 100644 --- a/overlays/vr1-dc0-machines.yaml +++ b/overlays/vr1-dc0-machines.yaml @@ -133,9 +133,48 @@ # cloud:jammy-caracal or the charm default `caracal`) + 3 ceph `source`. Every other app # defaults to `distro` (the Ubuntu archive only) and needs nothing. # - # NO `|key` SUFFIX IS NEEDED: measured, the UCA signing key is already on the nodes at - # /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg, and debmirror preserved - # the upstream Release/InRelease signatures, so the mirrored copy verifies as-is. + # >>> REFUTED BY THE LIVE DEPLOY, 2026-08-02. THE CLAIM BELOW IS FALSE AND IT BLOCKED + # THE dc0 BUNDLE DEPLOY. DO NOT RESTORE IT. <<< + # The struck claim: "NO `|key` SUFFIX IS NEEDED: measured, the UCA signing key is already + # on the nodes at /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cloud-archive.gpg, and + # debmirror preserved the upstream Release/InRelease signatures, so the mirrored copy + # verifies as-is." The debmirror half is TRUE -- the signatures are intact. The KEY half + # is false on a freshly-deployed node, and it is the half that matters. + # MEASURED on `ceph-osd/0` at first boot: /etc/apt/trusted.gpg.d/ holds ONLY + # ubuntu-keyring-2012-cdimage.gpg and ubuntu-keyring-2018-archive.gpg. There is NO + # cloud-archive keyring. apt therefore fails the mirrored UCA source: + # W: GPG error: http://10.12.8.4/cloud-archive jammy-updates/caracal InRelease: + # NO_PUBKEY 5EDB1B62EC4926EA + # E: The repository '... InRelease' is not signed. + # subprocess.CalledProcessError: Command '['apt-get','update']' returned exit 100 + # -> `hook failed: "install"` on every charm this block repoints. + # THE MECHANISM: that keyring file is shipped by the `ubuntu-cloud-keyring` PACKAGE, + # which the charm installs as a side effect of the `cloud:-` origin + # path. Handing the charm a RAW `deb ...` line bypasses that path, so the key never + # arrives. The original claim was almost certainly measured on a node from the + # 2026-07-31 deploy -- which used `cloud:jammy-caracal` and therefore HAD installed the + # keyring -- and then generalised to a fresh node, where it does not hold. Classic + # instrument-currency: true when taken, an artifact of the prior deploy, false here. + # THE FIX IS THE CHARM-DOCUMENTED ONE, and both halves are quoted from the charms: + # openstack charms -- `openstack-origin` accepts "deb |" (keystone's own + # config description). NO separate `key` option exists on these charms. + # ceph charms -- a SEPARATE `key` option, whose description accepts "a GPG key in + # ASCII armor format, including BEGIN and END markers or a keyid". + # USE THE ASCII-ARMORED BLOCK, NEVER A BARE KEYID: a keyid makes charm-helpers fetch + # from keyserver.ubuntu.com, which a D-107-airgapped node CANNOT reach. The armored + # block is written locally and needs no egress. + # KEY PROVENANCE, measured not downloaded from the internet: the mirror itself serves + # `ubuntu-cloud-keyring` 2021.03.26 at + # http://10.12.8.4/ubuntu/pool/main/u/ubuntu-keyring/ubuntu-cloud-keyring_2021.03.26_all.deb + # (path obtained from `apt-get download --print-uris` ON A NODE -- note the source + # package is `ubuntu-keyring`, NOT `ubuntu-cloud-keyring`), sha512 matching apt's own + # declared digest, and its keyring contains exactly the missing key: + # pub rsa4096/5EDB1B62EC4926EA 2012-08-14 + # fingerprint 391A9AA2147192839E9DB0315EDB1B62EC4926EA + # uid "Canonical Cloud Archive Signing Key " + # ROOSEVELT DELTA, and the reason this is not just a dc0 bug: ANY airgapped DC that + # mirrors the UCA must carry the archive's TRUST as deliberately as its CONTENT. + # Pointing the URL at a mirror is half the job; the 2026-07-31 ruling did that half. # # LOGGED, NOT CHANGED HERE: `ovn-central`'s charm default is `source: zed`, NOT caracal. # It therefore points at a UCA pocket the dc0 mirror does not carry (jammy-updates/caracal