| 2026-07-13 |
SAFETY SWEEP: mark the config.xml push path as a LIVE HAZARD (no behaviour change)
...
After D-113(a2) was proven (Office1's config is now API-managed), every instruction in the repo
that still said "render a config.xml and push it to the edge" became ACTIVELY DANGEROUS: a full
config.xml push replaces /conf/config.xml wholesale and drops ~667 migration-populated elements
(measured), including the only 2 firewall pass rules on the box. Following the old runbook steps
against the live edge would clobber it.
Warnings and headers only. No behaviour changes, no live system touched.
Marked (5 files):
runbooks/dc-dc-phase1-office1-standup.md DANGER banner; the config.xml render + config-ISO
sub-steps removed as RUNNABLE instructions (they
would clobber the live edge). Image-prep retained;
history preserved in git + the build changelog.
runbooks/dc-dc-phase2-tofu-dc-substrate.md STOP banner on Step 4.
scripts/opnsense-render-config.sh DANGER header at point-of-use. NOT dead: under
D-113(a2) it is to be reduced to a MINIMAL bootstrap
render. Until then, safe only for a brand-new,
not-yet-booted edge.
scripts/opnsense-build-config-iso.sh RETIRED header quoting the upstream source proving
the importer can never fire on a nano image.
docs/dc-dc-deployment-workflow.md STALE-CONTENT warning atop the tooling gap register.
THE SWEEP ALSO CAUGHT A PRE-EXISTING LANDMINE, unrelated to today's work: dc-dc-phase2 Step 4
still instructs building a CONFIG ISO for DC1's edge -- but D-112 established that ISO can never
be read (opnsense-importer probes for a read-only root; on a pre-installed nano the root is
writable and a factory /conf/config.xml already exists, so it bootstrap_and_exit 0's without
enumerating a single device). That runbook has been telling anyone who follows it to build an
inert artifact and then wonder why the edge came up on factory defaults -- which is EXACTLY the
day lost on 2026-07-12. It was never corrected at the source. Now it is. Its WAN_IF/LAN_IF
"chicken-and-egg" discussion is moot for the same reason: that problem only exists if you try to
seed a full config before first boot, and D-112(c) measures vtnetN AFTER boot.
Changelogs deliberately NOT touched -- they are history, not instructions. Marking history is
noise; marking instructions is safety.
Verification: opnsense-render-config 24 PASS, opnsense-build-config-iso 2 PASS, opnsense-api
21 PASS; repo-lint 0 fail.
STILL OPEN (the actual D-113(a2) migration): reduce the template to a minimal bootstrap, retire
the config-ISO path and the opnsense-edge module's config_seed/cdrom wiring, and rewrite Stage 3's
edge steps around the API. This sweep makes the repo SAFE, not FINISHED.
Revert: git revert this commit (restores the previous, dangerous instructions verbatim).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
CHECKPOINT 2026-07-13: session close (edge routing + serving DHCP; D-113(a2) proven)
...
Session-ledger checkpoint so a fresh session can resume cold.
LIVE STATE (measured): office1-opnsense is UP -- routing, NAT, egress 0.0% loss, serial console,
SSH key management, and kea-dhcp4 serving DHCP on udp/67 (pool .100-.199). 8 LAN pass rules in
pf. Nothing broken, nothing mid-flight.
LANDED TODAY:
1. DOCFIX-193 APPLIED -- DHCP live (was built-but-unapplied at last close)
2. SEC-005 -- GitBucket credential stored; unattended git push now WORKS (the branch had sat
12 commits ahead of origin for two days -- a real data-loss exposure, now closed)
3. SEC-006/007 -- the two known credential exposures put ON the ledger (prose only before)
4. D-113 RULED (a2): stay on OPNsense, move config to the REST API
5. D-113(a2) PROVEN END TO END, read AND write: API -> OPNsense -> Kea -> daemon.
scripts/opnsense-api.sh + harness (21 PASS); gauntlet 53 ALL GREEN.
THE ONE TRAP: DHCP on Office1 is now API-MANAGED. A full rendered config.xml push WOULD CLOBBER
IT. Do NOT run the old scp/install/reboot path against this edge until the template is reduced.
NEXT: reduce opnsense-config.xml.tmpl to a minimal bootstrap (sshd + root key + console) and move
DHCP/firewall/interfaces to the API. Repo work; no live mutation needed to land it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) write path PROVEN on the live edge: API -> OPNsense -> Kea -> daemon
...
The read half was proven earlier today; this closes the other half. Configuration written
through the REST API reaches the running daemon. (a2) is now demonstrated end to end.
Executed (live, operator-approved, deliberately IDEMPOTENT -- the payload was the exact values
read back from the API moments before, so the edge's behaviour should not change; the point was
to exercise the WRITE path):
POST kea/dhcpv4/set_subnet/93473635-... -> {"result":"saved"}
POST kea/service/reconfigure -> {"status":"ok"}
GROUND TRUTH after the write (the API's {"result":"saved"} is only the service's own verdict):
kea-dhcp4 pid 30027 (RESTARTED), bound udp4 10.10.0.1:67
Kea's OWN generated /usr/local/etc/kea/kea-dhcp4.conf:
interfaces ['vtnet0'], subnet 10.10.0.0/24, pools ['10.10.0.100-10.10.0.199'],
routers 10.10.0.1, domain-name-servers 10.10.0.1
Router unaffected: WAN 172.30.1.2, LAN 10.10.0.1, default route 172.30.1.1, 8 LAN pass rules in
pf, egress to 1.1.1.1 at 0.0% packet loss.
Payload gotcha, measured -- do NOT infer this: a GET response is not directly re-POSTable.
OPNsense GET returns select fields as {value: {selected: 0|1}}; SET expects the comma-joined
selected values, so the GET must be flattened first. And kea/service/reconfigure is the apply
verb -- without it the config saves but the daemon never reloads.
CONSEQUENCE: the config.xml push path is now RETIRED IN PRACTICE for Office1. DHCP is
API-managed, so pushing a full rendered config.xml WOULD CLOBBER IT. Do not run the old
scp/install/reboot path against this edge. Reducing the template to a minimal bootstrap
(sshd + root key + console) is the next step under D-113(a2) and is NOT yet done.
Revert: re-POST the previous values (unchanged, so nothing to undo), or edit in the GUI at
Services > Kea DHCP > DHCPv4 > Subnets. Pre-API config.xml backups remain in /conf/backup/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) PROVEN: the OPNsense REST API round-trips the edge config
...
The API path adopted in D-113 is now demonstrated working against the LIVE Office1 edge,
closing the question that gated the ruling.
Measured:
GET core/firmware/status -> HTTP 200, OPNsense 26.1 (auth works)
GET kea/dhcpv4/get -> dhcpv4.general.enabled=1, interfaces=lan
POST kea/dhcpv4/search_subnet -> 1 row: subnet 10.10.0.0/24, pools 10.10.0.100-10.10.0.199,
option_data.routers 10.10.0.1, dns 10.10.0.1,
domain office1.vr1.cloud.neumatrix.local, descr "LAN DHCP"
The API reads back EXACTLY what we pushed via config.xml yesterday, and exposes it as TYPED,
NAMED FIELDS -- not free-form XML. That is the (a2) argument made concrete: DOCFIX-191 (missing
sshd), DOCFIX-192 (silenced console) and DOCFIX-193 (an inert ISC <dhcpd> block against a Kea
backend) are NOT EXPRESSIBLE here. You cannot omit a typed field that has a default, and you
cannot address the wrong DHCP backend when the endpoint IS the backend.
GUI gotcha worth recording (26.1): API keys are NOT created in the user-edit dialog, and NOT in
the ApiKeys tab -- that tab wires only search_api_key + del_api_key, so it can list and delete
but CANNOT create. Creation is a ROW ACTION on the Users list: the fa-ticket icon, "Create and
download API key for this user" (source: mvc/app/views/OPNsense/Auth/user.volt). Any instruction
saying "edit user -> API keys -> +" describes the OLD UI and is wrong for 26.1. ApiKeyField::add()
mints key+secret as base64(random_bytes(60)) and stores key|crypt(secret,'$6$') -- the secret is
shown EXACTLY ONCE.
API gotchas: response root key is `dhcpv4`, NOT `dhcp4` (the config XML element name differs from
the API key name -- do not infer one from the other). camelCase actions map to snake_case URLs:
searchSubnetAction -> POST /api/kea/dhcpv4/search_subnet.
SEC-007 amended: ~/vr1-office1-creds/ now also holds the OPNsense API key/secret (0600).
NOT DONE: no WRITE through the API yet -- that is a live mutation and stays gated. And the
config.xml template is still in place: DO NOT push a full rendered config.xml now, it would
clobber API-managed state. Next step is reducing the template to a MINIMAL bootstrap (sshd +
root key + console) and moving DHCP/firewall/interfaces to the API.
Revert: read-only against the edge + docs; nothing to undo. To revoke the credential: GUI ->
System > Access > Users > root row > ApiKeys tab -> delete, then rm ~/vr1-office1-creds/opnsense-api.txt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113(a2) step 1: thin OPNsense REST API client + harness
...
First delivery under the D-113 ruling (stay on OPNsense, move config off hand-authored
config.xml and onto the documented REST API).
scripts/opnsense-api.sh -- [--dry-run] <GET|POST> <api-path> [json-body]
tests/opnsense-api/run-tests.sh -- offline harness, 21 PASS, no network, no real key
Gauntlet ALL GREEN (53 harnesses, was 52). repo-lint 0 fail.
Design decisions that the harness now enforces:
- THE SECRET NEVER REACHES ARGV. Credentials are read from a file and handed to curl via
--config on STDIN. `curl -u key:secret` would put the secret in argv, where any user on the
box can read it out of `ps`. Harness T9 stubs curl with an argv recorder and asserts the
secret is absent from argv and present on stdin -- so a future "simplification" to `curl -u`
turns the harness red. That case is the point of the file.
- The API host is NEVER inferred (hard rule 2): $OPNSENSE_API_HOST unset is a loud failure, not
a silent default of 10.10.0.1 (T3).
- --insecure is deliberate and scoped: the edge's self-signed cert is REGENERATED ON EVERY BOOT
(measured -- "Created web GUI TLS certificate" appears in the config revision trail after each
reboot), so pinning it is pointless. Acceptable ONLY on the private lab LAN leg; never to be
copied to a tenant-facing surface.
- Dry-run reads no credentials (T11), which is what lets the harness prove URL construction
offline.
NOT RUN against the edge yet, and blocked on one thing: the API is alive (lighttpd on 443
answers 401) but root has 0 API keys. Minting one is a GUI action (System > Access > Users >
root > API keys), deliberately NOT automated -- writing apikeys into config.xml by hand is the
exact anti-pattern D-113 just retired. Key lands in ~/vr1-office1-creds/opnsense-api.txt,
operator-only, never read into agent context (SEC-007 covers that directory).
Proof-of-path once the key exists: drive yesterday's DHCP subnet through the API and confirm it
round-trips. If it does, the template retires to a minimal bootstrap config (sshd + root key +
console + apikey) and DHCP/firewall/interfaces move to the API. If it cannot express it, that is
the signal to revisit D-113.
Revert: git rm scripts/opnsense-api.sh && git rm -r tests/opnsense-api/ (nothing live touched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113 ADOPTED: option (a2) -- stay on OPNsense, move config to the REST API
...
Operator ruling 2026-07-13. Two constraints surfaced during the discussion narrowed the field
decisively, and both are recorded so the reasoning is not re-run from the wrong premise:
1. GUI ACCESS IS A REQUIREMENT (operator). This eliminates VyOS and plain-Linux outright --
neither has a GUI -- and both had been the leading candidates until it was stated.
2. VyOS LTS binaries are subscription-gated (verified against vyos.net/get and the Software
Access Subscription page). Free VyOS means the ROLLING release, a moving target that fights
this repo's appendix-B version-pinning discipline. Real friction, independent of the GUI.
Field reduced to OPNsense / pfSense CE / OpenWrt / IPFire. pfSense rejected (same GUI-owned XML
lineage -- a migration that buys nothing). OpenWrt was the credible alternative: its UCI config
is text-first BY DESIGN, structurally unlike OPNsense's GUI-owned config.xml. It loses on cost:
a new image pipeline, a fresh bootstrap problem, and dnsmasq instead of Kea -- all to escape a
problem that is fixable in place.
The ruling rests on this: the bug class was never OPNsense. It was hand-authoring config.xml
when OPNsense ships a documented REST API with mature Ansible collections that model exactly
what we got wrong (dhcp_subnet, firewall rules, system settings as typed resources). NONE of
DOCFIX-191/192/193 is expressible through the API -- you cannot forget to enable sshd in a
format where sshd is a typed field with a default. (a2) keeps the GUI, keeps a working
routing+DHCP edge, keeps the libvirt fixes, and removes the defect source.
(a1) config.xml templating is now explicitly REJECTED: its cost compounds per feature and it
depends on an undocumented self-heal of an internal format (the 667-element finding).
Live-edge recon (read-only): lighttpd listening on 443/80, API answers 401 (alive), root has
0 API keys. Minting a key via the GUI is the bootstrap step -- hand-editing apikeys into
config.xml would be the exact anti-pattern being retired.
Revert: git revert this commit (docs only; nothing migrated, nothing built).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
D-113 (PROPOSED): is OPNsense the right edge platform? -- and a confound in D-112
...
Operator asked, after a five-bug day, whether something is better suited than OPNsense for
the simulated edges. Written up with alternatives; NOT RULED.
Honest cost accounting: of the five bugs, TWO (DOCFIX-188 memory_unit, DOCFIX-190 ACPI) were
libvirt module bugs that would have hit any guest and whose fixes now protect every VR1 VM --
not OPNsense's fault. The other three (DOCFIX-191 no sshd/key, DOCFIX-192 no console,
DOCFIX-193 no DHCP) share ONE root cause: hand-authoring the appliance's internal, GUI-owned
config.xml. 2026-07-13 adds a fourth of the same kind -- a full-config push drops ~667
migration-populated elements including the only two firewall pass rules, and works ONLY
because OPNsense regenerates them on boot. We depend on an undocumented self-heal of an
internal format. That, not the boot bugs, is the ongoing cost.
THE CONFOUND, and the reason this is worth ruling rather than waving through: D-112 adopted
option (c) with the explicit rationale "C since it is the way people automate opnsense" --
i.e. the REST API (opn-cli, the Ansible collection). What we actually BUILT is config.xml
templating delivered over scp. That satisfies (c)'s letter (provision over the network, post
boot) but NOT the rationale it was ruled on. Every OPNsense-specific bug we are charging
against the platform is an artifact of hand-authoring XML -- precisely what the API exists to
avoid. So the fair comparison is not OPNsense vs VyOS; it is:
(a1) OPNsense + config.xml templating -- status quo; cost COMPOUNDS per feature
(a2) OPNsense + REST API -- what D-112's rationale actually called for
(b) VyOS -- blank-sheet best; discards a working edge
(c) plain Linux (nftables/Kea/FRR + cloud-init) -- max fidelity to our own tooling
(d) pfSense -- REJECT, same config-format lineage
Recommendation (operator rules): stay on OPNsense, and the real choice is a1 vs a2 -- I lean
a2. The hard part is done and measured working, and edges 2/3 reuse the module, so the cost
paid was one-time. But (a1) compounds, and (a2) is the honest reading of D-112's own ruling.
Rule BEFORE Stage 3: it builds two more edges from this module, so ruling later means
migrating three edges instead of one.
Revert: git revert this commit (docs only; nothing built or migrated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
SEC-005: store a GitBucket credential on the jumphost so pushes stop failing
...
The jumphost clone could not push (could not read Username for https://git.baldurkeep.com),
which left this branch 12 commits ahead of origin for two days -- all the D-112 / OPNsense
edge work existed only on this box. Now fixed: credential.helper store + a dedicated PAT at
~/.git-credentials (mode 600), written by the operator via a hidden prompt. The token was
never read into agent context; the credential is verified by USING it, never by printing it.
Two GitBucket behaviours measured and recorded, because both cost a cycle:
1. The git username is NOT the web-UI login. The operator signs in with an email; that is
rejected for git transport. Measured against info/refs?service=git-receive-pack:
jesse.austin@neumatrix.com -> 401, JANeumatrix (the git user.name) -> 401,
jesse.austin -> 200 CAN PUSH. Note GitBucket 401s PAT basic-auth on /api/v3/ across the
board while accepting it for git -- so validating a credential against the API gives a
FALSE NEGATIVE. Validate against the receive-pack endpoint.
2. credential.helper store DELETES the stored credential when the server rejects it (git
calls credential reject on a 401). The first bad-username push therefore emptied
~/.git-credentials to 0 bytes, and the follow-up diagnostics were authenticating with an
empty string -- which read as a corrupt file and misdirected the investigation. If the
file mysteriously empties: the credential was rejected, it did not fail to write.
Tooling consequence: validate BEFORE writing.
SEC-005 is OPEN by operator ruling: a long-lived, account-wide PAT now sits in plaintext on
the jumphost (GitBucket has no per-repo token scoping, so the control is rotation, not
scope). Revoke/rotate at v1 close, or immediately if the jumphost is shared or rebuilt.
Revert: git config --global --unset credential.helper && rm -f ~/.git-credentials,
then revoke the token in GitBucket -> Account Settings -> Applications.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
DOCFIX-193 APPLIED: the Office1 edge now serves DHCP (kea-dhcp4 on udp/67)
...
Executed the live apply that DOCFIX-193 deliberately left pending. Measured end state
on root@10.10.0.1 after reboot: kea-dhcp4 running, bound udp4 10.10.0.1:67, subnet
10.10.0.0/24, pool .100-.199, routers/DNS 10.10.0.1. Router unaffected -- WAN 172.30.1.2,
LAN 10.10.0.1, default route 172.30.1.1, NAT automatic, egress 1.1.1.1 at 0.0% loss,
serial console + getty alive (DOCFIX-192 holds).
Path: render -> scp -> SHA-256 verify on guest (matched) -> install /conf/config.xml ->
reboot. Edge back in ~30s.
The finding that nearly bit us: a full-config push drops ~667 migration-populated
elements, including the ONLY 2 pass rules on the box ("Default allow LAN to any" + IPv6 --
legacy /filter/rule count is 0 in 26.1) against a pf base policy of block-drop-all.
Dropping them = an edge that boots but does not route. Settled NOT by inference but from
the box's own /conf/backup/ trail: three prior pushes of a rule-less config each came back
with both rules regenerated by run_migrations. Post-apply re-confirms it -- 791 elements,
both LAN rules back, webgui TLS cert back, and our Kea subnet4 survived the migration.
The overwrite is self-healing; do not hand-merge to preserve them.
Defect logged, not fixed here: root's shell on OPNsense is tcsh, so the pre-install
snapshot step (a $(...) inside a quoted remote command) died with "Illegal variable name"
and silently no-op'd -- the install ran without its rollback point. Config was recoverable
via /conf/config.xml.prev + /conf/backup/, so no harm, but the guard was absent when
relied upon. Remote commands must be fed to sh -s. The apply path wants to become a
harnessed scripts/opnsense-apply-config.sh; LOGGED as an open gap, out of scope here.
Not verified end-to-end: no client has taken a lease (office1-local has no host yet).
Revert: ssh root@10.10.0.1 'sh -s' <<< 'cp /conf/config.xml.prev /conf/config.xml && reboot'
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6Fzre1CxCY8tzwFCudu57
|
| 2026-07-12 |
D-112(c) EXECUTED: the Office1 OPNsense edge is UP and ROUTING
...
End state, measured over SSH: office1-opnsense, OPNsense 26.1, WAN 172.30.1.2/24 static,
LAN 10.10.0.1/24, default route via 172.30.1.1, NAT active, and egress to the internet
WORKS through the simulated ISP (ping 1.1.1.1, 0% loss). Serial console restored
(kern.console=ttyu0, getty running). Managed over SSH with the service key.
Bootstrap: console (pexpect + virsh console) -> factory login -> config streamed as
base64, SHA-256 verified byte-exact, factory config backed up -> reboot. The DOCFIX-192
fix was then pushed OVER SSH -- the D-112(c) steady-state path proving itself.
Gotchas recorded in the ledger because each cost real time:
- Console settings need TWO reboots. OPNsense regenerates /boot.config and
/boot/loader.conf FROM config.xml *during* boot, so they only take effect on the NEXT
boot. After reboot #1 the loader files were already correct while the running kernel
was still on video. Do not conclude "the fix failed" after one reboot.
- /etc/ttys uses `onifconsole`: getty starts on ttyu0 only if ttyu0 IS the kernel
console. `sysctl kern.console` is ground truth.
- The guest's root shell is CSH (both via console and ssh). It rejects $(...) with
"Illegal variable name." Pipe POSIX scripts to `sh -s`.
- `virsh console` REFUSES a unix-socket chardev. The serial must be a pty + log file.
LEDGER CORRECTION: the 2026-07-12 entry claimed the rendered config provided
"DHCP .100-.199". That is FALSE -- there is NO DHCP configuration in the template at all
(no <dhcpd>, no <Kea>), and nothing serves DHCP on office1-local (measured: no
kea/dhcpd/dnsmasq process, nothing on udp/67). Another optimistic claim from the
max-context session. The operator's goal was "router + DHCP": the ROUTER half is DONE,
the DHCP half was never built. OPNsense 26.1 uses Kea (ISC dhcpd is gone), so this needs
the Kea schema verified upstream -- do not invent it.
repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
D-112 ADOPTED: option (c) post-boot network provisioning (operator ruling)
...
Operator ruled (c): "C since it is the way people automate opnsense" -- prefer the
mainstream automation path (the OPNsense REST API, as used by opn-cli and the
Ansible OPNsense collection) over image-baking or a media-import trick. The
rendered config (opnsense-config.xml.tmpl) is REUSED unchanged; only the DELIVERY
mechanism changes.
IMPLEMENTATION NOT STARTED -- deliberately. "Push the config over the API" is not
actually reachable from a factory-default OPNsense, and the bootstrap sub-choices
are unruled. Recorded in D-112 so the next session does not discover this the hard
way:
1. NO ROUTE. Factory-default LAN is 192.168.1.1/24 on office1-local, an ISOLATED
libvirt network with no host leg on virbr2 (measured). The WAN side is reachable
at 172.30.1.126 but OPNsense blocks all inbound on WAN by default, so it is not
a way in.
2. NO CREDENTIALED API. The REST API authenticates with an API key+secret (not the
root password), minted via the GUI. SSH is disabled in the factory default.
3. OUR SERIAL IS WRITE-ONLY. The module attaches <serial type='file'> -- a one-way
capture that cannot be typed into. An interactive console means type='pty' +
virsh console (module change + recreate).
Sub-choices B1 (how to reach it) and B2 (how to authenticate first) are left
UNRULED in D-112 rather than picked.
SELF-CORRECTION recorded: I had earlier described option (b) as making the ISO
mechanism "work as designed, unchanged". That was over-claimed -- even with a
read-only root the importer prompts with a 7s timeout and asks for a device name,
so (b) may have been (d) in disguise. Logged so nobody revives (b) on my bad
summary. Note a console LOGIN SHELL is not the rejected (d): (d) was rejected for
racing the importer's 7s timeout; a login shell has no timing race.
The config-ISO path (opnsense-build-config-iso.sh + harness, the module's
config_seed volume + cdrom disk) is RETIRED but left in place -- it is inert, not
harmful. Remove it in the same change that lands (c), not before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
D-112 (PROPOSED): OPNsense Config Importer cannot fire on a nano image -- by design
...
Bug 3 of the Office1 edge boot. With memory (DOCFIX-188) and ACPI (DOCFIX-190)
fixed, OPNsense boots -- but on FACTORY DEFAULTS. The config ISO was never read.
Root cause is architectural, not a bug. rc.syshook.d/import/20-importer runs
`opnsense-importer -b`, whose first act is:
INSTALL="/.probe.for.readonly"
touch ${INSTALL} 2> /dev/null
if [ -f ${INSTALL} -a -f /conf/config.xml ]; then
bootstrap_and_exit 0
fi
That touch is a PROBE FOR A READ-ONLY ROOT. On installer media root is read-only,
the touch fails, the marker is absent, and the importer scans attached media
(cd9660 included). On a pre-installed nano the root is WRITABLE and a factory
/conf/config.xml already exists -- so both conditions hold and it exits
immediately, never enumerating a single device.
Our ISO was correct all along: verified ISO9660 'OPNSENSE_CFG' containing
CONF/CONFIG.XML with our 10.10.0.1, and the guest sees it as cd0 at exactly the
right byte size. Nothing was ever going to read it.
The README's research correctly established that the Importer's scan path supports
ISO9660 -- but never that the Importer RUNS on a nano image. It does not. The
module header has always flagged this mechanism UNVERIFIED; that flag is now
discharged NEGATIVE.
Recorded as D-112 PROPOSED with four options (bake config into the image / switch
to the installer image / post-boot API push / serial expect-scripting). NOT
implemented -- this replaces the config-delivery mechanism of a built surface and
also gates Stage 3, so it is the operator's ruling to make.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-190: second boot bug -- libvirt domains had ACPI disabled (FreeBSD kernel panic)
...
With DOCFIX-188's memory fix in place the guest finally got its 2 GiB, the kernel
reached interrupt initialisation, and hit a SECOND defect the first bug had masked:
panic: running without device atpic requires a local APIC
apic_init() ... mi_startup() ... db>
None of the three VM modules emitted a `features` block, so libvirt rendered
`-machine pc-i440fx-noble,...,acpi=off`. FreeBSD gets its local APIC from ACPI's
MADT table and the OPNsense kernel has no atpic fallback, so with ACPI off it has
no usable interrupt controller. The guest parks in the ddb debugger -- which is
the 100%-of-one-core spin, while virsh still reports the domain as "running".
Fix: features = { acpi = true, apic = {} } in opnsense-edge, cloudinit-vm and
node-vm. node-vm matters most long term: MAAS drives power via ACPI signalling,
so without it graceful shutdown / MAAS power-off would never have worked.
Guard S2 added. Like memory_unit, `tofu validate` cannot catch this -- the whole
features block is optional, so its absence is schema-valid. Flag renamed
--static-only (--check-memory-unit kept as a back-compat alias). Harness 4 -> 6
PASS; fixtures/s2-bad PASSES S1 and still FAILS S2, proving the guards catch
independent classes rather than one masking the other.
LIVE RESULT (measured): acpi=on; kernel boots to userland; CPU idle, no spin;
DHCP lease 172.30.1.126 hostname "OPNsense" on office1-wan. Both boot bugs closed.
STILL OPEN: the Configuration Importer did not apply -- OPNsense came up on
factory defaults. The ISO is verified well-formed (ISO9660 OPNSENSE_CFG containing
CONF/CONFIG.XML with our 10.10.0.1). This is the mechanism the module header has
always flagged UNVERIFIED. Do not guess the fix; read the Importer's console
output. The unanswered WAN ping is NOT a fault -- OPNsense blocks inbound on WAN.
Gauntlet-relevant suites green; repo-lint 0 fail; tofu fmt/validate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-189: re-audit the max-context session's troubleshooting + docs
...
Operator-instructed re-evaluation. The 2026-07-12 session authored its docs and
module comments WHILE WORKING FROM A WRONG DIAGNOSIS, at a context limit, so its
reasoning was re-checked rather than just its one-line bug (DOCFIX-188).
Headline: the settings it changed are mostly fine and are RETAINED. The reasoning
it recorded was not -- it attributed the fault to whatever knob was being turned,
on no evidence, four separate times. It also left one silently-dead knob.
FALSE CAUSAL CLAIMS (corrected at the source, opnsense-edge/main.tf). The disk
shape, q35->i440fx, the serial console and svm-disable were EACH written up as a
contributing cause. None changed the symptom -- the fault sat at a deterministic
262 bytes through all of them. All four settings RETAINED (each defensible on its
own merits); only the false justifications are gone. The serial console earned its
keep, but it REVEALED the fault, it did not change it.
DEAD KNOB (real functional defect). `disk_size_bytes = 16 GiB` did nothing once
DOCFIX-187 made the disk a direct copy; unused variables are legal HCL so neither
validate nor plan complained. MEASURED: the live disk is 11.00 GiB, not 16.
Removed from main.tf + module variables; real sizing is opnsense-prep-image.sh's
GROW (+8G). Proof it was dead: tofu plan is byte-identical before/after removal.
STALE/UNSOURCED FACTS. "Opteron_G3" is a red herring -- the host is really an AMD
EPYC 9965 (Zen 5, family 26); libvirt's CPU-model DB doesn't know family 26 and
falls back to the oldest name, and the prior session partly built its nested-virt
theory on that artifact. "OPNsense's 3 GB min" is unsourced -- annotated, do not
propagate. DOCFIX-186's backing_store parenthetical is stale (same conclusion,
different mechanism). DOCFIX-187's changelog gains a SUPERSEDED-IN-PART header.
KEPT: DOCFIX-186's apparmor + config-iso-staging findings are real and measured.
DOCFIX-185 (operator transport ruling) untouched.
STANDING LESSON: when a change does not fix the symptom, do not write it up as
though it did. A fault that is deterministic and immune to every knob you turn
means you have not yet touched the cause.
Gauntlet ALL GREEN (52); repo-lint 0 fail; tofu fmt/validate/plan clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-188 followup: correct the pass criteria (anchor on virsh dominfo, not the QEMU -m string)
...
libvirt normalizes MiB to KiB in the QEMU cmdline, so '-m size=2097152k' would be
SUCCESS, not failure -- the original criterion would have read a correct boot as a
failed one. Anchor on 'virsh dominfo' Max memory, which is unambiguous. Also: re-run
plan AFTER the undefine (the captured in-place plan is not what executes), and
pre-stage the two fallbacks (provider ignores memory_unit -> *1024; boots but no
router -> sizing, not a regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-188: root-cause the OPNsense boot triple-fault -- libvirt memory_unit (guest had 2 MiB)
...
The 2026-07-12 triple-fault was never a CPU/nesting/machine/console problem.
dmacvicar/libvirt >=0.9 reinterprets `memory` as raw libvirt units (KiB), not
MiB (the 0.8-era meaning the modules were authored against). With no
`memory_unit`, `memory = 2048` rendered `<memory unit='KiB'>2048</memory>` ->
QEMU `-m size=2048k` = 2 MiB. boot2 fits in 2 MiB and echoes /boot.config;
/boot/loader does not -- hence the deterministic 262-byte fault, immune to every
knob the prior window turned.
Measured every hop: module input -> tofu state -> domain XML -> virsh dominfo
(Max memory: 2048 KiB) -> live QEMU cmdline.
FOUNDATIONAL: the identical defect was latent in all three VM modules --
opnsense-edge (live blocker), cloudinit-vm (MAAS/NetBox/GitBucket, the next
step), node-vm (Stage 3). Every VR1 VM would have gotten 1024x too little RAM.
- memory_unit = "MiB" on all three libvirt_domain resources (memory_mib = 2048
value untouched -- it now means the 2 GiB always intended).
- Corrected the disproven "svm is ROOT CAUSE" comment in opnsense-edge; the
setting is retained as legitimate hardening, the false claim is not.
- Guard: opentofu-validate.sh S1 + --check-memory-unit flag. `tofu validate`
cannot catch this (memory_unit is optional => omitting it is schema-valid,
which is exactly how it shipped). Harness T3 is a true negative test: the
fixture IS the shipped defect. Harness 1 -> 4 PASS.
- Incident report gains a READ-THIS-FIRST box so nobody re-chases the CPU theories.
Gauntlet ALL GREEN (52 harnesses); repo-lint 0 fail; tofu fmt/validate clean.
Live boot verification remains a separately-gated operator step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y117t1F525ba9r1vXSBETk
|
DOCFIX-187: opnsense-edge module boot fixes + boot-triple-fault incident report
...
Office1 OPNsense edge first-boot debugging. Several correct module fixes made;
the boot is NOT yet resolved (BTX-loader triple-fault under double-nested virt),
documented as an OPEN incident for the next session.
Module (opentofu/modules/opnsense-edge/main.tf), all kept (a working config
needs them, none resolved the fault):
- serial console (nano is serial-only; the module had none) + boot-log capture
- machine q35 -> i440fx (pc)
- disk COW-overlay -> direct per-VM copy (matches virt-install --import)
- CPU host-passthrough with AMD svm disabled (documented AMD nested-virt fix;
note the provider key is `features` plural, not `feature`)
Incident report docs/incident-20260712-opnsense-edge-boot-triplefault.md: full
symptom (boot2 echoes /boot.config then triple-faults loading /boot/loader, a
consistent 262-byte fault confirmed fresh each boot), environment (double-nested
AMD Opteron_G3), everything tried, and ranked next steps (full CPU flags
+kvm_pv_eoi/+kvm_pv_unhalt, add a video device for the -D dual console, mem 4G,
UEFI/OVMF, outer-hypervisor CPU). Ledger checkpoint added; session at context
limit.
repo-lint 0 fail; module tofu validate Success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Office1 OPNsense edge build: real-ISP-router config + first-boot fixes
...
Bringing the Office1 OPNsense edge online (router + DHCP for office1-local)
before the Office1 NetBox VM. First-ever OPNsense boot in this repo; several
first-execution findings fixed.
DOCFIX-184: opnsense-prep-image.sh decompresses the .img.bz2 via python3's
stdlib bz2 when bunzip2 is absent (no sudo install needed). Harness 3/3.
DOCFIX-185: the OPNsense edge is a normal simulated-ISP router, not an
egress-airgap. Operator clarified the transport model -- each site (DCs +
Office) has its own real ISP connection; dark fiber is East-West/replication
only; D-107 node-airgap is a separate DC concern. Stripped the WAN
egress-control rules (seq 20/21/99) + MIRROR_* tokens from the config template,
render script, and harness (8/8). Flagged a D-100/D-107 amendment note as
follow-up.
DOCFIX-186: instantiate module "office1_opnsense" in main.tf (first real
validation of modules/opnsense-edge). Two infra findings: (1) config_iso_path
must be OUTSIDE the pool dir (create.content.url collides with its own target
volume) -> stage the ISO; (2) apparmor blocks the custom /var/lib/libvirt/vr1/
pool path (foundational -- all VR1 VMs) -> needs a one-time local apparmor rule
(operator sudo). office1-wan NAT network created via virsh (D-103 debt).
repo-lint 0 fail; opnsense-prep-image 3/3; opnsense-render-config 8/8.
Boot blocked only on the apparmor fix (pending operator sudo).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-11 |
D-111: align VR1 v6 subcarve to deployed NN mnemonic; fix ULA-gen doc bug
...
Basing VR1 NetBox on the real netbox.baldurkeep.com modeling (operator
directive). A read-only live export confirmed VR0-DC0/Willamette use an NN
net-byte mnemonic with /60-per-plane + /64-active; dc-dc-prefixes-import.py
did not match it (arbitrary contiguous indices, /64-only).
DOCFIX-182 / D-111: carve_v6 replaces carve_ula/carve_gua -- provider :10
(+API-VIP :11) GUA, metal /60 shared by admin :20 / internal :21, data :30,
storage :40, repl :50, all /60+/64 ULA; the ULA /56 is indexed to the GUA site
nibble so the 4th hextet reads DC.NN in both families. Direct-math _sub_at (no
subnet enumeration -- preserves the DOCFIX-181 no-hang property). Harness
40 -> 53 PASS. D-111 ADOPTED in design-decisions.md; scope-doc sub-decision #1
ratified.
DOCFIX-183: fix the invalid-IPv6 ULA-generation sed in the netem/ULA proposal
doc (4+6 -> 2+4+4 hextets, caught by actually running it); ratified VR1 ULA is
fd50:840e:74e2::/48.
repo-lint 0 fail; tests/dc-dc-prefixes-import 53/53.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
| 2026-07-10 |
DOCFIX-181: fix carve_gua subnet-enumeration hang (gauntlet was stalling)
...
netbox/dc-dc-prefixes-import.py::carve_gua() list()-ed every /64 subnet of the
GUA prefix (2**24 = ~16.7M for D-101's /40 example) only to use the first ->
hung/OOM. Real import runs would hang identically, not just the test. Fixed to
take the first /64 lazily via next() (O(1)).
Added a faulthandler watchdog to tests/dc-dc-prefixes-import/test_logic.py so a
future blocking regression self-aborts with a traceback after 30s instead of
silently stalling the whole run-tests-all.sh gauntlet -- the false-green that
let the ledger claim "40/40" for a test that never actually completed.
tests/dc-dc-prefixes-import: 40/40 in 0.05s (was: hung indefinitely).
Full gauntlet: ALL GREEN (52 harnesses). repo-lint 0 fail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DC-DC Stage 1 (dc-dc-phase0): first real execution + scaffold/tooling fixes
...
First end-to-end run of a DC-DC runbook against real infrastructure (vcloud host,
OpenTofu v1.12.3). Applied 13 libvirt objects (6 DC1 planes, dc1/office1/dc2
pools, office1-local, 3 mesh legs, all MTU 9000); prior VR0 nets torn down (wan
kept as a gap-#17 model); DC2 planes deferred (NetBox supernet), DC2 storage wired.
DOCFIX-178: scripts/prereqs/ idempotent workstation prereq installers + a
Prerequisites section in the runbook + tests/prereqs/ (25/25). Runbooks must
not assume prereq runtimes.
DOCFIX-179: OpenTofu scaffold first validation + apply. Per-module
required_providers (child modules do NOT inherit provider SOURCE -> OpenTofu
inferred nonexistent hashicorp/libvirt); tofu fmt -recursive; provider "maas"
deferred to Stage 3 (kept a sensitive key out of Stage-1 plan/state);
repo-lint now skips the .terraform/ provider cache; .terraform.lock.hcl added.
DOCFIX-180: dc-dc-phase0 runbook as-executed corrections folded in at phase
close (two MTU domains, non-pristine-host handling, Step-9 13-resource plan,
-input=false, Known-Gap resolved).
repo-lint 0 fail; prereqs harness 25/25; repo-lint harness 34/34.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
DOCFIX-177: make the skill and runbooks/README.md aware the VR1 DC-DC track exists
...
Found while the operator was heading to the vcloud host to begin real
execution: SKILL.md and runbooks/README.md (what its routing table points
to) had zero mentions of the entire VR1 DC-DC track, so a fresh session
would be routed to VR0's phase-01..08 runbooks with no indication today's
actual mission (dc-dc-phase0..6 + the new teardown runbook) exists.
Added a "VR1 DC-DC track" index to runbooks/README.md, VR1-specific rows
to SKILL.md's routing table and standard loops, and pointers in
references/environment.md and references/troubleshooting.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-175/176: OpenTofu state file security + VR1 teardown/rollback runbook
...
Found while reviewing what's missing from the VR1 buildout ahead of
tomorrow's execution.
DOCFIX-175: terraform.tfstate stores sensitive=true variables (like
maas_api_key) in plaintext regardless of the flag, which only redacts
CLI/plan output (verified against HashiCorp's own docs). Added .gitignore
coverage, a "State file handling" section in opentofu/README.md, an inline
caveat on maas_api_key's description, and a "SECURE THE STATE FILE" callout
at Stage 1 Step 10 -- the actual first real tofu apply (an early draft
wrongly assumed Stage 3; corrected after reading the runbooks directly).
DOCFIX-176: new runbooks/dc-dc-teardown-rollback.md for the OpenTofu/
libvirt/MAAS-vm_host layer, which D-061's existing teardown scripts don't
cover (different layer, juju/MAAS-machine). Two paths (scoped -target
teardown or full VR1 destroy), a rollback decision tree favoring
fix-forward over destroy, and explicit mesh-link shared-infrastructure
handling.
Also folded in: a fix to docs/changelog-20260710-ledger-scan-self-inflicted-falsepositive-fix.md
after re-running ledger-scan.sh caught that its own narration (and the
session-ledger mirror) had reintroduced the exact false-positive class
DOCFIX-174 had just fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-174: fix self-inflicted false positive in ledger-scan.sh next-free scan
...
DOCFIX-173's own changelog narrated its {3}->{3,} regex fix using a literal
illustrative example ("DOCFIX-1004"), which the widened regex then matched
as a real assignment -- inflating DOCFIX next-free from 174 to 1005.
Reworded the illustrative examples in prose, added a reproduc-line
exclusion to nextfree_mentions() as defense-in-depth, and added a
regression fixture to the test harness (37/37 PASS).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-170: full-project sweep -- VR0 runbook + OpenTofu doc fixes
...
Part of an 8-dimension parallel sweep for errors/architectural problems
across the whole repo, requested after DOCFIX-169. phase-08-workload-
cluster-acceptance.md had the same cascading-staleness class as DOCFIX-169
(Designate framing never updated after DOCFIX-167's bundle.yaml edit) --
fixed with a nuanced correction distinguishing "charms now active/idle"
from "D-011.8's real acceptance criterion still needs Stage 7's overlay."
Also fixed a stale opentofu/modules/base-image/outputs.tf comment
referencing nonexistent cloudinit-vm variables. Session ledger updated
with the full 8-dimension sweep summary and two orphaned DOCFIX-number
closures (164, 167) caught as integration misses.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-169: fix stale bundle.yaml expectations in phase-01-bundle-deploy.md
...
DOCFIX-167 (Designate + D-108 DR mechanism reactivation) edited the same
shared bundle.yaml this existing, live-cloud-relevant VR0 runbook deploys,
but didn't update this runbook's own documented expectations. Found during
a full-project sweep: "expected plan: 50 apps, 97 relations" was stale
(real count 56/108); "NO designate (D-019)" GATE text was flatly wrong.
Fixed both live-gate references; left the historical 2026-06-03 as-built
record unchanged (append-only audit trail). Flagged, rather than decided,
a real architectural tension: this bundle now serves both a plain VR0
redeploy and DC1's buildout, so any future phase-01 run gets the new
charms regardless of intent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-168: proposed netem parameters + ULA/GUA generation guidance (gaps #4(d)+11); tracker sync
...
docs/dc-dc-netem-and-ula-gua-proposal.md: a concrete, reasoned netem
parameter proposal (1ms/0.2ms jitter/0.01% loss/uncapped rate) for operator
ratification, plus the exact safe command (openssl rand -hex 5, RFC
4193-compliant) for the operator to generate the real org ULA /48
themselves. Deliberately does not generate the GUA carve or DC2's supernet
-- both need real ARIN/NetBox coordination this session has no authority
or visibility to perform.
Also finalizes docs/dc-dc-deployment-workflow.md and docs/session-ledger.md
across all seven gap-closure deliveries tonight (DOCFIX-162 through 168):
tooling gap register items #4(d), #5, #6, #7, #11, #12, #13, #14, #15, #16
updated to their real closed/partial state, new gap #17 added, harness-debt
item #10 corrected. Full test gauntlet re-run clean: same pre-existing
environment-gap failures as every earlier check tonight (jq missing on
this workstation), zero regressions across all seven new script families.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-161: final consolidation pass -- one more gap found, stale refs fixed
...
Last delivery of tonight's autonomous session. A final full read-through
of the deployment workflow tracker for cross-stage consistency found one
more real gap: no stage explicitly owns creating Office1's own OPNsense
edge VM, despite the topology being explicit that the design is per-site
(three sites, three edges). Logged as gap #16. Also fixed two stale
cross-references left over from earlier in the evening. Companion
Artifact redeployed with all seven runbooks reflected as written, the
gap register current, and the corrected 40/40 test count. Final full
gauntlet re-run: same 23 pre-existing environment-gap failures as every
earlier check tonight, zero regressions across the entire session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-160: adversarial review pass finds and fixes 3 real issues
...
Ran a fresh-eyes review subagent across tonight's full deliverable set
(lib-net.sh/lib-hosts.sh selectors, the NetBox DC-DC pipeline, all seven
Stage 1-7 runbooks). Found and fixed:
- A real Python control-flow bug in tests/dc-dc-prefixes-import/
test_logic.py: a `return` inside `try` made the `else` clause
unreachable, silently leaving 3 happy-path assertions uncounted.
37/37 -> 40/40 after the fix (independently reproduced before trusting
the finding).
- A self-contradictory pair of comments in dc-dc-prefixes-import.py about
/19-vs-/22 subnet arithmetic (no functional bug, wrong prose).
- Stale self-referential text in the Stage 4 runbook, written before
Stage 3 existed and never reconciled once it did.
Corrected the 40/40 count in the workflow tracker and session ledger.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
DOCFIX-157/158/159: Stage 4/6/7 runbooks -- MAAS deploy, DR drill, Designate/COS/Magnum
...
Completes all seven Stage 1-7 DC-DC runbooks (tooling gap register item #9
CLOSED):
- Stage 4 (MAAS enlist/commission/deploy, per DC): adapts phase-00's
validated single-rack sequence per DC. Mandates the $DC selector
convention (DOCFIX-151) and honestly documents that lib_hosts_select_dc
will keep failing until this stage's own enrollment populates real host
data. Names 7 real multi-rack gaps rather than working around them.
- Stage 6 (DR wiring and failover drill): implements Section 8's
failover/failback skeleton, one-way-then-two-way per D-108. Found three
real bundle.yaml gaps (missing cinder-backup/ceph-rbd-mirror charms,
wrong-plane rbd-mirror binding) and makes fixing them the runbook's own
first gated step. Includes a negative-control drill before the real one.
- Stage 7 (Designate, COS, Magnum/CAPI): quotes the bundle's actual
current "NO designate" text before describing the D-106 reversal;
identifies the reversal also affects the bundle's separate
os-public-hostname posture; leaves COS's mechanism as an operator
decision rather than inventing one.
Four new tooling-gap-register items (#12-15) added for cross-cutting
findings this runbook-authoring pass surfaced. None of the seven runbooks
have been executed against real infrastructure yet -- prep-only session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QrcJx8TUar7pYAvpGJw57A
|
