Skip to content

DR drill: apps01 + obs01 rebuild via stable_instance_id (2026-08-09)

Result: success, but unplanned. This didn't start as a drill — it was the real rollout of homelab-platform#3 (stable_instance_id, to stop cloud-init regenerating SSH host keys on every Proxmox reboot). It turned into a live, no-warning DR exercise partway through: a Terraform state gap, a Proxmox permission gap, and a provider ForceNew surprise combined to destroy both apps01 and obs01 mid-apply, with no snapshot and no plan to be operating at this scale. Everything came back, but this wasn't apps01-rebuild.md's deliberate "no rollback net, honestly staged" design — this one wasn't a choice, and both hosts were briefly down for real.

Scope

Turn on stable_instance_id on both existing shared hosts (apps01, obs01), landing homelab-platform#3. Ended up also exercising, for the first time: redeploying a service with its own secrets from scratch (homelab-observability's GRAFANA_ADMIN_PASSWORD, via a BWS_ACCESS_TOKEN CI machine account) — the exact gap apps01-rebuild.md's finding #4 flagged as untested — plus a full migration of Terraform state off a local file and onto HCP Terraform, plus simultaneous recovery of two hosts instead of one.

Target state before this started: apps01 ran homepage + code-server + the host-level alloy-agent; obs01 ran the homelab-observability stack (Grafana, Prometheus, Loki, Tempo, gateway Alloy) plus its own alloy-agent. Blast radius here: both hosts, everything on the shared VM fleet, at once — worse than the prior single-host drill, not by design but by what stable_instance_id needed to touch on hosts that already existed.

Steps run

1. Land the code changes (homelab-platform#3, PR #13)

Provider ssh {} block, module bump to homelab-iac-common@v0.1.13, stable_instance_id = true set fleet-wide in main.tf. Merged clean.

2. First terraform plan, from the wrong checkout — caught before applying

Run from an Orca-managed worktree of homelab-platform, not the checkout that had ever run apply. Terraform state was a gitignored local file (terraform/terraform.tfstate) that only ever existed in the canonical clone — this worktree had none. The plan came back wanting to create apps01/obs01 from scratch (4 to add, 0 to change, 0 to destroy, but for brand-new VMs at already-live VMIDs), which would have collided with the real running hosts. Not applied.

3. Migrate Terraform state to HCP Terraform (PR #14)

Added a literal cloud {} block (org codyecsl-homelab, workspace homelab-platform, execution mode Local — same pattern as tailscale-iac), hcp_terraform_token to secrets.json, TF_TOKEN_app_terraform_io passthrough in docker-compose.yml. From the canonical clone:

source ./scripts/bws-load.sh
docker compose run --rm terraform init

Terraform detected the backend change and prompted to copy existing local state into the new workspace — accepted. A follow-up plan from the canonical clone, now against real migrated state, came back 4 to add, 0 to change, 2 to destroy for apps01/obs01: enabling stable_instance_id on an already-existing VM sets initialization.meta_data_file_id, which is ForceNew in bpg/proxmox — not the single extra reboot homelab-platform#3 assumed, a full destroy-and-recreate. Discussed the options (fix the module upstream first, accept the rebuild, or scope to one host); decided to accept the rebuild on both hosts.

4. terraform apply — both VMs destroyed, recreate failed, both hosts down

module.host["obs01"]...: Destruction complete after 2s
module.host["apps01"]...: Destruction complete after 2s

then, creating the stable_instance_id meta-data snippet on each host:

Error: failed to get datastore: error reading datastore: received an
HTTP 403 response - Reason: Permission check failed (/storage/local,
Datastore.Allocate)

Datastore.Allocate is not datastore-scoped — despite the error naming /storage/local, this was a gap in the API token's role definition, not an ACL entry for that one storage. The token's role (CaddyIac, defined under Datacenter → Permissions → Roles) already granted Datastore.AllocateSpace — the privilege VM disk creation on local-lvm needs — but not the separate Datastore.Allocate privilege, which nothing had ever needed until this snippet write. Both apps01 and obs01 did not exist in Proxmox at this point — an unplanned, live outage of every service on the shared fleet. Fixed by adding Datastore.Allocate to the CaddyIac role itself (Datacenter → Permissions → Roles → CaddyIac → Add), not any path-specific pveum acl change.

5. Second apply attempt — new blocker, still both hosts down

Past the permission error, hit a different one immediately:

Error: failed to open SSH client: unable to authenticate user "root" over
SSH to "192.168.1.10:22" ... attempted methods [none password]

stable_instance_id's meta-data snippet is written to the Proxmox node's filesystem over SSH — content_type = "snippets" has no API upload path in bpg/proxmox — and this exposed three stacked gaps, not one:

  1. The toolbox container's ssh { agent = true } had no agent socket forwarded into it at all, only a static key file (mounted for git module fetches). Fixed in PR #15: bind-mount $SSH_AUTH_SOCK into the terraform service, SSH_AUTH_SOCK=/ssh-agent inside it.
  2. The host shell itself had no ssh-agent running (ssh-add -l → "Could not open a connection to your authentication agent") — needed eval "$(ssh-agent -s)" first.
  3. root@ the Proxmox node had no key-based access authorized at all. Password auth was the only thing that had ever worked there. Fixed with ssh-copy-id root@192.168.1.10 (prompted for and accepted the root password interactively) to actually install a key in root's authorized_keys before any agent-based flow had a chance of working.

6. Clean apply

docker compose run --rm terraform plan     # 4 to add, 0 to change, 0 to destroy
docker compose run --rm terraform apply

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
host_ips = { "apps01" = "192.168.10.101", "obs01" = "192.168.10.102" }

Same IPs as before — derived from vm_id, not discovered.

7. Clear stale host keys, rebaseline

ssh-keygen -R 192.168.10.101
ssh-keygen -R 192.168.10.102
SSH_KEY_PATH=~/.ssh/homelab-ci docker compose run --rm ansible \
  ansible-playbook -i inventory/hosts.yml site.yml

Clean run, failed=0 unreachable=0 on both hosts; alloy_agent correctly scoped to apps01 only. Note: the key baked into cloud-init for the debian user is ~/.ssh/homelab-ci ("homelab-ci deploy"), not docker-compose.yml's default ~/.ssh/id_ed25519 — needed the explicit override.

8. Verify stable_instance_id actually holds

Rebooted both hosts a second time and confirmed the SSH host key did not change on either — the real proof, since the first boot after recreation already used the pinned instance-id from the start.

9. Redeploy every service, via its own CI, on both hosts

gh workflow run deploy.yml --repo CodyECSL/homelab-homepage
gh workflow run deploy.yml --repo CodyECSL/homelab-code-server
gh workflow run deploy.yml --repo CodyECSL/homelab-observability

All three workflow_dispatch runs succeeded unmodified. Verified live:

homepage    :7990 -> 200
code-server :7991 -> 302  (auth redirect, expected)
grafana     :3000 -> 302  (login redirect, expected)
gateway     :4318 -> 404  (bare GET on an OTLP-only path, expected)

homelab-observability's GRAFANA_ADMIN_PASSWORD came back correctly via its CI's BWS_ACCESS_TOKEN — the first time any service's own secrets have been exercised end-to-end through a from-scratch host rebuild.

Findings

  1. Local, per-checkout Terraform state was a live landmine, not a theoretical one. It took an ordinary worktree without the state file to expose it (step 2) — no malice or unusual setup required, just a second place terraform got run from. Fixed by moving to HCP Terraform; this should have been true from this repo's first apply, the same way tailscale-iac was already set up.
  2. stable_instance_id forces a full VM rebuild on existing hosts, not a reboot. homelab-platform#3 assumed the latter; the module's initialization.meta_data_file_id is ForceNew in bpg/proxmox. Confirmed the hard way on both apps01 and obs01. caddy-iac still owes the equivalent fix per that issue and will hit the same thing. Worth fixing upstream in homelab-iac-common — decouple the meta-data attach from the VM resource (e.g. qm set --cicustom over SSH, post-create) so it stops forcing replacement.
  3. A role-permission gap and a three-layer SSH-access gap were both invisible until the exact moment they were needed, and they landed back-to-back, with both hosts already destroyed in between. The API token's CaddyIac role had Datastore.AllocateSpace but not the separate Datastore.Allocate privilege; neither that, nor a forwarded ssh-agent, nor — it turned out — any key-based access to root@ the Proxmox node at all (only ever password auth) had ever been exercised before this change. Nothing about the fleet's steady-state operation would have surfaced any of it ahead of time; the node had simply never needed to be reached this way before.
  4. Recovering two hosts at once, blind, is meaningfully worse than one. apps01-rebuild.md's finding #2 ("no bulk-redeploy primitive") is still true and now demonstrated at 2x — three separate gh workflow run invocations across three repos, by hand, in sequence. At the fleet's stated capacity (10-15 services on apps01 alone), an unplanned two-host outage like this one would mean redeploying well over a dozen services by hand, under time pressure, with no single "what was running here" manifest to work from.
  5. Secrets-through-a-rebuild is now verified, not just designed. homelab-observability's GRAFANA_ADMIN_PASSWORD round-tripped through BWS_ACCESS_TOKEN correctly on a service that had never been deployed fresh from nothing before. Closes the gap apps01-rebuild.md finding #4 explicitly flagged as the next thing worth testing.
  6. The IP-derivation and per-host blast-radius design held again, this time under two simultaneous host rebuilds rather than one — both VMs came back on their original IPs (.101/.102), and nothing about config/hosts.yaml's for_each needed touching to make that true.

Rough timing

Not tracked live with a stopwatch, but reconstructed afterward from two sources that turned out to have second-precision timestamps anyway: the system journal's podman container lifecycle events (every docker compose run --rm terraform|ansible ... invocation logs create/start/died/remove) and the GitHub API's PR-merge and workflow-trigger timestamps. Bash history itself had none — no HISTTIMEFORMAT was set, so nothing to reconstruct from there; worth turning on for future drills.

Time (EDT) Event Source
21:57 PR #13 merged — issue #3 code lands GitHub
21:59:55–22:00:09 First plan, wrong checkout — no state, would've recreated both VMs from scratch. Not applied. journal
22:28:51 PR #14 merged — HCP Terraform backend GitHub
22:31:30–22:31:46 terraform init — state migrated to HCP journal
22:32:23–22:32:24 plan against real state — reveals the 2 to destroy ForceNew surprise journal
22:36:42–22:36:58 apply — both VMs destroyed, recreate fails on Datastore.Allocate journal
22:37:58 Retry, fails fast (SSH access gap) journal
22:50:56 PR #15 merged — ssh-agent forwarding fix GitHub
22:53:09–22:54:39 Clean plan + successful apply — both VMs back journal
22:57:42–22:58:50 Ansible site.yml rebaseline (68s — matches the ~64-task recap) journal
23:13:33 homelab-observability GHA deploy triggered GitHub
23:24:43 This doc's own PR merged GitHub

Both hosts fully absent from Proxmox: ~18 minutes (22:36:58, destroy completes but recreate fails, to 22:54:39, successful recreate). Total incident, first merged PR to docs merged: ~87 minutes. The two blocking-error diagnose-and-fix cycles (permission, then SSH agent) were the dominant cost within that window, each requiring a code change, a merge, and a retry rather than a same-terminal fix — not the VM create/destroy operations themselves, which were consistently single-digit to low-double-digit seconds, comparable to apps01-rebuild.md's single-host timing.

One honest gap: which container run maps to which specific command is inferred from duration and sequence, not directly logged — podman's journal entries don't include argv. The window boundaries above (outage start/end, total span) don't depend on that inference and are solid.