DR drill: rebuild `apps01` from scratch (2026-08-06)¶

Result: success. First time this fleet's core DR claim - "rebuildable from git + Bitwarden alone" - was actually exercised end to end, rather than just designed for. homelab-iac-common/RUNBOOK.md used to note that no service had ever been stood up from scratch on this pattern; that's no longer true after this drill.

Scope¶

Destroy and recreate the apps01 VM itself (not just redeploy a stack onto the existing host), then rebaseline it and redeploy the one real service on it (homelab-homepage) via a clean run of its own CI - no manual shortcuts.

Target state before the drill (confirmed live via SSH): apps01 (192.168.10.101) ran exactly two stacks - homepage (deployed via GitHub Actions) and alloy-agent (the host-level observability agent). Small, low-risk blast radius: losing the host 502s homepage.flomico.com and stops this host's logs/metrics reaching obs01; nothing else depends on it.

Deliberately no rollback net. No Proxmox snapshot was taken before the destroy. The point of the drill is "new hardware, nothing to fall back on" - a snapshot would have made the test easier but less honest to that scenario.

Steps run¶

1. Drift check¶

cd homelab-platform
source ./scripts/bw-load.sh
docker compose run --rm terraform init
docker compose run --rm terraform plan

Confirmed no changes before touching anything, so anything that broke later would be attributable to the rebuild itself, not pre-existing drift.

2. Destroy and recreate the VM¶

docker compose run --rm terraform apply \
  -replace='module.host["apps01"].proxmox_virtual_environment_vm.this'

Plan came back scoped exactly as expected: 1 to add, 0 to change, 1 to destroy for apps01, and obs01 untouched. This is the direct evidence that config/hosts.yaml's for_each actually does isolate one host's rebuild from the rest of the fleet - not just a design intention.

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

host_ips = {
  "apps01" = "192.168.10.101"
  "obs01" = "192.168.10.102"
}

Same IP as before - the module derives it from vm_id, it isn't discovered.

3. Clear the stale SSH host key¶

ssh-keygen -R 192.168.10.101

Expected and already documented in RUNBOOK.md's Triage table (REMOTE HOST IDENTIFICATION HAS CHANGED → rebuilt VM reusing an IP). Ran it proactively rather than waiting to hit the error.

A brief pause/retry loop (ssh debian@192.168.10.101 'echo ok') was needed before the next step - terraform apply completing just means the Proxmox API accepted the request, not that cloud-init has finished on the guest (agent_enabled = false means Terraform doesn't wait for the guest agent).

4. Reapply the Ansible baseline¶

docker compose run --rm ansible ansible-playbook -i inventory/hosts.yml site.yml

PLAY RECAP
apps01   : ok=37  changed=15  unreachable=0  failed=0  skipped=11
obs01    : ok=19  changed=0   unreachable=0  failed=0  skipped=12

apps01's 15 changes are expected on a fresh host (docker install, unattended-upgrades config, /opt//srv layout, the alloy_agent role). obs01's zero changes confirm the blast radius stayed scoped correctly.

5. Redeploy the service via its own CI¶

cd ../homelab-homepage
gh workflow run deploy.yml
gh run watch

This is the step that had never actually happened before - homelab-homepage had only ever been converted onto this pattern, never deployed fresh onto a host that didn't already have it. The workflow succeeded unmodified: joined the tailnet, waited for the host to be reachable, ran compose_stack.

6. Verify¶

ssh debian@192.168.10.101 'docker ps'
ssh debian@192.168.10.101 'cat /opt/.stacks/homepage.json /opt/.stacks/alloy-agent.json'
curl -sk -o /dev/null -w '%{http_code}\n' https://homepage.flomico.com

homepage.json showed a fresh git_sha, deployed_by: github-actions, and a timestamp matching the CI run. curl to the plain :443 hostname (the public-facing port - :8443 is an internal listener only reachable from the guest-network IP, per caddy-iac's Caddyfile comments) returned 200.

Findings¶

Biggest gap: no runbook scenario covers rebuilding the host itself. RUNBOOK.md's Scenarios 1-4 all assume apps01 already exists. The sequence used here was assembled from three separate sources - this drill's own plan, homelab-platform/CLAUDE.md's "after adding or resizing a host" section (written for a new host, not a same-IP rebuild), and one row of the Triage table. Worth becoming an explicit scenario.
A host rebuild orphans every service on it, with no bulk-redeploy primitive. Cheap here - one gh workflow run call, because apps01 currently runs exactly one app. At the host's stated capacity of 10-15 services, this becomes 10-15 manual commands with no "redeploy everything that was on this host" tool.
The design held up under the one test that actually stresses it. config/hosts.yaml's for_each scoped the blast radius to precisely apps01, confirmed independently by both the Terraform plan and the Ansible recap. "Adding/rebuilding a host is a config edit, not a code edit" is now a verified property, not just a stated intention.
Scope limit of this drill: no secrets were exercised. homelab-homepage has no secrets.json, so this run never tested recreating a service's CI secrets from scratch - that's homelab-iac-common issue #1's territory, and remains unverified. A future drill of a service with secrets (e.g. homelab-observability, which needs a Grafana admin password) would be the next thing to test.
Minor, not new: alloy-agent's manifest still shows deployed_by: unknown after the rebuild - pre-existing behavior of that role (a raw Ansible play, not routed through compose_stack's CI-attributed deploy path), not something this rebuild changed.

Rough timing¶

VM replace (Terraform accepting the request): ~11s. Full drill, wall clock from destroy to a 200 through Caddy: a few minutes, dominated by waiting for cloud-init/boot and the Ansible run - not measured precisely this time, worth timing properly on the next drill if RTO is a question worth answering.

DR drill: rebuild apps01 from scratch (2026-08-06)¶