Skip to content

Runbook

Operator-facing procedures for the flomico.com homelab. The per-repo READMEs explain what each repo is; this explains how a change actually travels from an edit to something running.

How the repos fit together

Build-time: what this repo publishes, and who pins it

Everything below is consumed at an explicit tag. Nothing tracks main, so a change here is inert until a consumer bumps its pin — see Scenario 4.

flowchart LR
  subgraph common["homelab-iac-common"]
    MOD["terraform module<br/>proxmox-vm"]
    COL["ansible collection<br/>flomico.homelab"]
    SHIM["bw-load / bw-seed<br/>shims"]
    IMG["toolbox image<br/>ghcr.io/codyecsl/homelab-toolbox"]
    WF["reusable workflow<br/>deploy-stack.yml"]
  end

  COL -->|baked in at image build| IMG

  MOD -->|git module source| PLAT["homelab-platform"]
  IMG --> PLAT
  SHIM --> PLAT

  IMG --> SVC["service repos<br/>homelab-homepage, ..."]
  WF --> SVC

  PLAT -->|terraform apply| HOST["apps01<br/>192.168.10.101"]
  SVC -->|deploy| HOST

caddy-iac and tailscale-iac still carry their own copies of the Terraform and Bitwarden plumbing — migrating them is outstanding work, not the design.

Deploy-time: what happens on merge

sequenceDiagram
  autonumber
  participant Dev
  participant Repo as service repo
  participant CI as GitHub runner
  participant Net as tailnet
  participant Host as apps01

  Dev->>Repo: merge to main
  Repo->>CI: deploy.yml calls deploy-stack.yml@vX.Y.Z
  CI->>Net: join as ephemeral node, tag:ci
  Note over Net: grant tag:ci -> target host :22<br/>lives in tailscale-iac
  CI->>CI: pull toolbox image from GHCR
  CI->>Host: ssh :22, ansible-playbook
  Host->>Host: compose_stack writes /opt/STACK and runs compose
  Host->>Host: manifest to /opt/.stacks/STACK.json
  CI->>Net: node deregisters

Traffic and the route registry

caddy-iac's Caddyfile is the port registry. Every service here is reverse-proxied, so that one file is the authoritative list of what runs on .101 and on which port. There is no second registry to keep honest.

flowchart TD
  CF["caddy-iac<br/>ansible/caddy/Caddyfile"]
  CADDY["Caddy VM<br/>.240 admin / .241 guest"]
  HOST["apps01<br/>192.168.10.101:PORT"]
  SYNC["homelab-homepage<br/>sync-caddy-routes.yml (daily)"]
  YAML["config/services.yaml"]
  DASH["homepage dashboard"]

  CF -->|ansible deploy| CADDY
  CADDY -->|reverse_proxy| HOST
  CF -->|generator opens a PR on drift| SYNC
  SYNC --> YAML
  YAML -->|merge triggers a homepage deploy| DASH

Scenario 1: stand up a new service

Steps 4-5 below - deploying onto a host via a service's own CI, and verifying it - were exercised for the first time in the 2026-08-06 apps01 rebuild drill, which destroyed and recreated the VM itself and then redeployed homelab-homepage onto it via a clean run of its own workflow.

Steps 1-3 - creating a new service repo from nothing and wiring its CI secrets - remain untested. Every service repo so far was created by copying an existing one, not by working through this section fresh. Expect to hit something not written down here; add it when you do.

1. Decide two things first

Port. Pick one not already in caddy-iac's Caddyfile. That file is the registry — check it, don't guess.

Guest-safe or admin-only. This determines the Caddyfile site block and is easy to get wrong:

Site block Reachable at Who
Guest-safe svc.flomico.com:8443 { } .241 anyone on the guest-web Tailscale profile
Admin-only svc.flomico.com { } .240 full profile only

The :8443 suffix is an internal listener port — clients always connect on plain :443. See caddy-iac's CLAUDE.md.

2. Create the repo

myservice/
├── stack/docker-compose.yml    # what runs on apps01
├── config/                     # app config, if any (optional)
├── ansible/
│   ├── inventory/hosts.yml
│   └── deploy.yml
├── docker-compose.yml          # toolbox, runs the deploy
└── .github/workflows/deploy.yml

Copy inventory/hosts.yml, the root docker-compose.yml, and .github/workflows/deploy.yml from homelab-homepage — they are boilerplate. ansible/deploy.yml is the only one needing thought:

- name: Deploy myservice
  hosts: apps01
  become: true
  gather_facts: true
  roles:
    - role: flomico.homelab.compose_stack
      vars:
        compose_stack_name: myservice
        compose_stack_src: ../stack
        compose_stack_config_src: ../config   # omit if there is no config/

Bind persistent state to /srv/myservice/ rather than a named Docker volume, so a whole-VM backup captures something legible.

Only if the service needs secrets: add a secrets.json (schema in homelab-iac-common's README) and copy the bws-load.sh / bws-run.sh / bws-env.sh shims from scripts/bws-load-shim.sh, scripts/bws-run-shim.sh, and scripts/bws-env-shim.sh there. Secrets live in the one shared homelab project in Bitwarden Secrets Manager, not a per-repo item — add the field under its existing name if another repo already uses it, or a <service>_-prefixed one otherwise. Most services need none — homelab-homepage has neither.

3. Wire up CI

gh repo create CodyECSL/myservice --private --source=. --remote=origin --push

gh secret set TS_OAUTH_CLIENT_ID --repo CodyECSL/myservice
gh secret set TS_OAUTH_SECRET    --repo CodyECSL/myservice
gh secret set DEPLOY_SSH_KEY     --repo CodyECSL/myservice < ~/.ssh/homelab-ci

~/.ssh/homelab-ci is the dedicated CI deploy key, authorized on every host in the homelab. It gets there two different ways, and the difference matters when you add a host:

Host Authorized by Mechanism
apps01, obs01 homelab-platform docker_host_deploy_keys in its inventory, applied by flomico.homelab.docker_host
Caddy VM (.240) caddy-iac an ansible.posix.authorized_key task in its own deploy-caddy.yml pre_tasks — that playbook never applies docker_host

The private half is backed up in Bitwarden as the SSH Key: homelab-ci secure note (private_key/public_key fields), for recovery on a machine that doesn't have it locally.

The public half is declared in two places: homelab-platform's inventory as a literal, and ssh_public_key in the shared bws project. Those are the same key. ssh_public_key is a slightly misleading name for it — it is the homelab-ci public key, not the operator's personal key — and it does double duty: Terraform hands it to cloud-init for every new VM, and caddy-iac's playbook reads it to authorize the key on the already-running Caddy VM. A rotation has to update both declarations.

A VM built today already trusts homelab-ci

Because ssh_public_key is that key, a freshly provisioned host accepts CI from birth and needs no authorize step. Only hosts created before that secret was switched need fixing up — which is exactly why the Caddy VM did and nothing else does.

Never set DEPLOY_SSH_KEY to your personal key. It is also your GitHub key, so a pipeline holding it has far more reach than a deploy needs. If a host does not accept homelab-ci yet, authorize it there rather than widening the secret.

Ordering trap — this is how you lock a pipeline out

The key is authorized by a playbook run, which itself needs to log in. On a host that does not have it yet, the only key that works is the one cloud-init baked in from TF_VAR_ssh_public_key — your personal key.

Always authorize first, swap the secret second. Who does the authorizing run depends on which key CI currently holds:

  • CI holds nothing that works (a brand-new host): run the playbook once from your workstation, then set DEPLOY_SSH_KEY.
  • CI already holds a working key (the caddy-iac case — its pipeline carried the personal key, which the VM accepted): CI can bootstrap itself. Merge the playbook change while the old secret is still in place; that deploy authorizes homelab-ci. Only then swap the secret.

Either way, setting the secret first means CI cannot get in to authorize the key that would have let it in. Verify with ssh -i ~/.ssh/homelab-ci debian@<host> 'echo ok' before swapping — that check is the whole gate.

Reuse the same OAuth client #2 values as every other service repo (see tailscale-iac's README). Do not generate a new one. See homelab-iac-common issue #1 — this copy-paste step is the thing that should be automated.

Only if ansible/deploy.yml's preflight assert needs a secret of its own (Grafana's admin password is the first example — homelab-observability) — add the field to this repo's secrets.json and set the BWS_ACCESS_TOKEN repo secret (the read-only Secrets Manager machine account's token — same value on every repo that needs one; see homelab-iac-common's README). deploy-stack.yml runs the playbook under bws-run.sh inside the toolbox container, which resolves this repo's own secrets.json there:

gh secret set BWS_ACCESS_TOKEN --repo CodyECSL/myservice

Most services need none of this — homelab-homepage has neither secrets.json nor BWS_ACCESS_TOKEN.

Then the UI-only step that is easy to forget: grant the new repo read access to the toolbox image, or the deploy fails on an image pull.

github.com/users/CodyECSL/packages/container/homelab-toolbox/settings → Manage Actions access → add CodyECSL/myservice → Read.

Reusable-workflow access is already enabled account-wide on this repo and does not need repeating.

tailscale-iac needs no change — the tag:ci grant already covers apps01, obs01 and the Caddy VM on :22, which every service repo shares. Only a new host would need one: an endpoint in config/endpoints.yaml plus that endpoint added to the tag:ci grant in terraform/policy.tf. Do that deliberately — each host added widens what a compromised runner inherits.

4. Add the route and deploy

Follow Scenario 3 for the Caddyfile, then:

git push origin main
gh run watch --repo CodyECSL/myservice

5. Verify

ssh debian@192.168.10.101 'cat /opt/.stacks/myservice.json'   # deployed_by: github-actions
curl -sI http://192.168.10.101:PORT | head -1                 # 200 from the host
curl -s -o /dev/null -w '%{http_code}\n' https://myservice.flomico.com/

If the last one fails but the second succeeds, it is Caddy or DNS, not the service — see Triage.

Scenario 2: update an existing service

Three kinds of change, and they do not all go the same way.

Change Path Action
App config (config/**) CI merge → auto-deploy
Stack definition (stack/**, image tag, ports) CI merge → auto-deploy
Hostname or upstream port caddy-iac too Scenario 3

For the first two:

git commit -am "..." && git push origin main
gh run watch --repo CodyECSL/myservice

deploy-stack.yml runs compose_stack, which always passes --force-recreate. That flag is load-bearing: these stacks bind-mount their config, and a plain docker compose up only inspects service definitions, so without it a deploy copies new config to disk and leaves the container serving the old version.

When to deploy by hand instead

The manual path is identical in effect — same role, same image:

docker compose run --rm ansible ansible-playbook -i inventory/hosts.yml deploy.yml

Reach for it when CI itself is what is broken (expired secret, GHCR access, tailnet), or when iterating fast enough that a commit per attempt is silly. It leaves deployed_by as your username in /opt/.stacks/, which is how you can tell afterwards which deploys were manual.

Changing what host a service runs on

host_ip in the service repo's .github/workflows/deploy.yml, and the inventory default. Both currently name 192.168.10.101 literally — LAN topology is committed by convention across these repos, not hidden.

Scenario 3: change a route

Adding a hostname, changing a port, or moving a service between guest and admin. This spans three repos and the order matters.

# 1. caddy-iac: edit the site block
cd ~/Documents/GitHubRepos/caddy-iac
source ./scripts/bws-load.sh
docker compose run --rm ansible \
  ansible-playbook -i inventory/hosts.yml playbooks/deploy-caddy.yml

A redeploy is required — the Caddyfile is bind-mounted, so a committed change that has not been deployed is not live.

# 2. verify the route before trusting DNS
curl -s -o /dev/null -w '%{http_code}\n' \
  --resolve svc.flomico.com:443:192.168.10.241 https://svc.flomico.com/   # .240 if admin-only

Certificates need no action: DNS-01 with a *.flomico.com wildcard means a new hostname works immediately after redeploy, with no DNS change.

# 3. homelab-homepage picks the route up automatically

sync-caddy-routes.yml runs daily and opens a PR when routes drift. To not wait, run npm run sync locally and commit. Merging that PR triggers a homepage deploy on its own.

Optionally add a config/service-meta.json entry for a proper display name, icon and group — without one the service still appears, under "Uncategorized" with a capitalised-subdomain name. A new route can never silently vanish from the dashboard; it just looks generic.

Finally, from a browser: flush DNS, or a cached negative lookup makes a working change look broken.

sudo systemctl restart systemd-resolved && sudo resolvectl flush-caches

Scenario 4: change something shared

Editing homelab-iac-common changes nothing anywhere until consumers bump their pins. That is the design — blast radius is opt-in.

# 1. verify locally first - all three CI jobs can be run by hand
./test/bw-load.test.sh
terraform -chdir=terraform/modules/proxmox-vm init -backend=false && \
terraform -chdir=terraform/modules/proxmox-vm validate
ansible-lint ansible/collections/ansible_collections/flomico/homelab

# 2. tag and push - the image only builds on tags
git push origin main
git tag v0.1.4 -m "v0.1.4" && git push origin v0.1.4

# 3. confirm the image tag exists before bumping anything
gh run view "$(gh run list --repo CodyECSL/homelab-iac-common \
  --workflow toolbox.yml --limit 1 --json databaseId --jq '.[0].databaseId')" \
  --repo CodyECSL/homelab-iac-common --log \
  | grep -oE 'homelab-toolbox:[a-zA-Z0-9._-]+' | sort -u

Then bump each consumer that needs it:

Consumer What to bump
service repos uses: ...deploy-stack.yml@vX.Y.Z
homelab-platform module ?ref=, image tag in docker-compose.yml, HOMELAB_COMMON_REF in the shims

Collection or role changes reach hosts through the image, so a consumer that only bumps the workflow ref keeps running the old roles until toolbox_tag moves too.

Never move a published tag. Cut a new one. v0.1.0 was force-moved once while nothing consumed it; that stopped being safe the moment the first deploy succeeded.

Triage

The failing step name usually identifies the cause outright.

Symptom Cause Fix
workflow was not found calling deploy-stack.yml Reusable-workflow access off, or a tag that does not exist gh api -X PUT repos/CodyECSL/homelab-iac-common/actions/permissions/access -f access_level=user
tailscale up fails OAuth client's tag or scope wrong Client needs auth_keys + tag:ci, nothing else
Wait for the host to be reachable times out Tailnet joined, ACL grant missing Check tag:ci → host :22 in tailscale-iac
invalid reference format ... must be lowercase Owner casing in an image reference Lowercase it — github.repository_owner preserves CodyECSL
manifest unknown Image tag does not exist Confirm the published tags; {{version}} strips the leading v
denied / unauthorized on pull Repo lacks package read access Manage Actions access on the package → add repo → Read
SSH auth failure in Deploy Wrong half of the keypair, or the public half isn't authorized on the host DEPLOY_SSH_KEY is the private key, and should be homelab-ci everywhere. The public half is declared in two places: docker_host_deploy_keys in homelab-platform/ansible/inventory/hosts.yml (apps01, obs01) and ssh_public_key in bws (the Caddy VM, via caddy-iac's playbook — despite the name, that secret holds the homelab-ci key). Compare the relevant one against ~/.ssh/homelab-ci.pub or the SSH Key: homelab-ci Bitwarden item — if the two declarations have drifted, that is the bug. On a host that has never had a playbook run, the key isn't authorized yet — run it once from a workstation
Unable to change directory ... /opt/STACK Only expected under --check Real runs create it first
Config edited but container serves the old version --force-recreate skipped It is in the role; suspect a stale image tag instead
REMOTE HOST IDENTIFICATION HAS CHANGED Rebuilt VM reusing an IP ssh-keygen -R <ip>
Service reachable on .101:PORT but not by hostname Caddy not redeployed, or stale DNS Scenario 3 steps 1 and 3

Note CI never consults a known_hosts file — the toolbox image sets ANSIBLE_HOST_KEY_CHECKING=False, since an ephemeral runner has nowhere durable to pin a key. A genuine host-key change would therefore not be noticed by a deploy. Accepted because the only path in is the tailnet, ACL-scoped to port 22 on one host.

Recurring maintenance

Item Cadence Notes
CADDY_IAC_TOKEN expires Fine-grained PAT on homelab-homepage for the route sync. Expires silently — the symptom is the sync workflow quietly no-opping
Tailscale OAuth client #2 on rotation Shared by every service repo, so rotation touches all of them — see homelab-iac-common issue #1
DEPLOY_SSH_KEY on rotation The dedicated homelab-ci key, backed up in Bitwarden as SSH Key: homelab-ci. Rotating it means updating the public half in both places it is declared — docker_host_deploy_keys in homelab-platform's inventory, and ssh_public_key in bws (which is the homelab-ci key, not a personal one) — then re-running homelab-platform's site.yml and caddy-iac's deploy, then re-setting the secret in every service repo. Authorize everywhere before swapping any secret
Toolbox base image as needed Rebuild by cutting a tag; picks up Debian and ansible-core patches
apps01 sizing when memory is tight config/hosts.yaml in homelab-platform, then terraform apply. Memory first — CPU overcommits fine
Unattended upgrades automatic docker_host enables security updates; it does not reboot for kernel updates

Open work

  • homelab-iac-common #1 — CI secrets are hand-copied per repo and recorded nowhere; rotation is O(repos) with no completeness check.
  • homelab-iac-common #2 — closed as of 2026-08-08. homelab-ci is authorized on apps01 and obs01 via docker_host_deploy_keys, and on the Caddy VM by an authorized_key task in caddy-iac's own playbook (that playbook does not apply docker_host, and Terraform's user_account.keys cannot touch a live VM). No pipeline carries the operator's personal key any more. Note the personal key is still authorized on every host — that is the operator's own access and the rollback path, deliberately kept.
  • homelab-iac-common #5 — deploy-stack.yml's toolbox_tag default silently falls behind every toolbox release.
  • proxmox-iac #1 — the Proxmox hypervisor itself (network, storage, VLANs) isn't source-controlled or documented as a from-scratch build.
  • No runbook scenario yet covers rebuilding a shared host itself (as opposed to a service on it) — see the apps01 rebuild drill's findings. Nor is there a way to bulk-redeploy every service on a host after one, beyond re-running each service's CI by hand.
  • caddy-iac and tailscale-iac still carry duplicated Terraform, Ansible and Bitwarden plumbing.
  • A future DR drill (or a real bare-metal rebuild) needs an explicit step to restore the homelab-ci SSH keypair before anything else touches apps01. docker_host_deploy_keys in homelab-platform's inventory authorizes this key's public half on apps01, and homelab-homepage's CI uses its private half as DEPLOY_SSH_KEY to deploy there. The private key exists only on the operator's machine and as a Bitwarden backup (SSH Key: homelab-ci secure note, private_key/public_key fields) — it is not in git and nothing pulls it down automatically. A restore also has two declarations of the public half to reconcile, not one: the literal in homelab-platform's inventory and ssh_public_key in bws. If they disagree, apps01/obs01 and the Caddy VM end up trusting different keys and only one repo's CI breaks — an annoying thing to diagnose under DR. On a lost-local-drives rebuild this step has to come before ansible-playbook site.yml, not after, since that inventory var is what authorizes the public half on the host in the first place: 
    bw get item "SSH Key: homelab-ci" | jq -r '.fields[] | select(.name=="private_key").value' > ~/.ssh/homelab-ci
chmod 600 ~/.ssh/homelab-ci
bw get item "SSH Key: homelab-ci" | jq -r '.fields[] | select(.name=="public_key").value' > ~/.ssh/homelab-ci.pub
    The apps01 rebuild drill predates this key's existence, so it didn't exercise this step — worth verifying on the next drill.