Rebuilding the home server with Ansible

After moving from Portainer to git, every Compose file and every piece of configuration was in git. But a git repository is only a description. Getting it onto the server was still a list of manual steps that I would have to remember after a disk failure. Probably at the worst possible moment, like the last time my RAID died while I was sitting in a bar.

Two repositories

The Compose repository stays as it is and describes what runs. A second repository with Ansible describes how it gets there:

playbooks/
  rebuild_omv.yml          the base: Docker, InfluxDB, users, SSH
  bootstrap_docker.yml
  bootstrap_influxdb.yml   InfluxDB runs as a host service, not in Docker
  install_omv.yml
  deploy_infra.yml         one playbook per stack
  deploy_monitoring.yml
  deploy_openhab.yml
  ...
roles/
  omv_install/             users, groups, sudo, SSH hardening
  omv_deploy_openhab/      openHAB and Mosquitto
  omv_deploy_monitoring/   Prometheus, Grafana, InfluxDB tasks, backups
  ...
inventory/
  group_vars/vault.yml     encrypted secrets (Ansible Vault)

All I need is a freshly installed OpenMediaVault with my user and SSH key. Then the base, and after it one playbook per stack:

ansible-playbook playbooks/rebuild_omv.yml -e allow_modifications=true --ask-vault-pass -K
ansible-playbook playbooks/deploy_infra.yml -e allow_modifications=true --ask-vault-pass -K
ansible-playbook playbooks/deploy_openhab.yml -e allow_modifications=true --ask-vault-pass -K
...

-K asks for the sudo password, my user has no passwordless sudo on the server.

What a role does

Take the openHAB role as an example. It runs these steps in order:

1. Preflight. Check that all required variables and vault values exist, and fail before touching anything.
2. Create the service users and groups (openhab, mosquitto) and read their IDs for the containers.
3. Create /srv/openhab/... with the right owners and permissions, and install my own add-ons.
4. Copy items, things, rules and sitemaps from the Compose repository, and render the files that contain credentials.
5. Write the .env file of the stack from role variables and vault values, with mode 0600.
6. docker compose up -d with the file from the Compose repository.
7. Check that the containers are actually running.

Every service runs as its own user, so the role does not hardcode IDs. It creates the user if needed, reads its UID and GID from the server and writes them into the .env file as PUID and PGID.

Secrets

All passwords and tokens live in a single file encrypted with Ansible Vault. Some openHAB things need credentials right inside their config file, like a device token or a bridge password. Those files are Jinja templates in the Compose repository:

Thing mideaac:ac:... "PortaSplit" [ ..., token="{{ openhab_midea_token }}", key="{{ openhab_midea_key }}" ]

The role renders them during the deployment. If one of the values is missing, the preflight step stops right away. Better than a thing that silently fails to connect.

Safe by default

Every playbook runs read-only, unless I pass -e allow_modifications=true. Together with Ansible’s --check mode, I can always see what would change before anything does:

ansible-playbook playbooks/deploy_openhab.yml --check --ask-vault-pass -K

Small changes, same path

The real value is not the rebuild I hopefully never need. It is that every small change takes the same path:

1. edit a rule in the Compose repository
2. commit (pre-commit scans it)
3. ansible-playbook playbooks/deploy_openhab.yml -e allow_modifications=true

openHAB picks up the changed file within seconds, and that is the whole deployment. The Flux tasks work the same way. The monitoring role reads the .flux files from git and creates or updates the InfluxDB tasks by name. And so do the Grafana dashboards. They are JSON files in the Compose repository now, the role copies them to Grafana’s provisioning folder and removes the ones I deleted in git.

Fast enough on a small CPU

The server has a Celeron J1900. The first version of the openHAB role copied every config file with its own task, and the deployment took a while. With the timing callback on, every Ansible module call took about 0.9 seconds. Not on the server, the server needs a fraction of that, but on my side, packing and sending the module. 35 config files meant 35 of those calls.

[defaults]
callbacks_enabled = ansible.posix.profile_tasks

Now the role packs the whole conf/ folder into one archive on my machine and unpacks it on the server in a single call:

- name: Pack OpenHAB configuration files
  community.general.archive:
    path: "{{ openhab_conf_source_dir }}/*"
    dest: "{{ playbook_dir }}/../.ansible/openhab-conf.tar.gz"
  delegate_to: localhost
  changed_when: false

- name: Deploy OpenHAB configuration files
  ansible.builtin.unarchive:
    src: "{{ playbook_dir }}/../.ansible/openhab-conf.tar.gz"
    dest: "{{ openhab_conf_dest_dir }}/"
    owner: "{{ openhab_user }}"
    group: "{{ openhab_group }}"
  become: true

It only reports a change when a file really differs, so a deployment without changes stays quiet. One thing to keep in mind: it adds and updates files, but never deletes one on the server. When I rename a rule file, I remove the old one by hand.

Things that were still in the OpenMediaVault UI

A rebuild only brings back what Ansible knows about. The weekly InfluxDB backup was a scheduled job in the OpenMediaVault UI, so it existed only in the OMV configuration of that very server. Now it is a systemd timer from the monitoring role:

[Timer]
OnCalendar=Sun *-*-* 00:00:00
Persistent=true

Persistent=true catches up a run that was missed while the server was off. Next to it, a small script checks every hour that the newest backup set is complete (every file from the manifest there, with the right size) and writes the result for the node exporter. Grafana alerts me via openHAB when the backup is older than 8 days or incomplete.

Guard rails

The Ansible repository gets the same checks as the Compose repository. pre-commit runs TruffleHog, ansible-lint and a check that every vault.yml is really encrypted, and GitHub Actions runs the lint again. One small problem: the playbooks load the vault, but CI must never get the vault password. So for the syntax check, CI simply copies the example vault (all values CHANGE_ME) over the real one:

- name: Use example vault
  run: cp inventory/group_vars/vault.example.yml inventory/group_vars/vault.yml
- name: Lint Ansible source
  run: ansible-lint roles playbooks requirements.yml

And we are done. A handful of commands for the whole server, and one command for every small change. Next up is what all of this was built for: heating the water when electricity is cheap.