From Portainer to git: Docker Compose for the home server

openHAB, Mosquitto and Grafana run as Docker containers on my OpenMediaVault server, next to a few other services (reverse proxy, VPN, file sync, backups). For a long time they were stacks in Portainer. Paste a Compose file into a text box, press deploy, forget about it.

The problem with clicking

Portainer works fine. Until you need to answer one of these questions:

• What did I change last month, when things started to break?
• Which version of the openHAB config belongs to which Compose file?
• The disk died. How exactly was this set up?

The Compose files only existed inside Portainer’s database, and the openHAB configuration only in a directory on the server. No history at all.

Step 1: get the Compose files out

Boring, but done in an afternoon. Open every stack in Portainer, copy the Compose file, commit it to a git repository. One directory per stack:

stacks/
  infra/        reverse proxy (SWAG), Authelia, WireGuard, Pi-hole, Duplicati, ...
  monitoring/   Prometheus, node exporter, Grafana, speedtest tracker
  openhab/      openHAB, Mosquitto
  owncloud/     ownCloud, MariaDB, Redis
  ...

Then everything around the containers followed: the openHAB items, things, rules and sitemaps, the Prometheus config, the Flux tasks, the SWAG proxy configs. If it is text and I wrote it, it goes into git.

Step 2: no secrets in the files

Copied from Portainer, the Compose files were full of passwords, user IDs and paths. Everything secret or specific to the host became a variable:

services:
  openhab:
    image: openhab/openhab:latest
    restart: unless-stopped
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /srv/openhab/addons:/openhab/addons
      - /srv/openhab/conf:/openhab/conf
      - /srv/openhab/userdata:/openhab/userdata
    environment:
      EXTRA_JAVA_OPTS: "-Duser.timezone=${TZ:-Europe/Berlin}"
      USER_ID: "${PUID_OPENHAB}"
      GROUP_ID: "${PGID_OPENHAB}"
    networks: [default, lsio]

  mosquitto:
    image: eclipse-mosquitto:latest
    restart: unless-stopped
    volumes:
      - /srv/mosquitto:/mosquitto
    ports:
      - 1883:1883

networks:
  lsio:
    external: true   # shared with the reverse proxy

The values live in a .env file next to the Compose file on the server, and .env is in .gitignore. Every service also runs as its own user (PUID_OPENHAB, …) and owns only its own directories under /srv.

A container that fetches pages from the internet gets a bit more. My price scraper, for example, has its own network, a read-only file system and no Linux capabilities at all:

  web-metrics-scraper:
    image: ghcr.io/herschdorfer/web-metrics-scraper:latest
    user: "${WEB_METRICS_PUID}:${WEB_METRICS_PGID}"
    volumes:
      - /srv/web-metrics-scraper/config.ini:/config/config.ini:ro
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    networks:
      - web-metrics

If someone ever gets code into it, there is not much to do in there.

Step 3: let the machine check it

A repository with configuration for things that have passwords needs some guard rails. Before every commit, pre-commit runs TruffleHog to look for secrets, plus the usual whitespace and YAML checks. And on every push, GitHub Actions validates the whole thing:

• docker compose config for every stack, so a broken Compose file never reaches the server
• promtool check config for Prometheus
• the Grafana dashboard exports must be valid JSON
• TruffleHog over the full git history

- name: Validate compose files
  run: |
    for f in stacks/*/docker-compose.yml
    do
      docker compose -f "$f" config -q --no-interpolate
    done

--no-interpolate is important here. CI has no .env files, and it does not need them to check the structure.

A check that checked nothing

My first CI workflow simply ran all pre-commit hooks, TruffleHog included. Green on every run. Great! Until I looked at how the hook is called:

trufflehog git file://. --since-commit HEAD --results=verified --fail

--since-commit HEAD means “scan what changed since the last commit”. On my machine, during a commit, that is exactly the new change. In CI everything is already committed, so it scanned nothing and passed every single time.

Now CI skips that hook and scans the full history instead:

- name: Scan history for secrets
  run: >-
    docker run --rm -v "$PWD:/repo:ro" trufflesecurity/trufflehog:3.90.5
    git file:///repo --results=verified,unknown --fail --no-update
- name: Run pre-commit hooks
  env:
    SKIP: trufflehog
  run: pre-commit run --all-files

The history was clean, luckily. Lesson learned: only trust a check you have seen fail at least once.

Updates

Most images use :latest, and Watchtower takes care of updates in two ways. Containers I trust to update on their own (Mosquitto, for example) carry an opt-in label, and Watchtower updates them once a week, on Sunday at 04:00. A second Watchtower instance only watches the rest, openHAB among them, and reports new versions.

labels:
  - com.centurylinklabs.watchtower.enable=true
  - com.centurylinklabs.watchtower.scope=infra-monitor

Why weekly? Checking 16 containers within a few seconds every night regularly got single requests rejected by the registry with toomanyrequests, and that container was then skipped until the next run.

And one thing I learned the hard way. “Monitor only” does not mean “nothing changes”. To report a new image, the watching instance downloads it, so the local openhab/openhab:latest already pointed to the new version. The next time I deployed a completely unrelated rule change, Compose noticed that the running container no longer matched its image and recreated it. openHAB went from 5.2.1 to 5.2.2 in the middle of the evening, and its first start got stuck. For twelve minutes every rule failed, until a simple docker restart openhab fixed it. With :latest, any deployment can be an upgrade.

The other way round is just as surprising. Without a watching Watchtower, a deployment never pulls a newer :latest by itself, it only pulls images that are missing. For my own scraper I want every release right away, so it gets pull_policy: always.

What was still missing

Now everything was described in git. But getting it onto the server was still manual work: copy files, create directories with the right owners, write the .env files, docker compose up -d. And a fresh server would need users, groups and permissions that only existed in my head.