The home server behind this series is an old board with a Celeron J1900, four slow cores from 2013. It runs openHAB, InfluxDB, Grafana, Prometheus, ownCloud and about 20 containers. It felt sluggish, and the load average sat at 1.3 even at night, when nothing happens. So I went looking for what it was doing.
The short answer: mostly waiting for itself. After a week of small changes the load average is at 0.35, and the CPU time of the busiest parts is down by more than half. Nothing I removed was actually needed.
How to see where the CPU time goes
top shows a moment. What I wanted was the sum since boot, per service. Every Docker container and every systemd service runs in its own cgroup, and each cgroup counts its CPU time in cpu.stat:
awk '/usage_usec/ {print $2 / 1e6 " s"}' /sys/fs/cgroup/system.slice/influxdb.service/cpu.stat
Divided by the uptime that gives the share of one core. On the first morning, the server used 65 % of one core on average, around the clock:
| What | Share of one core |
|---|---|
| openHAB | 14 % |
| ownCloud | 10.5 % |
| InfluxDB | 10 % |
| node exporter | 9.5 % |
| kdamond (kernel) | 3 % |
| pihole | 3 % |
| containerd | 3 % |
| speedtest-tracker | 3 % |
The sum of the running processes only explained about half of the total. The rest were processes that live for less than a second. The server started 13.5 new processes per second.
ownCloud: a cron job every minute
The ownCloud image runs its background jobs every minute, and every run starts two PHP processes (occ system:cron and occ dav:cleanup-chunks). On top of that the health check ran every 30 seconds and loaded all entrypoint scripts before it asked the web server. For a cloud with two users that is a lot of work for nothing. ownCloud itself recommends 15 minutes:
environment: - OWNCLOUD_CROND_SCHEDULE=*/15 * * * * healthcheck: interval: 5m
ownCloud went from 10.5 % to 0.7 % of one core.
Health checks: 13,000 checks a day
Many images come with a Docker health check every 30 seconds. Each check starts a process in the container through containerd, and on this CPU that is not free. I set all of them to 5 minutes. Nothing restarts a container because of its health status here, so the check only feeds the status display:
healthcheck: interval: 5m
That is about 2,000 instead of 13,000 checks a day.
The node exporter collected things nobody looked at
The Prometheus node exporter had four extra collectors enabled for a big dashboard: systemd, processes, interrupts and tcpstat. The systemd collector asks systemd over D-Bus about every unit, every time Prometheus scrapes. I removed all four and set the scrape interval from 15 to 60 seconds. The node exporter went from 9.5 % to 0.3 %.
One collector had to stay: the textfile collector. A backup check writes its result into a file there, and two alerts depend on it.
openHAB stored every value every two seconds
The inverters report through AhoyDTU every two seconds, and openHAB stored every single update of every item: AC voltage, DC voltage, DC current, frequency, irradiation, efficiency. That was about 52,000 points per hour, and most of them were only on a dashboard I never opened.
I went through the ten most frequent measurements. The voltages and currents went away completely, the per-inverter values got one point per minute, temperature and frequency one every five minutes. The persistence rate now lives on the items as a group, so the persistence file stays short:
Items {
*, !gPersistMinute*, !gPersist5Min* : strategy = everyUpdate
gPersistence*, gPersistMinute* : strategy = everyMinute
gPersist5Min* : strategy = every5Minutes
}
A full day in InfluxDB went from 695,890 to 89,610 points, and openHAB from 14 % to about 5 % of one core. The total power, the energy counters and everything the calculations use still get every update.
InfluxDB: 687 shards for 600 MB
InfluxDB was the strangest case. It used 10 % of one core while it had nothing to do, and it ran 8,180 goroutines.
InfluxDB stores every bucket in shards, one per time span. For buckets that keep data forever, the default span is one week. After nine years that made 687 shards, 254 of them practically empty, left over from data I had deleted years ago. Every shard has its own background routines that wake up regularly, whether there is anything to do or not.
The fix was to copy each bucket into a new one with one shard per year and swap the names. Two things I learned on the way. InfluxDB refuses a shard duration together with infinite retention, both on create and on update, so I had to set it through the API:
curl -X PATCH "$URL/api/v2/buckets/$ID" -H "Authorization: Token $TOKEN" \
-d '{"retentionRules":[{"type":"expire","everySeconds":0,"shardGroupDurationSeconds":31449600}]}'
And when copying the last hours after the swap, range(stop: now()) misses everything in the future. The electricity prices for tomorrow are stored as a forecast, with future timestamps, and the best-price windows depend on them.
117.6 million points later, 687 shards were 11. InfluxDB went from 8,180 to 1,330 goroutines and from about 10 % to almost nothing.
A kernel thread nobody asked for
kdamond used 3 % of one core all the time. It belongs to DAMON, a part of the Linux kernel that watches which memory is used how often. Debian switched on its statistics module damon_stat by default. On big servers it costs a fraction of a percent, on this CPU 3 %, and nothing here reads its numbers. A kernel parameter turns it off:
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT damon_stat.enabled=N"
What did not work: powertop –auto-tune
powertop --auto-tune switches on every power saving option it knows. Within minutes the server felt slower than before. It had also enabled SATA link power management on the second SATA controller, where the system SSD sits, so every disk access first had to wake up the link. A reboot undid it, and I left it like that.
The result
| What | Before | After |
|---|---|---|
| Load average | 1.3 | 0.35 |
| ownCloud | 10.5 % | 0.7 % |
| node exporter | 9.5 % | 0.3 % |
| InfluxDB | 10 % | ~0 % |
| openHAB | 14 % | ~5 % |
| kdamond | 3 % | 0 % |
Shares of one core. The server feels quicker, and the InfluxDB backup now has 17 files instead of 701.
What it did not change is the power draw. The lowest quarter hour at night stayed at 63 to 65 W. A J1900 at 40 % or at 15 % of one core is a difference of a watt or two, lost in the noise of the fridge and the router. The disks are the bigger part, and whether their spindown works is the next thing I am measuring.