The last post about the home server ended with a disappointment. I had cut its CPU load by two thirds, and the power draw did not move. A Celeron J1900 at 40 % or at 15 % of one core is a watt or two. The bigger part are the two 4 TB NAS disks of the RAID, which spin around the clock, although most of the day nobody touches them. The system and all containers live on an SSD, the RAID only holds files, the ownCloud data and the backups.
So the disks should sleep. That turned out to be one setting and a week of finding out who keeps waking them.
The setting
OpenMediaVault sets the spindown per disk under Storage, Disks. I chose standby after 60 minutes and APM level 127, the lowest level that still allows standby. OpenMediaVault does not write this into /etc/hdparm.conf, it generates a small script per disk that a service runs at boot. Good to know when you look for the setting and find an old hdparm.conf instead.
How to see whether the disks sleep
hdparm -C asks a disk for its power state without waking it. A timer runs it every five minutes and writes the result for the Prometheus node exporter, so the state ends up in Grafana like any other metric:
state=$(hdparm -C "$disk" | sed -n 's/.*drive state is: *//p')
case $state in
standby) value=1 ;;
*) value=0 ;;
esac
echo "node_disk_standby{disk=\"$(basename "$disk")\"} $value"
The file goes into the directory of the textfile collector. With that I could see when the disks slept and, more interesting, when they woke up.
They never slept
The first night showed a straight line at 0. From 04:07 to 10:40 there was not a single read or write on the RAID, and still the disks never went to standby.
The reason was my own monitoring. Prometheus scrapes the node exporter every 60 seconds, and its hwmon collector reads every temperature sensor it finds. That includes the disks: the kernel module drivetemp makes their temperature available as a sensor, and reading it is a command to the disk. The kernel documentation says it plainly, reading the temperature resets the standby timer. So the disks were asked for their temperature every minute and never reached 60 minutes of rest.
The fix is one line in the node exporter command, which leaves out the two RAID disks and keeps the SSD:
- '--collector.hwmon.chip-exclude=^(thermal_thermal_zone[0-9]+|target[01]:0:0_[01]:0:0:0)$$'
The disk temperatures are still watched by smartd, which checks them without waking a sleeping disk. The next night both disks reported standby.
Backups all over the night
Now the disks slept, but not for long. Every backup job had its own time, picked years ago without thinking about it: the web server backup, the ownCloud database dump, the pull of the web server backup, the InfluxDB backup and its check, and Duplicati. Each of them woke the RAID, and after each one the disks spun for another hour until the standby timer ran out.
I moved all of them into one block in the night:
| Time | Job |
|---|---|
| 02:20 | web server backup (on the web server) |
| 02:45 | InfluxDB backup (Sundays) |
| 02:50 | ownCloud database dump, web server pull |
| 03:00 | Duplicati |
| 03:15 | check of the InfluxDB backup |
Now the disks wake up once at about 02:55 and go back to standby at about 04:10. There was also a read at 02:46 every night that I never found in any log. Since the bundling it is gone, so it was probably one of the jobs after all.
The monthly RAID check
In the first days of a month Debian checks the whole RAID (mdcheck). It starts on the first Sunday and runs in slices of six hours until both disks are read completely, about seven hours in total. On those days the disks are awake at random times. I left it like that, a few days a month are fine.
A side effect: no more SMART self-tests
While looking for something else I noticed that the disks had not run a SMART self-test since the day the spindown started to work. Until then a short test ran every day.
smartd only checks the disks once a day, and with the power mode set to standby it skips a sleeping disk, including the self-test that is due. Once a day at the same minute, while the disks sleep, means never. So the check has to run more often, and the tests belong in the hour when the disks are awake anyway for the backups.
My first try was a check every 30 minutes. A sleeping disk is skipped, so I thought it would cost nothing. But on a disk that is awake, every check is a command, and every command starts the 60 minute standby timer again. With a check every 30 minutes the disks never reached it. They stayed awake for a whole day until I noticed it in Grafana.
The check interval has to be longer than the standby time. These are the settings in OpenMediaVault under Storage, S.M.A.R.T.:
| Setting | Value |
|---|---|
| Check interval | 3720 seconds (62 minutes) |
| Power mode | Standby |
| Short test | every day at 03:00 |
| Long test | once a month at 03:00, one disk a day |
Between two checks a disk can fall asleep, and at least one check falls into the hour after 03:00 when the backups keep the disks awake. Since then a short test runs every night again.
One side effect stays. A check that lands while a disk is awake is a command too, so after every wake the disks can stay up to an hour longer than needed.
The graphics card that checked for a monitor
The server has no monitor, but the kernel log had two lines every ten seconds, around the clock:
i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io,decodes=io:owns=io+mem
No process caused it. The board has a VGA port, and VGA has no way to tell the computer that a monitor was plugged in. So the graphics driver checks every ten seconds whether something is there, and every check wakes the graphics chip for a moment. On a server without a monitor that is useless, and a kernel parameter turns it off:
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT drm_kms_helper.poll=0"
After the change the messages stopped. A monitor plugged in later is not detected automatically any more, which I can live with.
Who wakes them at noon?
The disks still woke up every day, around noon and often again at about 15:00. The wake at 12:55 had no read or write on the disks at all, so it was a command, not a file access. smartd was not it (no self-test ran in those days), Samba was not it, and the journal had nothing.
So I asked the kernel. Its tracing can log every SCSI command with the process that sent it:
cd /sys/kernel/tracing echo 1 > events/scsi/scsi_dispatch_cmd_start/enable cat trace_pipe > /root/scsi-trace.txt
The first day of trace was huge, because my own hdparm -C checks every five minutes show up there too. They ask for the power state without waking the disk, so I filtered them out. What was left were two culprits, both from ownCloud.
At 15:00 and in the night, ownCloud wrote to its log file, which lies in the data directory on the RAID. Three background jobs failed every time they ran, and each run added a warning. I deleted the jobs and moved the log onto the SSD:
environment: - OWNCLOUD_LOG_FILE=/mnt/data/owncloud.log
The noon wake is ownCloud’s daily update check. It reads a certificate file from the data directory, so it has to wake the RAID. I could switch the update check off, but I want to know about ownCloud updates, so the disks may wake up once a day for that.
The result
Now the disks sleep most of the day. On a normal day they are awake from about 02:50 to 04:10 for the backups and the self-test, and for one and a half to two and a half hours in the early afternoon, about three hours a day.
The night base load of the whole flat, the lowest quarter hour of the night, was 63.7 W in the median of the six nights before the spindown and 59.9 W in the nine nights after it. A night where I forgot something switched on is in neither.
The savings are small. About 3.5 W are about 30 kWh a year, at my new fixed price around 4 euros. I did it anyway, because a disk that does nothing should not spin, and on the way I found two things that were actually broken: the missing SMART self-tests and my own monitoring keeping the disks awake. Measuring the server alone with a plug is next, the numbers above are the base load of the whole flat at night.
