short version first: I'm looking for Linux compatible software which is able to transparently cache HDD writes using an SSD. However, I only want to spin up the HDD once or twice a day (to write the cached data to the HDD). The rest of the time, the HDD should not be spinning due to noise concerns.
Now the longer version: I have built a completely silent computer running Xubuntu. It has a A10-6700T APU, huge fanless cooler, fanless PSU, SSD. The problem is: it also has (and needs) a noisy HDD and I want to forbid spinning it up during the night. All writes should be cached on the SSD, reads are not needed in the night.
Throughout every day, this computer will automatically download about 5 GB of data which will be retained for about a year, giving a total needed disk capacity of slightly less than 2 TB. This data is currently stored on a 3 TB noisy hard disk drive which is spinning day and night. Sometimes, I'll need to access some data from several months ago. However, most times I'll only need data from the last 14 days, which would fit on the SSD. Ideally, I'd like a transparent solution (all data on one filesystem) which caches all writes to the SSD, writing to the HDD only once a day. Reads would be served by the cache if they were still on the SSD, else the HDD would have to spin up.
I have tried bcache without much success (using cache_mode=writeback, writeback_running=0, writeback_delay=86400, sequential_cutoff=0, congested_write_threshold_us=0 – anything missing?) and I read about ZFS ZIL/L2ARC but I'm not sure I can achieve my goal with ZFS. Any pointers?
If all else fails, I will simply use some scripts to automatically copy files over to the big drive while deleting the oldest files from the SSD.
Edit: Cache really is the wrong term for what I wanted. It seems the original intended solution is not possible for me at this time. Thanks to everybody who contributed!
Best Answer
Thats not really how or why caching works - caching is meant to take advantage of the faster speed and better random access of the SSD or other memory, rather than minimising writes to the hdd. Caching is designed to maximise speed, not reduce write, by having a smaller, faster buffer backing a large, slow storage device.
In fact, bcache, which is part of the mainline linux kernel, will simply pass on sequential writes to the hdd, rather than passing it through the SSD since there's no performance advantage.
ZIL/L2Arc uses the SSD to store logs (in the case of ZIL) and commonly used file clusters (with L2Arc). ZIL speeds up synchronous writes by storing them till they're all ready to be written. L2Arc stores commonly accessed files on faster storage. Neither of these will let you do what you need.
Fusion drive also uses a SSD and a HDD transparently putting commonly used files on the SSD for faster reads, and HDDs for bulk storage of less often used files. It too doesn't let you use the SSD as scratch space, and back up files to the main drive every so often
What you're looking at isn't caching at all, but rather periodic backups of the system to the HDD. I'm guessing ou might be able to fudge something together with Aufs -the first branch on the HDD with larger files (so it takes precidence), and a branch on the SSD, then run a script that moves files from the SSD to the HDD periodically, while having it accessable from the same location. I've not tested this out yet, but unlike caching, periodically moving files and using aufs will likely be exactly what you want.
The simplest way to sort files by age is ls -tr - t sorts by time (newest to oldest) and r reverses the order. (If you have a specific range in mind
find . -mtime nworks great n as a specific number gives you files modified n days ago, -n gives you files modified in the last n days and + n gives you files modified n days ago). You can find some ideas on how to use it hereSince quietness is your real goal, you can check if your drive supports silent mode with
hdparm -M /dev/sda- this should output something likeacoustic = 254 (128=quiet ... 254=fast), in which case you can make your hard drive quieter with the commandhdparm -M 128 /dev/sda. Run the opposite commandhdparm -M 1254 /dev/sdawhen you need more speed.