I've long been meaning to use a spare SSD as a cache for my spinning hard drives but I've been put off by the lack of good integration of SSD caching solutions by default on Ubuntu.
I've tried enhanceio in the past, but it seems to have gone unmaintained so today I'm going to give bcache a try.
$ sudo lvcreate -n jaja -l100%PVS vg0 /dev/sda1:0-51196 /dev/sdb1:0-51196 /dev/sde1:0-51196 /dev/sdf1:0-51196
Logical volume "jaja" created
I benchmarked the resulting volume using disks, just to get a rough idea of the throughput level:
Now that we have the LVM volume, let's make a filesystem on it. I'm gonna use ext4:
$ sudo mkfs.ext4 /dev/vg0/jaja
mke2fs 1.42.9 (4-Feb-2014)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
52428800 inodes, 209702912 blocks
10485145 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
6400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Let's see what the performance is like while copying some 8G of documents on the new filesystem. For that, we'll run the copy operation 3 times and drop the memory caches in between the runs:
$ echo 2 | sudo tee /proc/sys/vm/drop_caches
2
$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m59.823s
user 0m0.279s
sys 0m12.332s
$ time rm -r /media/shock/JAJA/test-files/
real 0m0.601s
user 0m0.006s
sys 0m0.590s
$ echo 2 | sudo tee /proc/sys/vm/drop_caches
2
$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m59.669s
user 0m0.270s
sys 0m12.152s
$ time rm -r /media/shock/JAJA/test-files/
real 0m0.598s
user 0m0.006s
sys 0m0.589s
$ echo 2 | sudo tee /proc/sys/vm/drop_caches
2
$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m59.023s
user 0m0.259s
sys 0m11.467s
$ free -h
total used free shared buffers cached
Mem: 15G 15G 184M 52M 13M 9.3G
-/+ buffers/cache: 6.1G 9.5G
Now we're gonna start using bcache.
$ sudo make-bcache -B /dev/vg0/jaja
UUID: c51f533d-c09a-4f6e-ab7f-b1527e0dd844
Set UUID: e0278651-343a-4a66-8690-7e0f40a7456d
version: 1
block_size: 1
data_offset: 16
$ sudo make-bcache -C /dev/sdd2
UUID: 396ac4ca-68a1-4a41-b2ff-631382a47baa
Set UUID: c4d1415e-d3d0-43bd-b101-2c5a39735bfa
version: 0
nbuckets: 131072
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1
$ cat /sys/block/dm-1/bcache/state
no cache
$ cat /sys/block/bcache0/bcache/state
no cache
$ echo c4d1415e-d3d0-43bd-b101-2c5a39735bfa | sudo tee /sys/block/bcache0/bcache/attach
c4d1415e-d3d0-43bd-b101-2c5a39735bfa
$ cat /sys/block/dm-1/bcache/state
clean
$ cat /sys/block/bcache0/bcache/state
clean
$ sudo mkfs.ext4 -L JAJA /dev/bcache0
mke2fs 1.42.9 (4-Feb-2014)
Discarding device blocks: done
Filesystem label=JAJA
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
52428800 inodes, 209702910 blocks
10485145 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
6400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
In this configuration we wouldn't see a significant difference in our copy benchmark, because writes are not cached, as you can see by issueing:
$ cat /sys/block/bcache0/bcache/cache_mode
[writethrough] writeback writearound none
but let's test that anyway:
$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m59.325s
user 0m0.276s
sys 0m12.301s
Now let's enable write caching and test with that:
$ echo writeback |sudo tee /sys/block/bcache0/bcache/cache_mode
[sudo] password for shock:
writeback
shock@shock-desktop:~$ cat /sys/block/bcache0/bcache/cache_mode
writethrough [writeback] writearound none
shock@shock-desktop:~$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m57.692s
user 0m0.286s
sys 0m12.820s
shock@shock-desktop:~$ time rm -r /media/shock/JAJA/test-files/
real 0m0.627s
user 0m0.006s
sys 0m0.617s
shock@shock-desktop:~$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m58.577s
user 0m0.293s
sys 0m13.035s
shock@shock-desktop:~$ cat /sys/fs/bcache/c4d1415e-d3d0-43bd-b101-2c5a39735bfa/cache_available_percent
100
shock@shock-desktop:~$ echo 0 | sudo tee /sys/block/bcache0/bcache/sequential_cutoff
0
shock@shock-desktop:~$ time rm -r /media/shock/JAJA/test-files/
real 0m0.618s
user 0m0.006s
sys 0m0.608s
shock@shock-desktop:~$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m15.007s
user 0m0.271s
sys 0m12.409s
shock@shock-desktop:~$ cat /sys/fs/bcache/c4d1415e-d3d0-43bd-b101-2c5a39735bfa/cache_available_percent
100
shock@shock-desktop:~$ time rm -r /media/shock/JAJA/test-files/
real 0m0.619s
user 0m0.006s
sys 0m0.609s
shock@shock-desktop:~$ time cp -R /media/shock/test-files/ /media/shock/JAJA/
real 1m15.086s
user 0m0.272s
sys 0m12.319s
Tip
whww