Curiosity drove me to return to disk benchmarks today to run some more tests on Digital Ocean vs Lionde’s systems. The numbers didn’t fit with my expectations and I wanted to understand why there was a discrepancy, was it just a bad day at Digital Ocean? Time to dig a little deeper…..
# ioping /tmp -c 10
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=1 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=2 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=3 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=4 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=5 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=6 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=7 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=8 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=9 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=10 time=0.3 ms
# ioping /tmp -c 10
4096 bytes from /tmp (ext3 /dev/root): request=1 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=2 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=3 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=4 time=0.1 ms
4096 bytes from /tmp (ext3 /dev/root): request=5 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=6 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=7 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=8 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=9 time=0.8 ms
4096 bytes from /tmp (ext3 /dev/root): request=10 time=0.2 ms
# ioping /tmp -RD
--- /tmp (ext4 /dev/disk/by-label/DOROOT) ioping statistics ---
11516 requests completed in 3000.1 ms, 8725 iops, 34.1 mb/s
min/avg/max/mdev = 0.1/0.1/12.0/0.2 ms
# ioping /tmp -RD
--- /tmp (ext3 /dev/root) ioping statistics ---
7757 requests completed in 3000.3 ms, 5002 iops, 19.5 mb/s
min/avg/max/mdev = 0.1/0.2/87.1/2.0 ms
# ioping /tmp -RC
--- /tmp (ext4 /dev/disk/by-label/DOROOT) ioping statistics ---
26603 requests completed in 3000.0 ms, 156733 iops, 612.2 mb/s
min/avg/max/mdev = 0.0/0.0/1.6/0.0 ms
# ioping /tmp -RC
--- /tmp (ext3 /dev/root) ioping statistics ---
16200 requests completed in 3000.1 ms, 93285 iops, 364.4 mb/s
min/avg/max/mdev = 0.0/0.0/0.0/0.0 ms
# ioping /tmp -RL
--- /tmp (ext4 /dev/disk/by-label/DOROOT) ioping statistics ---
4826 requests completed in 3000.1 ms, 2332 iops, 582.9 mb/s
min/avg/max/mdev = 0.3/0.4/12.6/0.3 ms
# ioping /tmp -RL
--- /tmp (ext3 /dev/root) ioping statistics ---
5162 requests completed in 3000.3 ms, 2677 iops, 669.1 mb/s
min/avg/max/mdev = 0.2/0.4/1.5/0.2 ms
# cat random-read-test.fio
[random-read]
rw=randread
size=128m
directory=/tmp/fio-test/data
# fio random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 1.59
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 128MB)
Jobs: 1 (f=1): [r] [100.0% done] [13355K/0K /s] [3260 /0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=29086
read : io=131072KB, bw=16419KB/s, iops=4104 , runt= 7983msec
clat (usec): min=72 , max=24936 , avg=233.35, stdev=744.50
lat (usec): min=72 , max=24937 , avg=233.68, stdev=744.51
bw (KB/s) : min=12570, max=26088, per=101.13%, avg=16602.93, stdev=4092.67
cpu : usr=4.51%, sys=47.01%, ctx=9367, majf=1, minf=23
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=32768/0/0, short=0/0/0
lat (usec): 100=2.88%, 250=92.12%, 500=2.27%, 750=0.29%, 1000=0.22%
lat (msec): 2=0.38%, 4=0.69%, 10=1.04%, 20=0.10%, 50=0.01%
Run status group 0 (all jobs):
READ: io=131072KB, aggrb=16418KB/s, minb=16812KB/s, maxb=16812KB/s, mint=7983msec, maxt=7983msec
Disk stats (read/write):
vda: ios=32396/5, merge=0/27, ticks=4200/0, in_queue=4164, util=51.99%
# cat random-read-test.fio
[random-read]
rw=randread
size=128m
directory=/tmp/fio-test/data
# fio random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 1.59
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 128MB)
Jobs: 1 (f=1): [r] [100.0% done] [32890K/0K /s] [8030 /0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=4115
read : io=131072KB, bw=32316KB/s, iops=8078 , runt= 4056msec
clat (usec): min=62 , max=413 , avg=114.31, stdev=17.83
lat (usec): min=63 , max=414 , avg=115.62, stdev=17.81
bw (KB/s) : min=31536, max=32880, per=100.03%, avg=32326.00, stdev=440.52
cpu : usr=0.00%, sys=27.10%, ctx=32770, majf=0, minf=24
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=32768/0/0, short=0/0/0
lat (usec): 100=10.91%, 250=89.06%, 500=0.04%
Run status group 0 (all jobs):
READ: io=131072KB, aggrb=32315KB/s, minb=33091KB/s, maxb=33091KB/s, mint=4056msec, maxt=4056msec
Disk stats (read/write):
xvda: ios=32379/0, merge=0/0, ticks=3344/0, in_queue=3343, util=81.46%
| Test @ Queue Depth = 1 | Digital Ocean | Linode |
|---|---|---|
| Completion Latency (usec) | 233 | 114 |
| Bandwidth (MB/s) | 16.2 | 31.57 |
| IO request complete @ 250usec | 95.00% | 99.97% |
| CPU utilisation | 52% | 27% |
I performed the same test again, but this time using asynchronous IO subsystems to issue up to 8 requests simultaneously. I wanted to put more pressure on Linodes systems due to the higher latencies associated with rotating disks.
[random-read]
rw=randread
size=128m
directory=/tmp
ioengine=libaio
iodepth=8
direct=1
invalidate=1
# fio rand-read-test-aio.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio 1.59
Starting 1 process
Jobs: 1 (f=1): [r] [-.-% done] [64877K/0K /s] [15.9K/0 iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=3288
read : io=131072KB, bw=62386KB/s, iops=15596 , runt= 2101msec
slat (usec): min=6 , max=4902 , avg=32.49, stdev=52.94
clat (usec): min=80 , max=5270 , avg=476.55, stdev=226.08
lat (usec): min=186 , max=5291 , avg=510.26, stdev=209.81
bw (KB/s) : min=60408, max=63512, per=100.20%, avg=62512.00, stdev=1419.21
cpu : usr=9.52%, sys=53.33%, ctx=3319, majf=0, minf=29
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=32768/0/0, short=0/0/0
lat (usec): 100=8.39%, 250=6.87%, 500=39.42%, 750=37.18%, 1000=7.12%
lat (msec): 2=0.99%, 4=0.01%, 10=0.04%
Run status group 0 (all jobs):
READ: io=131072KB, aggrb=62385KB/s, minb=63882KB/s, maxb=63882KB/s, mint=2101msec, maxt=2101msec
Disk stats (read/write):
vda: ios=29769/0, merge=0/0, ticks=7424/0, in_queue=7392, util=78.60%
# cat rand-read-test-aio.fio
[random-read]
rw=randread
size=128m
directory=/tmp
ioengine=libaio
iodepth=8
direct=1
invalidate=1
# fio rand-read-test-aio.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio 1.59
Starting 1 process
random-read: (groupid=0, jobs=1): err= 0: pid=4196
read : io=131072KB, bw=181792KB/s, iops=45447 , runt= 721msec
slat (usec): min=6 , max=67 , avg= 9.19, stdev= 3.38
clat (usec): min=67 , max=3870 , avg=162.13, stdev=68.84
lat (usec): min=82 , max=3884 , avg=172.48, stdev=68.91
bw (KB/s) : min=177560, max=177560, per=97.67%, avg=177560.00, stdev= 0.00
cpu : usr=23.47%, sys=56.39%, ctx=4750, majf=0, minf=29
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=32768/0/0, short=0/0/0
lat (usec): 100=9.11%, 250=83.39%, 500=7.45%, 750=0.01%, 1000=0.01%
lat (msec): 2=0.02%, 4=0.03%
Run status group 0 (all jobs):
READ: io=131072KB, aggrb=181791KB/s, minb=186154KB/s, maxb=186154KB/s, mint=721msec, maxt=721msec
Disk stats (read/write):
xvda: ios=29033/0, merge=0/0, ticks=4487/0, in_queue=4467, util=84.79%
| Test @ Queue Depth = 8 | Digital Ocean | Linode |
|---|---|---|
| Completion Latency (usec) | 476 | 162 |
| Bandwidth (MB/s) | 61.0 | 173.4 |
| IO request complete @ 250usec | 15.26% | 92.5% |
| CPU utilisation | 63% | 80% |
I also tested with 32 deep IO queues to see how the throughput would change. Digital Ocean stalled at around 80MB/s at 60% CPU where Linodes system rose to 240MB/s by which point it was reading 99% CPU utilisation - thats 300% Digital Oceans disk bandwidth!
Its apparent that Linodes disk systems beats Digital Oceans in just about every area and not by a small margin. The results from multiple tests across both serial and random nature operations show the results clearly. Closer analysis of the data via ‘fio’ highlights that Digital Oceans systems are struggling to deal with all disk operations in a timely manner, the range of timings between the quickest and slowest operations is killing the overall bandwidth. Linode on the other hand serve all requests quickly and as such hold together better under strain of these tests. What exactly causes the large deviation I can only guess but my suspicions are its server loading possibly as a result of over selling to enable the low prices. I don’t know if this is right but if it is, it will be interesting to see how Digital Oceans performance fares in the future as they are definitely the hot new kid on the block, have a great marketing program and are hoovering up lots of new accounts on a daily basis. As always, it pays to look beyond marketing, Digital Oceans claim of “Blazing Fast 20GB SSD” might not be entirely misleading, especially compared to Amazons performance, but Linode’s old fashioned rotating platters have them bested right now.