nguvu

VPS disk performance, Digital Ocean vs Linode part II

2 February, 2014

Curiosity drove me to return to disk benchmarks today to run some more tests on Digital Ocean vs Lionde’s systems. The numbers didn’t fit with my expectations and I wanted to understand why there was a discrepancy, was it just a bad day at Digital Ocean? Time to dig a little deeper…..

I/O IOPings

Digital Ocean

# ioping /tmp -c 10
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=1 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=2 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=3 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=4 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=5 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=6 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=7 time=0.2 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=8 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=9 time=0.3 ms
4096 bytes from /tmp (ext4 /dev/disk/by-label/DOROOT): request=10 time=0.3 ms

Linode

# ioping /tmp -c 10
4096 bytes from /tmp (ext3 /dev/root): request=1 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=2 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=3 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=4 time=0.1 ms
4096 bytes from /tmp (ext3 /dev/root): request=5 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=6 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=7 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=8 time=0.2 ms
4096 bytes from /tmp (ext3 /dev/root): request=9 time=0.8 ms
4096 bytes from /tmp (ext3 /dev/root): request=10 time=0.2 ms

I/O Seek Test (no cache)

Digital Ocean

# ioping /tmp -RD
--- /tmp (ext4 /dev/disk/by-label/DOROOT) ioping statistics ---
11516 requests completed in 3000.1 ms, 8725 iops, 34.1 mb/s
min/avg/max/mdev = 0.1/0.1/12.0/0.2 ms

Linode

# ioping /tmp -RD
--- /tmp (ext3 /dev/root) ioping statistics ---
7757 requests completed in 3000.3 ms, 5002 iops, 19.5 mb/s
min/avg/max/mdev = 0.1/0.2/87.1/2.0 ms

I/O Reads (cached)

Digital Ocean

# ioping /tmp -RC
--- /tmp (ext4 /dev/disk/by-label/DOROOT) ioping statistics ---
26603 requests completed in 3000.0 ms, 156733 iops, 612.2 mb/s
min/avg/max/mdev = 0.0/0.0/1.6/0.0 ms

Linode

# ioping /tmp -RC
--- /tmp (ext3 /dev/root) ioping statistics ---
16200 requests completed in 3000.1 ms, 93285 iops, 364.4 mb/s
min/avg/max/mdev = 0.0/0.0/0.0/0.0 ms

I/O Reads - Sequential

Digital Ocean

# ioping /tmp -RL
--- /tmp (ext4 /dev/disk/by-label/DOROOT) ioping statistics ---
4826 requests completed in 3000.1 ms, 2332 iops, 582.9 mb/s
min/avg/max/mdev = 0.3/0.4/12.6/0.3 ms

Linode

# ioping /tmp -RL
--- /tmp (ext3 /dev/root) ioping statistics ---
5162 requests completed in 3000.3 ms, 2677 iops, 669.1 mb/s
min/avg/max/mdev = 0.2/0.4/1.5/0.2 ms

FIO - Random read test, queue depth=1.

Digital Ocean

# cat random-read-test.fio
[random-read]
rw=randread
size=128m
directory=/tmp/fio-test/data
# fio random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 1.59
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 128MB)
Jobs: 1 (f=1): [r] [100.0% done] [13355K/0K /s] [3260 /0  iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=29086
  read : io=131072KB, bw=16419KB/s, iops=4104 , runt=  7983msec
clat (usec): min=72 , max=24936 , avg=233.35, stdev=744.50
 lat (usec): min=72 , max=24937 , avg=233.68, stdev=744.51
bw (KB/s) : min=12570, max=26088, per=101.13%, avg=16602.93, stdev=4092.67
  cpu          : usr=4.51%, sys=47.01%, ctx=9367, majf=1, minf=23
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued r/w/d: total=32768/0/0, short=0/0/0
 lat (usec): 100=2.88%, 250=92.12%, 500=2.27%, 750=0.29%, 1000=0.22%
 lat (msec): 2=0.38%, 4=0.69%, 10=1.04%, 20=0.10%, 50=0.01%

Run status group 0 (all jobs):
READ: io=131072KB, aggrb=16418KB/s, minb=16812KB/s, maxb=16812KB/s, mint=7983msec, maxt=7983msec

Disk stats (read/write):
vda: ios=32396/5, merge=0/27, ticks=4200/0, in_queue=4164, util=51.99%

Linode

# cat random-read-test.fio
[random-read]
rw=randread
size=128m
directory=/tmp/fio-test/data
# fio random-read-test.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=sync, iodepth=1
fio 1.59
Starting 1 process
random-read: Laying out IO file(s) (1 file(s) / 128MB)
Jobs: 1 (f=1): [r] [100.0% done] [32890K/0K /s] [8030 /0  iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=4115
  read : io=131072KB, bw=32316KB/s, iops=8078 , runt=  4056msec
clat (usec): min=62 , max=413 , avg=114.31, stdev=17.83
 lat (usec): min=63 , max=414 , avg=115.62, stdev=17.81
bw (KB/s) : min=31536, max=32880, per=100.03%, avg=32326.00, stdev=440.52
  cpu          : usr=0.00%, sys=27.10%, ctx=32770, majf=0, minf=24
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued r/w/d: total=32768/0/0, short=0/0/0
 lat (usec): 100=10.91%, 250=89.06%, 500=0.04%

Run status group 0 (all jobs):
READ: io=131072KB, aggrb=32315KB/s, minb=33091KB/s, maxb=33091KB/s, mint=4056msec, maxt=4056msec

Disk stats (read/write):
xvda: ios=32379/0, merge=0/0, ticks=3344/0, in_queue=3343, util=81.46%   

Summary of 1 deep queue testing

Test @ Queue Depth = 1 Digital Ocean Linode
Completion Latency (usec) 233 114
Bandwidth (MB/s) 16.2 31.57
IO request complete @ 250usec 95.00% 99.97%
CPU utilisation 52% 27%

FIO - Random read test, queue depth=8.

I performed the same test again, but this time using asynchronous IO subsystems to issue up to 8 requests simultaneously. I wanted to put more pressure on Linodes systems due to the higher latencies associated with rotating disks.

Digital Ocean

[random-read]
rw=randread
size=128m
directory=/tmp
ioengine=libaio
iodepth=8
direct=1
invalidate=1
# fio rand-read-test-aio.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio 1.59
Starting 1 process
Jobs: 1 (f=1): [r] [-.-% done] [64877K/0K /s] [15.9K/0  iops] [eta 00m:00s]
random-read: (groupid=0, jobs=1): err= 0: pid=3288
  read : io=131072KB, bw=62386KB/s, iops=15596 , runt=  2101msec
slat (usec): min=6 , max=4902 , avg=32.49, stdev=52.94
clat (usec): min=80 , max=5270 , avg=476.55, stdev=226.08
 lat (usec): min=186 , max=5291 , avg=510.26, stdev=209.81
bw (KB/s) : min=60408, max=63512, per=100.20%, avg=62512.00, stdev=1419.21
  cpu          : usr=9.52%, sys=53.33%, ctx=3319, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued r/w/d: total=32768/0/0, short=0/0/0
 lat (usec): 100=8.39%, 250=6.87%, 500=39.42%, 750=37.18%, 1000=7.12%
 lat (msec): 2=0.99%, 4=0.01%, 10=0.04%

Run status group 0 (all jobs):
READ: io=131072KB, aggrb=62385KB/s, minb=63882KB/s, maxb=63882KB/s, mint=2101msec, maxt=2101msec

Disk stats (read/write):
vda: ios=29769/0, merge=0/0, ticks=7424/0, in_queue=7392, util=78.60% 

Linode

# cat rand-read-test-aio.fio
[random-read]
rw=randread
size=128m
directory=/tmp
ioengine=libaio
iodepth=8
direct=1
invalidate=1
# fio rand-read-test-aio.fio
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=8
fio 1.59
Starting 1 process

random-read: (groupid=0, jobs=1): err= 0: pid=4196
  read : io=131072KB, bw=181792KB/s, iops=45447 , runt=   721msec
slat (usec): min=6 , max=67 , avg= 9.19, stdev= 3.38
clat (usec): min=67 , max=3870 , avg=162.13, stdev=68.84
 lat (usec): min=82 , max=3884 , avg=172.48, stdev=68.91
bw (KB/s) : min=177560, max=177560, per=97.67%, avg=177560.00, stdev= 0.00
  cpu          : usr=23.47%, sys=56.39%, ctx=4750, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued r/w/d: total=32768/0/0, short=0/0/0
 lat (usec): 100=9.11%, 250=83.39%, 500=7.45%, 750=0.01%, 1000=0.01%
 lat (msec): 2=0.02%, 4=0.03%

Run status group 0 (all jobs):
   READ: io=131072KB, aggrb=181791KB/s, minb=186154KB/s, maxb=186154KB/s, mint=721msec, maxt=721msec

Disk stats (read/write):
  xvda: ios=29033/0, merge=0/0, ticks=4487/0, in_queue=4467, util=84.79%

Summary of 8 deep queue testing

Test @ Queue Depth = 8 Digital Ocean Linode
Completion Latency (usec) 476 162
Bandwidth (MB/s) 61.0 173.4
IO request complete @ 250usec 15.26% 92.5%
CPU utilisation 63% 80%

I also tested with 32 deep IO queues to see how the throughput would change. Digital Ocean stalled at around 80MB/s at 60% CPU where Linodes system rose to 240MB/s by which point it was reading 99% CPU utilisation - thats 300% Digital Oceans disk bandwidth!

Summary

Its apparent that Linodes disk systems beats Digital Oceans in just about every area and not by a small margin. The results from multiple tests across both serial and random nature operations show the results clearly. Closer analysis of the data via ‘fio’ highlights that Digital Oceans systems are struggling to deal with all disk operations in a timely manner, the range of timings between the quickest and slowest operations is killing the overall bandwidth. Linode on the other hand serve all requests quickly and as such hold together better under strain of these tests. What exactly causes the large deviation I can only guess but my suspicions are its server loading possibly as a result of over selling to enable the low prices. I don’t know if this is right but if it is, it will be interesting to see how Digital Oceans performance fares in the future as they are definitely the hot new kid on the block, have a great marketing program and are hoovering up lots of new accounts on a daily basis. As always, it pays to look beyond marketing, Digital Oceans claim of “Blazing Fast 20GB SSD” might not be entirely misleading, especially compared to Amazons performance, but Linode’s old fashioned rotating platters have them bested right now.