SSD vs. TB benchmarks

testing 1 NVME, the Adata SX8200, against a RAID5 array of 7 old HDDs (WD Red 3GB).

not sure which settings or stats to focus on here. all 4k randread, with varying depth queue (DQ) and test size.

DQ1 1GB (10x advantage to NVME)

1. IOPS: NVME 12k RAID5 1.1k
2. bandwidth: NVME 50MB/s RAID5 5MB/s

DQ1 latency test 256MB (70x advantage to NVME)

1. NVME 189k RAID5 2.7k
2. NVME 775MB/s RAID5 11MB/s

DQ64 1GB (100x to NVME)

1. IOPS: NVME 297k RAID5 2.6k
2. bandwidth: NVME 1200MB/s RAID5 11MB/s

DQ64 16GB (170x to NVME)

1. NVME 288k RAID5 1.7k
2. NVME 1200MB/s RAID5 7.1MB/s

4k randread depth queue 1, 1GB

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read.fio --bs=4k --iodepth=1 --size=1G --readwrite=randread

for the NVME (Adata SX8200)

test: (groupid=0, jobs=1): err= 0: pid=3056070: Sun Feb 16 21:50:12 2020
  read: IOPS=11.9k, BW=46.7MiB/s (48.9MB/s)(256MiB/5484msec)
   bw (  KiB/s): min=44376, max=50696, per=100.00%, avg=48058.30, stdev=2340.39, samples=10
   iops        : min=11094, max=12674, avg=12014.50, stdev=585.14, samples=10

 for the RAID5 (7 HDDs)

 test: (groupid=0, jobs=1): err= 0: pid=3055955: Sun Feb 16 21:50:02 2020
  read: IOPS=1162, BW=4650KiB/s (4762kB/s)(256MiB/56370msec)
   bw (  KiB/s): min= 1261, max=14640, per=99.54%, avg=4628.40, stdev=1391.54, samples=112
   iops        : min=  315, max= 3660, avg=1157.10, stdev=347.89, samples=112

4k randread depth queue 1, 256MB latency test

fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=random_read.fio --bs=4k --iodepth=1 --size=256MB --readwrite=randread

for the NVME (Adata SX8200)

  read: IOPS=189k, BW=738MiB/s (774MB/s)(256MiB/347msec)
    clat (nsec): min=180, max=34881, avg=210.56, stdev=139.

for the RAID5 (7 HDDs)

   read: IOPS=2706, BW=10.6MiB/s (11.1MB/s)(256MiB/24213msec)
    clat (usec): min=54, max=88073, avg=365.22, stdev=1524.43

4k randread depth queue 64, 1GB

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read.fio --bs=4k --iodepth=64 --size=1G --readwrite=randread

for the NVME (Adata SX8200)

 test: (groupid=0, jobs=1): err= 0: pid=3053833: Sun Feb 16 21:31:07 2020
  read: IOPS=297k, BW=1160MiB/s (1216MB/s)(1024MiB/883msec)
   bw (  MiB/s): min= 1141, max= 1141, per=98.43%, avg=1141.44, stdev= 0.00, samples=1
   iops        : min=292209, max=292209, avg=292209.00, stdev= 0.00, samples=1

for the RAID5 (7 HDDs)

test: (groupid=0, jobs=1): err= 0: pid=3053935: Sun Feb 16 21:33:39 2020
  read: IOPS=2631, BW=10.3MiB/s (10.8MB/s)(1024MiB/99634msec)
   bw (  KiB/s): min= 6252, max=27320, per=99.30%, avg=10450.44, stdev=2091.69, samples=198
   iops        : min= 1563, max= 6830, avg=2612.61, stdev=522.92, samples=198

4k randread depth queue 64, 16GB

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read.fio --bs=4k --iodepth=64 --size=16G --readwrite=randread

 for the NVME (Adata SX8200)

for the NVME (Adata SX8200)
test: (groupid=0, jobs=1): err= 0: pid=3056599: Sun Feb 16 21:56:22 2020
  read: IOPS=288k, BW=1125MiB/s (1179MB/s)(16.0GiB/14568msec)
   bw (  MiB/s): min=  973, max= 1226, per=99.94%, avg=1124.02, stdev=74.61, samples=28
   iops        : min=249234, max=314076, avg=287749.50, stdev=19099.27, samples=28

for the RAID5 (7 HDDs)

 test: (groupid=0, jobs=1): err= 0: pid=3056790: Sun Feb 16 22:38:16 2020
  read: IOPS=1735, BW=6940KiB/s (7107kB/s)(16.0GiB/2417443msec)
   bw (  KiB/s): min= 3784, max=27408, per=100.00%, avg=6950.02, stdev=961.95, samples=4827
   iops        : min=  946, max= 6852, avg=1737.50, stdev=240.49, samples=4827