randReadWrite: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32
...
randReadWrite/sync: (g=1): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32
...
fio-3.16
Starting 32 processes

randReadWrite: (groupid=0, jobs=16): err= 0: pid=312100: Sat Oct 10 00:18:17 2020
  read: IOPS=1515k, BW=11.6GiB/s (12.4GB/s)(44.8GiB/3875msec)
    slat (nsec): min=781, max=445947, avg=6676.08, stdev=16766.22
    clat (nsec): min=801, max=4118.6k, avg=222522.62, stdev=315539.15
     lat (usec): min=2, max=4458, avg=229.25, stdev=328.68
    clat percentiles (usec):
     |  1.00th=[   93],  5.00th=[   98], 10.00th=[  102], 20.00th=[  106],
     | 30.00th=[  110], 40.00th=[  113], 50.00th=[  116], 60.00th=[  120],
     | 70.00th=[  129], 80.00th=[  176], 90.00th=[  449], 95.00th=[  881],
     | 99.00th=[ 1778], 99.50th=[ 1958], 99.90th=[ 2212], 99.95th=[ 2311],
     | 99.99th=[ 2507]
   bw (  MiB/s): min= 1344, max=20366, per=79.37%, avg=9392.96, stdev=489.16, samples=112
   iops        : min=172118, max=2606889, avg=1202292.86, stdev=62611.91, samples=112
  write: IOPS=650k, BW=5078MiB/s (5325MB/s)(19.2GiB/3875msec); 0 zone resets
    slat (nsec): min=1002, max=161664, avg=4324.36, stdev=3499.69
    clat (nsec): min=832, max=4383.1k, avg=222437.88, stdev=315227.32
     lat (usec): min=3, max=4421, avg=226.82, stdev=317.02
    clat percentiles (usec):
     |  1.00th=[   93],  5.00th=[   99], 10.00th=[  102], 20.00th=[  106],
     | 30.00th=[  110], 40.00th=[  113], 50.00th=[  116], 60.00th=[  120],
     | 70.00th=[  129], 80.00th=[  176], 90.00th=[  449], 95.00th=[  881],
     | 99.00th=[ 1778], 99.50th=[ 1958], 99.90th=[ 2212], 99.95th=[ 2311],
     | 99.99th=[ 2507]
   bw (  MiB/s): min=  581, max= 8731, per=79.39%, avg=4031.70, stdev=209.98, samples=112
   iops        : min=74390, max=1117684, avg=516050.71, stdev=26877.05, samples=112
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=7.38%, 250=77.19%, 500=6.43%, 750=3.09%, 1000=1.59%
  lat (msec)   : 2=3.91%, 4=0.42%, 10=0.01%
  cpu          : usr=16.11%, sys=66.94%, ctx=368256, majf=0, minf=231
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=5869812,2518796,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
randReadWrite/sync: (groupid=1, jobs=16): err= 0: pid=359938: Sat Oct 10 00:18:17 2020
  read: IOPS=42.6k, BW=333MiB/s (349MB/s)(38.0GiB/120003msec)
    slat (nsec): min=1613, max=15273k, avg=7565.89, stdev=28584.51
    clat (nsec): min=1393, max=182314k, avg=8131310.76, stdev=4270955.18
     lat (usec): min=6, max=182319, avg=8139.00, stdev=4273.36
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    7], 10.00th=[    7], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    8], 80.00th=[    9], 90.00th=[    9], 95.00th=[   11],
     | 99.00th=[   20], 99.50th=[   30], 99.90th=[   85], 99.95th=[  102],
     | 99.99th=[  126]
   bw (  KiB/s): min=125856, max=403904, per=99.96%, avg=340611.70, stdev=3651.41, samples=3831
   iops        : min=15732, max=50488, avg=42576.25, stdev=456.43, samples=3831
  write: IOPS=18.2k, BW=142MiB/s (149MB/s)(16.7GiB/120003msec); 0 zone resets
    slat (usec): min=2, max=3766, avg=21.00, stdev=14.60
    clat (usec): min=245, max=181827, avg=8206.89, stdev=4313.65
     lat (usec): min=266, max=181841, avg=8228.02, stdev=4312.57
    clat percentiles (msec):
     |  1.00th=[    7],  5.00th=[    7], 10.00th=[    7], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    8], 60.00th=[    8],
     | 70.00th=[    9], 80.00th=[    9], 90.00th=[   10], 95.00th=[   11],
     | 99.00th=[   21], 99.50th=[   30], 99.90th=[   86], 99.95th=[  102],
     | 99.99th=[  126]
   bw (  KiB/s): min=53499, max=176384, per=99.97%, avg=145866.30, stdev=1583.69, samples=3831
   iops        : min= 6687, max=22048, avg=18232.99, stdev=197.97, samples=3831
  lat (usec)   : 2=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=94.05%, 20=4.96%, 50=0.79%
  lat (msec)   : 100=0.14%, 250=0.05%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=20, max=1100.3k, avg=102.25, stdev=775.61
    sync percentiles (nsec):
     |  1.00th=[   50],  5.00th=[   60], 10.00th=[   60], 20.00th=[   70],
     | 30.00th=[   70], 40.00th=[   80], 50.00th=[   80], 60.00th=[   90],
     | 70.00th=[  100], 80.00th=[  120], 90.00th=[  161], 95.00th=[  211],
     | 99.00th=[  290], 99.50th=[  350], 99.90th=[  628], 99.95th=[  772],
     | 99.99th=[ 3792]
  cpu          : usr=1.47%, sys=34.49%, ctx=30054527, majf=0, minf=279
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=200.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=5111123,2188608,0,7299249 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=11.6GiB/s (12.4GB/s), 11.6GiB/s-11.6GiB/s (12.4GB/s-12.4GB/s), io=44.8GiB (48.1GB), run=3875-3875msec
  WRITE: bw=5078MiB/s (5325MB/s), 5078MiB/s-5078MiB/s (5325MB/s-5325MB/s), io=19.2GiB (20.6GB), run=3875-3875msec

Run status group 1 (all jobs):
   READ: bw=333MiB/s (349MB/s), 333MiB/s-333MiB/s (349MB/s-349MB/s), io=38.0GiB (41.9GB), run=120003-120003msec
  WRITE: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=16.7GiB (17.9GB), run=120003-120003msec