randReadWrite: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64
...
randReadWrite/sync: (g=1): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=64
...
fio-3.16
Starting 4 processes

randReadWrite: (groupid=0, jobs=2): err= 0: pid=2827376: Sat Oct 10 02:00:54 2020
  read: IOPS=18.1k, BW=142MiB/s (149MB/s)(140GiB/1012161msec)
    slat (nsec): min=1042, max=5184.3M, avg=103638.06, stdev=7030847.75
    clat (usec): min=2, max=7279.0k, avg=4856.72, stdev=64335.13
     lat (usec): min=28, max=7279.2k, avg=4960.57, stdev=65085.08
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    6], 95.00th=[    6],
     | 99.00th=[    7], 99.50th=[    8], 99.90th=[   11], 99.95th=[   14],
     | 99.99th=[ 4799]
   bw (  KiB/s): min=   24, max=253185, per=100.00%, avg=146582.42, stdev=20914.68, samples=3551
   iops        : min=    2, max=31648, avg=18321.94, stdev=2614.35, samples=3551
  write: IOPS=7771, BW=60.7MiB/s (63.7MB/s)(60.0GiB/1012161msec); 0 zone resets
    slat (nsec): min=1252, max=1339.9k, avg=6395.14, stdev=2123.35
    clat (usec): min=2, max=7279.0k, avg=4872.19, stdev=64964.15
     lat (usec): min=7, max=7279.0k, avg=4878.72, stdev=64964.20
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    4], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    6], 95.00th=[    6],
     | 99.00th=[    7], 99.50th=[    8], 99.90th=[   11], 99.95th=[   14],
     | 99.99th=[ 4799]
   bw (  KiB/s): min=   25, max=109907, per=100.00%, avg=63458.06, stdev=8468.22, samples=3516
   iops        : min=    2, max=13738, avg=7931.39, stdev=1058.54, samples=3516
  lat (usec)   : 4=0.01%, 20=0.01%, 50=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.88%, 4=47.62%, 10=51.37%, 20=0.10%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2000=0.01%, >=2000=0.01%
  cpu          : usr=3.92%, sys=10.08%, ctx=10932552, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=18348730,7865670,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
randReadWrite/sync: (groupid=1, jobs=2): err= 0: pid=1312314: Sat Oct 10 02:00:54 2020
  read: IOPS=5347, BW=41.8MiB/s (43.8MB/s)(140GiB/3431769msec)
    slat (nsec): min=1452, max=21304k, avg=145312.20, stdev=182136.92
    clat (usec): min=2, max=128830, avg=16396.39, stdev=3795.23
     lat (usec): min=262, max=129149, avg=16541.98, stdev=3824.74
    clat percentiles (usec):
     |  1.00th=[11207],  5.00th=[12387], 10.00th=[13042], 20.00th=[13829],
     | 30.00th=[14484], 40.00th=[15008], 50.00th=[15533], 60.00th=[16188],
     | 70.00th=[16909], 80.00th=[18220], 90.00th=[20841], 95.00th=[22938],
     | 99.00th=[29754], 99.50th=[36439], 99.90th=[46400], 99.95th=[50070],
     | 99.99th=[59507]
   bw (  KiB/s): min=24496, max=60208, per=100.00%, avg=42787.33, stdev=2936.27, samples=13723
   iops        : min= 3062, max= 7526, avg=5348.37, stdev=367.03, samples=13723
  write: IOPS=2290, BW=17.9MiB/s (18.8MB/s)(59.0GiB/3431769msec); 0 zone resets
    slat (usec): min=3, max=5070, avg= 9.31, stdev= 4.30
    clat (usec): min=569, max=128929, avg=16714.72, stdev=3831.48
     lat (usec): min=580, max=128940, avg=16724.21, stdev=3831.93
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12649], 10.00th=[13304], 20.00th=[14091],
     | 30.00th=[14746], 40.00th=[15270], 50.00th=[15926], 60.00th=[16581],
     | 70.00th=[17433], 80.00th=[18482], 90.00th=[21103], 95.00th=[23462],
     | 99.00th=[30278], 99.50th=[36963], 99.90th=[46924], 99.95th=[51119],
     | 99.99th=[60031]
   bw (  KiB/s): min=10848, max=26448, per=100.00%, avg=18327.40, stdev=1265.63, samples=13723
   iops        : min= 1356, max= 3306, avg=2290.87, stdev=158.20, samples=13723
  lat (usec)   : 4=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.09%, 20=87.18%, 50=12.67%
  lat (msec)   : 100=0.05%, 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=20, max=155352, avg=110.54, stdev=96.17
    sync percentiles (nsec):
     |  1.00th=[   70],  5.00th=[   80], 10.00th=[   80], 20.00th=[   90],
     | 30.00th=[   90], 40.00th=[   90], 50.00th=[  100], 60.00th=[  100],
     | 70.00th=[  110], 80.00th=[  131], 90.00th=[  151], 95.00th=[  191],
     | 99.00th=[  262], 99.50th=[  310], 99.90th=[  470], 99.95th=[  580],
     | 99.99th=[ 1128]
  cpu          : usr=2.08%, sys=13.91%, ctx=53866394, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=200.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=18353047,7861353,0,26214274 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=140GiB (150GB), run=1012161-1012161msec
  WRITE: bw=60.7MiB/s (63.7MB/s), 60.7MiB/s-60.7MiB/s (63.7MB/s-63.7MB/s), io=60.0GiB (64.4GB), run=1012161-1012161msec

Run status group 1 (all jobs):
   READ: bw=41.8MiB/s (43.8MB/s), 41.8MiB/s-41.8MiB/s (43.8MB/s-43.8MB/s), io=140GiB (150GB), run=3431769-3431769msec
  WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=59.0GiB (64.4GB), run=3431769-3431769msec