randReadWrite: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
randReadWrite/sync: (g=1): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
fio-3.12
Starting 2 processes

randReadWrite: (groupid=0, jobs=1): err= 0: pid=8630: Wed Apr  7 21:14:30 2021
  read: IOPS=11.0k, BW=93.6MiB/s (98.2MB/s)(10.0GiB/120001msec)
    slat (usec): min=52, max=3077, avg=80.43, stdev=32.37
    clat (nsec): min=1030, max=22744k, avg=876180.61, stdev=292953.48
     lat (usec): min=55, max=25010, avg=956.73, stdev=313.45
    clat percentiles (usec):
     |  1.00th=[  529],  5.00th=[  635], 10.00th=[  685], 20.00th=[  750],
     | 30.00th=[  799], 40.00th=[  832], 50.00th=[  865], 60.00th=[  898],
     | 70.00th=[  938], 80.00th=[  979], 90.00th=[ 1037], 95.00th=[ 1074],
     | 99.00th=[ 1172], 99.50th=[ 1237], 99.90th=[ 5080], 99.95th=[ 6063],
     | 99.99th=[ 9896]
   bw (  KiB/s): min=17669, max=97137, per=77.28%, avg=74107.38, stdev=7828.27, samples=239
   iops        : min= 2208, max=12142, avg=9262.99, stdev=978.55, samples=239
  write: IOPS=5138, BW=40.1MiB/s (42.1MB/s)(4818MiB/120001msec); 0 zone resets
    slat (nsec): min=2200, max=51519, avg=3211.74, stdev=714.99
    clat (usec): min=55, max=24055, avg=876.08, stdev=290.05
     lat (usec): min=59, max=24062, avg=879.35, stdev=290.16
    clat percentiles (usec):
     |  1.00th=[  529],  5.00th=[  635], 10.00th=[  685], 20.00th=[  750],
     | 30.00th=[  799], 40.00th=[  832], 50.00th=[  865], 60.00th=[  898],
     | 70.00th=[  938], 80.00th=[  979], 90.00th=[ 1037], 95.00th=[ 1074],
     | 99.00th=[ 1172], 99.50th=[ 1237], 99.90th=[ 5080], 99.95th=[ 5997],
     | 99.99th=[ 9896]
   bw (  KiB/s): min= 7901, max=41867, per=77.29%, avg=31772.86, stdev=3468.08, samples=239
   iops        : min=  987, max= 5233, avg=3971.16, stdev=433.49, samples=239
  lat (usec)   : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.49%, 750=20.14%
  lat (usec)   : 1000=63.62%
  lat (msec)   : 2=15.36%, 4=0.21%, 10=0.18%, 20=0.01%, 50=0.01%
  cpu          : usr=2.15%, sys=7.98%, ctx=1438485, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1438418,616666,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
randReadWrite/sync: (groupid=1, jobs=1): err= 0: pid=8953: Wed Apr  7 21:14:30 2021
  read: IOPS=10.6k, BW=82.9MiB/s (86.9MB/s)(9949MiB/120001msec)
    slat (usec): min=52, max=2544, avg=82.36, stdev=24.50
    clat (nsec): min=880, max=5744.3k, avg=981632.09, stdev=133511.93
     lat (usec): min=79, max=5973, avg=1064.12, stdev=137.56
    clat percentiles (usec):
     |  1.00th=[  701],  5.00th=[  783], 10.00th=[  824], 20.00th=[  873],
     | 30.00th=[  914], 40.00th=[  947], 50.00th=[  979], 60.00th=[ 1004],
     | 70.00th=[ 1037], 80.00th=[ 1074], 90.00th=[ 1139], 95.00th=[ 1205],
     | 99.00th=[ 1352], 99.50th=[ 1401], 99.90th=[ 1549], 99.95th=[ 1663],
     | 99.99th=[ 3163]
   bw (  KiB/s): min=80752, max=86224, per=100.00%, avg=84896.45, stdev=599.50, samples=239
   iops        : min=10094, max=10778, avg=10612.05, stdev=74.95, samples=239
  write: IOPS=4556, BW=35.6MiB/s (37.3MB/s)(4272MiB/120001msec); 0 zone resets
    slat (nsec): min=2370, max=40009, avg=3233.03, stdev=665.03
    clat (usec): min=438, max=5986, avg=1006.56, stdev=141.58
     lat (usec): min=442, max=5989, avg=1009.85, stdev=141.59
    clat percentiles (usec):
     |  1.00th=[  717],  5.00th=[  799], 10.00th=[  848], 20.00th=[  898],
     | 30.00th=[  938], 40.00th=[  971], 50.00th=[  996], 60.00th=[ 1029],
     | 70.00th=[ 1057], 80.00th=[ 1106], 90.00th=[ 1172], 95.00th=[ 1237],
     | 99.00th=[ 1401], 99.50th=[ 1467], 99.90th=[ 1631], 99.95th=[ 1778],
     | 99.99th=[ 3294]
   bw (  KiB/s): min=34624, max=38464, per=100.00%, avg=36457.28, stdev=763.79, samples=239
   iops        : min= 4328, max= 4808, avg=4557.15, stdev=95.47, samples=239
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=2.52%, 1000=53.29%
  lat (msec)   : 2=44.15%, 4=0.03%, 10=0.01%
  fsync/fdatasync/sync_file_range:
    sync (nsec): min=9, max=14180, avg=28.77, stdev=14.94
    sync percentiles (nsec):
     |  1.00th=[   20],  5.00th=[   20], 10.00th=[   20], 20.00th=[   20],
     | 30.00th=[   20], 40.00th=[   20], 50.00th=[   30], 60.00th=[   30],
     | 70.00th=[   30], 80.00th=[   40], 90.00th=[   40], 95.00th=[   50],
     | 99.00th=[   50], 99.50th=[   60], 99.90th=[   60], 99.95th=[   70],
     | 99.99th=[  110]
  cpu          : usr=2.59%, sys=9.60%, ctx=1820328, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=200.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1273507,546816,0,1820308 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=93.6MiB/s (98.2MB/s), 93.6MiB/s-93.6MiB/s (98.2MB/s-98.2MB/s), io=10.0GiB (11.8GB), run=120001-120001msec
  WRITE: bw=40.1MiB/s (42.1MB/s), 40.1MiB/s-40.1MiB/s (42.1MB/s-42.1MB/s), io=4818MiB (5052MB), run=120001-120001msec

Run status group 1 (all jobs):
   READ: bw=82.9MiB/s (86.9MB/s), 82.9MiB/s-82.9MiB/s (86.9MB/s-86.9MB/s), io=9949MiB (10.4GB), run=120001-120001msec
  WRITE: bw=35.6MiB/s (37.3MB/s), 35.6MiB/s-35.6MiB/s (37.3MB/s-37.3MB/s), io=4272MiB (4480MB), run=120001-120001msec

Disk stats (read/write):
  nvme0n1: ios=2711528/1139581, merge=0/1186769, ticks=209230/195070, in_queue=0, util=99.03%