randReadWrite: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=256 ... randReadWrite/sync: (g=1): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=256 ... fio-3.16 Starting 128 processes randReadWrite: (groupid=0, jobs=64): err= 0: pid=452813: Fri Oct 9 20:12:33 2020 read: IOPS=93.4k, BW=729MiB/s (765MB/s)(85.5GiB/120006msec) slat (usec): min=5, max=32148, avg=306.77, stdev=651.59 clat (usec): min=3, max=494224, avg=122392.17, stdev=80997.77 lat (usec): min=109, max=494472, avg=122699.14, stdev=81082.31 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 78], 60.00th=[ 107], | 70.00th=[ 163], 80.00th=[ 199], 90.00th=[ 249], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 368], 99.90th=[ 414], 99.95th=[ 426], | 99.99th=[ 451] bw ( KiB/s): min=223935, max=1098555, per=86.45%, avg=645607.56, stdev=3366.38, samples=15296 iops : min=27964, max=137294, avg=80673.18, stdev=420.80, samples=15296 write: IOPS=40.0k, BW=313MiB/s (328MB/s)(36.6GiB/120006msec); 0 zone resets slat (usec): min=9, max=44314, avg=874.65, stdev=1724.67 clat (usec): min=3, max=494247, avg=122200.26, stdev=80832.93 lat (usec): min=335, max=494623, avg=123075.15, stdev=81447.68 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 78], 60.00th=[ 106], | 70.00th=[ 163], 80.00th=[ 199], 90.00th=[ 249], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 368], 99.90th=[ 409], 99.95th=[ 422], | 99.99th=[ 451] bw ( KiB/s): min=93762, max=478289, per=86.44%, avg=276700.58, stdev=1453.16, samples=15296 iops : min=11690, max=59758, avg=34559.84, stdev=181.65, samples=15296 lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=6.44% lat (msec) : 100=52.67%, 250=31.16%, 500=9.70% cpu : usr=0.59%, sys=32.74%, ctx=8988205, majf=0, minf=834 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=11203063,4801791,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 randReadWrite/sync: (groupid=1, jobs=64): err= 0: pid=1603048: Fri Oct 9 20:12:33 2020 read: IOPS=60.0k, BW=476MiB/s (500MB/s)(55.8GiB/120002msec) slat (usec): min=6, max=20289, avg=106.14, stdev=262.04 clat (usec): min=3, max=541781, avg=187066.42, stdev=97366.08 lat (usec): min=26, max=541807, avg=187172.77, stdev=97409.50 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 128], 10.00th=[ 130], 20.00th=[ 132], | 30.00th=[ 133], 40.00th=[ 136], 50.00th=[ 138], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 255], 90.00th=[ 363], 95.00th=[ 418], | 99.00th=[ 485], 99.50th=[ 506], 99.90th=[ 518], 99.95th=[ 523], | 99.99th=[ 531] bw ( KiB/s): min=180919, max=728786, per=99.85%, avg=487041.51, stdev=2600.09, samples=15304 iops : min=22614, max=91098, avg=60878.33, stdev=325.01, samples=15304 write: IOPS=26.1k, BW=204MiB/s (214MB/s)(23.9GiB/120002msec); 0 zone resets slat (usec): min=10, max=36448, avg=313.82, stdev=1039.13 clat (usec): min=3, max=541392, avg=187047.12, stdev=97305.44 lat (usec): min=58, max=541515, avg=187361.18, stdev=97614.00 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 128], 10.00th=[ 130], 20.00th=[ 132], | 30.00th=[ 133], 40.00th=[ 136], 50.00th=[ 138], 60.00th=[ 140], | 70.00th=[ 150], 80.00th=[ 255], 90.00th=[ 363], 95.00th=[ 418], | 99.00th=[ 485], 99.50th=[ 506], 99.90th=[ 518], 99.95th=[ 523], | 99.99th=[ 531] bw ( KiB/s): min=72469, max=328036, per=99.85%, avg=208649.11, stdev=1125.08, samples=15304 iops : min= 9056, max=41000, avg=26079.10, stdev=140.63, samples=15304 lat (usec) : 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04% lat (msec) : 100=0.12%, 250=79.44%, 500=19.78%, 750=0.60% fsync/fdatasync/sync_file_range: sync (nsec): min=20, max=1481.3k, avg=158.48, stdev=2814.86 sync percentiles (nsec): | 1.00th=[ 50], 5.00th=[ 60], 10.00th=[ 70], 20.00th=[ 71], | 30.00th=[ 80], 40.00th=[ 81], 50.00th=[ 90], 60.00th=[ 101], | 70.00th=[ 141], 80.00th=[ 191], 90.00th=[ 221], 95.00th=[ 270], | 99.00th=[ 652], 99.50th=[ 828], 99.90th=[ 3504], 99.95th=[ 4448], | 99.99th=[24960] cpu : usr=0.64%, sys=41.45%, ctx=17025807, majf=0, minf=1018 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=199.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=7317084,3134613,0,10435433 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=729MiB/s (765MB/s), 729MiB/s-729MiB/s (765MB/s-765MB/s), io=85.5GiB (91.8GB), run=120006-120006msec WRITE: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=36.6GiB (39.3GB), run=120006-120006msec Run status group 1 (all jobs): READ: bw=476MiB/s (500MB/s), 476MiB/s-476MiB/s (500MB/s-500MB/s), io=55.8GiB (59.9GB), run=120002-120002msec WRITE: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=23.9GiB (25.7GB), run=120002-120002msec