randRead: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32 ... randReadWrite: (g=1): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32 ... randReadWrite/sync: (g=2): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32 ... fio-3.16 Starting 48 processes randRead: (groupid=0, jobs=16): err= 0: pid=2199576: Sat Oct 10 12:35:16 2020 read: IOPS=1983k, BW=15.1GiB/s (16.2GB/s)(256GiB/16917msec) slat (nsec): min=852, max=4075.2k, avg=7359.80, stdev=20512.06 clat (nsec): min=741, max=5742.5k, avg=249544.19, stdev=498034.33 lat (nsec): min=1933, max=6058.3k, avg=256961.20, stdev=513738.89 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 76], 10.00th=[ 77], 20.00th=[ 79], | 30.00th=[ 81], 40.00th=[ 82], 50.00th=[ 83], 60.00th=[ 85], | 70.00th=[ 88], 80.00th=[ 143], 90.00th=[ 562], 95.00th=[ 1418], | 99.00th=[ 2704], 99.50th=[ 2900], 99.90th=[ 3064], 99.95th=[ 3097], | 99.99th=[ 3163] bw ( MiB/s): min= 1119, max=40095, per=80.42%, avg=12461.11, stdev=971.86, samples=528 iops : min=143237, max=5132238, avg=1595015.85, stdev=124397.78, samples=528 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=75.00%, 250=10.35%, 500=4.13%, 750=1.99%, 1000=1.57% lat (msec) : 2=4.06%, 4=2.90%, 10=0.01% cpu : usr=8.32%, sys=71.93%, ctx=2102552, majf=0, minf=1181 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=33554432,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 randReadWrite: (groupid=1, jobs=16): err= 0: pid=2205768: Sat Oct 10 12:35:16 2020 read: IOPS=203k, BW=1589MiB/s (1666MB/s)(179GiB/115471msec) slat (nsec): min=802, max=9114.1M, avg=33754.32, stdev=11563851.37 clat (nsec): min=822, max=14035M, avg=1648459.40, stdev=58793210.38 lat (usec): min=2, max=14073k, avg=1682.29, stdev=60320.94 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 118], 10.00th=[ 127], | 20.00th=[ 141], 30.00th=[ 155], 40.00th=[ 172], | 50.00th=[ 186], 60.00th=[ 204], 70.00th=[ 227], | 80.00th=[ 277], 90.00th=[ 578], 95.00th=[ 1270], | 99.00th=[ 41681], 99.50th=[ 45351], 99.90th=[ 53216], | 99.95th=[ 61604], 99.99th=[1920992] bw ( MiB/s): min= 1, max=11804, per=100.00%, avg=1923.20, stdev=161.10, samples=2488 iops : min= 167, max=1511020, avg=246163.27, stdev=20621.00, samples=2488 write: IOPS=87.2k, BW=681MiB/s (714MB/s)(76.8GiB/115471msec); 0 zone resets slat (nsec): min=1122, max=6420.1M, avg=93569.32, stdev=2782000.29 clat (nsec): min=871, max=14073M, avg=1659168.96, stdev=59795521.64 lat (usec): min=3, max=14073k, avg=1752.82, stdev=59858.08 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 118], 10.00th=[ 127], | 20.00th=[ 141], 30.00th=[ 157], 40.00th=[ 172], | 50.00th=[ 186], 60.00th=[ 204], 70.00th=[ 227], | 80.00th=[ 277], 90.00th=[ 578], 95.00th=[ 1270], | 99.00th=[ 41681], 99.50th=[ 45351], 99.90th=[ 53216], | 99.95th=[ 61604], 99.99th=[1920992] bw ( KiB/s): min= 712, max=5183830, per=100.00%, avg=843856.71, stdev=70677.32, samples=2488 iops : min= 82, max=647974, avg=105475.00, stdev=8834.67, samples=2488 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.52%, 250=75.65%, 500=12.34%, 750=3.52%, 1000=1.64% lat (msec) : 2=3.92%, 4=0.47%, 10=0.01%, 20=0.03%, 50=1.77% lat (msec) : 100=0.12%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=2.43%, sys=12.04%, ctx=1500197, majf=0, minf=253 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=23489902,10064530,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 randReadWrite/sync: (groupid=2, jobs=16): err= 0: pid=2829709: Sat Oct 10 12:35:16 2020 read: IOPS=38.9k, BW=304MiB/s (319MB/s)(35.6GiB/120001msec) slat (nsec): min=1793, max=17198k, avg=20666.88, stdev=48700.35 clat (usec): min=241, max=134963, avg=8904.88, stdev=4344.26 lat (usec): min=245, max=134968, avg=8925.72, stdev=4350.62 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 26], 99.50th=[ 36], 99.90th=[ 70], 99.95th=[ 80], | 99.99th=[ 103] bw ( KiB/s): min=109616, max=385230, per=99.94%, avg=310953.43, stdev=3786.29, samples=3830 iops : min=13702, max=48152, avg=38868.79, stdev=473.30, samples=3830 write: IOPS=16.7k, BW=130MiB/s (137MB/s)(15.3GiB/120001msec); 0 zone resets slat (usec): min=3, max=4161, avg=24.99, stdev=17.26 clat (usec): min=235, max=144505, avg=8980.03, stdev=4383.25 lat (usec): min=246, max=144515, avg=9005.17, stdev=4381.89 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 8], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 26], 99.50th=[ 36], 99.90th=[ 70], 99.95th=[ 81], | 99.99th=[ 103] bw ( KiB/s): min=47472, max=168283, per=99.95%, avg=133272.96, stdev=1633.52, samples=3830 iops : min= 5934, max=21035, avg=16658.67, stdev=204.19, samples=3830 lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=88.37%, 20=10.03%, 50=1.32% lat (msec) : 100=0.27%, 250=0.01% fsync/fdatasync/sync_file_range: sync (nsec): min=10, max=145103, avg=105.67, stdev=244.68 sync percentiles (nsec): | 1.00th=[ 50], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 70], 40.00th=[ 80], 50.00th=[ 90], 60.00th=[ 100], | 70.00th=[ 120], 80.00th=[ 131], 90.00th=[ 171], 95.00th=[ 201], | 99.00th=[ 282], 99.50th=[ 350], 99.90th=[ 612], 99.95th=[ 724], | 99.99th=[ 2928] cpu : usr=1.43%, sys=34.35%, ctx=29397201, majf=0, minf=232 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=200.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=4667068,2000196,0,6666784 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=15.1GiB/s (16.2GB/s), 15.1GiB/s-15.1GiB/s (16.2GB/s-16.2GB/s), io=256GiB (275GB), run=16917-16917msec Run status group 1 (all jobs): READ: bw=1589MiB/s (1666MB/s), 1589MiB/s-1589MiB/s (1666MB/s-1666MB/s), io=179GiB (192GB), run=115471-115471msec WRITE: bw=681MiB/s (714MB/s), 681MiB/s-681MiB/s (714MB/s-714MB/s), io=76.8GiB (82.4GB), run=115471-115471msec Run status group 2 (all jobs): READ: bw=304MiB/s (319MB/s), 304MiB/s-304MiB/s (319MB/s-319MB/s), io=35.6GiB (38.2GB), run=120001-120001msec WRITE: bw=130MiB/s (137MB/s), 130MiB/s-130MiB/s (137MB/s-137MB/s), io=15.3GiB (16.4GB), run=120001-120001msec