randReadWrite: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=256 ... randReadWrite/sync: (g=1): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=256 ... randReadWrite/fsync: (g=2): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=256 ... fio-3.12 Starting 24 processes randReadWrite: (groupid=0, jobs=8): err= 0: pid=3810: Tue May 18 17:49:04 2021 read: IOPS=2188, BW=2189MiB/s (2295MB/s)(257GiB/120006msec) slat (usec): min=245, max=45887, avg=3374.71, stdev=2210.16 clat (usec): min=19, max=1175.9k, avg=650369.02, stdev=108201.77 lat (usec): min=544, max=1182.1k, avg=653745.92, stdev=108559.49 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 550], 10.00th=[ 567], 20.00th=[ 584], | 30.00th=[ 600], 40.00th=[ 617], 50.00th=[ 625], 60.00th=[ 642], | 70.00th=[ 659], 80.00th=[ 676], 90.00th=[ 726], 95.00th=[ 944], | 99.00th=[ 1053], 99.50th=[ 1083], 99.90th=[ 1116], 99.95th=[ 1133], | 99.99th=[ 1150] bw ( KiB/s): min=36044, max=293273, per=9.45%, avg=211819.79, stdev=36795.30, samples=1912 iops : min= 35, max= 286, avg=206.37, stdev=35.93, samples=1912 write: IOPS=941, BW=942MiB/s (987MB/s)(110GiB/120006msec); 0 zone resets slat (usec): min=156, max=28282, avg=352.08, stdev=238.73 clat (msec): min=4, max=1166, avg=650.95, stdev=108.07 lat (msec): min=4, max=1167, avg=651.31, stdev=108.07 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 550], 10.00th=[ 567], 20.00th=[ 584], | 30.00th=[ 600], 40.00th=[ 617], 50.00th=[ 625], 60.00th=[ 642], | 70.00th=[ 659], 80.00th=[ 676], 90.00th=[ 726], 95.00th=[ 944], | 99.00th=[ 1062], 99.50th=[ 1083], 99.90th=[ 1116], 99.95th=[ 1133], | 99.99th=[ 1150] bw ( KiB/s): min=22937, max=137846, per=9.45%, avg=91116.53, stdev=17943.49, samples=1912 iops : min= 22, max= 134, avg=88.51, stdev=17.52, samples=1912 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 750=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02% lat (msec) : 100=0.03%, 250=0.08%, 500=0.35%, 750=90.42%, 1000=6.21% cpu : usr=3.36%, sys=17.86%, ctx=206948, majf=0, minf=6521430 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=262688,112989,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 randReadWrite/sync: (groupid=1, jobs=8): err= 0: pid=57658: Tue May 18 17:49:04 2021 read: IOPS=1489, BW=1489MiB/s (1562MB/s)(175GiB/120004msec) slat (usec): min=234, max=11674, avg=2731.74, stdev=1540.16 clat (usec): min=10, max=1587.4k, avg=954381.92, stdev=114047.05 lat (usec): min=1755, max=1590.7k, avg=957115.32, stdev=114249.99 clat percentiles (msec): | 1.00th=[ 818], 5.00th=[ 869], 10.00th=[ 885], 20.00th=[ 902], | 30.00th=[ 919], 40.00th=[ 927], 50.00th=[ 936], 60.00th=[ 953], | 70.00th=[ 961], 80.00th=[ 978], 90.00th=[ 1011], 95.00th=[ 1150], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1536], 99.95th=[ 1552], | 99.99th=[ 1569] bw ( KiB/s): min=14948, max=224233, per=10.79%, avg=164571.30, stdev=22162.17, samples=1904 iops : min= 14, max= 218, avg=160.22, stdev=21.63, samples=1904 write: IOPS=638, BW=639MiB/s (670MB/s)(74.9GiB/120004msec); 0 zone resets slat (usec): min=1659, max=150274, avg=5871.14, stdev=2051.27 clat (usec): min=11, max=1585.2k, avg=954110.16, stdev=113948.42 lat (usec): min=1829, max=1592.6k, avg=959983.69, stdev=114408.67 clat percentiles (msec): | 1.00th=[ 818], 5.00th=[ 869], 10.00th=[ 885], 20.00th=[ 902], | 30.00th=[ 919], 40.00th=[ 927], 50.00th=[ 936], 60.00th=[ 953], | 70.00th=[ 961], 80.00th=[ 978], 90.00th=[ 1011], 95.00th=[ 1150], | 99.00th=[ 1435], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1569] bw ( KiB/s): min= 7474, max=113589, per=10.79%, avg=70594.29, stdev=12687.42, samples=1904 iops : min= 7, max= 110, avg=68.44, stdev=12.40, samples=1904 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03% lat (msec) : 100=0.04%, 250=0.13%, 500=0.22%, 750=0.22%, 1000=87.57% cpu : usr=2.20%, sys=11.95%, ctx=290072, majf=0, minf=4608815 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=178718,76650,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 randReadWrite/fsync: (groupid=2, jobs=8): err= 0: pid=58087: Tue May 18 17:49:04 2021 read: IOPS=1500, BW=1501MiB/s (1574MB/s)(176GiB/120002msec) slat (usec): min=236, max=13497, avg=2397.21, stdev=1338.75 clat (usec): min=17, max=1564.6k, avg=948657.13, stdev=121812.96 lat (usec): min=1002, max=1567.9k, avg=951056.04, stdev=122008.54 clat percentiles (msec): | 1.00th=[ 793], 5.00th=[ 869], 10.00th=[ 885], 20.00th=[ 894], | 30.00th=[ 911], 40.00th=[ 919], 50.00th=[ 919], 60.00th=[ 927], | 70.00th=[ 944], 80.00th=[ 961], 90.00th=[ 1028], 95.00th=[ 1267], | 99.00th=[ 1401], 99.50th=[ 1435], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1552] bw ( KiB/s): min= 2043, max=243712, per=12.42%, avg=190869.15, stdev=25815.48, samples=1911 iops : min= 1, max= 238, avg=186.36, stdev=25.22, samples=1911 write: IOPS=643, BW=644MiB/s (675MB/s)(75.4GiB/120002msec); 0 zone resets slat (usec): min=159, max=4149, avg=347.11, stdev=98.86 clat (usec): min=10, max=1563.6k, avg=951508.02, stdev=123142.02 lat (usec): min=265, max=1564.1k, avg=951856.08, stdev=123145.03 clat percentiles (msec): | 1.00th=[ 802], 5.00th=[ 877], 10.00th=[ 885], 20.00th=[ 902], | 30.00th=[ 911], 40.00th=[ 919], 50.00th=[ 927], 60.00th=[ 936], | 70.00th=[ 944], 80.00th=[ 961], 90.00th=[ 1036], 95.00th=[ 1284], | 99.00th=[ 1401], 99.50th=[ 1435], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1552] bw ( KiB/s): min= 2048, max=129024, per=12.42%, avg=81877.58, stdev=14235.43, samples=1909 iops : min= 2, max= 126, avg=79.91, stdev=13.90, samples=1909 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 500=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03% lat (msec) : 100=0.05%, 250=0.12%, 500=0.22%, 750=0.45%, 1000=88.03% fsync/fdatasync/sync_file_range: sync (nsec): min=50, max=24812, avg=431.79, stdev=584.59 sync percentiles (nsec): | 1.00th=[ 102], 5.00th=[ 124], 10.00th=[ 171], 20.00th=[ 245], | 30.00th=[ 314], 40.00th=[ 370], 50.00th=[ 402], 60.00th=[ 426], | 70.00th=[ 462], 80.00th=[ 532], 90.00th=[ 644], 95.00th=[ 756], | 99.00th=[ 1004], 99.50th=[ 1144], 99.90th=[12608], 99.95th=[13504], | 99.99th=[14400] cpu : usr=2.41%, sys=13.18%, ctx=462713, majf=0, minf=5149245 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=199.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=180111,77245,0,255318 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=2189MiB/s (2295MB/s), 2189MiB/s-2189MiB/s (2295MB/s-2295MB/s), io=257GiB (275GB), run=120006-120006msec WRITE: bw=942MiB/s (987MB/s), 942MiB/s-942MiB/s (987MB/s-987MB/s), io=110GiB (118GB), run=120006-120006msec Run status group 1 (all jobs): READ: bw=1489MiB/s (1562MB/s), 1489MiB/s-1489MiB/s (1562MB/s-1562MB/s), io=175GiB (187GB), run=120004-120004msec WRITE: bw=639MiB/s (670MB/s), 639MiB/s-639MiB/s (670MB/s-670MB/s), io=74.9GiB (80.4GB), run=120004-120004msec Run status group 2 (all jobs): READ: bw=1501MiB/s (1574MB/s), 1501MiB/s-1501MiB/s (1574MB/s-1574MB/s), io=176GiB (189GB), run=120002-120002msec WRITE: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=75.4GiB (80.0GB), run=120002-120002msec