Да, https://s4u.com.ua/hp-p420-controller-osobennosti.html

Судя по вашим скринам система во время тестов больше же ничем не нагружалась - правильно?

Верно

4 ssh терминала,думаю это не всчёт

На этом же контроллере или на другом?

raidz2 Run status group 0 (all jobs): READ: bw=45.4MiB/s (47.6MB/s), 45.4MiB/s-45.4MiB/s (47.6MB/s-47.6MB/s), io=6141MiB (6440MB), run=135415-135415msec WRITE: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=2051MiB (2150MB), run=135415-135415msec raidz1 Run status group 0 (all jobs): READ: bw=49.1MiB/s (51.5MB/s), 49.1MiB/s-49.1MiB/s (51.5MB/s-51.5MB/s), io=6141MiB (6440MB), run=125108-125108msec WRITE: bw=16.4MiB/s (17.2MB/s), 16.4MiB/s-16.4MiB/s (17.2MB/s-17.2MB/s), io=2051MiB (2150MB), run=125108-125108msec raid10 Run status group 0 (all jobs): READ: bw=49.2MiB/s (51.6MB/s), 49.2MiB/s-49.2MiB/s (51.6MB/s-51.6MB/s), io=6141MiB (6440MB), run=124904-124904msec WRITE: bw=16.4MiB/s (17.2MB/s), 16.4MiB/s-16.4MiB/s (17.2MB/s-17.2MB/s), io=2051MiB (2150MB), run=124904-124904msec один диск ext4 Run status group 0 (all jobs): READ: bw=44.0MiB/s (46.2MB/s), 44.0MiB/s-44.0MiB/s (46.2MB/s-46.2MB/s), io=6141MiB (6440MB), run=139455-139455msec WRITE: bw=14.7MiB/s (15.4MB/s), 14.7MiB/s-14.7MiB/s (15.4MB/s-15.4MB/s), io=2051MiB (2150MB), run=139455-139455msec Disk stats (read/write): sdi: ios=1568152/527084, merge=2436/284, ticks=7199412/1691648, in_queue=4581688, util=99.99%

Тест для ext4 ☝️

Не совсем понимаю что значит сырой девайс. Без файловой системы, не монтируя никуда, через fio ?

плюс минус так-же

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=/dev/sdi --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 --max-jobs=20 fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=48.6MiB/s,w=16.3MiB/s][r=12.4k,w=4166 IOPS][eta 00m:00s] fiotest: (groupid=0, jobs=1): err= 0: pid=12499: Tue May 18 10:52:59 2021 read: IOPS=11.6k, BW=45.2MiB/s (47.3MB/s)(6141MiB/136004msec) bw ( KiB/s): min=45136, max=52808, per=100.00%, avg=46236.02, stdev=591.58, samples=272 iops : min=11284, max=13202, avg=11559.00, stdev=147.90, samples=272 write: IOPS=3860, BW=15.1MiB/s (15.8MB/s)(2051MiB/136004msec); 0 zone resets bw ( KiB/s): min=14488, max=17544, per=100.00%, avg=15439.83, stdev=332.99, samples=272 iops : min= 3622, max= 4386, avg=3859.96, stdev=83.25, samples=272 cpu : usr=4.02%, sys=7.12%, ctx=1049391, majf=0, minf=173 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=45.2MiB/s (47.3MB/s), 45.2MiB/s-45.2MiB/s (47.3MB/s-47.3MB/s), io=6141MiB (6440MB), run=136004-136004msec WRITE: bw=15.1MiB/s (15.8MB/s), 15.1MiB/s-15.1MiB/s (15.8MB/s-15.8MB/s), io=2051MiB (2150MB), run=136004-136004msec Disk stats (read/write): sdi: ios=1568030/524237, merge=2502/192, ticks=6980847/1672985, in_queue=4290188, util=99.98%

диски как-то особо не блещут показателями. Это что-то консьюмерское?

не думаю что тут дело в дисках. Сейчас попробую подключить к машине с виндой и проверить диски на скорость.

не думаю что тут дело в дисках. Сейчас попробую подключить к машине с виндой и проверить диски на скорость.

Disk model: KINGSTON SEDC500

SEDC500R

SEDC500R

я думаю, что он больше и не сможет

в 10 рейде долен же быть прирост производительности в сравнении с одним диском

в 10 рейде долен же быть прирост производительности в сравнении с одним диском

я сейчас про сырой девайс

pool: tank state: ONLINE config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 errors: No known data errors

здесь все 8 дисков - это модель SDC 500R / 960?

к сырому девайсу вопросов нет

да

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=/dev/sdi --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 --max-jobs=20 fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=48.6MiB/s,w=16.3MiB/s][r=12.4k,w=4166 IOPS][eta 00m:00s] fiotest: (groupid=0, jobs=1): err= 0: pid=12499: Tue May 18 10:52:59 2021 read: IOPS=11.6k, BW=45.2MiB/s (47.3MB/s)(6141MiB/136004msec) bw ( KiB/s): min=45136, max=52808, per=100.00%, avg=46236.02, stdev=591.58, samples=272 iops : min=11284, max=13202, avg=11559.00, stdev=147.90, samples=272 write: IOPS=3860, BW=15.1MiB/s (15.8MB/s)(2051MiB/136004msec); 0 zone resets bw ( KiB/s): min=14488, max=17544, per=100.00%, avg=15439.83, stdev=332.99, samples=272 iops : min= 3622, max= 4386, avg=3859.96, stdev=83.25, samples=272 cpu : usr=4.02%, sys=7.12%, ctx=1049391, majf=0, minf=173 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=45.2MiB/s (47.3MB/s), 45.2MiB/s-45.2MiB/s (47.3MB/s-47.3MB/s), io=6141MiB (6440MB), run=136004-136004msec WRITE: bw=15.1MiB/s (15.8MB/s), 15.1MiB/s-15.1MiB/s (15.8MB/s-15.8MB/s), io=2051MiB (2150MB), run=136004-136004msec Disk stats (read/write): sdi: ios=1568030/524237, merge=2502/192, ticks=6980847/1672985, in_queue=4290188, util=99.98%

а где можно посмотреть результат подобного теста на raid10?

raidz2 Run status group 0 (all jobs): READ: bw=45.4MiB/s (47.6MB/s), 45.4MiB/s-45.4MiB/s (47.6MB/s-47.6MB/s), io=6141MiB (6440MB), run=135415-135415msec WRITE: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=2051MiB (2150MB), run=135415-135415msec raidz1 Run status group 0 (all jobs): READ: bw=49.1MiB/s (51.5MB/s), 49.1MiB/s-49.1MiB/s (51.5MB/s-51.5MB/s), io=6141MiB (6440MB), run=125108-125108msec WRITE: bw=16.4MiB/s (17.2MB/s), 16.4MiB/s-16.4MiB/s (17.2MB/s-17.2MB/s), io=2051MiB (2150MB), run=125108-125108msec raid10 Run status group 0 (all jobs): READ: bw=49.2MiB/s (51.6MB/s), 49.2MiB/s-49.2MiB/s (51.6MB/s-51.6MB/s), io=6141MiB (6440MB), run=124904-124904msec WRITE: bw=16.4MiB/s (17.2MB/s), 16.4MiB/s-16.4MiB/s (17.2MB/s-17.2MB/s), io=2051MiB (2150MB), run=124904-124904msec один диск ext4 Run status group 0 (all jobs): READ: bw=44.0MiB/s (46.2MB/s), 44.0MiB/s-44.0MiB/s (46.2MB/s-46.2MB/s), io=6141MiB (6440MB), run=139455-139455msec WRITE: bw=14.7MiB/s (15.4MB/s), 14.7MiB/s-14.7MiB/s (15.4MB/s-15.4MB/s), io=2051MiB (2150MB), run=139455-139455msec Disk stats (read/write): sdi: ios=1568152/527084, merge=2436/284, ticks=7199412/1691648, in_queue=4581688, util=99.99%

raidz2 Run status group 0 (all jobs): READ: bw=45.4MiB/s (47.6MB/s), 45.4MiB/s-45.4MiB/s (47.6MB/s-47.6MB/s), io=6141MiB (6440MB), run=135415-135415msec WRITE: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=2051MiB (2150MB), run=135415-135415msec raidz1 Run status group 0 (all jobs): READ: bw=49.1MiB/s (51.5MB/s), 49.1MiB/s-49.1MiB/s (51.5MB/s-51.5MB/s), io=6141MiB (6440MB), run=125108-125108msec WRITE: bw=16.4MiB/s (17.2MB/s), 16.4MiB/s-16.4MiB/s (17.2MB/s-17.2MB/s), io=2051MiB (2150MB), run=125108-125108msec raid10 Run status group 0 (all jobs): READ: bw=49.2MiB/s (51.6MB/s), 49.2MiB/s-49.2MiB/s (51.6MB/s-51.6MB/s), io=6141MiB (6440MB), run=124904-124904msec WRITE: bw=16.4MiB/s (17.2MB/s), 16.4MiB/s-16.4MiB/s (17.2MB/s-17.2MB/s), io=2051MiB (2150MB), run=124904-124904msec один диск ext4 Run status group 0 (all jobs): READ: bw=44.0MiB/s (46.2MB/s), 44.0MiB/s-44.0MiB/s (46.2MB/s-46.2MB/s), io=6141MiB (6440MB), run=139455-139455msec WRITE: bw=14.7MiB/s (15.4MB/s), 14.7MiB/s-14.7MiB/s (15.4MB/s-15.4MB/s), io=2051MiB (2150MB), run=139455-139455msec Disk stats (read/write): sdi: ios=1568152/527084, merge=2436/284, ticks=7199412/1691648, in_queue=4581688, util=99.99%

контроллер p420i, режим работы hba

а есть возможность подключиться к вашему серверу и посмотреть конфигурацию пула и погонять тесты (fio)?

вчитался в ветку, Владимир, вроде бы вы гоняли тесты через dd большим блоком? т. е. пытались замерять линейный throughput?. а с помощью fio замеряете iops.. по поводу настроек fio - опция direct должна быть выставлена в 0. опцию ioengine я выставляю либо в posixaio, либо в sync (если мне надо проверить работу slog).

fio --directory=/mnt/datastore/tank --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 --max-jobs=20 fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process Jobs: 1 (f=1): [m(1)][99.2%][r=107MiB/s,w=35.7MiB/s][r=27.3k,w=9139 IOPS][eta 00m:01s] fiotest: (groupid=0, jobs=1): err= 0: pid=12820: Tue May 18 11:51:09 2021 read: IOPS=12.0k, BW=50.7MiB/s (53.2MB/s)(6141MiB/121153msec) bw ( KiB/s): min=40064, max=140064, per=99.66%, avg=51730.68, stdev=10231.54, samples=242 iops : min=10016, max=35016, avg=12932.64, stdev=2557.90, samples=242 write: IOPS=4333, BW=16.9MiB/s (17.7MB/s)(2051MiB/121153msec); 0 zone resets bw ( KiB/s): min=13440, max=47080, per=99.67%, avg=17275.05, stdev=3400.23, samples=242 iops : min= 3360, max=11770, avg=4318.72, stdev=850.06, samples=242 cpu : usr=5.56%, sys=87.08%, ctx=24960, majf=0, minf=189 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=50.7MiB/s (53.2MB/s), 50.7MiB/s-50.7MiB/s (53.2MB/s-53.2MB/s), io=6141MiB (6440MB), run=121153-121153msec WRITE: bw=16.9MiB/s (17.7MB/s), 16.9MiB/s-16.9MiB/s (17.7MB/s-17.7MB/s), io=2051MiB (2150MB), run=121153-121153msec

pool: tank state: ONLINE config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 errors: No known data errors

fio --directory=/mnt/datastore/tank --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 --max-jobs=20 fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process Jobs: 1 (f=1): [m(1)][99.2%][r=107MiB/s,w=35.7MiB/s][r=27.3k,w=9139 IOPS][eta 00m:01s] fiotest: (groupid=0, jobs=1): err= 0: pid=12820: Tue May 18 11:51:09 2021 read: IOPS=12.0k, BW=50.7MiB/s (53.2MB/s)(6141MiB/121153msec) bw ( KiB/s): min=40064, max=140064, per=99.66%, avg=51730.68, stdev=10231.54, samples=242 iops : min=10016, max=35016, avg=12932.64, stdev=2557.90, samples=242 write: IOPS=4333, BW=16.9MiB/s (17.7MB/s)(2051MiB/121153msec); 0 zone resets bw ( KiB/s): min=13440, max=47080, per=99.67%, avg=17275.05, stdev=3400.23, samples=242 iops : min= 3360, max=11770, avg=4318.72, stdev=850.06, samples=242 cpu : usr=5.56%, sys=87.08%, ctx=24960, majf=0, minf=189 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=50.7MiB/s (53.2MB/s), 50.7MiB/s-50.7MiB/s (53.2MB/s-53.2MB/s), io=6141MiB (6440MB), run=121153-121153msec WRITE: bw=16.9MiB/s (17.7MB/s), 16.9MiB/s-16.9MiB/s (17.7MB/s-17.7MB/s), io=2051MiB (2150MB), run=121153-121153msec

сейчас у вас собран zpool а-ля raid10? можете еще раз конфигурацию показать?

А модель ssd из которых собираете?

диски: SEDC500R Raid контроллер p420i

получилось прогнать тест с теми параметрами, которые я прислал?

Чуть позже, не хочу мешать Сергею.

Чуть позже, не хочу мешать Сергею.

также интересно поиграться с режимом sync для пула, когда он выставлен в always, можно еще добавить опцию end_fsync=1, вроде так она выглядит если я правильно помню

sync=1 и fsync=1

мне интересно на сколько просядет..

мм.. интересно в разрезе сравнить свои результаты на пуле собранном из enterprise ssd и результаты у Владимира, на пуле собранном из фактически consumer ssd