Unsolved
This post is more than 5 years old
21 Posts
0
2479
July 21st, 2015 11:00
ScaleIO 1.32.403.2 in ESXi 5.5 U2 Performance questions
I have 4 node cluster in ESXi 5.5 U2, each node has 3 LSI 9300 8i controllers connected to 12Gb directed backplane. The backplane has 6 connectors and each LSI 9300 8i has 2 12Gb cables connected to the backplane.
Each server has 6 SATA SSDs. I connected 2 SATA SSDs per LSI 9300 8i controller to avoid IO bottleneck even one Controller can handle up to 12 SSDs. It sounds crazy but anyway.
Each SSD has 4K random Read at 65000 IOPS as raw data so the 24SSDs will provide 1.5 million IOPS as raw data. Do you believe the logical volume will provide 750000 IOPS? The 750K IOPS is too low.
I create one VM per host and mapped 6x32GB volumes to each host. I added 6 volumes as Raw to the VM. I assigned 2 disk per controller as VMware Paravirtual
I saw the XtreamIO single brick with 24-25 SAS SSDs only provide 250K IOPS.
spectrum17
21 Posts
0
July 21st, 2015 13:00
you think i should get 1 million IOPS.
It has 2 10Gb cards. Each card is assigned to different CPU which is NUMA.
Do you have any hints for performance tuning?
I already did all these command in ESXi 5.5 U2 hosts.
# esxcli system module parameters list -m scini | grep netConSchedThrd
netConSchedThrd int Number of network scheduler threads.Default 2. Minimum 1. Maximum 10
# esxcli system module parameters list -m scini | grep mapTgtSockets
mapTgtSockets int Number of sockets per target end-point.Default 1. Minimum 1. Maximum 8
# esxcli system module parameters list -m scini | grep netSockRcvBufSize
netSockRcvBufSize int Socket buffer receive size. Default 512K. Minimum 8K
esxcli system module parameters list -m scini | grep netSockSndBufSize
netSockSndBufSize int Socket buffer send size. Default 512K. Minimum 8K
esxcli system module parameters set -m scini -p "netConSchedThrd=4 mapTgtSockets=4 netSockRcvBufSize=4194304 netSockSndBufSize=4194304"
DanAharoni
16 Posts
0
July 21st, 2015 13:00
I have not tested such a system, but I do not expect you will get more than 200K per node if it were in Linux and with ESX overhead a little less than that (depending on the exact drives etc). you are better off having less SSD drives per node and going wider if the intent is to get max IOPs. in terms of the compare to XIO, keep in mind that each brick is really built out of two nodes and it does not scale as far as ScaleIO.
spectrum17
21 Posts
0
July 21st, 2015 14:00
I also changee the MTU to 9000 on vswitch, SVM NICs, switch port.
The iperf is showing 9.8Gb/s between SVM hosts.
alexkh
60 Posts
0
July 22nd, 2015 13:00
And from all 4 nodes simultaneously?
alexkh
60 Posts
0
July 22nd, 2015 13:00
How many IOPs do you get for running IO from single node?
spectrum17
21 Posts
0
July 22nd, 2015 13:00
I got around 260K IOPS per node, 4K, Random.
I just check the volume.
[root@scaleio1 ~]# scli --query_sdc_volume_limits --volume_name vol01 --sdc_id 4da4d27f00000002
Successfully queried limits from SDC with ID 4da4d27f00000002 to volume vol01: IOPS limit is unlimited. Bandwidth limit is unlimited.
spectrum17
21 Posts
0
July 22nd, 2015 13:00
Hopefully, someone form community can help me to answer these questions.
The servers have 6 SATA SSDs connected to LSI 9300 8i.
There are 4 nodes cluster. Server has 2 socket CPU, Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz and 512GB RAM.
There are 2 10Gb NICs connected to different NUMA node to get max performance.
The performance is dropping 80% when the RAM is enabled.
I don`t see much of the IOPS if I have add 6 more SATA SSds per nodes.
Does someone have any idea?
How can i increase the IO on the cluster with 4 nodes?
spectrum17
21 Posts
0
July 22nd, 2015 19:00
It is around 170K IOPS from single node.
spectrum17
21 Posts
0
July 22nd, 2015 19:00
From all 4 nodes simultaneously, it is between 600K-700K IOPS.
I think it is too low.
I run fio on 6 SATA with raw device, it gave me 260K IOPS. The 4 nodes IOPS from 24 raw devices gave me 1.04 millions IOPS. I expect to get 820K-900K IOPS.
I also installed RHEL 6.6 64-bit on 4 nodes and I got the same number as ESXi because I use passthrough mode for LSI 93008I to SVM.
Please let me know how I am going to increase the IOPS to 820K-900K IOPS.
In Linux, I also followed the performance tuning guide.
sbin/insmod $DRV_BIN netConSchedThrd=8
netSockSndBufSize=4194304 netSockRcvBufSize=4194304
mapTgtSockets=4
service scini restart
Change MTU to 9000
ifconfig txqueuelen 10000
alexkh
60 Posts
0
July 22nd, 2015 22:00
Can you copy the content of a file from SVM: /opt/emc/scaleio/sds/cfg/conf.txt ?
DanAharoni
16 Posts
1
July 23rd, 2015 13:00
I have to say, 170K IOPs per node with ESX is not too bad at all :-)
spectrum17
21 Posts
0
July 23rd, 2015 17:00
I also run this command on 4 ESXi hosts.
esxcli system module parameters set -m scini -p "netConSchedThrd=4 mapTgtSockets=4 netSockRcvBufSize=4194304 netSockSndBufSize=4194304"
spectrum17
21 Posts
0
July 23rd, 2015 17:00
tgt_rep__name=/opt/emc/scaleio/sds/cfg/rep_tgt.txt
tgt_net__recv_buffer=4096
tgt_net__send_buffer=4096
tgt_thread__ini_io=500
tgt_thread__tgt_io_main=500
tgt_umt_num=1500
tgt_umt_os_thrd=8
tgt_net__worker_thread=8
tgt_asyncio_max_req_per_file=400
numa_affinity=1000
numa_memory_affinity=0
rcache_allowed_high_memory_percentage=95
rcache_allowed_low_memory_percentage=95
spectrum17
21 Posts
0
July 23rd, 2015 18:00
I know ScaleIO is scaled out smoothly and linearly.
Can it scale out smoothly and linearly with adding more devices on each node?
The RAM cache does not work well with all SSDs. Why? Have someone tested RAM cache with SSDs?
After i rebooted SVM, the IOPS is dropping to 735K IOPS. Each node IOSP is around 183K IOPS.
Thanks
spectrum17
21 Posts
0
July 23rd, 2015 18:00
I run this command on ESXi hosts.
esxcli system module parameters set -m scini -p "netConSchedThrd=10 mapTgtSockets=8 netSockRcvBufSize=4194304 netSockSndBufSize=4194304"
I also add these options to /etc/grub.conf on vms which are running fio.
vmw_pvscsi.cmd_per_lun=254 vmw_pvscsi.ring_pages=32
cat /sys/module/vmw_pvscsi/parameters/cmd_per_lun
254
cat /sys/module/vmw_pvscsi/parameters/ring_pages
32
Each vm with fio, 208K IOPS, 4K, 100%Read, 100%Random.
The total IOPS is around 833K IOPS.
The RAW IOPS of 4 nodes is 1.04 millions IOPS.
Do you think 833K IOPS is good number or low?