error in volume.log

Hey Team,

Got a customer trying to integrate ScaleIO with OpenStack Juno and is getting the below error:

Below error in /var/log/cinder/volume.log in openstack controller node.
2015-11-16 14:04:29.383 81577 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/scaleio.py", line 1011, in get_volume_stats
2015-11-16 14:04:29.383 81577 TRACE cinder.openstack.common.threadgroup self._update_volume_stats()
2015-11-16 14:04:29.383 81577 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/emc/scaleio.py", line 887, in _update_volume_stats

This is his current setup:

RHEL 7.1

Red hat Openstack 6

ScaleIO R1_32.403.2

SDC successful connect to SDS:-

ScaleIO-10-222-88-15:~ # scli --mdm_ip 10.222.88.15 --query_all_sds

Query-all-SDS returned 3 SDS nodes.

Protection Domain 644bbe3d00000000 Name: SIO_DM1

SDS ID: 9050e7a100000002 Name: hkgesxdr04.oocl.com-ESX State: Connected, Joined IP: 10.222.88.14,192.168.202.114 Port: 7072

SDS ID: 9050e7a600000007 Name: hkgesxdr06.oocl.com-ESX State: Connected, Joined IP: 10.222.88.16,192.168.202.116 Port: 7072

SDS ID: 9050e7a500000006 Name: hkgesxdr05.oocl.com-ESX State: Connected, Joined IP: 10.222.88.15,192.168.202.115 Port: 7072

ScaleIO-10-222-88-15:~ # scli --mdm_ip 10.222.88.15 --query_all_sdc

MDM restricted SDC mode: Disabled

Query all SDC returned 14 SDC nodes.

SDC ID: 429ffea700000000 Name: ESX-146.222.8.219 IP: 146.222.8.219 State: Connected GUID: 61165164-39E3-47C4-8319-112B1FB7F046

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  1 IOPS 1.0 KB (1024 Bytes) per-second

SDC ID: 429ffea800000001 Name: ESX-146.222.8.217 IP: 146.222.8.217 State: Connected GUID: 1C0A47DF-7D53-4B8F-9B89-5E91FA5DFE56

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  1 IOPS 1.0 KB (1024 Bytes) per-second

SDC ID: 429ffea900000002 Name: ESX-146.222.8.218 IP: 146.222.8.218 State: Connected GUID: E697B873-26FD-48D3-AB98-603F8492B2AE

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  1 IOPS 1.0 KB (1024 Bytes) per-second

SDC ID: 42a04cc800000007 Name: hkgopcdv001 IP: 192.168.202.12 State: Connected GUID: EEC819B7-1EA5-4F81-B88A-6BB91A5D770E

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04cc900000008 Name: hkgopcdv002 IP: 192.168.202.13 State: Connected GUID: 9BE4C6A1-D393-4DD6-AB2C-161C0F143471

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04cca00000009 Name: hkgopcdv003 IP: 192.168.202.14 State: Connected GUID: 62BB7CA2-B692-4B85-81E6-C96CB0881751

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04ccb0000000a Name: hkgopndv001 IP: 192.168.202.15 State: Connected GUID: EB13944F-7BFE-47A6-8215-1162A7285915

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04ccc0000000b Name: hkgopndv002 IP: 192.168.202.16 State: Connected GUID: E986360C-CC36-4CBC-81BB-40568F3D566F

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04ccd0000000c Name: hkgopndv003 IP: 192.168.202.17 State: Connected GUID: 50DFC0CF-4EFB-471F-887D-8BA0BC50233F

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04cce0000000d Name: hkgopndv004 IP: 192.168.202.18 State: Connected GUID: 85096685-F596-4D18-BB74-F086017351D5

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04ccf0000000e Name: hkgopndv005 IP: 192.168.202.19 State: Connected GUID: BB93C62E-CAEA-41C8-A565-3A9454F29423

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04cd00000000f Name: hkgopndv006 IP: 192.168.202.20 State: Connected GUID: 559D5CFA-A606-4D3B-A924-04D0A0A7BD5C

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth:  0 IOPS 0 Bytes per-second

SDC ID: 42a04cd100000010 Name: hkgopndv007 IP: 192.168.202.21 State: Connected GUID: D5535C0D-1B1B-4029-A367-8DA48A79D31F

    Read bandwidth:  0 IOPS 0 Bytes per-second

    Write bandwidth: 0 IOPS 0 Bytes per-second

SDC ID: 42a04cd200000011 Name: hkgopndv008 IP: 192.168.202.22 State: Connected GUID: 2AFA23EB-E8D4-4B7D-B922-B581AA15458E

    Read bandwidth:  0 IOPS 0 Bytes per-second

Write bandwidth:  0 IOPS 0 Bytes per-second

[root@hkgopcdv001 cinder]# cat cinder.conf

[DEFAULT]

amqp_durable_queues=False

notification_driver=cinder.openstack.common.notifier.rpc_notifier

rpc_backend=cinder.openstack.common.rpc.impl_kombu

control_exchange=openstack

osapi_volume_listen=192.168.202.12

osapi_volume_workers=24

api_paste_config=/etc/cinder/api-paste.ini

glance_host=192.168.202.71

host=ha-controller

storage_availability_zone=nova

default_availability_zone=nova

auth_strategy=keystone

enabled_backends=VNX5600_FC,ScaleIO

debug=True

verbose=True

log_dir=/var/log/cinder

use_syslog=False

volume_backend_name=DEFAULT

volume_clear=none

default_timeout=100

nfs_shares_config=/etc/cinder/shares-nfs.conf

nfs_used_ratio=0.95

nfs_oversub_ratio=1.0

nfs_mount_options=nosharecache

rabbit_hosts=192.168.202.12:5672,192.168.202.13:5672,192.168.202.14:5672

rabbit_use_ssl=False

rabbit_userid=openstack

rabbit_password=redhat123

rabbit_virtual_host=/

rabbit_ha_queues=True

max_retries=-1

[BRCD_FABRIC_EXAMPLE]

[CISCO_FABRIC_EXAMPLE]

[database]

connection=mysql://cinder:redhat123@192.168.202.70/cinder

idle_timeout=3600

min_pool_size=1

max_pool_size=20

max_retries=10

retry_interval=10

max_overflow=30

[fc-zone-manager]

[keymgr]

[keystone_authtoken]

[matchmaker_redis]

[matchmaker_ring]

[oslo_messaging_amqp]

[profiler]

[ssl]

[VNX5600_FC]

san_ip=146.222.8.55

san_secondary_ip=146.222.8.84

san_login=osadmin

san_password=Openstack1000

storage_vnx_authentication_type=local

naviseccli_path=/opt/Navisphere/bin/naviseccli

storage_vnx_pool_name=OS-POOL-CAPACITY

volume_backend_name=VNX5600_FC

volume_driver=cinder.volume.drivers.emc.emc_cli_fc.EMCCLIFCDriver

[ScaleIO]

volume_driver=cinder.volume.drivers.emc.scaleio.ScaleIODriver

cinder_scaleio_config_file=/etc/cinder/cinder_scaleio.config

volume_backend_name=ScaleIO

I'm not an expert with OpenStack so any ideas how to fix this issue will be greatly appreciated!

Cheers!