Investigating high network latency

Hi all,

We are testing ScaleIO 2.0.0.3 on a mix of HPE ProLiant servers (mostly DL385p Gen8) with 1 TB Samsung Evo 850 SSDs, 10 GbE adapters (HPE NC552SFP) and Hyper-V on Win 2012 R2 (hyper-converged).

We get approx. 200 MB/s and 5,500 IOPS max., and want to investigate how to improve performance.

Bandwidth seems to be OK, e.g.

    SDS 9e5af21d00000002 10.99.99.53 bandwidth 1.0 GB (1024 MB) per-second

    SDS 9e5b192d00000006 10.99.99.54 bandwidth 978.3 MB (1001744 KB) per-second

    SDS 9e5acb110000000c 10.99.99.56 bandwidth 974.5 MB (997931 KB) per-second

    SDS 9e5b403d0000000e 10.99.99.61 bandwidth 1.1 GB (1120 MB) per-second

    SDS 9e5b674f00000010 10.99.99.63 bandwidth 946.4 MB (969109 KB) per-second

    SDS 9e5b675000000011 10.99.99.55 bandwidth 1.1 GB (1101 MB) per-second

    SDS 9e5b675100000012 10.99.99.52 bandwidth 929.6 MB (951952 KB) per-second

    SDS 9e5b675200000013 10.99.99.51 bandwidth 1.1 GB (1124 MB) per-second

    SDS 9e5b675300000014 10.99.99.57 bandwidth 939.7 MB (962217 KB) per-second

But latencies seem to be very inconsistent and much too high, e.g.

SDS 10.99.99.63

  Average IO size: 65.0 KB (66560 Bytes)

  Average latency (micro seconds): 29402

SDS 10.99.99.63

  Average IO size: 52.0 KB (53248 Bytes)

  Average latency (micro seconds): 20303

SDS 10.99.99.63

  Average IO size: 19.0 KB (19456 Bytes)

  Average latency (micro seconds): 7638

SDS 10.99.99.63

  Average IO size: 4.0 KB (4096 Bytes)

  Average latency (micro seconds): 1930

SDS 10.99.99.63

  Average IO size: 8.0 KB (8192 Bytes)

  Average latency (micro seconds): 720

SDS 10.99.99.63

  Average IO size: 0 Bytes

  Average latency (micro seconds): 0

SDS 10.99.99.63

  Average IO size: 20.0 KB (20480 Bytes)

  Average latency (micro seconds): 10100

SDS 10.99.99.63

  Average IO size: 0 Bytes

  Average latency (micro seconds): 0

SDS 10.99.99.63

  Average IO size: 42.0 KB (43008 Bytes)

  Average latency (micro seconds): 52027

"scli --query_sds_network_test_results" shows the IP addresses of all the partner SDS (which is helpful) but "scli --query_network_latency_meters" only ever shows its own IP address: is this a bug or on purpose?

Thanks a lot for any help,

Frank