S5232F - local port 100g latency issues

Hello,

I have a S5232F (firmware 10.5.2.6) switch with 100G ports, I have 10 ports connected to ConnectX-5 Ex 100G cards.

The ports are configured as following:

interface ethernet1/1/19
  no shutdown
  switchport mode trunk
  switchport access vlan 5
  switchport trunk allowed vlan 4
  mtu 9216
  flowcontrol receive off
  flowcontrol transmit off

 

I experience high latency in application that they crawl to a slow. I can see those slow-downs between the hosts too.
Here is an example ping: (ping -i 0.01 -r 192.168.0.2 -D)

[1675173015.287308] 64 bytes from 192.168.0.2: icmp_seq=90 ttl=64 time=0.967 ms
[1675173015.298742] 64 bytes from 192.168.0.2: icmp_seq=91 ttl=64 time=0.395 ms
[1675173015.309413] 64 bytes from 192.168.0.2: icmp_seq=92 ttl=64 time=0.057 ms
[1675173015.320425] 64 bytes from 192.168.0.2: icmp_seq=93 ttl=64 time=0.083 ms
[1675173015.331463] 64 bytes from 192.168.0.2: icmp_seq=94 ttl=64 time=0.093 ms
[1675173015.342657] 64 bytes from 192.168.0.2: icmp_seq=95 ttl=64 time=0.263 ms
[1675173015.353484] 64 bytes from 192.168.0.2: icmp_seq=96 ttl=64 time=0.137 ms
[1675173015.364457] 64 bytes from 192.168.0.2: icmp_seq=97 ttl=64 time=0.116 ms
[1675173015.375637] 64 bytes from 192.168.0.2: icmp_seq=98 ttl=64 time=0.037 ms
[1675173015.386457] 64 bytes from 192.168.0.2: icmp_seq=99 ttl=64 time=0.114 ms
[1675173015.397391] 64 bytes from 192.168.0.2: icmp_seq=100 ttl=64 time=0.052 ms
[1675173015.408393] 64 bytes from 192.168.0.2: icmp_seq=101 ttl=64 time=0.045 ms
[1675173015.419478] 64 bytes from 192.168.0.2: icmp_seq=102 ttl=64 time=0.127 ms
[1675173015.431295] 64 bytes from 192.168.0.2: icmp_seq=103 ttl=64 time=0.947 ms
[1675173015.441391] 64 bytes from 192.168.0.2: icmp_seq=104 ttl=64 time=0.045 ms
[1675173015.452567] 64 bytes from 192.168.0.2: icmp_seq=105 ttl=64 time=0.214 ms
[1675173015.463969] 64 bytes from 192.168.0.2: icmp_seq=106 ttl=64 time=0.617 ms
[1675173015.475303] 64 bytes from 192.168.0.2: icmp_seq=107 ttl=64 time=0.954 ms
[1675173015.485470] 64 bytes from 192.168.0.2: icmp_seq=108 ttl=64 time=0.130 ms
[1675173015.498590] 64 bytes from 192.168.0.2: icmp_seq=109 ttl=64 time=1.24 ms
[1675173015.562198] 64 bytes from 192.168.0.2: icmp_seq=110 ttl=64 time=54.8 ms
[1675173015.562318] 64 bytes from 192.168.0.2: icmp_seq=111 ttl=64 time=43.9 ms
[1675173015.562405] 64 bytes from 192.168.0.2: icmp_seq=112 ttl=64 time=33.0 ms
[1675173015.562511] 64 bytes from 192.168.0.2: icmp_seq=113 ttl=64 time=22.1 ms
[1675173015.562673] 64 bytes from 192.168.0.2: icmp_seq=114 ttl=64 time=11.3 ms
[1675173015.562837] 64 bytes from 192.168.0.2: icmp_seq=115 ttl=64 time=0.612 ms
[1675173015.573595] 64 bytes from 192.168.0.2: icmp_seq=116 ttl=64 time=0.251 ms
[1675173015.584498] 64 bytes from 192.168.0.2: icmp_seq=117 ttl=64 time=0.135 ms
[1675173015.599933] 64 bytes from 192.168.0.2: icmp_seq=118 ttl=64 time=4.57 ms
[1675173015.606121] 64 bytes from 192.168.0.2: icmp_seq=119 ttl=64 time=0.771 ms
[1675173015.617587] 64 bytes from 192.168.0.2: icmp_seq=120 ttl=64 time=1.23 ms
[1675173015.627308] 64 bytes from 192.168.0.2: icmp_seq=121 ttl=64 time=0.956 ms
[1675173015.637462] 64 bytes from 192.168.0.2: icmp_seq=122 ttl=64 time=0.108 ms
[1675173015.665999] 64 bytes from 192.168.0.2: icmp_seq=123 ttl=64 time=17.6 ms
[1675173015.677382] 64 bytes from 192.168.0.2: icmp_seq=124 ttl=64 time=18.0 ms
[1675173015.683778] 64 bytes from 192.168.0.2: icmp_seq=125 ttl=64 time=7.70 ms
[1675173015.689168] 64 bytes from 192.168.0.2: icmp_seq=126 ttl=64 time=3.08 ms
[1675173015.697356] 64 bytes from 192.168.0.2: icmp_seq=127 ttl=64 time=1.27 ms
[1675173015.707409] 64 bytes from 192.168.0.2: icmp_seq=128 ttl=64 time=1.33 ms
[1675173015.716607] 64 bytes from 192.168.0.2: icmp_seq=129 ttl=64 time=0.519 ms
[1675173015.741055] 64 bytes from 192.168.0.2: icmp_seq=130 ttl=64 time=13.6 ms
[1675173015.744595] 64 bytes from 192.168.0.2: icmp_seq=131 ttl=64 time=6.24 ms
[1675173015.753054] 64 bytes from 192.168.0.2: icmp_seq=132 ttl=64 time=4.70 ms
[1675173015.758957] 64 bytes from 192.168.0.2: icmp_seq=133 ttl=64 time=0.607 ms
[1675173015.769663] 64 bytes from 192.168.0.2: icmp_seq=134 ttl=64 time=0.281 ms
[1675173015.786908] 64 bytes from 192.168.0.2: icmp_seq=135 ttl=64 time=6.52 ms
[1675173015.805923] 64 bytes from 192.168.0.2: icmp_seq=136 ttl=64 time=15.5 ms
[1675173015.807850] 64 bytes from 192.168.0.2: icmp_seq=137 ttl=64 time=6.50 ms
[1675173015.812628] 64 bytes from 192.168.0.2: icmp_seq=138 ttl=64 time=1.28 ms
[1675173015.821623] 64 bytes from 192.168.0.2: icmp_seq=139 ttl=64 time=0.277 ms
[1675173015.839062] 64 bytes from 192.168.0.2: icmp_seq=140 ttl=64 time=5.70 ms
[1675173015.850236] 64 bytes from 192.168.0.2: icmp_seq=141 ttl=64 time=6.88 ms
[1675173015.863290] 64 bytes from 192.168.0.2: icmp_seq=142 ttl=64 time=9.93 ms
[1675173015.863541] 64 bytes from 192.168.0.2: icmp_seq=143 ttl=64 time=0.189 ms
[1675173015.895116] 64 bytes from 192.168.0.2: icmp_seq=144 ttl=64 time=20.7 ms
[1675173015.898615] 64 bytes from 192.168.0.2: icmp_seq=145 ttl=64 time=13.2 ms
[1675173015.901702] 64 bytes from 192.168.0.2: icmp_seq=146 ttl=64 time=3.07 ms
[1675173015.967955] 64 bytes from 192.168.0.2: icmp_seq=147 ttl=64 time=59.2 ms
[1675173015.968394] 64 bytes from 192.168.0.2: icmp_seq=148 ttl=64 time=49.0 ms
[1675173015.968432] 64 bytes from 192.168.0.2: icmp_seq=149 ttl=64 time=38.0 ms
[1675173015.968459] 64 bytes from 192.168.0.2: icmp_seq=150 ttl=64 time=27.1 ms
[1675173015.968486] 64 bytes from 192.168.0.2: icmp_seq=151 ttl=64 time=16.1 ms
[1675173015.968490] 64 bytes from 192.168.0.2: icmp_seq=152 ttl=64 time=5.14 ms
[1675173015.974895] 64 bytes from 192.168.0.2: icmp_seq=153 ttl=64 time=1.54 ms
[1675173015.983385] 64 bytes from 192.168.0.2: icmp_seq=154 ttl=64 time=0.036 ms
[1675173015.994380] 64 bytes from 192.168.0.2: icmp_seq=155 ttl=64 time=0.030 ms
[1675173016.005390] 64 bytes from 192.168.0.2: icmp_seq=156 ttl=64 time=0.044 ms
[1675173016.017376] 64 bytes from 192.168.0.2: icmp_seq=157 ttl=64 time=0.030 ms
[1675173016.028388] 64 bytes from 192.168.0.2: icmp_seq=158 ttl=64 time=0.039 ms
[1675173016.057442] 64 bytes from 192.168.0.2: icmp_seq=159 ttl=64 time=18.0 ms
[1675173016.057612] 64 bytes from 192.168.0.2: icmp_seq=160 ttl=64 time=7.26 ms
[1675173016.109974] 64 bytes from 192.168.0.2: icmp_seq=161 ttl=64 time=49.5 ms
[1675173016.109994] 64 bytes from 192.168.0.2: icmp_seq=162 ttl=64 time=38.6 ms
[1675173016.110690] 64 bytes from 192.168.0.2: icmp_seq=163 ttl=64 time=28.3 ms
[1675173016.110756] 64 bytes from 192.168.0.2: icmp_seq=164 ttl=64 time=17.4 ms
[1675173016.110760] 64 bytes from 192.168.0.2: icmp_seq=165 ttl=64 time=6.41 ms
[1675173016.120046] 64 bytes from 192.168.0.2: icmp_seq=166 ttl=64 time=5.70 ms
[1675173016.127138] 64 bytes from 192.168.0.2: icmp_seq=167 ttl=64 time=2.79 ms
[1675173016.135776] 64 bytes from 192.168.0.2: icmp_seq=168 ttl=64 time=1.43 ms
[1675173016.145209] 64 bytes from 192.168.0.2: icmp_seq=169 ttl=64 time=0.863 ms
[1675173016.155711] 64 bytes from 192.168.0.2: icmp_seq=170 ttl=64 time=0.365 ms
[1675173016.166540] 64 bytes from 192.168.0.2: icmp_seq=171 ttl=64 time=0.189 ms
[1675173016.178774] 64 bytes from 192.168.0.2: icmp_seq=172 ttl=64 time=1.42 ms
[1675173016.190147] 64 bytes from 192.168.0.2: icmp_seq=173 ttl=64 time=2.79 ms
[1675173016.201439] 64 bytes from 192.168.0.2: icmp_seq=174 ttl=64 time=4.09 ms
[1675173016.224064] 64 bytes from 192.168.0.2: icmp_seq=175 ttl=64 time=16.6 ms
[1675173016.224242] 64 bytes from 192.168.0.2: icmp_seq=176 ttl=64 time=4.90 ms
[1675173016.231233] 64 bytes from 192.168.0.2: icmp_seq=177 ttl=64 time=1.88 ms
[1675173016.249589] 64 bytes from 192.168.0.2: icmp_seq=178 ttl=64 time=10.2 ms
[1675173016.268370] 64 bytes from 192.168.0.2: icmp_seq=179 ttl=64 time=18.7 ms
[1675173016.271008] 64 bytes from 192.168.0.2: icmp_seq=180 ttl=64 time=10.6 ms
[1675173016.275313] 64 bytes from 192.168.0.2: icmp_seq=181 ttl=64 time=4.29 ms
[1675173016.282357] 64 bytes from 192.168.0.2: icmp_seq=182 ttl=64 time=1.33 ms
[1675173016.292733] 64 bytes from 192.168.0.2: icmp_seq=183 ttl=64 time=1.71 ms
[1675173016.302145] 64 bytes from 192.168.0.2: icmp_seq=184 ttl=64 time=1.11 ms
[1675173016.311107] 64 bytes from 192.168.0.2: icmp_seq=185 ttl=64 time=0.089 ms
[1675173016.321836] 64 bytes from 192.168.0.2: icmp_seq=186 ttl=64 time=0.490 ms
[1675173016.371736] 64 bytes from 192.168.0.2: icmp_seq=187 ttl=64 time=39.3 ms
[1675173016.378944] 64 bytes from 192.168.0.2: icmp_seq=188 ttl=64 time=35.6 ms
[1675173016.435794] 64 bytes from 192.168.0.2: icmp_seq=189 ttl=64 time=81.4 ms
[1675173016.443453] 64 bytes from 192.168.0.2: icmp_seq=190 ttl=64 time=78.1 ms
[1675173016.524635] 64 bytes from 192.168.0.2: icmp_seq=191 ttl=64 time=145 ms

 

Running the same ping via a 1G interfaces seems fine, even though there are the occasional 1 ms blips, but nothing as on the 100G interface.

[1675173082.393453] 64 bytes from 10.1.13.44: icmp_seq=138 ttl=64 time=0.110 ms
[1675173082.404448] 64 bytes from 10.1.13.44: icmp_seq=139 ttl=64 time=0.108 ms
[1675173082.417221] 64 bytes from 10.1.13.44: icmp_seq=140 ttl=64 time=1.86 ms
[1675173082.426623] 64 bytes from 10.1.13.44: icmp_seq=141 ttl=64 time=1.26 ms
[1675173082.436077] 64 bytes from 10.1.13.44: icmp_seq=142 ttl=64 time=0.724 ms
[1675173082.446456] 64 bytes from 10.1.13.44: icmp_seq=143 ttl=64 time=0.110 ms
[1675173082.457451] 64 bytes from 10.1.13.44: icmp_seq=144 ttl=64 time=0.111 ms
[1675173082.468458] 64 bytes from 10.1.13.44: icmp_seq=145 ttl=64 time=0.113 ms
[1675173082.480496] 64 bytes from 10.1.13.44: icmp_seq=146 ttl=64 time=0.123 ms
[1675173082.492509] 64 bytes from 10.1.13.44: icmp_seq=147 ttl=64 time=0.144 ms
[1675173082.503460] 64 bytes from 10.1.13.44: icmp_seq=148 ttl=64 time=0.112 ms
[1675173082.515452] 64 bytes from 10.1.13.44: icmp_seq=149 ttl=64 time=0.112 ms
[1675173082.526450] 64 bytes from 10.1.13.44: icmp_seq=150 ttl=64 time=0.108 ms
[1675173082.538472] 64 bytes from 10.1.13.44: icmp_seq=151 ttl=64 time=0.117 ms
[1675173082.549466] 64 bytes from 10.1.13.44: icmp_seq=152 ttl=64 time=0.114 ms
[1675173082.560548] 64 bytes from 10.1.13.44: icmp_seq=153 ttl=64 time=0.200 ms
[1675173082.571490] 64 bytes from 10.1.13.44: icmp_seq=154 ttl=64 time=0.117 ms
[1675173082.582479] 64 bytes from 10.1.13.44: icmp_seq=155 ttl=64 time=0.132 ms
[1675173082.593464] 64 bytes from 10.1.13.44: icmp_seq=156 ttl=64 time=0.111 ms
[1675173082.605164] 64 bytes from 10.1.13.44: icmp_seq=157 ttl=64 time=0.816 ms
[1675173082.618573] 64 bytes from 10.1.13.44: icmp_seq=158 ttl=64 time=2.21 ms
[1675173082.628269] 64 bytes from 10.1.13.44: icmp_seq=159 ttl=64 time=1.91 ms
[1675173082.638089] 64 bytes from 10.1.13.44: icmp_seq=160 ttl=64 time=1.73 ms
[1675173082.646502] 64 bytes from 10.1.13.44: icmp_seq=161 ttl=64 time=0.154 ms
[1675173082.658471] 64 bytes from 10.1.13.44: icmp_seq=162 ttl=64 time=0.117 ms
[1675173082.669465] 64 bytes from 10.1.13.44: icmp_seq=163 ttl=64 time=0.114 ms
[1675173082.681494] 64 bytes from 10.1.13.44: icmp_seq=164 ttl=64 time=0.144 ms
[1675173082.693460] 64 bytes from 10.1.13.44: icmp_seq=165 ttl=64 time=0.113 ms
[1675173082.705500] 64 bytes from 10.1.13.44: icmp_seq=166 ttl=64 time=0.135 ms
[1675173082.717926] 64 bytes from 10.1.13.44: icmp_seq=167 ttl=64 time=1.57 ms
[1675173082.727582] 64 bytes from 10.1.13.44: icmp_seq=168 ttl=64 time=1.22 ms
[1675173082.736857] 64 bytes from 10.1.13.44: icmp_seq=169 ttl=64 time=0.502 ms
[1675173082.747861] 64 bytes from 10.1.13.44: icmp_seq=170 ttl=64 time=0.512 ms
[1675173082.758455] 64 bytes from 10.1.13.44: icmp_seq=171 ttl=64 time=0.110 ms
[1675173082.769514] 64 bytes from 10.1.13.44: icmp_seq=172 ttl=64 time=0.171 ms
[1675173082.780518] 64 bytes from 10.1.13.44: icmp_seq=173 ttl=64 time=0.168 ms
[1675173082.791454] 64 bytes from 10.1.13.44: icmp_seq=174 ttl=64 time=0.111 ms
[1675173082.802528] 64 bytes from 10.1.13.44: icmp_seq=175 ttl=64 time=0.186 ms
[1675173082.813474] 64 bytes from 10.1.13.44: icmp_seq=176 ttl=64 time=0.133 ms
[1675173082.824450] 64 bytes from 10.1.13.44: icmp_seq=177 ttl=64 time=0.110 ms
[1675173082.836544] 64 bytes from 10.1.13.44: icmp_seq=178 ttl=64 time=0.112 ms
[1675173082.849002] 64 bytes from 10.1.13.44: icmp_seq=179 ttl=64 time=1.65 ms
[1675173082.858476] 64 bytes from 10.1.13.44: icmp_seq=180 ttl=64 time=1.13 ms
[1675173082.867954] 64 bytes from 10.1.13.44: icmp_seq=181 ttl=64 time=0.607 ms
[1675173082.878464] 64 bytes from 10.1.13.44: icmp_seq=182 ttl=64 time=0.120 ms
[1675173082.889451] 64 bytes from 10.1.13.44: icmp_seq=183 ttl=64 time=0.110 ms
[1675173082.900458] 64 bytes from 10.1.13.44: icmp_seq=184 ttl=64 time=0.108 ms
[1675173082.911450] 64 bytes from 10.1.13.44: icmp_seq=185 ttl=64 time=0.110 ms
[1675173082.922454] 64 bytes from 10.1.13.44: icmp_seq=186 ttl=64 time=0.112 ms
[1675173082.933463] 64 bytes from 10.1.13.44: icmp_seq=187 ttl=64 time=0.114 ms
[1675173082.944523] 64 bytes from 10.1.13.44: icmp_seq=188 ttl=64 time=0.111 ms
[1675173082.955465] 64 bytes from 10.1.13.44: icmp_seq=189 ttl=64 time=0.109 ms
[1675173082.967957] 64 bytes from 10.1.13.44: icmp_seq=190 ttl=64 time=1.59 ms
[1675173082.976694] 64 bytes from 10.1.13.44: icmp_seq=191 ttl=64 time=0.347 ms
[1675173082.987457] 64 bytes from 10.1.13.44: icmp_seq=192 ttl=64 time=0.112 ms
[1675173082.998479] 64 bytes from 10.1.13.44: icmp_seq=193 ttl=64 time=0.136 ms
[1675173083.009618] 64 bytes from 10.1.13.44: icmp_seq=194 ttl=64 time=0.275 ms
[1675173083.020456] 64 bytes from 10.1.13.44: icmp_seq=195 ttl=64 time=0.110 ms
[1675173083.031453] 64 bytes from 10.1.13.44: icmp_seq=196 ttl=64 time=0.108 ms
[1675173083.042476] 64 bytes from 10.1.13.44: icmp_seq=197 ttl=64 time=0.129 ms
[1675173083.053465] 64 bytes from 10.1.13.44: icmp_seq=198 ttl=64 time=0.117 ms
[1675173083.064456] 64 bytes from 10.1.13.44: icmp_seq=199 ttl=64 time=0.110 ms
[1675173083.075474] 64 bytes from 10.1.13.44: icmp_seq=200 ttl=64 time=0.132 ms
[1675173083.086457] 64 bytes from 10.1.13.44: icmp_seq=201 ttl=64 time=0.114 ms
[1675173083.098445] 64 bytes from 10.1.13.44: icmp_seq=202 ttl=64 time=0.108 ms
[1675173083.110461] 64 bytes from 10.1.13.44: icmp_seq=203 ttl=64 time=0.117 ms
[1675173083.122452] 64 bytes from 10.1.13.44: icmp_seq=204 ttl=64 time=0.110 ms
[1675173083.134458] 64 bytes from 10.1.13.44: icmp_seq=205 ttl=64 time=0.108 ms
[1675173083.145453] 64 bytes from 10.1.13.44: icmp_seq=206 ttl=64 time=0.113 ms
[1675173083.156451] 64 bytes from 10.1.13.44: icmp_seq=207 ttl=64 time=0.108 ms
[1675173083.167655] 64 bytes from 10.1.13.44: icmp_seq=208 ttl=64 time=0.313 ms
[1675173083.178468] 64 bytes from 10.1.13.44: icmp_seq=209 ttl=64 time=0.125 ms
[1675173083.189451] 64 bytes from 10.1.13.44: icmp_seq=210 ttl=64 time=0.107 ms
[1675173083.200450] 64 bytes from 10.1.13.44: icmp_seq=211 ttl=64 time=0.107 ms
[1675173083.211464] 64 bytes from 10.1.13.44: icmp_seq=212 ttl=64 time=0.115 ms
[1675173083.222468] 64 bytes from 10.1.13.44: icmp_seq=213 ttl=64 time=0.113 ms
[1675173083.234456] 64 bytes from 10.1.13.44: icmp_seq=214 ttl=64 time=0.113 ms
[1675173083.245461] 64 bytes from 10.1.13.44: icmp_seq=215 ttl=64 time=0.115 ms
[1675173083.256613] 64 bytes from 10.1.13.44: icmp_seq=216 ttl=64 time=0.264 ms
[1675173083.268470] 64 bytes from 10.1.13.44: icmp_seq=217 ttl=64 time=0.119 ms
[1675173083.279454] 64 bytes from 10.1.13.44: icmp_seq=218 ttl=64 time=0.115 ms
[1675173083.290449] 64 bytes from 10.1.13.44: icmp_seq=219 ttl=64 time=0.107 ms
[1675173083.302723] 64 bytes from 10.1.13.44: icmp_seq=220 ttl=64 time=0.379 ms
[1675173083.315176] 64 bytes from 10.1.13.44: icmp_seq=221 ttl=64 time=1.83 ms
[1675173083.324377] 64 bytes from 10.1.13.44: icmp_seq=222 ttl=64 time=1.03 ms
[1675173083.333549] 64 bytes from 10.1.13.44: icmp_seq=223 ttl=64 time=0.122 ms
[1675173083.344456] 64 bytes from 10.1.13.44: icmp_seq=224 ttl=64 time=0.114 ms
[1675173083.355450] 64 bytes from 10.1.13.44: icmp_seq=225 ttl=64 time=0.108 ms
[1675173083.366453] 64 bytes from 10.1.13.44: icmp_seq=226 ttl=64 time=0.109 ms
[1675173083.377448] 64 bytes from 10.1.13.44: icmp_seq=227 ttl=64 time=0.107 ms
[1675173083.388456] 64 bytes from 10.1.13.44: icmp_seq=228 ttl=64 time=0.109 ms
[1675173083.399452] 64 bytes from 10.1.13.44: icmp_seq=229 ttl=64 time=0.106 ms
[1675173083.410475] 64 bytes from 10.1.13.44: icmp_seq=230 ttl=64 time=0.106 ms
[1675173083.421450] 64 bytes from 10.1.13.44: icmp_seq=231 ttl=64 time=0.109 ms
[1675173083.433449] 64 bytes from 10.1.13.44: icmp_seq=232 ttl=64 time=0.110 ms
[1675173083.445450] 64 bytes from 10.1.13.44: icmp_seq=233 ttl=64 time=0.110 ms
[1675173083.457447] 64 bytes from 10.1.13.44: icmp_seq=234 ttl=64 time=0.107 ms
[1675173083.468451] 64 bytes from 10.1.13.44: icmp_seq=235 ttl=64 time=0.112 ms
[1675173083.480451] 64 bytes from 10.1.13.44: icmp_seq=236 ttl=64 time=0.112 ms
[1675173083.492446] 64 bytes from 10.1.13.44: icmp_seq=237 ttl=64 time=0.108 ms
[1675173083.503455] 64 bytes from 10.1.13.44: icmp_seq=238 ttl=64 time=0.110 ms
[1675173083.514449] 64 bytes from 10.1.13.44: icmp_seq=239 ttl=64 time=0.109 ms
[1675173083.527153] 64 bytes from 10.1.13.44: icmp_seq=240 ttl=64 time=1.80 ms
[1675173083.537368] 64 bytes from 10.1.13.44: icmp_seq=241 ttl=64 time=2.01 ms
[1675173083.546933] 64 bytes from 10.1.13.44: icmp_seq=242 ttl=64 time=1.58 ms
[1675173083.557957] 64 bytes from 10.1.13.44: icmp_seq=243 ttl=64 time=2.60 ms
[1675173083.565906] 64 bytes from 10.1.13.44: icmp_seq=244 ttl=64 time=0.556 ms
[1675173083.576804] 64 bytes from 10.1.13.44: icmp_seq=245 ttl=64 time=0.457 ms
[1675173083.587463] 64 bytes from 10.1.13.44: icmp_seq=246 ttl=64 time=0.117 ms
[1675173083.598485] 64 bytes from 10.1.13.44: icmp_seq=247 ttl=64 time=0.124 ms
[1675173083.609483] 64 bytes from 10.1.13.44: icmp_seq=248 ttl=64 time=0.134 ms
[1675173083.620496] 64 bytes from 10.1.13.44: icmp_seq=249 ttl=64 time=0.124 ms
[1675173083.631448] 64 bytes from 10.1.13.44: icmp_seq=250 ttl=64 time=0.108 ms
[1675173083.642508] 64 bytes from 10.1.13.44: icmp_seq=251 ttl=64 time=0.166 ms
[1675173083.653458] 64 bytes from 10.1.13.44: icmp_seq=252 ttl=64 time=0.109 ms
[1675173083.664460] 64 bytes from 10.1.13.44: icmp_seq=253 ttl=64 time=0.115 ms
[1675173083.675525] 64 bytes from 10.1.13.44: icmp_seq=254 ttl=64 time=0.157 ms
[1675173083.686509] 64 bytes from 10.1.13.44: icmp_seq=255 ttl=64 time=0.167 ms
[1675173083.697459] 64 bytes from 10.1.13.44: icmp_seq=256 ttl=64 time=0.112 ms
[1675173083.708447] 64 bytes from 10.1.13.44: icmp_seq=257 ttl=64 time=0.107 ms
[1675173083.719460] 64 bytes from 10.1.13.44: icmp_seq=258 ttl=64 time=0.115 ms
[1675173083.730459] 64 bytes from 10.1.13.44: icmp_seq=259 ttl=64 time=0.108 ms
[1675173083.742450] 64 bytes from 10.1.13.44: icmp_seq=260 ttl=64 time=0.107 ms
[1675173083.753446] 64 bytes from 10.1.13.44: icmp_seq=261 ttl=64 time=0.106 ms
[1675173083.764454] 64 bytes from 10.1.13.44: icmp_seq=262 ttl=64 time=0.109 ms
[1675173083.775448] 64 bytes from 10.1.13.44: icmp_seq=263 ttl=64 time=0.106 ms
[1675173083.786458] 64 bytes from 10.1.13.44: icmp_seq=264 ttl=64 time=0.113 ms
[1675173083.797448] 64 bytes from 10.1.13.44: icmp_seq=265 ttl=64 time=0.108 ms
[1675173083.808449] 64 bytes from 10.1.13.44: icmp_seq=266 ttl=64 time=0.107 ms
[1675173083.819458] 64 bytes from 10.1.13.44: icmp_seq=267 ttl=64 time=0.111 ms
[1675173083.830457] 64 bytes from 10.1.13.44: icmp_seq=268 ttl=64 time=0.110 ms
[1675173083.841455] 64 bytes from 10.1.13.44: icmp_seq=269 ttl=64 time=0.108 ms
[1675173083.852452] 64 bytes from 10.1.13.44: icmp_seq=270 ttl=64 time=0.109 ms
[1675173083.863473] 64 bytes from 10.1.13.44: icmp_seq=271 ttl=64 time=0.125 ms
[1675173083.874473] 64 bytes from 10.1.13.44: icmp_seq=272 ttl=64 time=0.134 ms

The 1G interface runs over a separate HP switch.

I have also captured a latency spike in tcpdump:

host1

10:26:14.012285 IP 192.168.0.1 > 192.168.0.2: ICMP echo request, id 63872, seq 283, length 64 
10:26:14.066509 IP 192.168.0.2 > 192.168.0.1: ICMP echo reply, id 63872, seq 283, length 64

host2

10:26:14.014328 IP 192.168.0.1 > 192.168.0.2: ICMP echo request, id 63872, seq 283, length 64 
10:26:14.014372 IP 192.168.0.2 > 192.168.0.1: ICMP echo reply, id 63872, seq 283, length 64

 

How can those latency spikes and issues be explained?
What can I do to solve them? How do I figure out what's causing them?