How to use iPerf to test and troubleshoot UDP packet loss

The content of this article is about how to use iPerf to test and troubleshoot UDP packet loss problems. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

Use iPerf to test and troubleshoot UDP packet loss issues

Phenomenon description

Use high-speed channels to open the same region (Region After setting up two ECS instances of the VPC network type under The rate shows an increasing trend. As shown below:

##Problem Analysis

Assume that the private IPs of two network type ECS instances are VPC ECS A (192.168.104.235) and ECS B (10.182.83.13), and use Netcat (NC) to monitor and send UDP data packets, then the communication link diagram between ECS instance A and instance B of the network type is as follows:

The data flow direction is:

ECS A(192.168.104.235)-> NC 1(100.105.59.3)-> VGW(10.141.166.253)-> NC 2(100.105.59.9)-> ECS B(10.182.83.13)

We need to troubleshoot and analyze its link to find out the ultimate cause of packet loss.

Solution

Note: Since only the communication between source Netcat (i.e. NC 1) and destination Netcat (i.e. NC 2) is seen , packet capture and troubleshooting should avoid misunderstandings, that is, arbitrarily judging that packet loss is caused by direct communication between Netcat (NC).

During the investigation, it will be found that the captured packet of eth0 at the source end is sent to VGW, but when the packet is captured at the destination end, it is found that the shell encapsulates the destination NC 2 IP, as in the example:

 [Time ] 17:32:07.130844   Point: `input `
 [ETHER] 24:4c:07:33:0e:02 -> 00:04:37:28:00:65, eth_type: 0x0800
 [IPv4 ] 100.105.59.3 -> 10.141.166.253
 proto: 17, ver: 04, ihl: 05, len: 1534, ident: 59824,R: 0, DF: 1, MF: 0, offset: 0, ttl: 60, chksum: 0xfe47
 [UDP  ] sport: 46703, dport: 250, size: 1514, chksum: 0x0000
 [VxLan] debug_flag: 0, vlan_tag: 0, payload_type: 0, version: 1, tunnel_id: 1878597, tos: 0, tof: 0
 [IPv4 ] 192.168.104.235 -> 10.182.83.13
 proto: 17, ver: 04, ihl: 05, len: 1498, ident: 55469,R: 0, DF: 1, MF: 0, offset: 0, ttl: 64, chksum: 0xd50e
 [UDP  ] sport: 36687, dport: 5001, size: 1478, chksum: 0xa0aa
 [Time ] 17:32:07.130854   Point: `output`
 [ETHER] 24:4c:07:33:0e:02 -> 00:04:37:28:00:65, eth_type: 0x0800
 [IPv4 ] 100.105.59.3 -> 100.105.59.9
 proto: 17, ver: 04, ihl: 05, len: 1534, ident: 59824,R: 0, DF: 1, MF: 0, offset: 0, ttl: 60, chksum: 0x0000
 [UDP  ] sport: 46703, dport: 250, size: 1514, chksum: 0x0000
 [VxLan] debug_flag: 0, vlan_tag: 0, payload_type: 0, version: 1, tunnel_id: 2125861, tos: 0, tof: 0
 [IPv4 ] 192.168.104.235 -> 10.182.83.13
 proto: 17, ver: 04, ihl: 05, len: 1498, ident: 55469,R: 0, DF: 1, MF: 0, offset: 0, ttl: 64, chksum: 0xd50e
 [UDP  ] sport: 36687, dport: 5001, size: 1478, chksum: 0xa0aa

Confirm that the data packet passes After VGW, start counting packet capture information:

ECS A sends UDP traffic through iPerf: iperf -c 10.182.83.13 -u -b 600m

ECS B receives through iPerf: iperf -u - s

Capture packets inside the instance.

ECS A：sudo tcpdump -w ~/client.pcap -n -i eth0 src host 192.168.104.25 and src port 1234
ECS B：sudo tcpdump -w ~/server.pcap -n -i eth0 src host 192.168.104.25 and src port 1234

Capture packets at two NC eth0 locations.

NC 1：sudo houyi-tcpdump -w /apsara/i-6we6pnh19n2q7srkgomd.pcap -nnK -i eth0
 udp and src inner_port 1234 and dst inner_host 10.182.83.13
NC 2：sudo houyi-tcpdump -B 4096 -w /apsara/i-6we53i9h3ducbju5rmuw.pap -nn -i eth0 
udp -K and src inner_host 192.168.104.235 and src inner_port 1234

Deploy flow systems in ASW and LSW.

100.105.59.3:46728 -> 10.141.166.253:250

Note: Since the destination packet shell automatically encapsulates the destination NC 1 IP, the message format of the VGW side data packet is: 100.105.59.3:46728 -> 100.105.59.9:250.

Analysis based on packet capture results.

ECS A packet loss/packet delivery: 171/510203

NC 1 eth0 packet delivery: 510204

ASW and LSW flow statistics packet delivery: 510204

NC 2 eth0 packet receiving: 510204

ECS B packet receiving: 510204, capture 507442, dropped by kernel 2162

The above analysis locates the packet loss of the instance protocol stack, and adjusts the internal UDP Buffer Sizes of the instance. Adjust the network stack (Stack). The default UDF Buffer Size is 212992 (208 KB), and you can adjust it to 2097152 (2 MB).

/proc/sys/net/core/rmem_default #默认的接收数据包内存大小
/proc/sys/net/core/rmem_max #最大的接收数据包内存大小

Test UDP packet loss after adjustment.

The above is the detailed content of How to use iPerf to test and troubleshoot UDP packet loss. For more information, please follow other related articles on the PHP Chinese website!