listen queue of socket fd: 3 Error analysis

Nowadays, Django applications are basically deployed using uwsgi. An error similar to the following listen queue of socket "127.0.0.1:9001" (fd: 3) has occurred twice. Let’s talk about these two errors below. the solution process.

Error scenario

• centos 6.6
• uwsgi2.0
• nginx1.6

Error log interception

<code><span>Tue</span><span>Jun</span><span>2</span><span>17</span>:<span>33</span>:<span>27</span><span>2015</span> - *** <span>uWSGI</span><span>listen</span><span>queue</span><span>of</span><span>socket</span><span>"127.0.0.1:9001"</span> (<span>fd</span>: <span>3</span>) <span>full</span><span>!</span><span>!</span><span>!</span> (<span>101</span>/<span>100</span>) ***
<span>Tue</span><span>Jun</span><span>2</span><span>17</span>:<span>33</span>:<span>28</span><span>2015</span> - *** <span>uWSGI</span><span>listen</span><span>queue</span><span>of</span><span>socket</span><span>"127.0.0.1:9001"</span> (<span>fd</span>: <span>3</span>) <span>full</span><span>!</span><span>!</span><span>!</span> (<span>101</span>/<span>100</span>) ***</code>

• The first time was because the firewall of China Unicom computer room was misconfigured, which restricted the server output, that is There is no problem in sending external packages to the server, but when the server returns the package to the outside, it is very slow and almost unavailable. At this time, a large number of errors appear in the uwsgi log

• The second time is after the concurrency volume increased sharply, the active link remained Around 6000, this error appeared in large numbers.

Analysis

Based on this error, we searched for relevant information. It should be a problem with system-level parameters. For details, please refer to linux man page listen(2).

lzzNote: A simple understanding is that each listener For sockets, before there is an accept, the length of the socket waiting for processing is 128 by default in Linux (at least in centos6.6), and the default is 100 in my compiled uwsgi, which means that before adjusting the system parameters, The highest is 128. So how can we adjust the length of the

? * System parameters must be adjusted to make it effective * You must adjust the uwsgi configuration and then restart the application

Operation

Modify the system parameters

The configuration file is modified directly here, and it will still be valid after restarting.

Modify the /etc/sysctl.conf file, add or modify these parameter values ​​

<code><span>#对于一个经常处理新连接的高负载 web服务环境来说，默认的 128 太小了</span>
net<span>.core</span><span>.somaxconn</span> = <span>262144</span>
?<span>#表示SYN<strong>队列</strong>的长度，默认为1024，加大<strong>队列</strong>长度为8192，可以容纳更多等待连接的网络连接数</span>
net<span>.ipv</span>4<span>.tcp</span>_max_syn_backlog = <span>8192</span><span>#网卡设备将请求放入<strong>队列</strong>的长度</span>
net<span>.core</span><span>.netdev</span>_max_backlog = <span>65536</span></code>

After the modification is completed, remember to

Reload the parameters uwsgi adjustment

Whether it is configuration or adding one on the command line options, for example, add the following configuration to the .ini file

<code><span>listen</span>=<span>1024</span></code>

, then restart the application and reload the configuration.

Summary

By modifying the configuration, this error has almost never occurred, and the system throughput and concurrency have been greatly improved. So system characteristics and tuning are very important to improve the overall service quality.

Reference

• listen(2) - Linux man page