Detailed explanation of the causes of Redis blocking

Discovered obstruction

The online application service is the first to sense it, and can add abnormal statistics on the application side and alert through email, SMS, and WeChat .

Use the log system to count exceptions and trigger alarm logic

Use the Redis monitoring system to discover blocking problems and trigger alarms. Recommended CacheCloud system.

Internal reasons

Unreasonable use of API or data structures

For high concurrency scenarios, avoid executing on large objects Commands whose algorithm complexity exceeds O(n)O(n).

Found slow query: slowlog get {n}

Found large object: redis-cli -h{ip} -p{port} bigkeys

CPU saturation

CPU saturation refers to redis running the single-core CPU to 100%.

The top command checks the CPU usage of the redis process

redis-cli -h{ip} -p{port} –stat obtains the current redis usage and determines whether the concurrency has reached the limit

info commandstats Analysis command consumes unreasonable time, possibly excessive memory optimization

Persistence blocking

1. Fork blocking

occurs in RDB or When AOF is rewritten, the redis main thread calls fork to generate a child process to complete the persistent file rewriting

Use the info stats command to obtain the lastest_fork_usec indicator, which indicates that the latest fork operation of redis took time

2. AOF Blocking of disk brushing

Enable AOF, and file disk flushing is usually done once per second. When the hard disk pressure is too high, fsync needs to wait for the writing to complete

Check the aof_delayed_fsync indicator in the redis log or info persistence statistics

If you can use iotop, which process may consume too much hard disk resources

3. HugePage write operation blocking

For operating systems that enable Transparent HugePages, each write command causes Copying the memory page unit from 4KB to 2MB

will slow down the execution time of write operations, resulting in a large number of slow queries for write operations

External reasons

CPU competition

1. Process competition: redis is a typical CPU-intensive application. Use the top and sar commands to locate the time point and process of CPU consumption

2. Bind the CPU: A common optimization is to bind the redis process to the CPU, which reduces the CPU context switching overhead. If the fork child process does If the CPU is bound, there will be fierce CPU competition between the parent and child processes, which will greatly affect the stability of redis.

Memory swap

If the operating system swaps out the memory used by redis to the hard disk, the performance of redis after the swap will drop sharply.

Checking method to identify redis memory exchange:

1. Query the redis process number

redis-cli info server | grep process_id

2. Query the memory exchange information based on the process number

cat /proc/{process_id}/smaps | grep Swap

If The exchange volume is all 0KB or 4KB in some cases, which is normal.

Prevent memory exchange:

1. Ensure that the machine has sufficient available memory

2. Ensure that all redis examples set the maximum available memory (maxmemory) to prevent redis memory under extreme circumstances Uncontrollable growth

3. Reduce the system’s swap priority, such as echo 10>/proc/sys/vm/swappiness

Network problems

1. Connection rejection

Network interruption: Generally in the case of network cutover or bandwidth exhaustion

Redis connection rejection:

Reject new connections when the number of connections is greater than maxclients When the connection is entered, the rejected_connections indicator of info stats

Try to use NIO long connections or connection pools when the client accesses redis

redis is used in scenarios where a large number of distributed nodes are accessed and the life cycle is short ( Such as Map/Reduce), it is recommended to set the tcp-keepalive and timeout parameters to allow redis to actively check and close invalid connections

Connection overflow:

Process limit: The maximum number of files that a process can open is controlled, ulimit -n, usually 1024, redis with a large number of connections needs to increase this value

Backlog queue overflow: The system uses the backlog queue to save tcp connections on specific ports, redis defaults to 511, system backlog defaults to 128, and cron can be used online. Regularly execute netstat -s | grep overflowed statistics

2. Network delay

Measure the network delay between machines

redis-cli -h{ip} -p{port} –latency
redis-cli -h{ip} -p{port} –latency-history 默认15秒完成一行统计，-i控制采样时间
redis-cli -h{ip} -p{port} –latency-dist 统计图展示，每1秒采样一次

3. Network card soft interrupt

Single The network card queue can only use one CPU. Under high concurrency, network card data interaction is concentrated on the same CPU, resulting in a situation where multi-core CPUs cannot be fully utilized.

Generally appears in scenarios with high network traffic throughput

For more redis knowledge, please pay attention to the redis introductory tutorial column.

The above is the detailed content of Detailed explanation of the causes of Redis blocking. For more information, please follow other related articles on the PHP Chinese website!