How Redis prevents frequent main database switching in sentry mode_Troubleshoot monitoring jitter and improve down-after-milliseconds

The root cause of Sentinel's misjudgment that the main database is offline is that network jitter or the response delay of the main database is mistakenly captured by down-after-milliseconds; subjective offline only relies on a single sentinel timeout, and objective offline requires majority consensus. However, simultaneous timeout of multiple sentinels will trigger false switching.

Why did Sentinel misjudge that the main database was offline?

Sentinel frequently switches the main library. The fundamental reason is not that the configuration is too aggressive, but that the network jitter or the main library response delay is mistakenly captured by down-after-milliseconds . The sentinel's judgment of subjective offline (sdown) only relies on the ping response timeout of a single sentinel without negotiation; while the objective offline (odown) requires the majority of sentinels to reach agreement - but if multiple sentinels time out due to network delay at the same time, a failover will be quickly triggered.

Common causes include: occasional packet loss on the intranet, short-term filling of the main database CPU leading to delayed INFO or PING responses, cross-machine room deployment of Sentinel and the main database, and excessive load on Sentinel itself (such as monitoring a large number of instances without tuning).

How to set down-after-milliseconds reasonably?

This value is not as small as possible, nor can it be set to 5000ms or 30000ms. It must be greater than 3~5 times the "average command processing time of the normal network RTT main library", and leave a buffer margin. The actual test recommendations are as follows:

• LAN (same computer room) deployment: start from 3000 , observe the frequency of sdown in the log, and gradually increase it to 5000 ~ 8000
• Cross-computer room or high-latency link: set to at least 15000 , and check sentinel ping-reply-timeout simultaneously (default 1000ms), increase to 2000 if necessary
• If the main database often has slow queries (such as unoptimized KEYS * ), it is necessary to combine slowlog-log-slower-than and monitoring to confirm the glitch cycle. down-after-milliseconds should be significantly longer than this cycle.

After modification, you must restart the sentinel process one by one (or use SENTINEL SET to take effect dynamically). Simply changing the configuration file will not take effect.

What monitoring indicators can detect jitter in advance?

Don't just stare at switch-master log. The really useful signals are hidden in Sentinel’s own indicators:

• sentinel_masters : A sudden drop to 0 means that all sentinels have lost contact with the main database, which is likely to be a network partition.
• sentinel_running_scripts : persistent > 0 indicates that the failover script is stuck, possibly due to notification service timeout.
• sentinel_tilt : A value of 1 indicates that the sentinel enters "tilt mode" (internal clock abnormality). At this time, all subjective offline determinations are suspended. System time synchronization ( ntpq -p ) and CPU load need to be checked.
• connected_clients and used_cpu_sys of the main library suddenly increase, and combined with FAIL message timestamp in the sentinel log, it can be determined whether the problem is caused by the main library itself.

It is recommended to use Prometheus to capture the redis_sentinel_* indicators, make histogram statistics on sentinel_odown_time , and identify whether batch triggering occurs within a certain few minutes.

In addition to parameter adjustment, what other hard circumvention methods are there?

Parameters are just the bottom line, the architectural level is more critical:

• Ensure that the number of sentinels is an odd number (3 or 5) and **dispersed in different physical machines/availability zones** to avoid single-point host failure and collapse of most sentinels.
• Disable the default value of sentinel failover-timeout (60000ms) and set it to more than 180000 to prevent the new master election from being forced before the network is restored.
• Turn off unnecessary sentinel notifications (such as sentinel notification-script ), or ensure that the script has idempotence and timeout control ( timeout 5s ./notify.sh )
• The main library turns on tcp-keepalive 60 to reduce false timeouts caused by idle disconnection of intermediate devices (such as cloud vendor SLB).

The most easily overlooked thing is the resource limit of Sentinel itself: each Sentinel can manage up to 50 master nodes by default. After exceeding this limit, the heartbeat detection delay increases, and the actual effect of down-after-milliseconds is compromised. Monitor sentinel_masters and note whether sentinel_monitors are close to the upper limit.

The above is the detailed content of How Redis prevents frequent main database switching in sentry mode_Troubleshoot monitoring jitter and improve down-after-milliseconds. For more information, please follow other related articles on the PHP Chinese website!