How to detect faults in Redis cluster?

Method for fault detection in Redis cluster: first modify the Python script and write a piece of data every 1 second; then write data to the Redis cluster in a loop, and forcefully kill a master node to observe the application connection situation; Finally, restart the downed 8001 node.

Method for fault detection in Redis cluster: automatic failover test

Modify the Python script and write every 1s The purpose of entering a piece of data is to facilitate observation of the impact on the application within the time period (about 1 second) when the master node goes down and the cluster automatically fails over, or the performance of the application before and after automatic failover.

The following script writes data to the Redis cluster in a loop. During execution, it forcibly kills a master node and observes the application connection status.
At the same time, if an exception occurs, suspend the application for 2s, because the cluster failover time configured above is 1s. If the application is suspended for 2s, the failover process can be skipped. When the failover is completed, the application It returned to normal state. Although node 8001 was down, the application continued to connect to node 8001, but the application was completely unaware.

Related learning recommendations: redis video tutorial

import timefrom time import ctime,sleepfrom rediscluster import StrictRedisCluster
startup_nodes = [
    {"host":"111.231.253.***", "port":8001},
    {"host":"111.231.253.***", "port":8002},
    {"host":"111.231.253.***", "port":8003},
    {"host":"111.231.253.***", "port":8004},
    {"host":"111.231.253.***", "port":8005},
    {"host":"111.231.253.***", "port":8006}
]
redis_conn= StrictRedisCluster(startup_nodes=startup_nodes, decode_responses=True,password="root")for i in range(0, 100000):    try:
        redis_conn.set('name' + str(i), str(i))        print('setting name' + str(i) +"--->" + time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())))
        time.sleep(1)    except:        print("connect to redis cluster error")
        time.sleep(2)

It was found that after killing the master node, only one connection error occurred, and then because of Redis The automatic failover of the cluster is successful and transparent to the program, so the application then works normally and is not affected by the failure of one of the master nodes.

The status of the cluster at this time is that node 8001 is down. Obviously, the slave node 8004 corresponding to 8001 takes over the master node, upgrades to master, and provides services to the outside world

Observe the logs of the 8004 instance that was upgraded to the master node. You will find that after the original 8001 master node is forcibly killed, within 1 second, 8001 is successfully upgraded to the master node.
If During the failover process, no application accesses Redis, and the application does not even know that a failover has occurred in the Redis cluster. As long as the master and slave nodes of a certain node in the cluster do not go down at the same time, there will be no problem in the entire cluster, and it will not affect the application. The process is completely transparent.

Then restart the downed 8001 node, and you will find that the 8001 node automatically becomes the slave node of its original slave node (8004)

The above is the detailed content of How to detect faults in Redis cluster?. For more information, please follow other related articles on the PHP Chinese website!