This article will talk about high availability and persistence in redis, and look at the persistence function and two methods of Redis (RDB and AOF). I hope it will be helpful to you!
In web servers, high availability means that the server can operate normally The access time is measured by how long it takes to provide normal services (99.9%, 99.99%, 99.999%, etc.). [Related recommendations: Redis Video Tutorial]
But in the context of Redis, the meaning of high availability seems to be broader, in addition to ensuring the provision of normal services (such as master-slave separation, rapid disaster recovery technology), it is also necessary to consider the expansion of data capacity, data security and no loss, etc.
In Redis, the technologies to achieve high availability mainly include persistence, master-slave separation, sentinels and clusters.
| High availability strategy | Description |
|---|---|
| Persistence | Persistence is the simplest high-availability method (sometimes not even classified as a high-availability method). Its main function is data backup, that is, storing data on the hard disk to ensure that the data will not be lost due to process exit. |
| Master-slave replication | Master-slave replication is the basis of high-availability Redis. Both Sentinel and Cluster achieve high availability based on master-slave replication. Master-slave replication mainly implements multi-machine backup of data, as well as load balancing and simple fault recovery for read operations. Defects: Failure recovery cannot be automated, write operations cannot be load balanced, and storage capacity is limited by a single machine. |
| Sentinel | Based on master-slave replication, Sentinel implements automated fault recovery. Disadvantages: Write operations cannot be load balanced, and storage capacity is limited by a single machine. |
| Cluster | Through clustering, Redis solves the problem that write operations cannot be load-balanced and storage capacity is limited by a single machine, achieving a relatively complete high-availability solution. |
Redis是内存数据库，数据都是存储在内存中，为了避免服务器断电等原因导致Redis进程异常退出后数据的永久丢失，需要定期将Redis中的数据以某种形式（数据或命令）从内存保存到硬盘；当下次Redis重启时，利用持久化文件实现数据恢复。除此之外，为了进行灾难备份，可以将持久化文件拷贝到一个远程位置。
RDB持久化是指在指定的时间间隔内将内存中当前进程中的数据生成快照保存到硬盘（因此也称为快照持久化），用二进制压缩存储，保存的文件后缀是rdb；当Redis重新启动时，可以读取快照文件恢复数据。
RDB持久化的触发分为手动触发和自动触发两种。
[root@localhost ~]# vim /etc/redis/6379.conf ##219行，以下三个save条件满足任意一个时，都会引起bgsave的调用save 900 1 ##当时间到900秒时，如果redis数据发生了至少1次变化，则执行bgsavesave 300 10 ##当时间到300秒时，如果redis数据发生了至少10次变化，则执行bgsavesave 60 10000 ##当时间到60秒时，如果redis数据发生了至少10000次变化，则执行bgsave##254行，指定RDB文件名dbfilename dump.rdb##264行，指定RDB文件和AOF文件所在目录dir /var/lib/redis/6379##242行，是否开启RDB文件压缩rdbcompression yes
除了save m n以外，还有一些其他情况会触发bgsave:
RDB文件的载入工作是在服务器启动时自动执行的，并没有专门的命令。但是由于AOF的优先级更高，因此当AOF开启时，Redis会优先载入AOF文件来恢复数据；只有当AOF关闭时，才会在Redis服务器启动时检测RDB文件，并自动载入。服务器载入RDB文件期间处于阻塞状态，直到载入完成为止。
Redis载入RDB文件时，会对RDB文件进行校验，如果文件损坏，则日志中会打印错误，Redis启动失败。
RDB持久化是将进程数据写入文件，而AOF持久化则是将Redis执行的每次写、删除命令记录到单独的日志文件中，查询操作不会记录；当Redis重启时再次执行AOF文件中的命令来恢复数据。
与RDB相比，AOF的实时性更好，因此已成为主流的持久化方案。
Redis服务器默认开启RDB，关闭AOF；要开启AOF，需要在配置文件中配置
[root@localhost ~]# vim /etc/redis/6379.conf ##700行，修改，开启AOFappendonly yes##704行，指定AOF文件名称appendfilename "appendonly.aof"##796行，是否忽略最后一条可能存在问题的指令aof-load-truncated yes[root@localhost ~]# /etc/init.d/redis_6379 restartStopping ... Redis stopped Starting Redis server...
由于需要记录Redis的每条写命令，因此AOF不需要触发，下面介绍AOF的执行流程。
AOF的执行流程包括：
Redis first appends the command to the buffer instead of writing it directly to the file, mainly to avoid directly writing a command every time Writing to the hard disk causes hard disk IO to become the bottleneck of Redis load.
The format appended by the command is the protocol format requested by the Redis command. It is a plain text format and has the advantages of good compatibility, strong readability, easy processing, simple operation, and avoidance of secondary overhead. In the AOF file, except for the select command used to specify the database (such as select 0 to select database No. 0), which is added by Redis, the others are write commands sent by the client.
Redis provides a variety of AOF buffer synchronization file strategies, which involve the write and fsync functions of the operating system. , the description is as follows:
In order to improve the efficiency of file writing, in modern operating systems, when the user calls the write function to write data to the file, the operating system usually temporarily stores the data into a memory buffer. When the buffer The data in the buffer is actually written to the hard disk only after it is filled or exceeds the specified time limit. Although such an operation improves efficiency, it also brings security issues: if the computer shuts down, the data in the memory buffer will be lost; therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to immediately transfer the data in the buffer to The data is written to the hard disk to ensure data security.
There are three synchronization methods for the synchronization file strategy of the AOF cache area, which are configured by modifying line 729 of /etc/redis/6379.conf.
Immediately after the command is written to aof_buf, the system fsync operation is called to synchronize to the AOF file. After fsync is completed, the thread returns. In this case, every write command must be synchronized to the AOF file, and hard disk IO becomes a performance bottleneck. Redis can only support about a few hundred TPS writes, seriously reducing the performance of Redis; even if a solid-state drive (SSD) is used, It can only handle tens of thousands of commands per second, and it will greatly reduce the life of the SSD.
After the command is written into aof_buf, the system write operation is called, and fsync synchronization of the AOF file is not performed; the synchronization is loaded by the operating system, and the synchronization period is usually 30 seconds. In this case, the file synchronization time is uncontrollable, and there will be a lot of data accumulated in the buffer, and data security cannot be guaranteed.
After the command is written to aof_buf, the system write operation is called. After the write is completed, the thread returns: the fsync synchronization file operation is called once per second by a dedicated thread. everysec is a compromise between the two aforementioned strategies, a balance between performance and data security. It is the default configuration of Redis and our recommended configuration.
As time goes by, the Redis server executes more and more write commands, and the AOF file will become larger and larger; an AOF that is too large will The file will not only affect the normal operation of the server, but also cause data recovery to take too long.
File rewriting refers to rewriting the AOF file regularly to reduce the size of the AOF file. It should be noted that AOF rewriting converts the data in the Redis process into write commands and synchronizes them to the new AOF file; no reading or writing operations are performed on the old AOF file.
Another point to note about file rewriting is: for AOF persistence, although file rewriting is highly recommended, it is not necessary; even without file rewriting, data can be persisted and stored in Redis is imported when it starts; therefore, in some realities, automatic file rewriting will be turned off, and then the scheduled task will be executed at a certain time every day.
The reason why file rewriting can compress AOF files is that:
It can be seen from the above reasons that since AOF executes fewer commands after rewriting, file rewriting can not only reduce the space occupied by the file, but also speed up the recovery speed.
File rewriting is divided into manual triggering and automatic triggering:
The automatically triggered configuration is located at lines 771 and 772 of /etc/redis/6379.conf
The file rewriting process is as follows:
Regarding the file rewriting process, there are two points that need special attention:
RDB persistence
Advantages: RDB files are compact, small in size, and fast in network transmission , suitable for full copy; the recovery speed is much faster than AOF. Of course, one of the most important advantages of RDB is that it has a relatively small impact on performance compared to AOF.
Disadvantages: The well-known disadvantage of RDB files is that the persistence method of data snapshots determines that real-time persistence cannot be achieved. Today, when data is becoming more and more important, a large amount of data loss is often unacceptable. Therefore, AOF persistence has become mainstream. In addition, RDB files need to meet a specific format and have poor compatibility (for example, the old version of Redis is not compatible with the new version of RDB files).
For RDB persistence, on the one hand, the Redis main process will be blocked when bgsave performs a fork operation. On the other hand, the sub-process writing data to the hard disk will also bring IO pressure.
AOF persistence
Corresponding to RDB persistence, the priority of AOF is to support second-level persistence and good compatibility. The disadvantage is that the file is large and the recovery speed is slow. , which has a great impact on performance.
For AOF persistence, the frequency of writing data to the hard disk is greatly increased (second level under everysec policy), the IO pressure is greater, and there may even be additional blocking problems in the AOF.
The rewriting of AOF files is similar to RDB's bgsave, and there will be problems with blocking during fork and IO pressure of the child process. Relatively speaking, because AOF writes data to the hard disk more frequently, it will have a greater impact on the performance of the Redis main process.
Generally speaking, it is recommended to turn off the automatic rewriting function of AOF, and set a scheduled task during the rewriting operation, and perform it in the early morning when the business volume is low, so as to reduce the impact of AOF on the performance of the main process and the read and write pressure of IO .
For more programming-related knowledge, please visit: Introduction to Programming! !
The above is the detailed content of A detailed explanation of high availability and persistence in redis. For more information, please follow other related articles on the PHP Chinese website!