Application of efficient log library under Linux

Due to the inherent characteristics of the log itself, records are inserted sequentially from left to right, which means that the records on the left are "older" than the records on the right. In other words, we do not need to rely on the system clock. This feature is very important for distribution. Very important for the system.

It is impossible to know when the log appeared. It may be that it is too simple in concept. In the database field, logs are more used to synchronize data and indexes when the system crashes, such as redo log in MySQL. Redo log is a disk-based data structure used to ensure data when the system hangs. The correctness and completeness of the system are also called write-ahead logs. For example, during the execution of a thing, the redo log will be written first, and then the actual changes will be applied. In this way, when the system recovers after a crash, it can be recreated based on the redo log. Put it back to restore the data (during the initialization process, there will be no client connection at this time). The log can also be used for synchronization between the database master and slave, because essentially, all operation records of the database have been written to the log. We only need to synchronize the log to the slave and replay it on the slave to achieve master-slave synchronization. Many other required components can also be implemented here. We can obtain all changes in the database by subscribing to the redo log, thereby implementing personalized business logic, such as auditing, cache synchronization, etc.

Distributed system services are essentially about state changes, which can be understood as state machines. Two independent processes (not dependent on the external environment, such as system clocks, external interfaces, etc.) will produce consistent outputs given consistent inputs. And ultimately maintain a consistent state, and the log does not rely on the system clock due to its inherent sequence, which can be used to solve the problem of change order.
We use this feature to solve many problems encountered in distributed systems. For example, in the standby node in RocketMQ, the main broker receives the client's request and records the log, and then synchronizes it to the slave in real time. The slave replays it locally. When the master hangs up, the slave can continue to process the request, such as rejecting the write request and continuing. Handle read requests. The log can not only record data, but also directly record operations, such as SQL statements.

The log is the key data structure to solve the consistency problem. The log is like an operation sequence. Each record represents an instruction. For example, the widely used Paxos and Raft protocols are all consistency protocols built based on the log.

Logs can be easily used to handle the inflow and outflow of data. Each data source can generate its own log. The data sources here can come from various aspects, such as an event stream (page click, cache refresh reminder, database binlog changes), we can centrally store logs in a cluster, and subscribers can read each record of the log based on offset, and apply their own changes based on the data and operations in each record.
The log here can be understood as a message queue, and the message queue can play the role of asynchronous decoupling and current limiting. Why do we say decoupling? Because for consumers and producers, the responsibilities of the two roles are very clear, they are responsible for producing messages and consuming messages, without caring about who is downstream or upstream, whether it is the change log of the database or a certain event. I don't need to care about a certain party at all. I only need to pay attention to the logs that interest me and each record in the logs.

We know that the QPS of the database is certain, and upper-layer applications can generally be expanded horizontally. At this time, if there is a sudden request scenario like Double 11, the database will be overwhelmed, then we can introduce message queues to combine the operations of each team's database Write to the log, and another application is responsible for consuming these log records and applying them to the database. Even if the database hangs, processing can continue from the position of the last message when recovering (both RocketMQ and Kafka support Exactly Once semantics ), here even if the speed of the producer is different from the speed of the consumer, there will be no impact. The log plays a buffering role here. It can store all records in the log and synchronize to the slave node regularly, so that the message The backlog capacity can be greatly improved because writing logs is processed by the master node. There are two types of read requests. One is tail-read, which means that the consumption speed can keep up with the writing speed. This kind of read You can go directly to the cache, and the other one is the consumer that lags behind the write request. This kind can be read from the slave node, through IO isolation and some file policies that come with the operating system, such as pagecache, cache read-ahead, etc. , the performance can be greatly improved.

Horizontal scalability is a very important feature in a distributed system. Problems that can be solved by adding machines are not a problem. So how to implement a message queue that can achieve horizontal expansion? If we have a stand-alone message queue, as the number of topics increases, IO, CPU, bandwidth, etc. will gradually become bottlenecks, and performance will slowly decrease. So how to proceed here? What about performance optimization?

1. topic/log sharding. Essentially, the messages written by the topic are the records of the log. As the number of writes increases, a single machine will slowly become a bottleneck. At this time, we can divide a single topic into multiple sub-topics. And allocate each topic to a different machine. In this way, those topics with a large amount of messages can be solved by adding machines, while some topics with a small amount of messages can be assigned to the same machine or not processed. Partition
2. Group commit, such as Kafka's producer client, when writing messages, first writes them to a local memory queue, then summarizes the messages according to each partition and node, and submits them in batches. For the server side or broker side, it can also be Using this method, the page cache is written first, and then the disk is flushed regularly. The method of flushing can be determined according to the business. For example, financial services may adopt a synchronous flushing method.
3. Avoid useless data copies
4. IO Isolation
   

Logs play a very important role in distributed systems and are the key to understanding various components of distributed systems. As our understanding deepens, we find that many distributed middleware are built based on logs, such as Zookeeper, HDFS, Kafka, RocketMQ, Google Spanner, etc., and even databases such as Redis, MySQL, etc., their master-slave is based on log synchronization. Relying on the shared log system, we can implement many systems: data synchronization and concurrency between nodes Update data order issues (consistency issues), persistence (the system can continue to provide services through other nodes when the system crashes), distributed lock services, etc. I believe that through practice and reading a large number of papers, there will be deeper insights. levels of understanding.

The above is the detailed content of Application of efficient log library under Linux. For more information, please follow other related articles on the PHP Chinese website!