The characteristics of Kafka are: 1. Provide high throughput for both publishing and subscription; 2. Can perform persistence operations and persist messages to disk, so it can be used for batch consumption; 3. Distributed system, Easy to expand outward; 4. Supports online and offline scenarios.
Kafka’s characteristics and usage scenarios
Kafka is a distributed publish-subscribe messaging system. It was originally developed by LinkedIn Corporation and later became part of the Apache project. Kafka is a distributed, partitionable, redundant and persistent log service.
It is mainly used to process active streaming data. In big data systems, we often encounter a problem. The entire big data is composed of various subsystems. Data needs to be continuously circulated in each subsystem with high performance and low latency.
Traditional enterprise messaging systems are not very suitable for large-scale data processing. In order to handle online applications (messages) and offline applications (data files, logs) at the same time, Kafka appeared. Kafka can play two roles:
Reduce the complexity of system networking.
Reduces programming complexity. Each subsystem no longer negotiates interfaces with each other. Each subsystem is plugged into a socket like a socket. Kafka assumes the role of a high-speed data bus
Kafka main features:
Provides high throughput for both publishing and subscription. It is understood that Kafka can produce about 250,000 messages (50 MB) per second and process 550,000 messages (110 MB) per second.
Persistence operations can be performed. Persist messages to disk so they can be used for batch consumption, such as ETL, as well as real-time applications. Prevent data loss by persisting data to hard disk and replication.
Distributed system, easy to expand outward. All producers, brokers and consumers will have multiple, and they are all distributed. Machines can be expanded without downtime.
The status of message processing is maintained on the consumer side, not on the server side. Automatically balances when failure occurs.
Supports online and offline scenarios.
The design points of Kafka:
1. Directly use the cache of the Linux file system to cache data efficiently.
2. Use linux Zero-Copy to improve sending performance. Traditional data sending requires 4 context switches. After using the sendfile system call, data is exchanged directly in the kernel state, and the system context switches are reduced to 2 times. According to the test results, the data sending performance can be improved by 60%.
3. The cost of data access on disk is O(1). Kafka uses topics for message management. Each topic contains multiple parts (itions). Each part corresponds to a logical log and is composed of multiple segments. Multiple messages are stored in each segment (see the figure below). The message ID is determined by its logical location, that is, the message ID can be directly located to the storage location of the message, avoiding additional mapping of ID to location. Each part corresponds to an index in the memory, and the offset of the first message in each segment is recorded. The message sent by the publisher to a certain topic will be evenly distributed to multiple parts (randomly or distributed according to the callback function specified by the user). The broker receives the published message and adds the message to the last segment of the corresponding part. When When the number of messages on a segment reaches the configured value or the message publishing time exceeds the threshold, the messages on the segment will be flushed to the disk. Only message subscribers flushed to the disk can subscribe to it. After the segment reaches a certain size, it will no longer be available. Data will be written to the segment again, and the broker will create a new segment.
4. Explicit distribution, that is, there will be multiple producers, brokers and consumers, all of which are distributed. There is no load balancing mechanism between Producer and broker. Zookeeper is used for load balancing between brokers and consumers.
All brokers and consumers will be registered in zookeeper, and zookeeper will save some of their metadata information. If a broker or consumer changes, all other brokers and consumers will be notified.
The above is the detailed content of What are the characteristics of Kafka?. For more information, please follow other related articles on the PHP Chinese website!