Real-time computing framework that can travel through time and space--Flink's processing of time

Flink is very important for the stream processing architecture. Kafka gives messages the ability to persist, and the ability to process data and even time travel depends on Flink.

In the Streaming-The Future of Big Data we know that the two most important things for streaming processing are correctness and time reasoning tools. Flink has very good support for both.

Flink guarantees correctness

For continuous event streaming data, since there may be events that have not yet arrived when we process them, the correctness of the data may be affected. The common practice of using high-latency offline computing is now guaranteed, but it also sacrifices low latency.

The correctness of Flink is reflected in the definition of the calculation window in line with the natural laws of data generation. For example, click stream event tracks the access status of 3 users A, B, and C. We see that there may be gaps in the data, which is the session window.

Using SparkStreaming's microbatch method (dashed lines are calculation windows, and solid lines are session windows), it is difficult to match the calculation windows and session windows. Using Flink's stream processing API, you can flexibly define the calculation window. For example, you can set a value, if it exceeds this value, the activity will be considered to be over.

Unlike general stream processing, Flink can adopt event time, which is very useful for correctness.

For the correctness of failure, the calculation state must be tracked. Most of the time, the state guarantee is completed by developers, but continuous stream processing calculations have no end. Flink uses checkpoint-checkpoint technology to solve this problem. At each checkpoint, the system records the intermediate calculation status, which is accurate to reset when a fault occurs. This method allows the system to have fault tolerance in a low overhead manner - when everything is normal, the checkpoint mechanism has a very small impact on the system.

The interface provided by Flink includes tracking computing tasks and using the same technology to implement stream processing and batch processing, simplifying operation and maintenance development work, which is also a guarantee of correctness.

Flink's time processing

The biggest difference between using stream processing and batch processing is the processing of time.

Using batch processing architecture

In this architecture, we can store data every once in a while, for example, in HDFS, and execute it regularly by the scheduler to output the results.

This architecture is feasible but there are several problems:

Too many separate parts. To calculate the number of events in the data, this architecture uses too many systems. Every system has learning costs and management costs, and there may be bugs. The method of handling time is unclear. Assume that it needs to be counted every 30 minutes instead. This change involves workflow scheduling logic (rather than application code logic), confusing DevOps problems with business requirements. Early warning. Assume that in addition to counting once an hour, you also need to receive a count warning as early as possible (such as when the number of events exceeds 10). To do this, Storm can be introduced to collect message flows in addition to periodically running batch jobs. Storm provides approximate counts in real time, and batch jobs provide accurate counts per hour. But this way, a system is added to the architecture, and a new programming model related to it. The above architecture is called the Lambda architecture.

​

Out-of-order event stream. In the real world, most event streams are out of order, that is, the actual order of events occurring is different from the order recorded by the data center. This means that events that originally belonged to the previous batch may be mistakenly classified into the current batch. Batch architectures are difficult to solve this problem, and most people choose to ignore it. The boundaries of batch jobs are unclear. Events around the segmentation time point may be classified into the previous batch or the current batch. Adopt stream processing

First, the message is written centrally to the message transmission system kafka, the event stream is provided by the message transmission system, and is processed only by a single Flink job.

Split the event stream into batches of tasks in units of time, and this logic is completely embedded in the application logic of the Flink program. The warning is generated by the same program, and out-of-order events are handled by Flink themselves. To change from grouping with fixed time to grouping according to the time period in which the data is generated, just modify the definition of the window in the Flink program. Also, if the application's code has been changed, you can replay the application by simply replaying the Kafka theme. Using a streaming architecture can significantly reduce systems that need to learn, manage, and write code. Flink application code example:

Code language: javascript Number of code runs: 0 run copy

 <code class="javascript">DataStream<LogEvent> stream = env // 通过Kafka生成数据流.addSource(new FlinkKafkaConsumer(...)) // 分组.keyBy("country") // 将时间窗口设为60分钟.timeWindow(Time.minutes(60)) // 针对每个时间窗口进行操作.apply(new CountPerWindowFunction());</code>

In stream processing, there are two main concepts of time:

Event time, that is, the time when the event actually occurs. More precisely, each event has a timestamp associated with it, and the timestamp is part of the data record.

Processing time, that is, the time when the event is processed. Processing time is actually the time measured by the machine that handles events.

Take the Star Wars series as an example. The first 3 movies released are the 4th, 5th and 6th in the series (this is the time of the event), and their release years are 1977, 1980 and 1983 (this is the time of the process). The 1st, 2nd, 3rd and 7th series released by the event time were then the corresponding processing times were 1999, 2002, 2005 and 2015 respectively. It can be seen that the order of event flow may be messy (although the order of years is generally not messy)

There is usually the third concept of time, namely, intake time, also known as entry time. It refers to the time when an event enters the stream processing framework. Data that lacks the time of the real event will be timestamped by the stream processor, that is, when the stream processor first sees it (this operation is done by the source function, which is the first processing point of the program).

In the real world, many factors (such as temporary connection interruption, network delay caused by different reasons, clock out of synchronization in distributed systems, steep increase in data rates, physical reasons, or bad luck) cause event time and processing time to be biased (i.e. event time deviation). Event chronological order and processing chronological order are often inconsistent, which means that events arrive at the stream processor in out of order.

Flink allows users to choose to adopt event time, processing time, or ingestion time definition windows based on the required semantics and requirements for accuracy.

window

The time window is the easiest and most useful one. It supports scrolling and sliding.

For example, a one-minute scrolling window collects the values ​​of the last minute and outputs the sum at the end of one minute:

The one-minute sliding window calculates the sum of the values ​​of the last minute, but slides every half minute and outputs the result:

In Flink, the one-minute scrolling window is defined as follows.

Code language: javascript Number of code runs: 0 run copy

 <code class="javascript">stream.timeWindow(Time.minutes(1))</code>

The one-minute sliding window that slides every half minute (i.e. 30 seconds) is shown below.

Code language: javascript Number of code runs: 0 run copy

 <code class="javascript">stream.timeWindow(Time.minutes(1), Time.seconds(30))</code>

Another common window supported by Flink is called the counting window. When using a count window, the grouping is no longer a timestamp, but the number of elements.

A sliding window can also be interpreted as a counting window consisting of 4 elements and slides once every two elements. The scrolling and sliding counting windows are defined as follows.

Code language: javascript Number of code runs: 0 run copy

 <code class="javascript">stream.countWindow(4) stream.countWindow(4, 2)</code>

Although the counting window is useful, its definition is not as rigorous as the time window, so it should be used with caution. Time does not stop, and the time window will always be "closed". But as far as counting windows are concerned, assuming that the number of elements defined is 100, and the corresponding elements of a key will never reach 100, the window will never close, and the memory occupied by the window will be wasted.

Another useful window supported by Flink is the session window. The session window is set by the timeout time, that is, how long does it take to wait before you think the session has ended. Examples are as follows:

Code language: javascript Number of code runs: 0 run copy

 <code class="javascript">stream.window(SessionWindows.withGap(Time.minutes(5))</code>

trigger

In addition to windows, Flink also provides a trigger mechanism. The trigger controls the time when the result is generated, that is, when the window content is aggregated and the result is returned to the user. Each default window has a trigger. For example, a time window that takes event time will be triggered when a watermark is received. For users, in addition to generating complete and accurate results when receiving the watermark, custom triggers can also be implemented.

Time backtrack

A core capability of stream processing architecture is the time backtracking mechanism. It means rewinding the data stream to a time in the past, restarting the handler until the current time is processed. Kafka supports this capability.

Real-time stream processing is always processing the most recent data (i.e., data of "current time" in the figure), while historical stream processing starts from the past and can be processed until the current time. Streaming processors support event time, which means "rewinding" the data stream and rerunning the same program with the same set of data will get the same result.

Watermark

Flink advances event time through watermarks. Watermark is a conventional record embedded in the stream. The calculation program uses the watermark to know that a certain time point has arrived. The window that receives the watermark will know that no more records earlier than that time will appear, because all events with timestamps less than or equal to that time have arrived. At this time, the window can safely calculate and give the result (sum). Watermark makes the event time completely independent of the processing time. Late watermarks ("late" is from the perspective of processing time) do not affect the correctness of the results, but only affect the speed of the results being received.

Watermarks are generated by application developers, which usually require a certain understanding of the corresponding fields. Perfect watermarks can never be wrong: Events whose timestamps are smaller than the watermark mark time will not appear again.

If the watermark is too late, the result may be received very slowly. The solution is to output an approximate result before the watermark arrives (Flink can do it). If the watermark arrives too early, you may receive an error result, but Flink's mechanism to process late data can solve this problem.

Related articles: Streaming-The Future of Big Data

The cornerstone of real-time computing big data processing - Google Dataflow

The Future of Data Architecture—A Brief Discussion on Streaming Processing Architecture

The above is the detailed content of Real-time computing framework that can travel through time and space--Flink's processing of time. For more information, please follow other related articles on the PHP Chinese website!