Introduce what is Apache Flink

Introduction to Apache Flink:

Apache Flink is a framework and distributed processing engine for stateful processing on unbounded and bounded data streams calculation. Flink runs in all common cluster environments and can compute at memory speeds and at any scale.

(Recommended tutorial:apache)

Next, let’s introduce the important aspects of the Flink architecture.

Handling unbounded and bounded data

Any type of data can form an event stream. Credit card transactions, sensor measurements, machine logs, user interaction records on a website or mobile app, all this data forms a stream.

Data can be processed as unbounded or bounded streams.

1. Unbounded flow The beginning of the flow is defined, but the end of the flow is not defined. They generate data endlessly. The data of unbounded flow must be processed continuously, that is, the data needs to be processed immediately after being ingested. We cannot wait until all the data arrives before processing because the input is infinite and will never be completed at any time. Processing unbounded data often requires ingesting events in a specific order, such as the order in which they occur, to be able to infer the completeness of the results.

2. Bounded flow defines the beginning of the flow and the end of the flow. Bounded streams allow calculations to be performed after all data has been ingested. All data in bounded streams can be sorted, so ordered ingestion is not required. Bounded stream processing is often called batch processing.

Apache Flink is good at processing unbounded and bounded data sets. Precise time control and statefulness enable Flink's runtime to run any application that handles unbounded streams. Bounded streams are processed internally by algorithms and data structures specifically designed for fixed-size data sets, resulting in excellent performance.

Deepen your understanding by exploring use cases built on top of Flink.

Deploy applications anywhere

Apache Flink is a distributed system that requires computing resources to execute applications. Flink integrates with all common cluster resource managers, such as Hadoop YARN, Apache Mesos and Kubernetes, but can also run as a standalone cluster.

Flink is designed to work well with each of the above resource managers, which is achieved through a resource-manager-specific deployment mode. Flink can interact with the current resource manager in a manner appropriate to it.

When you deploy a Flink application, Flink automatically identifies the required resources based on the application's configured parallelism and requests these resources from the resource manager. In the event of a failure, Flink replaces the failed container by requesting new resources. All communication to submit or control applications occurs through REST calls, which simplifies the integration of Flink into a variety of environments.

Run applications at any scale

Flink is designed to run stateful streaming applications at any scale. Therefore, the application is parallelized into potentially thousands of tasks that are distributed across the cluster and executed concurrently. So applications can take advantage of endless CPU, memory, disk, and network IO. And Flink makes it easy to maintain very large application state. Its asynchronous and incremental checkpointing algorithm has minimal impact on processing latency while ensuring the consistency of exactly-once state.

Flink users have reported some impressive scalability numbers in their production environments

Processing trillions of events per day, applications maintaining terabytes of state, and applications running on data Run on thousands of cores.

Exploiting memory performance

Stateful Flink programs are optimized for local state access. The state of a task is always maintained in memory or, if the state size exceeds available memory, is saved in an on-disk data structure that can be accessed efficiently. Tasks perform all computations by accessing local (usually in-memory) state, resulting in very low processing latency. Flink ensures exact-once state consistency in failure scenarios by regularly and asynchronously persisting local state storage.

The above is the detailed content of Introduce what is Apache Flink. For more information, please follow other related articles on the PHP Chinese website!