Let’s talk about how to parse Apache Avro data (explanation with examples)
青灯夜游forward
2022-02-22 10:47:163031browse
How to parse Apache Avro data? This article will introduce you to the methods of serializing to generate Avro data, deserializing to parse Avro data, and using FlinkSQL to parse Avro data. I hope it will be helpful to you!
With the rapid development of the Internet, cutting-edge technologies such as cloud computing, big data, artificial intelligence AI, and the Internet of Things have become mainstream high-tech technologies in today's era, such as e-commerce websites , face recognition, driverless driving, smart homes, smart cities, etc., not only facilitate people's daily necessities, food, housing and transportation, but behind the scenes, there is always a large amount of data being collected, cleared and analyzed by various system platforms. , and it is particularly important to ensure low latency, high throughput, and security of data. Apache Avro itself is serialized through Schema for binary transmission. On the one hand, it ensures high-speed transmission of data, and on the other hand, it ensures data security. , avro is currently used more and more widely in various industries. How to process and parse avro data is particularly important. This article will demonstrate how to generate avro data through serialization and use FlinkSQL for analysis.
This article is a demo of avro parsing. Currently, FlinkSQL is only suitable for simple avro data parsing. Complex nested avro data is not supported for the time being.
Scene introduction
This article mainly introduces the following three key contents:
How to serialize and generate Avro data
How to deserialize and parse Avro data
How to use FlinkSQL to parse Avro data
Prerequisites
To understand what avro is, please refer to the apache avro official website quick start guide
Understand avro application scenarios
Operation steps
1. Create a new avro maven project and configure the pom dependency
Note: The above pom file is configured to be automatically generated The path to the class, i.e.##project.basedir/src/main/avro/and{project.basedir}/src/main/java/, after this configuration, when executing the mvn command, this plug-in will automatically generate a class file from the avsc schema in this directory and put it in the latter Under contents. If the avro directory is not generated, just create it manually.2. Define schema
Use JSON to define schema for Avro. The schema consists of basic types (null, boolean, int, long, float, double, bytes, and string) and complex types (record, enum, array, map, union, and fixed). For example, the following defines a user's schema, creates an avro directory in the main directory, and then creates a new file user.avsc in the avro directory:
User user1 = new User(); user1.setName("Alyssa"); user1.setFavoriteNumber(256); // Leave favorite col or null // Alternate constructor User user2 = new User("Ben", 7, "red"); // Construct via builder User user3 = User.newBuilder() .setName("Charlie") .setFavoriteColor("blue") .setFavoriteNumber(null) .build(); // Serialize user1, user2 and user3 to disk DatumWriter userDatumWriter = new SpecificDatumWriter(User.class); DataFileWriter dataFileWriter = new DataFileWriter(userDatumWriter); dataFileWriter.create(user1.getSchema(), new File("user_generic.avro")); dataFileWriter.append(user1); dataFileWriter.append(user2); dataFileWriter.append(user3); dataFileWriter.close();
// Deserialize Users from disk DatumReader userDatumReader = new SpecificDatumReader(User.class); DataFileReader dataFileReader = new DataFileReader(new File("user_generic.avro"), userDatumReader); User user = null; while (dataFileReader.hasNext()) { // Reuse user object by passing it to next(). This saves us from // allocating and garbage collecting many objects for files with // many items. user = dataFileReader.next(user); System.out.println(user); }
The above is the detailed content of Let’s talk about how to parse Apache Avro data (explanation with examples). For more information, please follow other related articles on the PHP Chinese website!
Statement：
This article is reproduced at:juejin.cn. If there is any infringement, please contact admin@php.cn delete
 "164549776425442Let’s talk about how to parse Apache Avro data (explanation with examples)")
 "164549776967215Let’s talk about how to parse Apache Avro data (explanation with examples)")
 "164549777398428Let’s talk about how to parse Apache Avro data (explanation with examples)")
 "164549778225747Let’s talk about how to parse Apache Avro data (explanation with examples)")
 "164549778617930Let’s talk about how to parse Apache Avro data (explanation with examples)")
 "164549779092701Let’s talk about how to parse Apache Avro data (explanation with examples)")
 "164549779474716Let’s talk about how to parse Apache Avro data (explanation with examples)")
 "164549779865203Let’s talk about how to parse Apache Avro data (explanation with examples)")