Parse and map multiple records in NDJSON file using Gson

This article details how to use the Gson library to efficiently parse and map multiple JSON records in NDJSON (Newline Delimited JSON) files in Java. To solve the problem that the traditional Gson `fromJson` method can only read the first record, the tutorial proposes an iterative parsing solution based on the `JsonReader` and `peek()` methods, and provides a complete Java code example to ensure that all independent JSON objects in the file can be successfully read and deserialized.

NDJSON (Newline Delimited JSON) is a common format when dealing with text files containing multiple independent JSON objects, where each JSON object occupies one line. However, when trying to use the Gson library to parse such files, developers often encounter the problem that only the first JSON record can be read successfully. This is because Gson's fromJson method treats the entire input stream as a complete JSON structure by default, and the input is considered to end once the first top-level JSON object is parsed.

In order to solve this problem, we need to use the JsonReader class provided by Gson to perform iterative parsing and read the JSON objects in the file one by one.

Understand the differences between NDJSON format and traditional JSON parsing

Each record in the NDJSON file is an independent, complete JSON object, separated by newlines. For example:

 {"id": 1, "name": "Alice"}
{"id": 2, "name": "Bob"}
{"id": 3, "name": "Charlie"}

If you use gson.fromJson(reader, MyObject.class) directly, Gson will read the first {"id": 1, "name": "Alice"} and map it successfully, but the following content will be ignored because Gson thinks that the file has been parsed.

Solution: Use JsonReader for iterative parsing

To correctly parse all records in an NDJSON file, a combination of JsonReader and a loop structure is required. JsonReader provides finer-grained control over JSON streaming.

core steps

1. Initialize JsonReader : Create a JsonReader instance from a file or input stream.
2. Set relaxed mode (setLenient(true)) : There is usually no comma separation between each JSON object in the NDJSON file. JsonReader by default will expect a single JSON document or a comma-separated JSON array. In order for JsonReader to tolerate this non-strict JSON format (i.e. multiple top-level JSON objects), it needs to be set to relaxed mode.
3. Iterative reading : Use a loop structure, combined with the reader.peek() method to determine whether the end of the file is reached. In each loop, call gson.fromJson(reader, YourDTO.class) to read a JSON object and add it to the list.

Sample code

Let's say we have a CustomerFeedDTO class that maps each customer record in an NDJSON file:

 import java.util.ArrayList;
import java.util.List;
import java.util.Map;

//Customer Data Transfer Object (DTO)
class CustomerFeedDTO {
    private Map<string> profile;
    private Map<string> phone;
    private ArrayList&gt; addresses;
    private Map<string> orders;
    private ArrayList&gt; customs;

    //Constructor, Getter and Setter methods (omitted here, but needed in actual applications)
    public Map<string> getProfile() { return profile; }
    public void setProfile(Map<string> profile) { this.profile = profile; }

    public Map<string> getPhone() { return phone; }
    public void setPhone(Map<string> phone) { this.phone = phone; }

    public ArrayList&gt; getAddresses() { return addresses; }
    public void setAddresses(ArrayList&gt; addresses) { this.addresses = addresses; }

    public Map<string> getOrders() { return orders; }
    public void setOrders(Map<string> orders) { this.orders = orders; }

    public ArrayList&gt; getCustoms() { return customs; }
    public void setCustoms(ArrayList&gt; customs) { this.customs = customs; }

    @Override
    public String toString() {
        return "CustomerFeedDTO{"  
               "profile=" profile  
               ", phone=" phone  
               ", addresses=" addresses  
               ", orders=" orders  
               ", customs=" customs  
               '}';
    }
}</string></string></string></string></string></string></string></string></string>

Using the above CustomerFeedDTO and NDJSON files, the following code demonstrates how to parse all records:

 import com.google.gson.Gson;
import com.google.gson.stream.JsonReader;
import com.google.gson.stream.JsonToken;

import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

public class NdJsonParser {

    public static void main(String[] args) {
        List<customerfeeddto> customerFeedDTOs = new ArrayList();
        Gson gson = new Gson();

        // Assume customer.json is your NDJSON file try (FileReader fileReader = new FileReader("customer.json");
             JsonReader reader = new JsonReader(fileReader)) {

            // Must be set to relaxed mode to handle multiple top-level JSON objects reader.setLenient(true);

            // Loop reading until the end of the document while (reader.peek() != JsonToken.END_DOCUMENT) {
                CustomerFeedDTO customerFeedDTO = gson.fromJson(reader, CustomerFeedDTO.class);
                customerFeedDTOs.add(customerFeedDTO);
            }

            // Print all parsed customer records for (int i = 0; i <h4> Choice between reader.peek() and reader.hasNext()</h4>
<p> In the loop condition, we used reader.peek() != JsonToken.END_DOCUMENT. The peek() method returns the type of the next token without actually consuming it. When peek() returns JsonToken.END_DOCUMENT, it means that the end of the JSON document has been reached.</p>
<p> Although JsonReader also has a hasNext() method, it may cause exceptions in NDJSON scenarios. hasNext() is mainly used to check if there are more elements in a JSON array or object, and may throw an IllegalStateException when it tries to read an unexpected token (for example, when there are no more structured JSON elements at the end of the document). Therefore, for a stream like NDJSON, which is composed of multiple independent JSON objects, it is more robust and recommended to use peek() == JsonToken.END_DOCUMENT to determine the loop termination condition.</p>
<h3> Things to note and best practices</h3>
<ol>
<li> <strong>Resource management</strong> : Always use the try-with-resources statement to ensure that FileReader and JsonReader can be closed correctly after use to avoid resource leaks.</li>
<li> <strong>Error handling</strong> : In actual applications, more detailed exception handling logic should be added, such as catching JsonSyntaxException to handle malformed JSON records.</li>
<li> <strong>DTO design</strong> : The CustomerFeedDTO in the example uses Map<string> and ArrayList&gt; to handle unknown or dynamic JSON structures. When the JSON structure is known, it is recommended to use specific Java types (such as String, Integer, custom objects, etc.) instead of Map and ArrayList&gt; to improve type safety and code readability. For example, profile can be defined as a specific ProfileDTO class.</string>
</li>
<li> <strong>Performance considerations</strong> : For very large NDJSON files, loading all records into a List at once can consume a lot of memory. In this case, you can consider processing records one by one, or using a streaming API (such as Java 8 Stream API) combined with a custom iterator to process the data to avoid loading all the data at once.</li>
</ol>
<h3> Summarize</h3>
<p> Through the iterative parsing mechanism of JsonReader, combined with reader.setLenient(true) and reader.peek() != JsonToken.END_DOCUMENT as a loop condition, all JSON records in the NDJSON file can be effectively parsed and mapped. This approach provides fine-grained control over the JSON flow and is the standard and recommended practice when processing multiple independent rows of JSON data. Proper understanding and application of these techniques ensures reliable processing of NDJSON data in Java applications.</p></customerfeeddto>

The above is the detailed content of Parse and map multiple records in NDJSON file using Gson. For more information, please follow other related articles on the PHP Chinese website!