A deep understanding of HTTP persistent connections: policies and practices for sending multiple requests on the same socket

This article explores in-depth the mechanism of sending multiple HTTP requests on the same TCP Socket, namely, HTTP persistent connection (Keep-Alive). The article clarifies the difference between HTTP/1.x and HTTP/2 protocols, emphasizes the importance of server-side support for persistent connections, and how to correctly handle Connection: close response headers. By analyzing common errors and providing best practices, we aim to help developers build efficient and robust HTTP clients.

Understand HTTP persistent connections (Keep-Alive)

HTTP persistent connections, also known as HTTP Keep-Alive, allow clients to send and receive multiple HTTP requests and responses on the same TCP connection without having to re-establish a new connection for each request. This significantly reduces the overhead of TCP handshake and slow start, thereby improving performance and efficiency. In HTTP/1.1, persistent connections are the default behavior unless Connection: close is explicitly specified. HTTP/2 goes a step further, processing multiple requests and responses simultaneously on a single connection through multiplexing.

However, implementing the client sends multiple requests on the same Socket requires the correct cooperation between the client and the server. The client sends Connection: keep-alive is just a request, and the server has the right to choose whether to accept it.

Common misunderstandings and protocol analysis

When trying to implement persistent connections, developers often encounter some misunderstandings:

1. Misuse of HTTP protocol version

The original code tries to send a request using the HTTP/2 protocol string:

 "GET /" x " HTTP/2\r\n"

This is a common misconception. HTTP/2 and HTTP/1.x are two completely different protocols:

• HTTP/1.x is a text-based protocol, and each request/response pair is usually processed sequentially.
• HTTP/2 is a binary protocol that introduces advanced features such as multiplexing, head compression, and server push. It requires protocol upgrades during the TLS handshake phase through Application Layer Protocol Negotiation (ALPN), rather than simply declaring HTTP/2 in the request line. Replacing HTTP/1.1 with HTTP/2 directly does not allow the HTTP/1.x server to understand HTTP/2 requests, but will instead lead to protocol errors. For HTTP/1.x servers, HTTP/1.1 or HTTP/1.0 should always be used.

2. Connection: The effect of close response header

When the server contains Connection: close in the response header, it explicitly instructs the client to close the TCP connection after receiving the current response. This means that even if the client sends Connection: keep-alive in the request, the server's decision is still final.

In the example output:

 HTTP/1.1 200 OK
Content-Length: 2
Content-Type: text/plain
Connection: close
Accept-Ranges: none

The server explicitly returns Connection: close, so after sending the current response body, it closes the Socket connection. After reading all data, the reader.readLine() of the client will return null because the connection is closed, causing the loop to end and subsequent requests cannot be sent on the same socket.

3. The importance of server-side support

The core of the implementation of persistent connections is whether the server supports and enables it. Especially on resource-constrained devices, such as ESP32 microcontrollers, their HTTP stack may choose not to implement or enable persistent connections in order to simplify and save resources. In this case, the server may insist on closing the connection regardless of how the client requests Connection: keep-alive.

Correctly handle HTTP responses and persistent connections

In order to correctly handle multiple HTTP requests on the same Socket, the client needs:

1. Send the correct HTTP/1.1 request header : Make sure the request line uses HTTP/1.1 and includes the Host header. If you expect a persistent connection, you can explicitly send Connection: keep-alive, but this is not mandatory, because HTTP/1.1 is a persistent connection by default.

    String request = String.format(
       "GET /%s HTTP/1.1\r\n" // Using HTTP/1.1
         "Host: %s\r\n"
         "Connection: keep-alive\r\n" // explicitly request a persistent connection "\r\n",
       x, hostname
   );
   output.write(request);
   output.flush();

2. Correctly reading HTTP response : This is the most critical step. HTTP response consists of a status line, a response header, and a response body. When reading the response, you cannot simply loop through readLine() until you return null, as this will block until the connection is closed. The correct way to do it is:

   • Read the response header line by line : until an empty line (\r\n) is encountered, the header ends.
   • Parsing response headers : In particular, Content-Length and Connection headers.
     • If Content-Length exists, the response body of the specified number of bytes is read.
     • If Transfer-Encoding: chunked exists, the response body is read by the chunking encoding rule.
     • Resolve Connection header: If the server responds to Connection: close, the client should close the socket after reading the current response. If you respond to Connection: keep-alive (or not specified, HTTP/1.1 default), you can try to send the next request.

Example: A more robust responsive read logic (simplified version, not fully parsed for all HTTP details)

The following code snippet shows how to read a response and decide whether to continue based on the Connection header:

 import java.io.*;
import java.net.Socket;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;

public class HttpClientRobust {

    private Socket socket;
    private PrintWriter output;
    private BufferedReader reader;
    private String hostname;
    private boolean keepAlive = true; // Assume that initially expects to keep the connection public HttpClientRobust() throws IOException {
        URL url = new URL("http://192.168.178.56"); // Replace with your ESP32 IP
        hostname = url.getHost();
        int port = 80;

        socket = new Socket(hostname, port);
        output = new PrintWriter(socket.getOutputStream());
        reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
    }

    public void sendAndReceive(String path) throws IOException, InterruptedException {
        if (!keepAlive) {
            System.out.println("Connection closed by server. Cannot send more requests on this socket.");
            return;
        }

        String request = String.format(
            "GET /%s HTTP/1.1\r\n"
              "Host: %s\r\n"
              "Connection: keep-alive\r\n" // The client requests to keep the connection "\r\n",
            path, hostname
        );

        System.out.println("--- Sending Request ---");
        System.out.println(request);
        output.write(request);
        output.flush();

        System.out.println("--- Receiving Response ---");
        Map<string string> headers = new HashMap();
        String line;
        int contentLength = -1;

        // Read the status line line = reader.readLine();
        if (line == null) {
            System.out.println("Server closed connection prematurely.");
            keepAlive = false;
            return;
        }
        System.out.println(line); // Print status line// Read and parse response header while ((line = reader.readLine()) != null && !line.isEmpty()) {
            System.out.println(line);
            int colonIndex = line.indexOf(':');
            if (colonIndex > 0) {
                String headerName = line.substring(0, colonIndex).trim();
                String headerValue = line.substring(colonIndex 1).trim();
                headers.put(headerName.toLowerCase(), headerValue);
            }
        }

        // Check Connection header if ("close".equalsIgnoreCase(headers.get("connection"))) {
            keepAlive = false;
            System.out.println("Server requested to close connection.");
        }

        // Check Content-Length
        if (headers.containsKey("content-length")) {
            try {
                contentLength = Integer.parseInt(headers.get("content-length"));
                System.out.println("Content-Length: " contentLength);
                // Read the response body char[] buffer = new char[contentLength];
                int bytesRead = 0;
                while (bytesRead  0) {
                System.out.println("Response Body: " responseBody.toString().trim());
            }
        }

        if (!keepAlive) {
            close();
        }
    }

    public void close() throws IOException {
        if (socket != null && !socket.isClosed()) {
            socket.close();
            System.out.println("Socket closed.");
        }
    }

    public static void main(String[] args) throws IOException, InterruptedException {
        HttpClientRobust client = null;
        try {
            client = new HttpClientRobust();
            for (int i = 1; i <p> <strong>Notes:</strong></p>
<ul>
<li> reader.read(buffer, bytesRead, contentLength - bytesRead) in the above example is the correct way to read a fixed-length response body.</li>
<li> reader.ready() may not be reliable enough when reading the response body, because the data may still be in transit but has not reached the buffer yet. More robust HTTP clients require more complex logic to handle various Transfer-Encodings (such as chunked transfers) and timeouts.</li>
<li> If the server responds to Connection: close, the keepAlive flag will be set to false and subsequent requests will not be sent.</li>
</ul>
<h3> Summary and best practices</h3>
<ol>
<li><p> <strong>Understanding protocol differences</strong> : HTTP/1.x and HTTP/2 are different protocols. Don't confuse their usage. For most simple web services, HTTP/1.1 is sufficient, and its persistent connection is the default behavior.</p></li>
<li><p> <strong>Respect the server intention</strong> : Client requests Connection: keep-alive just a suggestion, and the server's Connection response header (especially Connection: close) has the final decision. Always check the server's response header.</p></li>
<li><p> <strong>Correctly parse the response</strong> : Do not rely on the readLine() loop until it returns null to determine the end of the response, which is wrong for persistent connections. Content-Length or Transfer-Encoding must be parsed to determine the boundaries of the response body.</p></li>
<li><p> <strong>Consider server capabilities</strong> : especially for embedded devices (such as ESP32), its HTTP stack may have limited functionality. If the server does not support persistent connections, the client should be ready to establish a new connection for each request.</p></li>
<li>
<p> <strong>Use mature HTTP client libraries</strong> : For complex HTTP communication, it is highly recommended to use Java built-in java.net.HttpURLConnection or third-party libraries such as Apache HttpClient, OkHttp, etc. These libraries have dealt with the complexity of the HTTP protocol, including persistent connections, redirection, authentication, SSL/TLS, timeouts, etc., greatly simplifying development work. For example:</p>
<pre class="brush:php;toolbar:false"> import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class ModernHttpClientExample {
    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newBuilder()
                .version(HttpClient.Version.HTTP_1_1) // Clearly use HTTP/1.1
                .connectTimeout(Duration.ofSeconds(10))
                .build();

        for (int i = 1; i  response = client.send(request, HttpResponse.BodyHandlers.ofString());

            System.out.println("Request " i " to /" path);
            System.out.println("Status Code: " response.statusCode());
            System.out.println("Response Body: " response.body());
            System.out.println("Connection Header: " response.headers().firstValue("connection").orElse("N/A"));

            // The HttpClient library automatically handles persistent connections. If the server returns Connection: close, it closes the current connection and establishes a new connection the next time it requests.
            Thread.sleep(1000);
        }
    }
}

By following these principles, developers can manage HTTP connections more effectively and build high-performance and reliable web applications.

The above is the detailed content of A deep understanding of HTTP persistent connections: policies and practices for sending multiple requests on the same socket. For more information, please follow other related articles on the PHP Chinese website!