A practical guide to efficiently sending Java application logs to the ELK Stack

This article aims to guide developers on how to send logs generated by Java applications (based on Spring Boot and slf4j) directly from Docker containers to ELK Stack (Elasticsearch, Logstash, Kibana). We will explore the best practices for using Filebeat as a log collector to avoid local file storage and achieve efficient log transmission and analysis.

Reasons to choose Filebeat

In a microservice architecture, efficient log management is crucial. Sending logs directly to the ELK Stack via HTTP is a feasible solution, but using Filebeat is often a better choice. Filebeat is a lightweight log collector specially designed for safe and reliable transmission of log data. It has the following advantages:

• Reliability: Filebeat can guarantee at least one transmission of the log, ensuring no data loss even in the event of network interruption.
• Low resource usage: Filebeat takes up very little resources and has almost negligible impact on the performance of Java applications.
• Simple configuration: Filebeat's configuration is relatively simple and easy to integrate into Docker containers.
• Deep integration with ELK Stack: Filebeat is tightly integrated with Elasticsearch and Logstash, making it easy to send log data to ELK Stack for processing and analysis.

Implementation steps

Here are the detailed steps to send Java application logs to ELK Stack:

1. Configuring log output for Java applications

   Make sure your Java application uses slf4j as the logging facade and configures a suitable logging implementation, such as Logback or Log4j2. Configure the log to output the log to the console (standard output). This is key because Docker containers capture the standard output and standard error streams by default.

   For example, using Logback, you can configure the following in the logback.xml file:

    <configuration>
       <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
           <encoder>
               <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
           </encoder>
       </appender>
   
       <root level="INFO">
           <appender-ref ref="STDOUT"></appender-ref>
       </root>
   </configuration>

2. Configure Filebeat

   Install Filebeat in your Docker container. You can install it through Dockerfile or other methods.

   Filebeat's configuration file (filebeat.yml) needs to be modified accordingly. Here is an example configuration:

    filebeat.inputs:
     - type: docker
       containers.ids: '*'
       # Uncomment to specify which containers logs to read
       #containers.ids: ["container_id1", "container_id2"]
   
   # Configure processors to enhance or manipulate events generated by the input.
   processors:
     - add_cloud_metadata: ~
     - add_docker_metadata: ~
     - add_host_metadata: ~
   
   output.logstash:
     hosts: ["logstash:5044"] # Replace with your Logstash host and port

   • filebeat.inputs.type: docker specifies Filebeat to read logs from the Docker container.
   • containers.ids: '*' specifies Filebeat to monitor the logs of all containers. You can specify a specific container ID if needed.
   • output.logstash.hosts specifies the address of Logstash. You need to replace this with your actual Logstash hostname and port.

3. Docker Compose configuration (optional)

   If your ELK Stack and Java applications are running in Docker containers, you can use Docker Compose to manage them. Here is a sample docker-compose.yml file:

    version: "3.7"
   services:
     elasticsearch:
       image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
       ports:
         - "9200:9200"
       environment:
         - "discovery.type=single-node"
   
     logstash:
       image: docker.elastic.co/logstash/logstash:7.17.0
       ports:
         - "5044:5044"
       depends_on:
         -elasticsearch
       volumes:
         - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
   
     kibana:
       image: docker.elastic.co/kibana/kibana:7.17.0
       ports:
         - "5601:5601"
       depends_on:
         -elasticsearch
   
     java-app:
       build: ./java-app
       depends_on:
         -elasticsearch
   
     filebeat:
       image: docker.elastic.co/beats/filebeat:7.17.0
       volumes:
         - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
         - /var/lib/docker/containers:/var/lib/docker/containers:ro # Mount the Docker container logs
       depends_on:
         -java-app
         - logstash
   

   Note: Dockerfile needs to be included in the java-app directory.

   You need to create the logstash.conf and filebeat.yml files and place them in the same directory as the docker-compose.yml file.

4. Logstash configuration (optional)

   Logstash can be used to parse and transform log data. If your log format is complex, you can use Logstash to extract key information. The following is a sample logstash.conf file:

    input {
     beats {
       port => 5044
     }
   }
   
   filter {
     grok {
       match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:thread}\] %{LOGLEVEL:level} %{DATA:logger} - %{GREEDYDATA:message}" }
     }
   }
   
   output {
     elasticsearch {
       hosts => ["elasticsearch:9200"] # Replace with your Elasticsearch host and port
       index => "java-app-%{ YYYY.MM.dd}"
     }
   }

   • input.beats.port specifies the port on which Logstash listens for log data sent by Filebeat.
   • filter.grok.match Use Grok filters to parse log messages. You need to modify the Grok expression according to your log format.
   • output.elasticsearch.hosts specifies the address of Elasticsearch.
   • output.elasticsearch.index specifies the index name of Elasticsearch.

5. Start the ELK Stack and Java applications

   Use the docker-compose up command to start the ELK Stack and Java applications.

6. Verify log data

   In Kibana, you can create an index schema and then search and analyze log data from Java applications.

Things to note

• Make sure Filebeat can access the Docker container's log directory. Typically, you need to mount the /var/lib/docker/containers directory into the Filebeat container.
• Modify Logstash's Grok expression according to your log format.
• Configure other options for Filebeat and Logstash according to your needs.
• Regularly check the Filebeat and Logstash logs to ensure they are functioning properly.
• Consider using container orchestration tools such as Kubernetes to manage ELK Stack and Java applications.

Summarize

Using Filebeat to send Java application logs to the ELK Stack is an efficient and reliable method. By configuring Filebeat and Logstash, you can easily collect, parse, and analyze log data of Java applications to better understand the running status of the application. Remember to adjust the configuration to your specific needs and monitor the health of your system regularly.

The above is the detailed content of A practical guide to efficiently sending Java application logs to the ELK Stack. For more information, please follow other related articles on the PHP Chinese website!