Methods and techniques for implementing log aggregation and statistics under Linux

Introduction:
In the process of application development and maintenance, logging is a very important task. By outputting logs, we can monitor the running status of the application in real time, troubleshoot problems, and perform performance analysis and optimization. However, in large systems, log files are usually scattered on different servers, making log search and analysis difficult. Therefore, it is very necessary to understand how to implement log aggregation and statistics under Linux.

1. Use rsyslog for log collection:
rsyslog is a popular log management software on Linux, which can help us collect, filter, process and forward logs. The following is a simple usage example:

1. Install rsyslog on server A:
   $ sudo apt-get install rsyslog
2. Configure the rsyslog.conf file:
   $ sudo vi /etc/rsyslog.conf
   Add the following content to the file:

Forward all logs to server B

. @serverBIP:514

3. Restart the rsyslog service:
   $ sudo service rsyslog restart

With the above configuration, all logs on server A will be sent to port 514 on server B.

2. Use ELK Stack for log analysis:
ELK Stack is a complete log analysis solution, including Elasticsearch, Logstash and Kibana. Here is a brief usage example:

1. Install Elasticsearch:
   $ sudo apt-get install default-jre
   $ wget -qO - https://artifacts.elastic.co/ GPG-KEY-elasticsearch | sudo apt-key add -
   $ sudo apt-get update && sudo apt-get install elasticsearch
2. Configure Elasticsearch:
   $ sudo vi /etc/elasticsearch/elasticsearch. yml
   Modify the following configuration items:

network.host: localhost
http.port: 9200

3. Start the Elasticsearch service:
   $ sudo service elasticsearch start
4. Install Logstash:
   $ sudo apt-get install logstash
5. Configure Logstash:
   $ sudo vi /etc/logstash/conf.d/logstash.conf
   Add the following:

input {
file {

path => "/var/log/nginx/access.log"

}
}

output {
elasticsearch {

hosts => ["localhost:9200"]
index => "nginx-access-logs"

}
}

6. Start Logstash service:
   $ sudo service logstash start
7. Install Kibana:
   $ wget -qO - https:// artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
   $ sudo apt-get install kibana
8. Configure Kibana:
   $ sudo vi /etc/kibana/kibana. yml
   Modify the following configuration items:

server.host: "localhost"
elasticsearch.url: "http://localhost:9200"

9. Start the Kibana service:
   $ sudo service kibana start

Through the above configuration and steps, we can view and analyze log data in real time in Kibana's web interface.

3. Use AWK for log statistics:
AWK is a powerful tool that can realize text analysis and processing, and is very useful in log statistics. Here is a simple example:

1. Use AWK to count the number of accesses for each IP address:
   $ awk '{ print $1 }' /var/log/nginx/access.log | sort | uniq -c
2. Use AWK to count the number of visits to each URL:
   $ awk '{ print $6 }' /var/log/nginx/access.log | sort | uniq -c

With the above command, we can easily count the number of visits to each IP address and URL.

Summary:
There are many methods and techniques to implement log aggregation and statistics under Linux. This article introduces simple examples using tools such as rsyslog, ELK Stack and AWK. Through these tools, we can better manage and analyze logs and improve the operating efficiency and stability of applications. Hope this article helps you!

The above is the detailed content of Methods and techniques for implementing log aggregation and statistics under Linux. For more information, please follow other related articles on the PHP Chinese website!