Python implements real-time monitoring of files

When business stability requirements are relatively high, in order to detect problems in time, operation and maintenance sometimes need to conduct real-time analysis of application logs. When certain conditions are met, an alarm will be issued immediately instead of passively waiting for problems to occur. To solve it, for example, you need to monitor nginx's $request_time and $upstream_response_time, analyze the most time-consuming requests, and then improve the code. At this time, you need to analyze the logs in real time. If you find long statements, you will alarm and remind them. Developers should pay attention. Of course, this is one of the application scenarios. This monitoring method can also be applied to any place where files need to be judged or analyzed. So today we will take a look at how to use python to monitor files in real time. I will give you three Method example:

The first one:

This is the simplest and easiest to understand, because everyone knows that there is a tail command under Linux, so You can directly use the Popen() function to call this command to execute and obtain the output. The code is as follows:

logfile='access.log'
command='tail -f ‘+logfile+'|grep “timeout”‘
popen=subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)
while True:
line=popen.stdout.readline().strip()
print line

Second type:

Use Python to operate files. Use the tell() and seek() methods of the file object to obtain the current file position and the position to be moved to respectively. The code is as follows:

import time
file = open(‘access.log')
while 1:
where = file.tell()
line = file.readline()
if not line:
time.sleep(1)
file.seek(where)
else:
print line,

The third method:

Use python’s yield to implement a generator function, and then call the generator function, so that when the log file changes When printing a new line, the code is as follows:

import time
def follow(thefile):
thefile.seek(0,2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
if __name__ == ‘__main__':
logfile = open(“access-log”,”r”)
loglines = follow(logfile)
for line in loglines:
print line,

Finally, explain the usage of the seek() function. This function receives 2 parameters: file.seek( off, whence=0), move off operation marks (file pointers) from the file, positive numbers move to the end direction, and negative numbers move to the start direction. If the whence parameter is set, the starting position set by whence shall prevail. 0 represents the beginning, 1 represents the current position, and 2 represents the end position of the file.

The above are three common methods. You can implement the specific log analysis code according to your own business logic. Complete.

The above is the entire content of this article. I hope it will be helpful to everyone's learning, and I also hope that everyone will subscribe to the PHP Chinese website.

For more articles related to python’s real-time monitoring of files, please pay attention to the PHP Chinese website!