How to Efficiently Read the Last Two Lines of a Large Log File in Go Every 10 Seconds?

Reading Last Lines from Large Log Files in Go Every 10 Seconds

When dealing with large log files, it becomes imperative to devise efficient methods to monitor and analyze the latest data without overloading memory. This article tackles this challenge by discussing an approach to read the last two lines of a log file every 10 seconds using Go.

To begin, we utilize a timer function (time.Tick) configured to trigger every 10 seconds. Within this function, the readFile function is invoked to request the latest lines from the log file.

To determine the starting point for reading the last lines, we employ the (os.File).Stat method to retrieve the file's size. Assuming that each line spans approximately 32 bytes, we calculate the start position as fileSize - 62* (for the last two lines).

Here's an example based on our assumptions:

package main

import (
    "fmt"
    "os"
    "time"
)

const MYFILE = "logfile.log"

func main() {
    c := time.Tick(10 * time.Second)
    for now := range c {
        readFile(MYFILE)
    }
}

func readFile(fname string) {
    file, err := os.Open(fname)
    if err != nil {
        panic(err)
    }
    defer file.Close()

    buf := make([]byte, 62)
    stat, statErr := file.Stat()
    if statErr != nil {
        panic(statErr)
    }
    start := stat.Size() - 62
    _, err = file.ReadAt(buf, start)
    if err == nil {
        fmt.Printf("%s\n", buf)
    }
}

This solution efficiently retrieves the last two lines of the log file without loading it into memory completely, ensuring efficient monitoring of large log files in real-time.

The above is the detailed content of How to Efficiently Read the Last Two Lines of a Large Log File in Go Every 10 Seconds?. For more information, please follow other related articles on the PHP Chinese website!