Two Golang large file reading solutions

The following column golang tutorial will introduce to you two solutions for reading very large files in Golang. I hope it will be helpful to friends in need!

Two solutions for reading very large files in Golang

1. Stream processing method

2. Film processing

In last year’s interview, I was asked how do you deal with very large files. I really didn’t think much about this question at the time. After I came back, I studied and discussed this issue carefully and made an analysis of reading large files.

For example, we have a log file that has been running for several years and is 100G in size. According to our previous operation, the code may be written like this:

func ReadFile(filePath string) []byte{
    content, err := ioutil.ReadFile(filePath)
    if err != nil {
        log.Println("Read error")
    }
    return content
}

The above code can read a file of several megabytes, but if it is larger than yourself and its memory, it will directly overturn. Because the above code reads all the contents of the file into the memory and then returns it. If you have a file of several megabytes, your memory is large enough to handle it, but once the file is hundreds of megabytes, it will not be so easy to process. Then, there are two correct methods. The first is to use stream processing. The code is as follows:

func ReadFile(filePath string, handle func(string)) error {
    f, err := os.Open(filePath)
    defer f.Close()
    if err != nil {
        return err
    }
    buf := bufio.NewReader(f)

    for {
        line, err := buf.ReadLine("\n")
        line = strings.TrimSpace(line)
        handle(line)
        if err != nil {
            if err == io.EOF{
                return nil
            }
            return err
        }
        return nil
    }
}

The second solution is to shard processing. When reading a binary file, When there is no newline character, use the following solution to process large files

func ReadBigFile(fileName string, handle func([]byte)) error {
    f, err := os.Open(fileName)
    if err != nil {
        fmt.Println("can't opened this file")
        return err
    }
    defer f.Close()
    s := make([]byte, 4096)
    for {
        switch nr, err := f.Read(s[:]); true {
        case nr < 0:
            fmt.Fprintf(os.Stderr, "cat: error reading: %s\n

For more related technical articles, please visit the go language tutorial column!

The above is the detailed content of Two Golang large file reading solutions. For more information, please follow other related articles on the PHP Chinese website!