How Can I Efficiently Read a Large Text File in Reverse Order Using Python?-Python Tutorial-php.cn

How Can I Efficiently Read a Large Text File in Reverse Order Using Python?

Barbara Streisand

Release： 2024-11-29 14:03:14

Original

194 people have browsed it

How Can I Efficiently Read a Large Text File in Reverse Order Using Python?

Reading a File in Reverse Order Using Python

Suppose you have a lengthy text file and desire to read its contents from the final line to the initial line. How can you achieve this in Python?

Answer:

The following Python generator function effectively reads a file in reverse order:

import os

def reverse_readline(filename, buf_size=8192):
    """A generator that returns the lines of a file in reverse order"""
    with open(filename, 'rb') as fh:
        segment = None
        offset = 0
        fh.seek(0, os.SEEK_END)
        file_size = remaining_size = fh.tell()
        while remaining_size > 0:
            offset = min(file_size, offset + buf_size)
            fh.seek(file_size - offset)
            buffer = fh.read(min(remaining_size, buf_size))
            # remove file's last "\n" if it exists, only for the first buffer
            if remaining_size == file_size and buffer[-1] == ord('\n'):
                buffer = buffer[:-1]
            remaining_size -= buf_size
            lines = buffer.split('\n'.encode())
            # append last chunk's segment to this chunk's last line
            if segment is not None:
                lines[-1] += segment
            segment = lines[0]
            lines = lines[1:]
            # yield lines in this chunk except the segment
            for line in reversed(lines):
                # only decode on a parsed line, to avoid utf-8 decode error
                yield line.decode()
        # Don't yield None if the file was empty
        if segment is not None:
            yield segment.decode()

Copy after login

This generator function exhibits a memory-efficient performance and is suitable for large files. It splits the file into manageable chunks and processes them backward line by line.

The above is the detailed content of How Can I Efficiently Read a Large Text File in Reverse Order Using Python?. For more information, please follow other related articles on the PHP Chinese website!