How Can Python Efficiently Extract Plain Text from HTML, Handling Entities and Unwanted Content?

Extracting Plain Text from HTML Using Python

In the pursuit of retrieving the text content from an HTML file, it is crucial to consider robust methods that handle HTML entities correctly and effectively. While solutions using regular expressions may prove limited, libraries like Beautiful Soup offer more sophisticated options. However, issues with capturing unwanted text and entity interpretation remain.

Beautiful Soup: A Powerful Tool with Caveats

Beautiful Soup is a popular choice for HTML parsing, yet it may retrieve additional elements like JavaScript source and fail to interpret HTML entities. For instance, the sequence ' in the source code is not converted to an apostrophe in the extracted text.

Enter html2text: A Promising Solution

Currently, html2text emerges as a compelling option. It handles HTML entities with ease and disregards unnecessary content like JavaScript. While it outputs markdown instead of plain text, this can be easily converted.

A Robust and Customizable Approach

The following code snippet leverages Beautiful Soup and offers enhanced control over the extraction process:

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
html = urlopen(url).read()
soup = BeautifulSoup(html, features="html.parser")

# Remove unwanted elements like scripts and styles
for script in soup(["script", "style"]):
    script.extract()

# Extract the text content
text = soup.get_text()

# Preprocess the text for improved readability
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
text = '\n'.join(chunk for chunk in chunks if chunk)

print(text)

By employing this approach, you can extract plain text effectively, handling both wanted and unwanted content as per your requirements.

The above is the detailed content of How Can Python Efficiently Extract Plain Text from HTML, Handling Entities and Unwanted Content?. For more information, please follow other related articles on the PHP Chinese website!