Scraping Webpages in Python With Beautiful Soup: Search and DOM Modification

This tutorial builds upon the previous introduction to Beautiful Soup, focusing on DOM manipulation beyond simple tree navigation. We'll explore efficient search methods and techniques for modifying HTML structure.

One common DOM search method is extend(). This method adds a list of elements to a tag, preserving their original order. However, simply extending with extend() might result in elements being added as strings rather than proper HTML tags.

import requests
from bs4 import BeautifulSoup

req = requests.get('https://en.wikipedia.org/wiki/Python_(programming_language)')
soup = BeautifulSoup(req.text, "lxml")

new_soup = BeautifulSoup("<ol></ol>", "lxml")

new_soup.ol.extend(['

The above example demonstrates this string-based addition. To add elements as proper HTML tags, use the wrap() method. Conversely, the unwrap() method removes tags while preserving their content. Let's illustrate with an example of removing bold and italic tags:

soup = BeautifulSoup("<p>We will <i>try</i> to get rid of <b>tags</b> that make text <b>bold</b> or <i>italic</i>. The content <i>within</i> the <b>tags</b> should still be <b>preserved</b>.</p>", "lxml")

for unwanted_tag in soup.find_all(["b", "i"]):
    unwanted_tag.unwrap()

print(soup.p.contents)
soup.p.smooth()
print(soup.p.contents)

Here, unwrap() removes the unwanted tags. Notice that after removing tags, the text fragments are stored as NavigableStrings. The smooth() method concatenates these strings into a single string, simplifying the output.

Conclusion

These tutorials provide a foundation for web scraping with Beautiful Soup. You've learned how to parse web pages, extract data, modify the HTML structure, and save the results locally. This empowers you to effectively process and manipulate web content for various applications.

The above is the detailed content of Scraping Webpages in Python With Beautiful Soup: Search and DOM Modification. For more information, please follow other related articles on the PHP Chinese website!