How to use Python crawler to crawl web page data using BeautifulSoup and Requests

1. Introduction

The implementation principle of the web crawler can be summarized into the following steps:

• Send HTTP request: The web crawler sends an HTTP request to the target website (Usually a GET request) Get the content of the web page. In Python, HTTP requests can be sent using the requests library.

• Parse HTML: After receiving the response from the target website, the crawler needs to parse the HTML content to extract useful information. HTML is a markup language used to describe the structure of web pages. It consists of a series of nested tags. The crawler can locate and extract the required data based on these tags and attributes. In Python, you can use libraries such as BeautifulSoup and lxml to parse HTML.

• Data extraction: After parsing the HTML, the crawler needs to extract the required data according to predetermined rules. These rules can be based on tag names, attributes, CSS selectors, XPath, etc. In Python, BeautifulSoup provides tag- and attribute-based data extraction capabilities, and lxml and cssselect can handle CSS selectors and XPath.

• Data storage: The data captured by the crawler usually needs to be stored in a file or database for subsequent processing. In Python, you can use file I/O operations, csv library or database connection library (such as sqlite3, pymysql, pymongo, etc.) to save data to a local file or database.

• Automatic traversal: The data of many websites is distributed on multiple pages, and crawlers need to automatically traverse these pages and extract data. The traversal process usually involves discovering new URLs, turning pages, etc. The crawler can look for new URLs while parsing the HTML, add them to the queue to be crawled, and continue with the steps above.

• Asynchronous and concurrency: In order to improve crawler efficiency, asynchronous and concurrency technologies can be used to process multiple requests at the same time. In Python, you can use multi-threading (threading), multi-process (multiprocessing), coroutine (asyncio) and other technologies to achieve concurrent crawling.

• Anti-crawler strategies and responses: Many websites have adopted anti-crawler strategies, such as limiting access speed, detecting User-Agent, verification codes, etc. In order to deal with these strategies, crawlers may need to use proxy IP, simulate browser User-Agent, automatically identify verification codes and other techniques. In Python, you can use the fake_useragent library to generate a random User-Agent, and use tools such as Selenium to simulate browser operations.

2. Basic concepts of web crawlers

A web crawler, also known as a web spider or web robot, is a program that automatically crawls web page information from the Internet. Crawlers usually follow certain rules to visit web pages and extract useful data.

3. Introduction to Beautiful Soup and Requests libraries

1. Beautiful Soup: A Python library for parsing HTML and XML documents, which provides a simple way to Extract data from web pages.

2. Requests: A simple and easy-to-use Python HTTP library for sending requests to websites and getting response content.

4. Select a target website

This article will take a page in Wikipedia as an example to capture the title and paragraph information in the page. To simplify the example, we will crawl the Wikipedia page of the Python language (https://en.wikipedia.org/wiki/Python_(programming_language).

5. Use Requests to obtain web content

First, install the Requests library:

pip install requests

Then, use Requests to send a GET request to the target URL and obtain the HTML content of the webpage:

import requests
 
url = "https://en.wikipedia.org/wiki/Python_(programming_language)"
response = requests.get(url)
html_content = response.text

6. Use Beautiful Soup to parse the webpage content

Install Beautiful Soup:

pip install beautifulsoup4

Next, use Beautiful Soup to parse the web content and extract the required data:

from bs4 import BeautifulSoup
 
soup = BeautifulSoup(html_content, "html.parser")
 
# 提取标题
title = soup.find("h2", class_="firstHeading").text
 
# 提取段落
paragraphs = soup.find_all("p")
paragraph_texts = [p.text for p in paragraphs]
 
# 打印提取到的数据
print("Title:", title)
print("Paragraphs:", paragraph_texts)

7. Extract the required data and save it

Save the extracted data to a text file:

with open("wiki_python.txt", "w", encoding="utf-8") as f:
    f.write(f"Title: {title}\n")
    f.write("Paragraphs:\n")
    for p in paragraph_texts:
        f.write(p)
        f.write("\n")

The above is the detailed content of How to use Python crawler to crawl web page data using BeautifulSoup and Requests. For more information, please follow other related articles on the PHP Chinese website!