Is it difficult to learn python crawler?

Simply speaking, the Internet is a large network composed of sites and network devices. We access the site through a browser, and the site returns HTML, JS, and CSS codes to the browser. These codes are parsed and rendered by the browser, and then Rich and colorful web pages appear before our eyes.

What is a crawler?

If we compare the Internet to a large spider web, data is stored in each node of the spider web, and a crawler is a small spider that crawls its own information along the network. A prey (data) crawler refers to a program that initiates a request to a website, obtains resources, analyzes and extracts useful data; from a technical perspective, it simulates the behavior of a browser requesting a site through a program, and converts the HTML code/JSON data returned by the site. /Binary data (pictures, videos) Climb locally, extract the data you need, and store it for use.

Basic process of crawler

How users obtain network data:

Method 1: Browser submits request--->Download web page code --->Parsed into a page

Method 2: Simulate the browser to send a request (get the web page code)->Extract useful data->Save it in a database or file

Crawler All you have to do is method 2;

Initiate a request

Use the http library to initiate a request to the target site, that is, send a Request

Request Contains: request header, request body, etc.

Request module defect: Unable to execute JS and CSS code

Get response content

If the server can respond normally, you will get a Response

Response includes: html, json, pictures, videos, etc.

Parsing content

Parsing html data: regular expressions (RE module), third-party parsing libraries such as Beautifulsoup, pyquery, etc.

Parse json data: json module

Parse binary data: write to file in wb mode

Save data

Database (MySQL, Mongdb , Redis)

File

The above is the detailed content of Is it difficult to learn python crawler?. For more information, please follow other related articles on the PHP Chinese website!