Understand Python crawler in one article

A crawler is simply a name for the process of using a program to obtain data on the Internet.

Principle of crawler

If we want to obtain data on the network, we need to give the crawler a website address (usually called URL in the program), the crawler sends an HTTP request to the server of the target web page, and the server returns the data To the client (that is, our crawler), the crawler then performs a series of operations such as data parsing and saving.

Process

Crawlers can save us time. For example, if I want to get the top 250 Douban movies, if we don’t use crawlers, we must first enter the URL of Douban movies on the browser, and the client ( The browser) finds the IP address of the server of the Douban Movie web page through analysis, and then establishes a connection with it. The browser creates an HTTP request and sends it to the Douban Movie server. After the server receives the request, it extracts the Top250 list from the database. , encapsulate it into an HTTP response, and then return the response result to the browser. The browser displays the response content and we see the data. Our crawler is also based on this process, but it is changed into code form.

HTTP request

HTTP request consists of request line, request header, blank line, and request body.

The request line consists of three parts:

1. Request method, common request methods are GET, POST, PUT, DELETE, HEAD
2. The resource path that the client wants to obtain
3. It is the HTTP protocol version number used by the client
The request header is a supplementary description of the request sent by the client to the server, such as the identity of the visitor, which will be discussed below .

The request body is the data submitted by the client to the server, such as the account and password information that needs to be improved when the user logs in. The request header and request body are separated by blank lines. The request body is not included in all requests. For example, general GET does not have a request body.

The picture above is the HTTP POST request sent to the server when the browser logs into Douban. The username and password are specified in the request body.

HTTP response

HTTP response format is very similar to the request format, and also consists of response lines, response headers, blank lines, and response bodies.

The response line also contains three parts, namely the HTTP version number of the server, the response status code and the status description.

There is a table of status codes here, corresponding to the meaning of each status code

## Part 2 It is the response header. The response header corresponds to the request header. It is some additional instructions from the server to the response, such as what is the format of the response content, how long is the response content, when is it returned to the client, and even some cookie information. will be placed in the response header.

The third part is the response body, which is the real response data. These data are actually the HTML source code of the web page.

How to write crawler code

Crawlers can use many languages such as Python, C, etc., but I think Python is the easiest,

because Python has ready-made libraries , has been encapsulated to almost perfection. Although

C also has ready-made libraries, its crawlers are still relatively niche. The only libraries are not simple enough, and the code is on various compilers, even on the same The compatibility between different versions of a compiler is not strong, so it is not particularly easy to use. So today we mainly introduce python crawler.

Install requests library

cmd run: pip install requests to install requests.

Then enter

import requests on IDLE or a compiler (personally recommend VS Code or Pycharm) to run. If no error is reported, the installation is successful.

The method to install most libraries is: pip install xxx (name of library)

Methods of requests

##requests.request() Construct a request and support the basic methods of each method requests.get() The main method to obtain HTML web pages, corresponding to HTTP's GET requests.post() To The method of submitting a POST request to an HTML web page, corresponding to HTTP's POST requests.put() The method of submitting a PUT request to an HTML web page, corresponding to HTTP's PUT requests.patch( ) Submit a partial modification request to the HTML web page, corresponding to HTTP's PATCT requests. delete() Submit a delete request to an HTML web page, corresponding to HTTP's DELETE

requests.head()	Method to obtain HTML web page header information, corresponding to HTTP HEAD

The most commonly used get method

r = requests .get(url)

Includes two important objects:

Constructs a Request object that requests resources from the server; returns a Response object containing server resources

r.status_code The return status of the HTTP request, 200 indicates successful connection, 404 indicates failure r.text The string form of the HTTP response content, that is, the page content corresponding to the url r.encoding The response content encoding method guessed from the HTTP header (if the header If charset does not exist, the encoding is considered to be ISO-8859-1) r.apparent_encoding The response content encoding method analyzed from the content (alternative encoding method ) r.content The binary form of the HTTP response content requests.ConnectionError Network connection error exception, such as DNS query failure, connection refused, etc. requests.HTTPError HTTP error exception requests.URLRequired URL missing exception requests.TooManyRedirects Exceeds the maximum number of redirects and generates Redirect exception requests.ConnectTimeout Timeout exception when connecting to the remote server requests.Timeout The request URL times out, resulting in a timeout exception Crawler small demo

requests is the most basic crawler library, but we can make a simple translation

I will first put the project structure of a small crawler project I made. The complete source code can be downloaded by private chatting with me.

The following is the source code of the translation part

import requests def English_Chinese(): url = "https://fanyi.baidu.com/sug" s = input("请输入要翻译的词(中/英):") dat = { "kw":s } resp = requests.post(url,data = dat)# 发送post请求 ch = resp.json() # 将服务器返回的内容直接处理成json => dict resp.close() dic_lenth = len(ch['data']) for i in range(dic_lenth): print("词:"+ch['data'][i]['k']+" "+"单词意思:"+ch['data'][i]['v'])

Detailed code explanation:

Import the requests module and set the url to the URL of the Baidu translation web page.

Then send the request through the post method, and then type the returned result into a dic (dictionary), but this time we printed it out and found that it is like this.

This is what a dictionary inside a list inside a dictionary looks like, probably like this

{ xx:xx , xx:

[{xx:xx} , {xx:xx} , {xx:xx} , {xx:xx}]}

The place I marked in red is information we need.

Suppose there are n dictionaries in the list marked blue, we can get the value of n through the len() function,

and use a for loop to traverse to get the result.

dic_lenth = len(ch['data'] for i in range(dic_lenth): print("词:"+ch['data'][i]['k']+" "+"单词意思:"+ch['data'][i]['v'])

Finally

Okay, that’s it for today’s sharing, bye~

Hey? I forgot one thing, let me give you another code to crawl the weather!

# -*- coding:utf-8 -*- import requests import bs4 def get_web(url): header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36 Edg/91.0.864.59"} res = requests.get(url, headers=header, timeout=5) # print(res.encoding) content = res.text.encode('ISO-8859-1') return content def parse_content(content): soup = bs4.BeautifulSoup(content, 'lxml') ''' 存放天气情况 ''' list_weather = [] weather_list = soup.find_all('p', class_='wea') for i in weather_list: list_weather.append(i.text) ''' 存放日期 ''' list_day = [] i = 0 day_list = soup.find_all('h1') for each in day_list: if i <= 6: list_day.append(each.text.strip()) i += 1 # print(list_day) ''' 存放温度：最高温度和最低温度 ''' tem_list = soup.find_all('p', class_='tem') i = 0 list_tem = [] for each in tem_list: if i == 0: list_tem.append(each.i.text) i += 1 elif i > 0: list_tem.append([each.span.text, each.i.text]) i += 1 # print(list_tem) ''' 存放风力 ''' list_wind = [] wind_list = soup.find_all('p', class_='win') for each in wind_list: list_wind.append(each.i.text.strip()) # print(list_wind) return list_day, list_weather, list_tem, list_wind def get_content(url): content = get_web(url) day, weather, tem, wind = parse_content(content) item = 0 for i in range(0, 7): if item == 0: print(day[i]+':\t') print(weather[i]+'\t') print("今日气温："+tem[i]+'\t') print("风力："+wind[i]+'\t') print('\n') item += 1 elif item > 0: print(day[i]+':\t') print(weather[i] + '\t') print("最高气温："+tem[i][0]+'\t') print("最低气温："+tem[i][1] + '\t') print("风力："+wind[i]+'\t') print('\n')