How to implement page turning in python crawler

Many friends who are new to python crawlers will encounter the problem of page turning in python crawlers. I will introduce one here.

The website that needs to be crawled is as shown in the picture

There are four pages of this plant queried. When we usually turn pages, our first thought is to click on the next page on the page. The same is true when writing a crawler. We want to extract links on the page for access. But doing so is cumbersome and inefficient.

There are generally two ways to turn pages:

1. Observe the link changes when the website turns pages

2. If the request method of writing a crawler is a post request, then It is necessary to observe whether the changes in post data are regular.

Today I will talk about the first method

As shown in the picture, when I click on the next page, I observe the link change in the address bar

We can be surprised to find that there is a page keyword in the link, and it happens to be 2. Let’s test it and change the value of page in the link to 3, and find that the access is normal. It's exactly the third page. Therefore, we found a way to turn the page and change the value of the keyword in the link.

So the steps of the page turning process are:

1. Get the total number of web pages

2. Use the for loop to change the value of page to turn the page.

The specific page turning code is implemented as follows:

Get the total page number

Network request method

Change the page value in the request header

For more Python related technical articles, please visit the Python Tutorial column to learn!

The above is the detailed content of How to implement page turning in python crawler. For more information, please follow other related articles on the PHP Chinese website!