Home>Article>Backend Development> What packages should be installed for python crawler?

What packages should be installed for python crawler?

silencement
silencement Original
2019-05-17 19:16:28 10484browse

What libraries need to be installed to learn Python web crawler?

What packages should be installed for python crawler?

The following are the relevant libraries involved in Python crawlers

Request library, parsing library, storage library, tool library

1. Request library: urllib/re/requests

(1) urllib/re is the library that comes with python by default. It can be verified by the following command:

What packages should be installed for python crawler?

No error message is output, indicating that the environment is normal

(2) requests installation

2.1 Open CMD, enter pip3 install requests

2.2 Wait for installation , Verification

What packages should be installed for python crawler?

(3) selenium installation (driving the browser for website access behavior)

3.1 Open CMD and enter pip3 install selenium

3.2 Install chromedriver

Website: https://npm.taobao.org/

Extract the downloaded compressed package and put the exe into D:\Python3.6.0 \Scripts\

This path only needs to be in the PATH variable

3.3 Wait for the installation to be completed, verify

What packages should be installed for python crawler?

and press Enter to pop up chrome browser interface

3.4 Install other browsers

Interfaceless browser phantomjs

Download URL: http://phantomjs.org/

Download After completion, unzip it, put the entire directory into D:\Python3.6.0\Scripts\, and add the path to the bin directory to the PATH variable

Verification:

Open CMD

What packages should be installed for python crawler?

2. Parsing library:

2.1 lxml (XPATH)

Open CMD and enter pip3 install lxml or download from https://pypi.python.org, For example, lxml-4.1.1-cp36-cp36m-win_amd64.whl (md5), first download the whl file, and execute pip3 install on the command line. File name.whl

2.2 beautifulsoup

To open CMD, you need to install lxml first

pip3 install beautifulsoup4

What packages should be installed for python crawler?

2.3 pyquery (similar to jquery syntax)

Open CMD , pip3 install pyquery

Verify the installation results

What packages should be installed for python crawler?

3. Repository

3.1 pymysql (operating MySQL, relational database)

Installation: pip3 install pymysql, test after installation:

What packages should be installed for python crawler?

3.2 pymongo (operating MongoDB, key-value)

Installation pip3 install pymongo

Verification

What packages should be installed for python crawler?

##3.3 redis (distributed crawler, maintain crawling queue) Installation: pip3 install redis


Verification:

What packages should be installed for python crawler?

4.1Django (distributed crawler maintenance system) pip3 install django


4.2jupyter (notepad running on the web page, supports markdown , you can run the code on the web page) Install pip3 install jupyter


Verification: Open CMD, jupyter notebook

and then you can create notepads, code blocks and Markdown blocks directly on the web page, Support printing

The above is the detailed content of What packages should be installed for python crawler?. For more information, please follow other related articles on the PHP Chinese website!

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn