Home >Backend Development >Python Tutorial >How to implement web crawler in python

How to implement web crawler in python

coldplay.xixi
coldplay.xixiOriginal
2020-10-21 15:09:0827610browse

How to implement a web crawler in python: 1. Use the get method in the request library to request the web page content of the url; 2. The [find()] and [find_all()] methods can traverse the html file and extract Specify information.

How to implement web crawler in python

How to implement a web crawler in python:

The first step: crawling

Use the get method in the request library to request the web page content of the url

Write code

[root@localhost demo]# touch demo.py
[root@localhost demo]# vim demo.py
#web爬虫学习 -- 分析
#获取页面信息
 
#输入:url
#处理:request库函数获取页面信息,并将网页内容转换成为人能看懂的编码格式
#输出:爬取到的内容
 
import requests
 
def getHTMLText(url):
    try:
        r = requests.get( url, timeout=30 )
        r.raise_for_status()    #如果状态码不是200,产生异常
        r.encoding = 'utf-8'    #字符编码格式改成 utf-8
        return r.text
    except:
        #异常处理
        return " error "
 
url = "http://www.baidu.com"
print( getHTMLText(url) )
[root@localhost demo]# python3 demo.py

How to implement web crawler in python

Step 2: Analysis

Use the BeautifulSoup class in the bs4 library to generate an object. The find() and find_all() methods can traverse this html file and extract specified information.

Writing code

[root@localhost demo]# touch demo1.py
[root@localhost demo]# vim demo1.py
#web爬虫学习 -- 分析
#获取页面信息
 
#输入:url
#处理:request库获取页面信息,并从爬取到的内容中提取关键信息
#输出:打印输出提取到的关键信息
 
import requests
from bs4 import BeautifulSoup
import re
 
def getHTMLText(url):
    try:
        r = requests.get( url, timeout=30 )
        r.raise_for_status()    #如果状态码不是200,产生异常
        r.encoding = 'utf-8'    #字符编码格式改成 utf-8
        return r.text
    except:
        #异常处理
        return " error "
 
def findHTMLText(text):
    soup = BeautifulSoup( text, "html.parser" )    #返回BeautifulSoup对象
    return soup.find_all(string=re.compile( '百度' )) #结合正则表达式,实现字符串片段匹配
 
url = "http://www.baidu.com"
text = getHTMLText(url)        #获取html文本内容
res = findHTMLText(text)    #匹配结果
 
print(res)        #打印输出
[root@localhost demo]# python3 demo1.py

How to implement web crawler in python

##Related free learning recommendations: python video tutorial

The above is the detailed content of How to implement web crawler in python. For more information, please follow other related articles on the PHP Chinese website!

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn