Writing a Python crawler from scratch - using the Scrapy framework to write a crawler

A web crawler is a program that crawls data on the Internet. It can be used to crawl the HTML data of a specific web page. Although we use some libraries to develop a crawler program, using a framework can greatly improve efficiency and shorten development time. Scrapy is written in Python, lightweight, simple and easy to use. Using Scrapy can easily complete the collection of online data. It has completed a lot of work for us without having to expend great efforts to develop it ourselves.

First of all, I need to answer a question.
Q: How many steps does it take to install a website into a crawler?
The answer is simple, four steps:
New Project (Project): Create a new crawler project
Clear Target (Items): Clarify the goals you want to crawl
Create a Spider (Spider): Create a crawler to start crawling web pages
Storage content (Pipeline): Design a pipeline to store crawled content

Okay, now that the basic process has been determined, just complete it step by step.

1. Create a new project (Project)
Hold down the Shift key and right-click in the empty directory, select "Open command window here", and enter the command:

scrapy startproject tutorial

Where tutorial is the project name.
You can see that a tutorial folder will be created, with the directory structure as follows:

tutorial/  
    scrapy.cfg  
    tutorial/  
        __init__.py  
        items.py  
        pipelines.py  
        settings.py  
        spiders/  
            __init__.py  
            ...

The following is a brief introduction to the role of each file:
scrapy.cfg: the project’s configuration file
tutorial/: the project’s Python module, which will be The code quoted here
tutorial/items.py: the project’s items file
tutorial/pipelines.py: the project’s pipelines file
tutorial/settings.py: the project’s settings file
tutorial/spiders/: the directory where the crawler is stored

2. Clear target (Item)
In Scrapy, items are containers used to load crawled content, a bit like Dic in Python, which is a dictionary, but provide some additional protection to reduce errors.
Generally speaking, items can be created using the scrapy.item.Item class, and properties can be defined using the scrapy.item.Field object (which can be understood as a mapping relationship similar to ORM).
Next, we start to build the item model.
First of all, the content we want is:
Name (name)
Link (url)
Description (description)

Modify the items.py file in the tutorial directory and add our own class after the original class.
Because we want to capture the content of the dmoz.org website, we can name it DmozItem:

# Define here the models for your scraped items  
#  
# See documentation in:  
# http://doc.scrapy.org/en/latest/topics/items.html  
  
from scrapy.item import Item, Field  
  
class TutorialItem(Item):  
    # define the fields for your item here like:  
    # name = Field()  
    pass  
  
class DmozItem(Item):  
    title = Field()  
    link = Field()  
    desc = Field()

It may seem a little confusing at first, but defining these items allows you to know your items when using other components what exactly is it.
Item can be simply understood as an encapsulated class object.

3. Make a crawler (Spider)

Making a crawler is generally divided into two steps: crawl first and then fetch.
That is to say, first you need to get all the content of the entire web page, and then take out the parts that are useful to you.
3.1 Crawling
Spider is a class written by the user to crawl information from a domain (or domain group).
They define the list of URLs for downloading, the scheme for tracking links, and the way to parse the content of the web page to extract items.
To build a Spider, you must create a subclass with scrapy.spider.BaseSpider and determine three mandatory attributes:
name: the identification name of the crawler, which must be unique. You must define different names in different crawlers. name.
start_urls: list of crawled URLs. The crawler starts fetching data from here, so the first download of data will start from these URLs. Other sub-URLs will be generated inherited from these starting URLs.
parse(): The parsing method. When calling, pass in the Response object returned from each URL as the only parameter. It is responsible for parsing and matching the captured data (parsing into items) and tracking more URLs.

Here you can refer to the ideas mentioned in the width crawler tutorial to help understand. Tutorial delivery: [Java] Zhihu Chin Episode 5: Using the HttpClient toolkit and width crawler.
That is to say, store the Url and gradually spread it as a starting point, grab all the webpage URLs that meet the conditions, store them and continue crawling.

Let’s write the first crawler, named dmoz_spider.py, and save it in the tutorialspiders directory.
dmoz_spider.py code is as follows:

from scrapy.spider import Spider  
  
class DmozSpider(Spider):  
    name = "dmoz"  
    allowed_domains = ["dmoz.org"]  
    start_urls = [  
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",  
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"  
    ]  
  
    def parse(self, response):  
        filename = response.url.split("/")[-2]  
        open(filename, 'wb').write(response.body)

allow_domains is the domain name range of the search, which is the constraint area of ​​the crawler. It stipulates that the crawler only crawls web pages under this domain name.
As can be seen from the parse function, the last two addresses of the link are taken out and stored as file names.
Then run it and take a look. Hold down shift and right-click in the tutorial directory, open the command window here, enter:

scrapy crawl dmoz

The running result is as shown in the figure:

The error is reported:
UnicodeDecodeError: 'ascii' codec can 't decode byte 0xb0 in position 1: ordinal not in range(128)
I got an error when I ran the first Scrapy project. It was really ill-fated.
There should be a coding problem. I googled and found the solution:
Create a new sitecustomize.py in the Libsite-packages folder of python:

import sys    
sys.setdefaultencoding('gb2312')

Run it again, OK, the problem is solved, take a look at the result:

最后一句INFO: Closing spider (finished)表明爬虫已经成功运行并且自行关闭了。
包含 [dmoz]的行 ，那对应着我们的爬虫运行的结果。
可以看到start_urls中定义的每个URL都有日志行。
还记得我们的start_urls吗？
http://www.dmoz.org/Computers/Programming/Languages/Python/Books
http://www.dmoz.org/Computers/Programming/Languages/Python/Resources
因为这些URL是起始页面，所以他们没有引用(referrers)，所以在它们的每行末尾你会看到 (referer: )。
在parse 方法的作用下，两个文件被创建：分别是 Books 和 Resources，这两个文件中有URL的页面内容。

那么在刚刚的电闪雷鸣之中到底发生了什么呢？
首先，Scrapy为爬虫的 start_urls属性中的每个URL创建了一个 scrapy.http.Request 对象 ，并将爬虫的parse 方法指定为回调函数。
然后，这些 Request被调度并执行，之后通过parse()方法返回scrapy.http.Response对象，并反馈给爬虫。

3.2取
爬取整个网页完毕，接下来的就是的取过程了。
光存储一整个网页还是不够用的。
在基础的爬虫里，这一步可以用正则表达式来抓。
在Scrapy里，使用一种叫做 XPath selectors的机制，它基于 XPath表达式。
如果你想了解更多selectors和其他机制你可以查阅资料：点我点我

这是一些XPath表达式的例子和他们的含义
/html/head/title: 选择HTML文档元素下面的'] In [3]: sel.xpath('//title/text()') Out[3]: [] In [4]: sel.xpath('//title/text()').extract() Out[4]: [u'Open Directory - Computers: Programming: Languages: Python: Books'] In [5]: sel.xpath('//title/text()').re('(\w+):') Out[5]: [u'Computers', u'Programming', u'Languages', u'Python']