Scrapy framework and database integration: how to implement dynamic data storage?

As the amount of Internet data continues to increase, how to quickly and accurately crawl, process, and store data has become a key issue in Internet application development. As an efficient crawler framework, the Scrapy framework is widely used in various data crawling scenarios due to its flexible and high-speed crawling methods.

However, just saving the crawled data to a file cannot meet the needs of most applications. Because in current applications, most data is stored, retrieved, and manipulated through databases. Therefore, how to integrate the Scrapy framework with the database to achieve fast and dynamic storage of data has become a new challenge.

This article will combine actual cases to introduce how the Scrapy framework integrates databases and implements dynamic data storage for reference by readers in need.

1. Preparation

Before the introduction, it is assumed that readers of this article have already understood the basic knowledge of the Python language and some methods of using the Scrapy framework, and can also use the Python language to create simple databases. operate. If you are not familiar with this, it is recommended to learn the relevant knowledge first and then read this article.

2. Select the database

Before starting to integrate the Scrapy framework with the database, we need to first choose a suitable database to store the data we crawled. Currently commonly used databases include MySQL, PostgreSQL, MongoDB and many other options.

These databases each have their own advantages and disadvantages, so you can choose according to your own needs. For example, when the amount of data is small, it is more convenient to use the MySQL database, and when massive data storage is required, MongoDB's document database is more suitable.

3. Configure database connection information

Before the specific operation, we need to configure the database connection information. For example, taking the MySQL database as an example, you can use the pymysql library in Python to connect.

In Scrapy, we usually configure it in settings.py:

MYSQL_HOST = 'localhost'
MYSQL_PORT = 3306
MYSQL_USER = 'root'
MYSQL_PASSWORD = '123456'
MYSQL_DBNAME = 'scrapy_demo'

In the above configuration, we configure the host name, port number, user name, and password where the MySQL database is located and database name. These information need to be modified according to the actual situation.

4. Writing the data storage Pipeline

In Scrapy, the data storage Pipeline is the key to realizing data storage. We need to write a Pipeline class and then set it in the Scrapy configuration file to store data.

Taking storage to MySQL as an example, we can write a MySQLPipeline class as follows:

import pymysql

class MySQLPipeline(object):

    def open_spider(self, spider):
        self.conn = pymysql.connect(host=spider.settings.get('MYSQL_HOST'),
                                    port=spider.settings.get('MYSQL_PORT'),
                                    user=spider.settings.get('MYSQL_USER'),
                                    password=spider.settings.get('MYSQL_PASSWORD'),
                                    db=spider.settings.get('MYSQL_DBNAME'))
        self.cur = self.conn.cursor()

    def close_spider(self, spider):
        self.conn.close()

    def process_item(self, item, spider):
        sql = 'INSERT INTO articles(title, url, content) VALUES(%s, %s, %s)'
        self.cur.execute(sql, (item['title'], item['url'], item['content']))
        self.conn.commit()

        return item

In the above code, we define a MySQLPipeline class to implement docking with the MySQL database, and Three methods open_spider, close_spider and process_item are defined.

Among them, the open_spider method is called when the entire crawler starts running to initialize the database connection; the close_spider method is called when the crawler ends and is used to close the database connection. Process_item is the method called every time the data is crawled to store the data in the database.

5. Enable Pipeline

After completing the writing of Pipeline, we also need to enable it in Scrapy's configuration file settings.py. Just add the Pipeline class to the ITEM_PIPELINES variable, as shown below:

ITEM_PIPELINES = {
    'myproject.pipelines.MySQLPipeline': 300,
}

In the above code, we added the MySQLPipeline class to the ITEM_PIPELINES variable and set the priority to 300, indicating that the Item is being processed , the Pipeline class will be the third one to be called.

6. Testing and Operation

After completing all configurations, we can run the Scrapy crawler and store the captured data in the MySQL database. The specific steps and commands are as follows:

1. Enter the directory where the Scrapy project is located and run the following command to create a Scrapy project:

scrapy startproject myproject

2. Create a Spider to test the data storage function of the Scrapy framework , and store the crawled data into the database. Run the following command in the myproject directory:

scrapy genspider test_spider baidu.com

The above command will generate a Spider named test_spider to crawl Baidu.

3. Write the Spider code. In the spiders directory of the test_sprider directory, open test_sprider.py and write the crawler code:

import scrapy
from myproject.items import ArticleItem

class TestSpider(scrapy.Spider):
    name = "test"
    allowed_domains = ["baidu.com"]
    start_urls = [
        "https://www.baidu.com",
    ]

    def parse(self, response):
        item = ArticleItem()
        item['title'] = 'MySQL Pipeline测试'
        item['url'] = response.url
        item['content'] = 'Scrapy框架与MySQL数据库整合测试'
        yield item

In the above code, we define a TestSpider class, inherited from Scrapy The built-in Spider class is used to handle crawler logic. In the parse method, we construct an Item object and set the three keywords 'content', 'url' and 'title'.

4. Create an items file in the myproject directory to define the data model:

import scrapy

class ArticleItem(scrapy.Item):
    title = scrapy.Field()
    url = scrapy.Field()
    content = scrapy.Field()

In the above code, we define an ArticleItem class to save the crawled articles. data.

5. Test code:

In the test_spider directory, run the following command to test your code:

scrapy crawl test

After executing the above command, Scrapy will start the TestSpider crawler , and store the data captured from Baidu homepage in the MySQL database.

7. Summary

This article briefly introduces how the Scrapy framework integrates with the database and implements dynamic data storage. I hope this article can help readers in need, and I also hope that readers can develop according to their actual needs to achieve more efficient and faster dynamic data storage functions.

The above is the detailed content of Scrapy framework and database integration: how to implement dynamic data storage?. For more information, please follow other related articles on the PHP Chinese website!