python - How to combine data from two different pages in scrapy and store them together

Question

1 For example, there is a blog list page a, to get the article list 2 and a blog content details page b, to get the article text 3 How to get the article + content to the database in sequence? 4 The current problem is mainly that I don’t know how to combine the data from two different pages

巴扎黑 · Answer

You can use request.meta to store additional information, such as


def parse_x1(self, response):
    extra = {}
    ...
    req = scrapy.Request(url2)
    req.callback = self.parse_x2
    ...
    req.meta['extra'] = extra 
    yield req
 
def parse_x2(self, response):
    item = {}
    extra = response.meta['extra']
    item.update(extra)
    ...
    
    yield item

Php8, I'm coming too

Learn website layout in 30 minutes

Shangguan Oracle Beginner to Proficient Video Tutorial

Your first line of UNI-APP code

Flutter from scratch to app launch

Brother Lian New Linux Video Tutorial

AXURE 9 Video Tutorial (Suitable for Product Manager Interactive Product Design UI)

Zero Basic Proficiency PS Video Tutorial

16 day UI video tutorial to get you started

PS Techniques and Slicing Techniques Video Tutorial

Alibaba Cloud Environment Construction and Project Launch Video Tutorial

Overview of Computer Networks - Basic Knowledge that Programmers Must Master

Essential Tutorial for Programmers - HTTP Protocol Explanation

Websocket Video Tutorial