python - Scrapy.FormRequest写对了吗?帮忙看看
大家讲道理
大家讲道理 2017-04-17 17:42:35
0
1
304
  • 从m.zhihu.com/topics找到所有的话题的内容,点击“更多”,发现请求的是'https://m.zhihu.com/node/TopicsPlazzaListV2', 并且发送的FormData为:

    def get_topic_url(self, response):
        topics = response.css('.item .blk > a[target=_blank]::attr(href)').extract()
        _xsrf = response.css('input[name="_xsrf"]::attr(value)').extract()[0]
        for topic in topics:
            print topic
        data = response.css('.zh-general-list::attr(data-init)').extract()
        import json
        param = json.loads(data[0])
        topic_id = param['params']['topic_id']
        hash_id = param['params']['hash_id']
        offset = param['params']['offset']
        yield scrapy.FormRequest(
                url="https://m.zhihu.com/node/TopicsPlazzaListV2",
                headers=headers,
                formdata={
                    "method":"next",
                    "params":{
                        "topic_id":topic_id,
                        "offset":offset,
                        "hash_id":hash_id,
                    },
                "_xsrf":_xsrf,
                },
                meta={
                    "proxy": proxy,
                    "cookiejar": response.meta["cookiejar"],
                },
                callback=self.get_topic_url,
        )

但是返回的是400代码,是不是代码哪里写错了?请指教

2016-05-08 10:43:52 [scrapy] DEBUG: Retrying <POST https://m.zhihu.com/node/TopicsPlazzaListV2> (failed 1 times): 400 Bad Request
2016-05-08 10:43:53 [scrapy] DEBUG: Retrying <POST https://m.zhihu.com/node/TopicsPlazzaListV2> (failed 2 times): 400 Bad Request
2016-05-08 10:43:53 [scrapy] DEBUG: Gave up retrying <POST https://m.zhihu.com/node/TopicsPlazzaListV2> (failed 3 times): 400 Bad Request
2016-05-08 10:43:53 [scrapy] DEBUG: Crawled (400) <POST https://m.zhihu.com/node/TopicsPlazzaListV2> (referer: https://m.zhihu.com/topics)
2016-05-08 10:43:53 [scrapy] DEBUG: Ignoring response <400 https://m.zhihu.com/node/TopicsPlazzaListV2>: HTTP status code is not handled or not allowed
大家讲道理
大家讲道理

光阴似箭催人老,日月如移越少年。

reply all(1)
阿神

Try setting the header to the header of the mobile browser.

Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template
About us Disclaimer Sitemap
php.cn:Public welfare online PHP training,Help PHP learners grow quickly!