python - Use scrapy framework to crawl Baidu pictures and get blocked

Question

The request address URL is the address of json obtained through firefox. It can be opened with a browser, but it was banned when crawling with scrapy. Please solve it. https://image.baidu.com/searc...

黄舟 · Answer

Try it at settings.py 将 ROBOTSTXT_OBEY = False.

某草草 · Answer

Try without adding hearingers

为情所困 · Answer

I agree with the upstairs, if there will still be a wall. The method of scrapy+selenium+phantomjs can be used.