Home > Backend Development > PHP Tutorial > Problem with crawler obtaining AJax data

Problem with crawler obtaining AJax data

WBOY
Release: 2016-07-06 13:52:21
Original
1782 people have browsed it

I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
Problem with crawler obtaining AJax data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Problem with crawler obtaining AJax data

Please tell me how to get json data

Reply content:

I want to write a crawler that crawls pictures on the petal network. I encountered a problem with Ajax getting data
Problem with crawler obtaining AJax data
The requested URL in the picture, the text type of the Response is json
, but when I access this url alone, it is html
Problem with crawler obtaining AJax data

Please tell me how to get json data

This is controlled by header
Problem with crawler obtaining AJax data

Just add the corresponding request header. Attached is my code

<code>#coding=utf-8

import requests as RQ

hburl = "http://huaban.com/boards/28195582/?ipva9fpx&max=759391397&limit=20&wfl=1"
heads = {
'Host':"huaban.com",
'Referer':"{http://huaban.com/boards/28195582/",
'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36",
'X-Request':"JSON",
'X-Requested-With':"XMLHttpRequest"
}


print RQ.get(hburl, headers=heads).content</code>
Copy after login

Referer is a common header used to prevent web pages from being obtained outside this site. When doing crawlers, be sure to ensure that the Referer is set

Related labels:
source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template