requests - 关于python模拟登陆抓取网站信息
怪我咯
怪我咯 2017-04-17 17:53:33
0
1
1208

我想用requests实现在www.lu.com模拟登陆后抓取一些产品信息,通过chrome的抓包工具我抓到了登陆时的form data和header(变量header),以及访问需要抓取的页面时的header(变量listheader),但运行完程序后我输入r.url发现返回的地址是https://list.lu.com/notFound.html,而不是我指定的具体页面,请问是怎么回事呢?

import requests,bs4
    
    s = requests.session()
    logindata={'userName':'XXXXXXX','password':'XXXXXXXX','isTrust':'Y','loginFlag':'1','validNum':'','loginagree':'on'}
    listheader={'Accept-Encoding':'gzip, deflate, sdch, br','Host':'list.lu.com','Upgrade-Insecure-Requests':'1','User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36'}
    header={'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36','X-Requested-With':'XMLHttpRequest','Referer':'https://user.lu.com/user/login?returnPostURL=https%3A%2F%2Fwww.lu.com%2F'}
    res=s.post('https://user.lu.com/user/login',logindata,headers=header)
    r=s.get('https://list.lu.com/list/productDetail/transferFa?riskFlag=false&productId=31398999',headers=listheader)

怪我咯
怪我咯

走同样的路,发现不同的人生

membalas semua(1)
大家讲道理
# coding=utf-8

__author__ = 'Rich'

import requests

session = requests.session()
session.headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36'
}
data = {
    'userName': 'XXXXXXX',
    'password': 'XXXXXXXX',
    'isTrust': 'Y',
    'loginFlag': '1',
    'validNum': '',
    'loginagree': 'on'
}

session.post('https://user.lu.com/user/login', data=data)

r = session.get('https://list.lu.com/list/productDetail/transferFa?riskFlag=false&productId=31398999')
print r.text
Muat turun terkini
Lagi>
kesan web
Kod sumber laman web
Bahan laman web
Templat hujung hadapan