Share an example of setting up python scrapy ip proxy

PHP中文网

Release： 2017-06-20 16:43:23

Original

2791 people have browsed it

Create a python directory at the same level as spider in the scrapy project and add a py file below with the content

# encoding: utf-8
import base64
proxyServer = 代理服务器地址 ##我的是‘:9010’

# 代理隧道验证信息  这个是在那个网站上申请的
proxyUser = 用户名
proxyPass = 密码
proxyAuth = "Basic " + base64.b64encode(proxyUser + ":" + proxyPass)

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta["proxy"] = proxyServer
        request.headers["Proxy-Authorization"] = proxyAuth

在setting文件中添加

Copy after login

DOWNLOADER_MIDDLEWARES = {
    '项目名.spider同级文件名.文件名.ProxyMiddleware': 543,
}

代码地址

Copy after login

<br>

Copy after login

The above is the detailed content of Share an example of setting up python scrapy ip proxy. For more information, please follow other related articles on the PHP Chinese website!

Related labels：

python scrapy acting set up

source：php.cn

Previous article：Detailed explanation of examples of Baidu API calls in Python Next article：How to use sublime text in python

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn