[Python] Web Crawler (4): Introduction and practical applications of Opener and Handler

Before starting the following content, let’s first explain the two methods in urllib2: info and geturl

The response object response (or HTTPError instance) returned by urlopen has two very useful methods info () and geturl()

1.geturl():

This returns the real URL obtained. This is very useful because urlopen (or used by the opener object) may There are redirects. The obtained URL may be different from the request URL.

Take a hyperlink in Renren as an example,

We build a urllib2_test10.py to compare the original URL and the redirected link:

from urllib2 import Request, urlopen, URLError, HTTPError old_url = 'http://rrurl.cn/b1UZuP' req = Request(old_url) response = urlopen(req) print 'Old url :' + old_url print 'Real url :' + response.geturl()

After running, you can see the URL pointed to by the real link:

2.info():

This returns a dictionary of objects Object, this dictionary describes the obtained page situation. Usually specific headers sent by the server. Currently an instance of httplib.HTTPMessage.

Classic headers include "Content-length", "Content-type", and other content.

We build a urllib2_test11.py to test the application of info:

from urllib2 import Request, urlopen, URLError, HTTPError old_url = 'http://www.baidu.com' req = Request(old_url) response = urlopen(req) print 'Info():' print response.info()

The running results are as follows, you can see the relevant information of the page :

Let’s talk about two important concepts in urllib2: Openers and Handlers.

1.Openers:

When you get a URL you use an opener (an instance of urllib2.OpenerDirector).

Normally, we use the default opener: through urlopen.

But you can create personalized openers.

2.Handles:

Openers use processor handlers, and all "heavy" work is handled by the handlers.

Each handler knows how to open URLs over a specific protocol, or how to handle various aspects of opening a URL.

For example, HTTP redirection or HTTP cookies.

You will want to create an opener if you want to fetch URLs with a specific handler, for example to get an opener that can handle cookies, or to get an opener that does not redirect.

To create an opener, instantiate an OpenerDirector,

and then call .add_handler(some_handler_instance).

Similarly, you can use build_opener, which is a more convenient function for creating opener objects. It only requires one function call.
build_opener adds several processors by default, but provides a quick way to add or update the default processors.

Other handlers You may want to handle proxies, authentication, and other common but somewhat special cases.

install_opener is used to create a (global) default opener. This means that calling urlopen will use the opener you installed.

The Opener object has an open method.

This method can be used directly to obtain urls like the urlopen function: it is usually not necessary to call install_opener, except for convenience.

After talking about the above two contents, let’s take a look at the basic authentication content. The Opener and Handler mentioned above will be used here.

Basic Authentication Basic Authentication

To demonstrate creating and installing a handler, we will use HTTPBasicAuthHandler.

When basic verification is required, the server sends a header (401 error code) to request verification. This specifies the scheme and a 'realm' and looks like this: Www-authenticate: SCHEME realm="REALM".

For example
Www-authenticate: Basic realm="cPanel Users"

The client must use a new request and include the correct name and password in the request header.

This is "basic authentication". In order to simplify this process, we can create an instance of HTTPBasicAuthHandler and let opener use this handler.

HTTPBasicAuthHandler uses a password management object to handle URLs and realms to map usernames and passwords.

If you know what realm (in the header sent from the server) is, you can use HTTPPasswordMgr.

Usually people don’t care what realm is. In that case, the convenient HTTPPasswordMgrWithDefaultRealm can be used.

This will specify a default username and password for your URL.

This will be provided when you provide an other combination for a specific realm.

We indicate this situation by specifying None for the realm parameter provided to add_password.

The highest-level URL is the first URL that requires authentication. Deeper URLs you pass to .add_password() will be equally suitable.

Having said so much nonsense, let’s use an example to demonstrate what is said above.

We build a urllib2_test12.py to test the info application:

# -*- coding: utf-8 -*- import urllib2 # 创建一个密码管理者 password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm() # 添加用户名和密码 top_level_url = "http://example.com/foo/" # 如果知道 realm, 我们可以使用他代替 ``None``. # password_mgr.add_password(None, top_level_url, username, password) password_mgr.add_password(None, top_level_url,'why', '1223') # 创建了一个新的handler handler = urllib2.HTTPBasicAuthHandler(password_mgr) # 创建 "opener" (OpenerDirector 实例) opener = urllib2.build_opener(handler) a_url = 'http://www.baidu.com/' # 使用 opener 获取一个URL opener.open(a_url) # 安装 opener. # 现在所有调用 urllib2.urlopen 将用我们的 opener. urllib2.install_opener(opener)

Note: In the above example, we only provide our HHTTPasicAuthHandler to build_opener.

The default openers have normal handlers: ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, HTTPErrorProcessor.

The top_level_url in the code can actually be a complete URL (including "http:", as well as the host name and optional port number).

For example: http://example.com/.

can also be an "authority" (i.e. hostname and optionally port number).

For example: "example.com" or "example.com:8080".

The latter contains the port number.

The above is the content of [Python] Web Crawler (4): Introduction and example applications of Opener and Handler. For more related content, please pay attention to the PHP Chinese website (m.sbmmt.com)!