Write a simple web crawler in Python to capture videos-Python Tutorial-php.cn

From the comments on the previous article, it seems that many children's shoes pay more attention to the source code of the crawler. All this article has made a very detailed record of using Python to write a simple web crawler to capture video download resources. Almost every step is introduced to everyone. I hope it can be helpful to everyone.

This is my first time coming into contact with crawlers. It was in May of this year that I wrote a blog search engine. The crawler used was quite intelligent, at least much higher than the crawler used by the website Here Comes the Movie!

Back to the topic of writing crawlers in Python.

Python has always been the main scripting language I use, bar none. Python's language is simple and flexible, and its standard library is powerful. It can be used as a calculator, text encoding conversion, image processing, batch downloading, batch text processing, etc. In short, I like it very much, and the more I use it, the better I get at it. I don’t tell most people about such a useful tool. . .

Because of its powerful string processing capabilities and the existence of modules such as urllib2, cookielib, re, and threading, it is simply easy to write a crawler in Python. How simple can it be? I told a classmate at that time that the total number of lines of script code for the several crawlers and scattered scripts I used to organize the movie did not exceed 1,000, and the website "Come to Write a Movie" only had about 150 lines. code. Because the code of the crawler is on another 64-bit Black Apple, I will not list it. I will only list the code of the website on the VPS. The code of the tornadoweb framework is

[xiaoxia@307232 movie_site]$ wc -l *.py template/*
  156 msite.py
   92 template/base.html
   79 template/category.html
   94 template/id.html
   47 template/index.html
   77 template/search.html

Copy after login

. Let’s directly show the code of the crawler. Write a process. The following content is for communication and learning purposes only and has no other meaning.

Take the latest video download resource of a certain bay as an example. Its URL is

http://piratebay.se/browse/200

because this web page There are a lot of advertisements, so I will only post part of the text:

For a python crawler, download the source code of this page, one line of code is enough. The urllib2 library is used here.

>>> import urllib2
>>> html = urllib2.urlopen(&#39;http://某piratebay.se/browse/200&#39;).read()
>>> print &#39;size is&#39;, len(html)
size is 52977

Copy after login

Of course, you can also use the system function in the os module to call the wget command to download web content. It is very convenient for students who have mastered the wget or curl tool.

Use Firebug to observe the structure of the web page, and you can know that the html in the main text is a table. Each resource is a tr tag.

For each resource, the information that needs to be extracted is:

1, video classification
2, resource name
3, resource Link
4, resource size
5, upload time

is enough, if necessary, you can increase it.

First extract a piece of code in the tr tag to observe.

<tr>
  <td class="vertTh">
   <center>
    <a href="/browse/200" title="此目录中更多">视频</a><br />
    (<a href="/browse/205" title="此目录中更多">电视</a>)
   </center>
  </td>
  <td>
<p class="detName">   <a href="/torrent/7782194/The_Walking_Dead_Season_3_Episodes_1-3_HDTV-x264" class="detLink" title="细节 The Walking Dead Season 3 Episodes 1-3 HDTV-x264">The Walking Dead Season 3 Episodes 1-3 HDTV-x264</a>
</p>
<a href="magnet:?xt=urn:btih:4f63d58e51c1a4a997c6f099b2b529bdbba72741&dn=The+Walking+Dead+Season+3+Episodes+1-3+HDTV-x264&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A6969&tr=udp%3A%2F%2Ftracker.ccc.de%3A80" title="Download this torrent using magnet"><img src="//static.某piratebay.se/img/icon-magnet.gif" alt="Magnet link" /></a>   <a href="//torrents.某piratebay.se/7782194/The_Walking_Dead_Season_3_Episodes_1-3_HDTV-x264.7782194.TPB.torrent" title="下载种子"><img src="//static.某piratebay.se/img/dl.gif" class="dl" alt="下载" /></a><img  src="//static.某piratebay.se/img/11x11p.png" / alt="Write a simple web crawler in Python to capture videos" ><img  src="//static.某piratebay.se/img/11x11p.png" / alt="Write a simple web crawler in Python to capture videos" >
   <font class="detDesc">已上传 <b>3 分钟前</b>, 大小 2 GiB, 上传者 <a class="detDesc" href="/user/paridha/" title="浏览 paridha">paridha</a></font>
  </td>
  <td align="right">0</td>
  <td align="right">0</td>
 </tr>

Copy after login

The following uses regular expressions to extract the content in the html code. Students who don’t know about regular expressions can go to http://docs.python.org/2/library/re.html to learn more.

There is a reason why you should use regular expressions instead of some other tools for parsing HTML or DOM trees. I tried using BeautifulSoup3 to extract content before, but later found that the speed was really slow. Being able to process 100 pieces of content in one second was already the limit of my computer. . . But by changing the regular expression and processing the content after compilation, the speed directly kills it!

How do I write my regular expression to extract so much content?

According to my past experience, ".*?" or ". ?" is very useful. But we should also pay attention to some small problems. You will know it when you actually use it

For the above tr tag code, I first need to make my expression match the symbol

indicates the beginning of the content. Of course, it can also be anything else, as long as you don’t miss the required content. Then the content I want to match is the following, to get the video classification.

(TV)

Then I have to match the resource link,

...

and then to other resource information,

font class="detDesc">Uploaded3 minutes ago, size 2 GiB, uploaded by

Last match

Done!

Of course, the final match does not need to be expressed in the regular expression. As long as the starting position is positioned correctly, the position where the information is obtained later will also be correct.

Friends who know more about regular expressions may know how to write them. Let me show you the expression processing process I wrote,

就这么简单，结果出来了，自我感觉挺欢喜的。

当然，这样设计的爬虫是有针对性的，定向爬取某一个站点的内容。也没有任何一个爬虫不会对收集到的链接进行筛选。通常可以使用BFS（宽度优先搜索算法）来爬取一个网站的所有页面链接。

完整的Python爬虫代码，爬取某湾最新的10页视频资源：

# coding: utf8
import urllib2
import re
import pymongo
db = pymongo.Connection().test
url = &#39;http://某piratebay.se/browse/200/%d/3&#39;
find_re = re.compile(r&#39;<tr>.+?\(.+?">(.+?)</a>.+?class="detLink".+?">(.+?)</a>.+?<a href="(magnet:.+?)" .+?已上传 <b>(.+?)</b>, 大小 (.+?),&#39;, re.DOTALL)
# 定向爬去10页最新的视频资源
for i in range(0, 10):
    u = url % (i)
    # 下载数据
    html = urllib2.urlopen(u).read()
    # 找到资源信息
    for x in find_re.findall(html):
        values = dict(
            category = x[0],
            name = x[1],
            magnet = x[2],
            time = x[3],
            size = x[4]
        )
        # 保存到数据库
        db.priate.save(values)
print &#39;Done!&#39;

Copy after login

以上代码仅供思路展示，实际运行使用到mongodb数据库，同时可能因为无法访问某湾网站而无法得到正常结果。

所以说，电影来了网站用到的爬虫不难写，难的是获得数据后如何整理获取有用信息。例如，如何匹配一个影片信息跟一个资源，如何在影片信息库和视频链接之间建立关联，这些都需要不断尝试各种方法，最后选出比较靠谱的。

曾有某同学发邮件想花钱也要得到我的爬虫的源代码。
要是我真的给了，我的爬虫就几百来行代码，一张A4纸，他不会说，坑爹啊！！！……

都说现在是信息爆炸的时代，所以比的还是谁的数据挖掘能力强