In [1]: hxs.select('//title') Out[1]: [] In [2]: hxs.select('//title').extract() Out[2]: [u'Open Directory - Computers: Programming: Languages: Python: Books'] In [3]: hxs.select('//title/text()') Out[3]: [] In [4]: hxs.select('//title/text()').extract() Out[4]: [u'Open Directory - Computers: Programming: Languages: Python: Books'] In [5]: hxs.select('//title/text()').re('(\w+):') Out[5]: [u'Computers', u'Programming', u'Languages', u'Python']
Copy after login
提取数据Extracting the data
现在我们开始尝试在这几个页面里提取真正的信息。
你能够在控制台里面输入response.body,检查源代码里面的XPaths是否与预期相同。然而,检查原始的HTML代码是一件非常枯燥乏味的事情。假如你想让你的工作变的简单,你使用Firefox扩展的插件例如Firebug来做这项任务。更多关于介绍信息请看Using Firebug for scraping和Using Firefox for scraping。
sites = hxs.select('//ul/li') for site in sites: title = site.select('a/text()').extract() link = site.select('a/@href').extract() desc = site.select('text()').extract() print title, link, desc Note
Copy after login
如果想了解更多的嵌套选择器,可以参考Nesting selectors和Working with relative XPaths相关的Selectors文档 将代码添加到我们spider中:
from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector class DmozSpider(BaseSpider): name = "dmoz" allowed_domains = ["dmoz.org"] start_urls = [ "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/", "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/" ] def parse(self, response): hxs = HtmlXPathSelector(response) sites = hxs.select('//ul/li') for site in sites: title = site.select('a/text()').extract() link = site.select('a/@href').extract() desc = site.select('text()').extract() print title, link, desc
[dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> {'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n], 'link': [u'http://gnosis.cx/TPiP/'], 'title': [u'Text Processing in Python']} [dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> {'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'title': [u'XML Processing with Python']}
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
