In the process of writing a crawler in Python (crawling Wikipedia entries), the url["href"] appeared during the output of the iterator. I thought it should be a usage in the iterator, but I couldn't find it. Please help. This usage means, thank you
#coding:utf-8 import urllib import urllib2 import re from bs4 import BeautifulSoup resp = urllib2.urlopen("https://en.wikipedia.org/wiki/Main_Page").read() soup = BeautifulSoup(resp,"html.parser") listurl = soup.findAll('a',href=re.compile("^/wiki/")) for url in listurl: print url.get_text(),"------>","https://en.wikipedia.org"+url["href"]
The url["href"] in the last line has a truncation effect on the crawled data. Before adding it, the output is:
print url
Output: Disclaimers
After adding it, the output is: :
print url["href"]
Output:/wiki/Wikipedia:General_disclaimer
Solution, thank you
As long as the class implements the __getitem__ method, you can use square brackets to get the value.