Python crawling article example tutorial

This article mainly introduces you to the relevant information of using python to crawl prose network articles. The introduction in the article is very detailed and has certain reference and learning value for everyone. Friends who need it can take a look below.

This article mainly introduces to you the relevant content about python crawling prose network articles. It is shared for your reference and study. Let’s take a look at the detailed introduction:

The rendering is as follows:

Configure python 2.7

 bs4

 requests

Installation using pipsudo pip install bs4

##

sudo pip install requests

Briefly explain the use of bs4 because it is crawling web pages, so I will introduce it. find and find_all

The difference between find and find_all is that they return different things. find returns the first matched tag and the content in the tag

find_all returns a list

For example, we write a test.html to test the difference between find and find_all.

The content is:

abc
three athree athree a
four
four p
four p
four p
 a

Then the code of test.py is:

from bs4 import BeautifulSoup
import lxml

if __name__=='__main__':
 s = BeautifulSoup(open('test.html'),'lxml')
 print s.prettify()
 print "------------------------------"
 print s.find('p')
 print s.find_all('p')
 print "------------------------------"
 print s.find('p',id='one')
 print s.find_all('p',id='one')
 print "------------------------------"
 print s.find('p',id="two")
 print s.find_all('p',id="two")
 print "------------------------------"
 print s.find('p',id="three")
 print s.find_all('p',id="three")
 print "------------------------------"
 print s.find('p',id="four")
 print s.find_all('p',id="four")
 print "------------------------------"

After running, we can see the result. When getting the specified tag, there is not much difference between the two. When getting a group of tags, the difference between the two will be displayed.

So when we use it, we must pay attention to what we want, otherwise an error will occur

The next step is to obtain web page information through requests. I don’t quite understand why others write heard. With other things

I directly access the web page, obtain several classified secondary web pages of Prose Network through the get method, and then crawl all the web pages through a group test

def get_html():
 url = "https://www.sanwen.net/"
 two_html = ['sanwen','shige','zawen','suibi','rizhi','novel']
 for doc in two_html:
 i=1
  if doc=='sanwen':
  print "running sanwen -----------------------------"
  if doc=='shige':
  print "running shige ------------------------------"
  if doc=='zawen':
  print 'running zawen -------------------------------'
  if doc=='suibi':
  print 'running suibi -------------------------------'
  if doc=='rizhi':
  print 'running ruzhi -------------------------------'
  if doc=='nove':
  print 'running xiaoxiaoshuo -------------------------'
 while(i<10):
 par = {'p':i}
 res = requests.get(url+doc+'/',params=par)
 if res.status_code==200:
  soup(res.text)
  i+=i

In this part of the code, I did not process the

res.status_code that is not 200. The resulting problem is that the error will not be displayed, and the crawled content will have lost. Then I analyzed the web page of prose website and found that it was www.sanwen.net/rizhi/&p=1

The maximum value of p is 10. I don’t understand this. The last time I crawled the disk, it was 100 pages. Forget it. Forget it and analyze it later. Then get the content of each page through the get method.

After getting the content of each page, we analyze the author and title. The code is like this

def soup(html_text):
 s = BeautifulSoup(html_text,'lxml')
 link = s.find('p',class_='categorylist').find_all('li')
 for i in link:
 if i!=s.find('li',class_='page'):
 title = i.find_all('a')[1]
 author = i.find_all('a')[2].text
 url = title.attrs['href']
 sign = re.compile(r'(//)|/')
 match = sign.search(title.text)
 file_name = title.text
 if match:
 file_name = sign.sub('a',str(title.text))

There is something wrong when getting the title. Please tell me. Guys, why do you add slashes in the title when writing prose? Not only one but also two. This problem directly caused an error in the file name when I wrote the file later, so I wrote a regular expression and I changed it for you. Bar.

The last step is to get the prose content. Through the analysis of each page, we get the article address, and then get the content directly. I originally wanted to get it one by one by changing the web page address, which saves trouble.

def get_content(url):
 res = requests.get('https://www.sanwen.net'+url)
 if res.status_code==200:
 soup = BeautifulSoup(res.text,'lxml')
 contents = soup.find('p',class_='content').find_all('p')
 content = ''
 for i in contents:
 content+=i.text+'\n'
 return content

The last thing is to write the file and save it ok

 f = open(file_name+'.txt','w')

 print 'running w txt'+file_name+'.txt'
 f.write(title.text+'\n')
 f.write(author+'\n')
 content=get_content(url) 
 f.write(content)
 f.close()

Three functions get the prose from the prose network, but there are The problem, the problem is that I don’t know why some prose is lost. I can only get about 400 articles. This is much different from the articles on Prose.com, but it is indeed obtained page by page. I hope this problem will be solved. Please help me. Maybe we should make the web page inaccessible. Of course, I think it has something to do with the broken network in my dormitory

 f = open(file_name+'.txt','w')
 print 'running w txt'+file_name+'.txt'
 f.write(title.text+'\n')
 f.write(author+'\n')
 content=get_content(url) 
 f.write(content)
 f.close()

Almost forgot about the rendering

There may be a timeout phenomenon. I can only say that you must choose a good Internet connection when going to college!

The above is the detailed content of Python crawling article example tutorial. For more information, please follow other related articles on the PHP Chinese website!