How to crawl the content in js using python

The content of this article is to share with you 3. How to use python to crawl the content in js. It has certain reference value. Friends in need can refer to it

1. When writing crawler software to obtain the required content, you may encounter that the required content is added by javascript and is empty when obtained. For example, when we obtain Sina News The number of comments cannot be obtained using ordinary methods

import requests
from bs4 import BeautifulSoup

res = requests.get('http://news.sina.com.cn/c/nd/2017-06-12/doc-ifyfzhac1650783.shtml')
res.encoding = 'utf-8'
soup = BeautifulSoup(res.text,'html.parser')
#取评论数
commentCount = soup.select_one('#commentCount1')
print(commentCount.text)

Copy after login

The result obtained at this time is empty. This is because the content is stored in a js file.

So we need to find the js that stores the comment content. After searching, we found that It is stored in js

Put the corresponding content into the json data viewer and we find that the total number of comments and the content of the comments are both there A json format is stored in the js file

import json
comments = requests.get('http://comment5.news.sina.com.cn/page/info?version=1&format=js&channel=gn&newsid=comos-fyfzhac1650783')
comments.encoding = 'utf-8'
print(comments)
jd = json.loads(comments.text.strip('var data=')) #移除改var data=将其变为json数据
print(jd['result']['count']['total'])

Comment: explained here Why do you need to remove var data=? Because the string prefix includes var data= when getting it, which does not conform to the json data format, so it needs to be removed from the request content during conversion

Why use jd[

]['count']['total'] when getting the total number of comments

The above is the detailed content of How to crawl the content in js using python. For more information, please follow other related articles on the PHP Chinese website!