How to crawl the content in js using python-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

How to crawl the content in js using python

零到壹度

Apr 10, 2018 am 09:54 AM

javascriptpythonCrawling

The content of this article is to share with you 3. How to use python to crawl the content in js. It has certain reference value. Friends in need can refer to it

1. When writing crawler software to obtain the required content, you may encounter that the required content is added by javascript and is empty when obtained. For example, when we obtain Sina News The number of comments cannot be obtained using ordinary methods

##Normal acquisition code example:

import requests
from bs4 import BeautifulSoup

res = requests.get(&#39;http://news.sina.com.cn/c/nd/2017-06-12/doc-ifyfzhac1650783.shtml&#39;)
res.encoding = &#39;utf-8&#39;
soup = BeautifulSoup(res.text,&#39;html.parser&#39;)
#取评论数
commentCount = soup.select_one(&#39;#commentCount1&#39;)
print(commentCount.text)

The result obtained at this time is empty. This is because the content is stored in a js file.

So we need to find the js that stores the comment content. After searching, we found that It is stored in js

Put the corresponding content into the json data viewer and we find that the total number of comments and the content of the comments are both there A json format is stored in the js file

#We can see the js in the message header File access path and request method

Code example

import json
comments = requests.get(&#39;http://comment5.news.sina.com.cn/page/info?version=1&format=js&channel=gn&newsid=comos-fyfzhac1650783&#39;)
comments.encoding = &#39;utf-8&#39;
print(comments)
jd = json.loads(comments.text.strip(&#39;var data=&#39;)) #移除改var data=将其变为json数据
print(jd[&#39;result&#39;][&#39;count&#39;][&#39;total&#39;])

Comment: explained here Why do you need to remove var data=? Because the string prefix includes var data= when getting it, which does not conform to the json data format, so it needs to be removed from the request content during conversion

Why use jd[

'result'

]['count']['total'] when getting the total number of comments

The above is the detailed content of How to crawl the content in js using python. For more information, please follow other related articles on the PHP Chinese website!

Statement

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

The Main Purpose of Python: Flexibility and Ease of UseApr 17, 2025 am 12:14 AM

Python's flexibility is reflected in multi-paradigm support and dynamic type systems, while ease of use comes from a simple syntax and rich standard library. 1. Flexibility: Supports object-oriented, functional and procedural programming, and dynamic type systems improve development efficiency. 2. Ease of use: The grammar is close to natural language, the standard library covers a wide range of functions, and simplifies the development process.

Python: The Power of Versatile ProgrammingApr 17, 2025 am 12:09 AM

Python is highly favored for its simplicity and power, suitable for all needs from beginners to advanced developers. Its versatility is reflected in: 1) Easy to learn and use, simple syntax; 2) Rich libraries and frameworks, such as NumPy, Pandas, etc.; 3) Cross-platform support, which can be run on a variety of operating systems; 4) Suitable for scripting and automation tasks to improve work efficiency.

Learning Python in 2 Hours a Day: A Practical GuideApr 17, 2025 am 12:05 AM

Yes, learn Python in two hours a day. 1. Develop a reasonable study plan, 2. Select the right learning resources, 3. Consolidate the knowledge learned through practice. These steps can help you master Python in a short time.

Python vs. C : Pros and Cons for DevelopersApr 17, 2025 am 12:04 AM

Python is suitable for rapid development and data processing, while C is suitable for high performance and underlying control. 1) Python is easy to use, with concise syntax, and is suitable for data science and web development. 2) C has high performance and accurate control, and is often used in gaming and system programming.

Python: Time Commitment and Learning PaceApr 17, 2025 am 12:03 AM

The time required to learn Python varies from person to person, mainly influenced by previous programming experience, learning motivation, learning resources and methods, and learning rhythm. Set realistic learning goals and learn best through practical projects.

Python: Automation, Scripting, and Task ManagementApr 16, 2025 am 12:14 AM

Python excels in automation, scripting, and task management. 1) Automation: File backup is realized through standard libraries such as os and shutil. 2) Script writing: Use the psutil library to monitor system resources. 3) Task management: Use the schedule library to schedule tasks. Python's ease of use and rich library support makes it the preferred tool in these areas.

Python and Time: Making the Most of Your Study TimeApr 14, 2025 am 12:02 AM

To maximize the efficiency of learning Python in a limited time, you can use Python's datetime, time, and schedule modules. 1. The datetime module is used to record and plan learning time. 2. The time module helps to set study and rest time. 3. The schedule module automatically arranges weekly learning tasks.

Python: Games, GUIs, and MoreApr 13, 2025 am 12:14 AM

Python excels in gaming and GUI development. 1) Game development uses Pygame, providing drawing, audio and other functions, which are suitable for creating 2D games. 2) GUI development can choose Tkinter or PyQt. Tkinter is simple and easy to use, PyQt has rich functions and is suitable for professional development.

See all articles