Backend Development
Python Tutorial
Python implements a method to capture HTML web pages and save them as PDF files
Python implements a method to capture HTML web pages and save them as PDF files
This article mainly introduces the method of Python to capture HTML web pages and save them in the form of PDF files. It analyzes the installation of the PyPDF2 module and the related operating skills of Python to capture HTML pages and generate PDF files based on the PyPDF2 module in the form of examples. , Friends who need it can refer to
The example in this article describes how Python can capture HTML web pages and save them as PDF files. Share it with everyone for your reference, the details are as follows:
1. Preface
Today I will introduce how to capture HTML web pages and save them as PDF , without further ado, go directly to the tutorial.
2. Preparation
1. Installation and use of PyPDF2 (used to merge PDF):
PyPDF2 version: 1.25.1
Installation:
pip install PyPDF2
Usage example:
from PyPDF2 import PdfFileMerger
merger = PdfFileMerger()
input1 = open("hql_1_20.pdf", "rb")
input2 = open("hql_21_40.pdf", "rb")
merger.append(input1)
merger.append(input2)
# Write to an output PDF document
output = open("hql_all.pdf", "wb")
merger.write(output)2. requests and beautifulsoup are two major artifacts of crawlers, reuqests is used for network requests, and beautifulsoup is used to operate html data. With these two shuttles, work can be done quickly. We don’t need crawler frameworks like scrapy. Using it on such a small program is a bit overkill. In addition, since you are converting html files to pdf, you must also have corresponding library support. wkhtmltopdf is a very useful tool that can convert html to pdf suitable for multiple platforms. pdfkit is the Python package of wkhtmltopdf. First install the following dependency packages
pip install requests pip install beautifulsoup4 pip install pdfkit
3. Install wkhtmltopdf
Windows platform directly at http:// wkhtmltopdf.org/downloads.html Download the stable version of wkhtmltopdf and install it. After the installation is completed, add the execution path of the program to the system environment $PATH variable. Otherwise, pdfkit cannot find wkhtmltopdf and the error "No wkhtmltopdf executable found" will appear. Ubuntu and CentOS can be installed directly using the command line
$ sudo apt-get install wkhtmltopdf # ubuntu $ sudo yum intsall wkhtmltopdf # centos
3. Data preparation
1. Get the url of each article
def get_url_list():
"""
获取所有URL目录列表
:return:
"""
response = requests.get("http://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000")
soup = BeautifulSoup(response.content, "html.parser")
menu_tag = soup.find_all(class_="uk-nav uk-nav-side")[1]
urls = []
for li in menu_tag.find_all("li"):
url = "http://www.liaoxuefeng.com" + li.a.get('href')
urls.append(url)
return urls2. Save the HTML of each article using a template through the article url File
html template:
html_template = """
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
</head>
<body>
{content}
</body>
</html>
"""Save:
def parse_url_to_html(url, name):
"""
解析URL，返回HTML内容
:param url:解析的url
:param name: 保存的html文件名
:return: html
"""
try:
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
# 正文
body = soup.find_all(class_="x-wiki-content")[0]
# 标题
title = soup.find('h4').get_text()
# 标题加入到正文的最前面，居中显示
center_tag = soup.new_tag("center")
title_tag = soup.new_tag('h1')
title_tag.string = title
center_tag.insert(1, title_tag)
body.insert(1, center_tag)
html = str(body)
# body中的img标签的src相对路径的改成绝对路径
pattern = "(<img .*?src=\")(.*?)(\")"
def func(m):
if not m.group(3).startswith("http"):
rtn = m.group(1) + "http://www.liaoxuefeng.com" + m.group(2) + m.group(3)
return rtn
else:
return m.group(1)+m.group(2)+m.group(3)
html = re.compile(pattern).sub(func, html)
html = html_template.format(content=html)
html = html.encode("utf-8")
with open(name, 'wb') as f:
f.write(html)
return name
except Exception as e:
logging.error("解析错误", exc_info=True)3. Convert html to pdf
def save_pdf(htmls, file_name):
"""
把所有html文件保存到pdf文件
:param htmls: html文件列表
:param file_name: pdf文件名
:return:
"""
options = {
'page-size': 'Letter',
'margin-top': '0.75in',
'margin-right': '0.75in',
'margin-bottom': '0.75in',
'margin-left': '0.75in',
'encoding': "UTF-8",
'custom-header': [
('Accept-Encoding', 'gzip')
],
'cookie': [
('cookie-name1', 'cookie-value1'),
('cookie-name2', 'cookie-value2'),
],
'outline-depth': 10,
}
pdfkit.from_file(htmls, file_name, options=options)4. Merge the converted single PDFs into one PDF
merger = PdfFileMerger() for pdf in pdfs: merger.append(open(pdf,'rb')) print u"合并完成第"+str(i)+'个pdf'+pdf
Full source code:
# coding=utf-8
import os
import re
import time
import logging
import pdfkit
import requests
from bs4 import BeautifulSoup
from PyPDF2 import PdfFileMerger
html_template = """
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
</head>
<body>
{content}
</body>
</html>
"""
def parse_url_to_html(url, name):
"""
解析URL，返回HTML内容
:param url:解析的url
:param name: 保存的html文件名
:return: html
"""
try:
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
# 正文
body = soup.find_all(class_="x-wiki-content")[0]
# 标题
title = soup.find('h4').get_text()
# 标题加入到正文的最前面，居中显示
center_tag = soup.new_tag("center")
title_tag = soup.new_tag('h1')
title_tag.string = title
center_tag.insert(1, title_tag)
body.insert(1, center_tag)
html = str(body)
# body中的img标签的src相对路径的改成绝对路径
pattern = "(<img .*?src=\")(.*?)(\")"
def func(m):
if not m.group(3).startswith("http"):
rtn = m.group(1) + "http://www.liaoxuefeng.com" + m.group(2) + m.group(3)
return rtn
else:
return m.group(1)+m.group(2)+m.group(3)
html = re.compile(pattern).sub(func, html)
html = html_template.format(content=html)
html = html.encode("utf-8")
with open(name, 'wb') as f:
f.write(html)
return name
except Exception as e:
logging.error("解析错误", exc_info=True)
def get_url_list():
"""
获取所有URL目录列表
:return:
"""
response = requests.get("http://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000")
soup = BeautifulSoup(response.content, "html.parser")
menu_tag = soup.find_all(class_="uk-nav uk-nav-side")[1]
urls = []
for li in menu_tag.find_all("li"):
url = "http://www.liaoxuefeng.com" + li.a.get('href')
urls.append(url)
return urls
def save_pdf(htmls, file_name):
"""
把所有html文件保存到pdf文件
:param htmls: html文件列表
:param file_name: pdf文件名
:return:
"""
options = {
'page-size': 'Letter',
'margin-top': '0.75in',
'margin-right': '0.75in',
'margin-bottom': '0.75in',
'margin-left': '0.75in',
'encoding': "UTF-8",
'custom-header': [
('Accept-Encoding', 'gzip')
],
'cookie': [
('cookie-name1', 'cookie-value1'),
('cookie-name2', 'cookie-value2'),
],
'outline-depth': 10,
}
pdfkit.from_file(htmls, file_name, options=options)
def main():
start = time.time()
file_name = u"liaoxuefeng_Python3_tutorial"
urls = get_url_list()
for index, url in enumerate(urls):
parse_url_to_html(url, str(index) + ".html")
htmls =[]
pdfs =[]
for i in range(0,124):
htmls.append(str(i)+'.html')
pdfs.append(file_name+str(i)+'.pdf')
save_pdf(str(i)+'.html', file_name+str(i)+'.pdf')
print u"转换完成第"+str(i)+'个html'
merger = PdfFileMerger()
for pdf in pdfs:
merger.append(open(pdf,'rb'))
print u"合并完成第"+str(i)+'个pdf'+pdf
output = open(u"廖雪峰Python_all.pdf", "wb")
merger.write(output)
print u"输出PDF成功！"
for html in htmls:
os.remove(html)
print u"删除临时文件"+html
for pdf in pdfs:
os.remove(pdf)
print u"删除临时文件"+pdf
total_time = time.time() - start
print(u"总共耗时：%f 秒" % total_time)
if __name__ == '__main__':
main()Related recommendations:
Python implements simple crawler sharing to capture links on the page
Python implements crawling the website title information of Baidu search results page
The above is the detailed content of Python implements a method to capture HTML web pages and save them as PDF files. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1378
52
How to use Debian Apache logs to improve website performance
Apr 12, 2025 pm 11:36 PM
This article will explain how to improve website performance by analyzing Apache logs under the Debian system. 1. Log Analysis Basics Apache log records the detailed information of all HTTP requests, including IP address, timestamp, request URL, HTTP method and response code. In Debian systems, these logs are usually located in the /var/log/apache2/access.log and /var/log/apache2/error.log directories. Understanding the log structure is the first step in effective analysis. 2. Log analysis tool You can use a variety of tools to analyze Apache logs: Command line tools: grep, awk, sed and other command line tools.
Python: Games, GUIs, and More
Apr 13, 2025 am 12:14 AM
Python excels in gaming and GUI development. 1) Game development uses Pygame, providing drawing, audio and other functions, which are suitable for creating 2D games. 2) GUI development can choose Tkinter or PyQt. Tkinter is simple and easy to use, PyQt has rich functions and is suitable for professional development.
PHP and Python: Comparing Two Popular Programming Languages
Apr 14, 2025 am 12:13 AM
PHP and Python each have their own advantages, and choose according to project requirements. 1.PHP is suitable for web development, especially for rapid development and maintenance of websites. 2. Python is suitable for data science, machine learning and artificial intelligence, with concise syntax and suitable for beginners.
From Text to Websites: The Power of HTML
Apr 13, 2025 am 12:07 AM
HTML is a language used to build web pages, defining web page structure and content through tags and attributes. 1) HTML organizes document structure through tags, such as,. 2) The browser parses HTML to build the DOM and renders the web page. 3) New features of HTML5, such as, enhance multimedia functions. 4) Common errors include unclosed labels and unquoted attribute values. 5) Optimization suggestions include using semantic tags and reducing file size.
The role of Debian Sniffer in DDoS attack detection
Apr 12, 2025 pm 10:42 PM
This article discusses the DDoS attack detection method. Although no direct application case of "DebianSniffer" was found, the following methods can be used for DDoS attack detection: Effective DDoS attack detection technology: Detection based on traffic analysis: identifying DDoS attacks by monitoring abnormal patterns of network traffic, such as sudden traffic growth, surge in connections on specific ports, etc. This can be achieved using a variety of tools, including but not limited to professional network monitoring systems and custom scripts. For example, Python scripts combined with pyshark and colorama libraries can monitor network traffic in real time and issue alerts. Detection based on statistical analysis: By analyzing statistical characteristics of network traffic, such as data
How debian readdir integrates with other tools
Apr 13, 2025 am 09:42 AM
The readdir function in the Debian system is a system call used to read directory contents and is often used in C programming. This article will explain how to integrate readdir with other tools to enhance its functionality. Method 1: Combining C language program and pipeline First, write a C program to call the readdir function and output the result: #include#include#include#includeintmain(intargc,char*argv[]){DIR*dir;structdirent*entry;if(argc!=2){
Nginx SSL Certificate Update Debian Tutorial
Apr 13, 2025 am 07:21 AM
This article will guide you on how to update your NginxSSL certificate on your Debian system. Step 1: Install Certbot First, make sure your system has certbot and python3-certbot-nginx packages installed. If not installed, please execute the following command: sudoapt-getupdatesudoapt-getinstallcertbotpython3-certbot-nginx Step 2: Obtain and configure the certificate Use the certbot command to obtain the Let'sEncrypt certificate and configure Nginx: sudocertbot--nginx Follow the prompts to select
Python and Time: Making the Most of Your Study Time
Apr 14, 2025 am 12:02 AM
To maximize the efficiency of learning Python in a limited time, you can use Python's datetime, time, and schedule modules. 1. The datetime module is used to record and plan learning time. 2. The time module helps to set study and rest time. 3. The schedule module automatically arranges weekly learning tasks.
