Web Front-endJS TutorialLet's talk about how to use third-party open source libraries to implement website crawling function in nodeLet's talk about how to use third-party open source libraries to implement website crawling function in node
This article will introduce to you how to easily implement the website crawling function with the help of third-party open source libraries in node. I hope it will be helpful to everyone!
nodejsImplementing website crawling function
Introduction to third-party libraries
request encapsulation of network requests
cheerio node version of jQuery
mkdirp creates multiple layers Folder directory
Implementation ideas
Through
requestGet the content of the specified urlThrough
cheerioFind the jump path in the page (remove duplicates)Through
mkdirpCreate directoryCreate a file through
fsand write the read content toGet Repeat the above steps to the path that is not accessed
Code implementation
const fs = require("fs");
const path = require("path");
const request = require("request");
const cheerio = require("cheerio");
const mkdirp = require("mkdirp");
// 定义入口url
const homeUrl = "https://www.baidu.com";
// 定义set存储已经访问过的路径，避免重复访问
const set = new Set([homeUrl]);
function grab(url) {
// 校验url规范性
if (!url) return;
// 去空格
url = url.trim();
// 自动补全url路径
if (url.endsWith("/")) {
url += "index.html";
}
const chunks = [];
// url可能存在一些符号或者中文，可以通过encodeURI编码
request(encodeURI(url))
.on("error", (e) => {
// 打印错误信息
console.log(e);
})
.on("data", (chunk) => {
// 接收响应内容
chunks.push(chunk);
})
.on("end", () => {
// 将相应内容转换成文本
const html = Buffer.concat(chunks).toString();
// 没有获取到内容
if (!html) return;
// 解析url
let { host, origin, pathname } = new URL(url);
pathname = decodeURI(pathname);
// 通过cheerio解析html
const $ = cheerio.load(html);
// 将路径作为目录
const dir = path.dirname(pathname);
// 创建目录
mkdirp.sync(path.join(__dirname, dir));
// 往文件写入内容
fs.writeFile(path.join(__dirname, pathname), html, "utf-8", (err) => {
// 打印错误信息
if (err) {
console.log(err);
return;
}
console.log(`[${url}]保存成功`);
});
// 获取到页面中所有a元素
const aTags = $("a");
Array.from(aTags).forEach((aTag) => {
// 获取到a标签中的路径
const href = $(aTag).attr("href");
// 此处可以校验href的合法或者控制爬去的网站范围，比如必须都是某个域名下的
// 排除空标签
if (!href) return;
// 排除锚点连接
if (href.startsWith("#")) return;
if (href.startsWith("mailto:")) return;
// 如果不想要保存图片可以过滤掉
// if (/\.(jpg|jpeg|png|gif|bit)$/.test(href)) return;
// href必须是入口url域名
let reg = new RegExp(`^https?:\/\/${host}`);
if (/^https?:\/\//.test(href) && !reg.test(href)) return;
// 可以根据情况增加更多逻辑
let newUrl = "";
if (/^https?:\/\//.test(href)) {
// 处理绝对路径
newUrl = href;
} else {
// 处理相对路径
newUrl = origin + path.join(dir, href);
}
// 判断是否访问过
if (set.has(newUrl)) return;
if (newUrl.endsWith("/") && set.has(newUrl + "index.html")) return;
if (newUrl.endsWith("/")) newUrl += "index.html";
set.add(newUrl);
grab(newUrl);
});
});
}
// 开始抓取
grab(homeUrl);Summary
The simple web crawler is complete. You can try changing homeUrl to the website you want to crawl.
For more node-related knowledge, please visit: nodejs tutorial! !
The above is the detailed content of Let's talk about how to use third-party open source libraries to implement website crawling function in node. For more information, please follow other related articles on the PHP Chinese website!
Python vs. JavaScript: Community, Libraries, and ResourcesApr 15, 2025 am 12:16 AMPython and JavaScript have their own advantages and disadvantages in terms of community, libraries and resources. 1) The Python community is friendly and suitable for beginners, but the front-end development resources are not as rich as JavaScript. 2) Python is powerful in data science and machine learning libraries, while JavaScript is better in front-end development libraries and frameworks. 3) Both have rich learning resources, but Python is suitable for starting with official documents, while JavaScript is better with MDNWebDocs. The choice should be based on project needs and personal interests.
From C/C to JavaScript: How It All WorksApr 14, 2025 am 12:05 AMThe shift from C/C to JavaScript requires adapting to dynamic typing, garbage collection and asynchronous programming. 1) C/C is a statically typed language that requires manual memory management, while JavaScript is dynamically typed and garbage collection is automatically processed. 2) C/C needs to be compiled into machine code, while JavaScript is an interpreted language. 3) JavaScript introduces concepts such as closures, prototype chains and Promise, which enhances flexibility and asynchronous programming capabilities.
JavaScript Engines: Comparing ImplementationsApr 13, 2025 am 12:05 AMDifferent JavaScript engines have different effects when parsing and executing JavaScript code, because the implementation principles and optimization strategies of each engine differ. 1. Lexical analysis: convert source code into lexical unit. 2. Grammar analysis: Generate an abstract syntax tree. 3. Optimization and compilation: Generate machine code through the JIT compiler. 4. Execute: Run the machine code. V8 engine optimizes through instant compilation and hidden class, SpiderMonkey uses a type inference system, resulting in different performance performance on the same code.
Beyond the Browser: JavaScript in the Real WorldApr 12, 2025 am 12:06 AMJavaScript's applications in the real world include server-side programming, mobile application development and Internet of Things control: 1. Server-side programming is realized through Node.js, suitable for high concurrent request processing. 2. Mobile application development is carried out through ReactNative and supports cross-platform deployment. 3. Used for IoT device control through Johnny-Five library, suitable for hardware interaction.
Building a Multi-Tenant SaaS Application with Next.js (Backend Integration)Apr 11, 2025 am 08:23 AMI built a functional multi-tenant SaaS application (an EdTech app) with your everyday tech tool and you can do the same. First, what’s a multi-tenant SaaS application? Multi-tenant SaaS applications let you serve multiple customers from a sing
How to Build a Multi-Tenant SaaS Application with Next.js (Frontend Integration)Apr 11, 2025 am 08:22 AMThis article demonstrates frontend integration with a backend secured by Permit, building a functional EdTech SaaS application using Next.js. The frontend fetches user permissions to control UI visibility and ensures API requests adhere to role-base
JavaScript: Exploring the Versatility of a Web LanguageApr 11, 2025 am 12:01 AMJavaScript is the core language of modern web development and is widely used for its diversity and flexibility. 1) Front-end development: build dynamic web pages and single-page applications through DOM operations and modern frameworks (such as React, Vue.js, Angular). 2) Server-side development: Node.js uses a non-blocking I/O model to handle high concurrency and real-time applications. 3) Mobile and desktop application development: cross-platform development is realized through ReactNative and Electron to improve development efficiency.
The Evolution of JavaScript: Current Trends and Future ProspectsApr 10, 2025 am 09:33 AMThe latest trends in JavaScript include the rise of TypeScript, the popularity of modern frameworks and libraries, and the application of WebAssembly. Future prospects cover more powerful type systems, the development of server-side JavaScript, the expansion of artificial intelligence and machine learning, and the potential of IoT and edge computing.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
SublimeText3 Chinese version
Chinese version, very easy to use
Atom editor mac version download
The most popular open source editor
VSCode Windows 64-bit Download
A free and powerful IDE editor launched by Microsoft
Zend Studio 13.0.1
Powerful PHP integrated development environment
DVWA
Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software
