How to implement breakpoint resumption in Node.js

青灯夜游
Release: 2021-06-18 18:18:45
forward
1637 people have browsed it

This article will introduce you to the method of implementingNode.js. It has certain reference value. Friends in need can refer to it. I hope it will be helpful to everyone.

How to implement breakpoint resumption in Node.js

Normal business needs: uploadpictures,Excel, etc. After all, the size of a few M can be uploaded to the server quickly.
For uploadingvideosand other large files that are hundreds of M or several G in size, you need to wait for a long time.
This produces corresponding solutions. Forpause,network disconnection,poor networksituations when uploading large files, useSlicingBreakpoint resume transfercan handle the above situation very well

Program analysis

  • Slice

    • is to segment the uploaded video. The specific operation is:
    • File.slice(start,end): Returns a new blob object
      • Copy The starting byte of the blob
      • Copy the ending byte of the blob
  • Resumable transfer

    • Before each slice is uploaded, request the server interface to read the number of uploaded slices of the same file
    • If a new file is uploaded, the server will return 0, otherwise it will return the number of uploaded slices

[Recommended learning: "nodejs tutorial"]

Specific solution process

This demo provides key Some ideas and methods, other functions such as:File restriction,lastModifiedDate verification file repeatability,Cache file regular clearingand other functional extensions can be based on this code Add on.

  • html
 
Copy after login
  • script
let count = 0; // 记录需要上传的文件下标 const handleVideo = async (event, name, url) => { // 阻止浏览器默认表单事件 event.preventDefault(); let currentSize = document.querySelector("h2"); let files = document.querySelector(name).files; // 默认切片数量 const sectionLength = 100; // 首先请求接口,获取服务器是否存在此文件 // count为0则是第一次上传,count不为0则服务器存在此文件,返回已上传的切片数 count = await handleCancel(files[0]); // 申明存放切片的数组对象 let fileCurrent = []; // 循环file文件对象 for (const file of [...files]) { // 得出每个切片的大小 let itemSize = Math.ceil(file.size / sectionLength); // 循环文件size,文件blob存入数组 let current = 0; for (current; current < file.size; current += itemSize) { fileCurrent.push({ file: file.slice(current, current + itemSize) }); } // axios模拟手动取消请求 const CancelToken = axios.CancelToken; const source = CancelToken.source(); // 当断点续传时,处理切片数量,已上传切片则不需要再次请求上传 fileCurrent = count === 0 ? fileCurrent : fileCurrent.slice(count, sectionLength); // 循环切片请求接口 for (const [index, item] of fileCurrent.entries()) { // 模拟请求暂停 || 网络断开 if (index > 90) { source.cancel("取消请求"); } // 存入文件相关信息 // file为切片blob对象 // filename为文件名 // index为当前切片数 // total为总切片数 let formData = new FormData(); formData.append("file", item.file); formData.append("filename", file.name); formData.append("total", sectionLength); formData.append("index", index + count + 1); await axios({ url: `http://localhost:8080/${url}`, method: "POST", data: formData, cancelToken: source.token, }) .then((response) => { // 返回数据显示进度 currentSize.innerHTML = `进度${response.data.size}%`; }) .catch((err) => { console.log(err); }); } } }; // 请求接口,查询上传文件是否存在 // count为0表示不存在,count不为0则已上传对应切片数 const handleCancel = (file) => { return axios({ method: "post", url: "http://localhost:8080/getSize", headers: { "Content-Type": "application/json; charset = utf-8" }, data: { fileName: file.name, }, }) .then((res) => { return res.data.count; }) .catch((err) => { console.log(err); }); };
Copy after login
  • node server
// 使用express构建服务器api const express = require("express"); // 引入上传文件逻辑代码 const upload = require("./upload_file"); // 处理所有响应,设置跨域 app.all("*", (req, res, next) => { res.header("Access-Control-Allow-Origin", "*"); res.header("Access-Control-Allow-Headers", "X-Requested-With"); res.header("Access-Control-Allow-Methods", "PUT,POST,GET,DELETE,OPTIONS"); res.header("Access-Control-Allow-Headers", "Content-Type, X-Requested-With "); res.header("X-Powered-By", " 3.2.1"); res.header("Content-Type", "application/json;charset=utf-8"); next(); }); const app = express(); app.use(bodyParser.json({ type: "application/*+json" })); // 视频上传(查询当前切片数) app.post("/getSize", upload.getSize); // 视频上传接口 app.post("/video", upload.video); // 开启本地端口侦听 app.listen(8080);
Copy after login
  • upload_file

// 文件上传模块 const formidable = require("formidable"); // 文件系统模块 const fs = require("fs"); // 系统路径模块 const path = require("path"); // 操作写入文件流 const handleStream = (item, writeStream) => { // 读取对应目录文件buffer const readFile = fs.readFileSync(item); // 将读取的buffer || chunk写入到stream中 writeStream.write(readFile); // 写入完后,清除暂存的切片文件 fs.unlink(item, () => {}); }; // 视频上传(切片) module.exports.video = (req, res) => { // 创建解析对象 const form = new formidable.IncomingForm(); // 设置视频文件上传路径 let dirPath = path.join(__dirname, "video"); form.uploadDir = dirPath; // 是否保留上传文件名后缀 form.keepExtensions = true; // err 错误对象 如果解析失败包含错误信息 // fields 包含除了二进制以外的formData的key-value对象 // file 对象类型 上传文件的信息 form.parse(req, async (err, fields, file) => { // 获取上传文件blob对象 let files = file.file; // 获取当前切片index let index = fields.index; // 获取总切片数 let total = fields.total; // 获取文件名 let filename = fields.filename; // 重写上传文件名,设置暂存目录 let url = dirPath + "/" + filename.split(".")[0] + `_${index}.` + filename.split(".")[1]; try { // 同步修改上传文件名 fs.renameSync(files.path, url); console.log(url); // 异步处理 setTimeout(() => { // 判断是否是最后一个切片上传完成,拼接写入全部视频 if (index === total) { // 同步创建新目录,用以存放完整视频 let newDir = __dirname + `/uploadFiles/${Date.now()}`; // 创建目录 fs.mkdirSync(newDir); // 创建可写流,用以写入文件 let writeStream = fs.createWriteStream(newDir + `/${filename}`); let fsList = []; // 取出所有切片文件,放入数组 for (let i = 0; i < total; i++) { const fsUrl = dirPath + "/" + filename.split(".")[0] + `_${i + 1}.` + filename.split(".")[1]; fsList.push(fsUrl); } // 循环切片文件数组,进行stream流的写入 for (let item of fsList) { handleStream(item, writeStream); } // 全部写入,关闭stream写入流 writeStream.end(); } }, 100); } catch (e) { console.log(e); } res.send({ code: 0, msg: "上传成功", size: index, }); }); }; // 获取文件切片数 module.exports.getSize = (req, res) => { let count = 0; req.setEncoding("utf8"); req.on("data", function (data) { let name = JSON.parse(data); let dirPath = path.join(__dirname, "video"); // 计算已上传的切片文件个数 let files = fs.readdirSync(dirPath); files.forEach((item, index) => { let url = name.fileName.split(".")[0] + `_${index + 1}.` + name.fileName.split(".")[1]; if (files.includes(url)) { ++count; } }); res.send({ code: 0, msg: "请继续上传", count, }); }); };
Copy after login

Logical analysis

  • First request to upload the query file The first upload, or the corresponding slice already exists
    • The first time the file is uploaded, the slices will start from 0
      • If the file already has the corresponding slice, the upload request will start from the number of slices
      Loop through the slice array and upload each slice file
    • A simulated manual pause request is used, and the request is canceled when the number of slices is greater than 90
    Server
  • Receives the query file filename, searches for the temporarily stored file address, and determines whether the corresponding uploaded file exists
    • This file has never been uploaded, then Return 0, the number of slices starts from 0
      • If the file has been uploaded, the corresponding number of slices will be returned
      Receive the uploaded file slices, and store the file in the temporary storage directory
    • Determine whether the slice has been uploaded by
        count
      • andtotalAfter the upload is completed, create a file storage directory and create a writable stream for writing operations
      • Extract the corresponding temporary files into the array, loop the file directory array, read and write to the file buffer in sequence
      • After writing is completed, close the writable stream.
Summary

The above code involves changes or deviations in specific business processes. It's just one of the specific implementation methods.

I hope this article can be helpful to everyone. If there is something wrong in the writing, I would like to give you some pointers.


The above code address: https://github.com/Surprise-ling/uploadFile
  • For more programming-related knowledge, please visit:
Programming Video

! !

The above is the detailed content of How to implement breakpoint resumption in Node.js. For more information, please follow other related articles on the PHP Chinese website!

Related labels:
source:juejin.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template
About us Disclaimer Sitemap
php.cn:Public welfare online PHP training,Help PHP learners grow quickly!