Home > Article > Web Front-end > Using Node.js to implement compression and decompression functions
This article mainly introduces the method of compression and decompression based on Node.js. Now I share it with you and give it as a reference.
Compression format
zip and gzip are the two most common compression formats we see. Of course, gzip is rarely touched under Windows.
tar is an archive format that does not compress by default. It needs to be combined with gzip to compress the final tar file into a tar.gz file in gzip format, which is usually abbreviated to tgz.
Why is rar not mentioned? Because it is a patent-protected algorithm, you can get the decompression tool for free, while the compression tool has to be paid. Therefore, in our general application scenarios, rar compressed files are rarely provided.
This article will introduce respectively how to realize the compression and decompression of gzip, tar, tgz and zip under Node.js.
Uncompressed file library
The uncompressed file library used in this article comes from urllib. You need to clone it first and go to the specified directory.
git clone https://github.com/node-modules/urllib.git nodejs-compressing-demo
gzip
on Linux In the world, the responsibilities of each tool will be very pure and very single. For example, gzip will only compress files. As for how the folder is packaged and compressed, it has nothing to do with it. That is what tar is responsible for.
gzip command line compresses a file
For example, if we want to gzip the nodejs-compressing-demo/lib/urllib.js file, we will get a urllib.js .gz file, the source file will be deleted.
$ ls -l nodejs-compressing-demo/lib/urllib.js -rw-r--r-- 1 a a 31318 Feb 12 11:27 nodejs-compressing-demo/lib/urllib.js $ gzip nodejs-compressing-demo/lib/urllib.js $ ls -l nodejs-compressing-demo/lib/urllib.js.gz -rw-r--r-- 1 a a 8909 Feb 12 11:27 nodejs-compressing-demo/lib/urllib.js.gz # 还原压缩文件 $ gunzip nodejs-compressing-demo/lib/urllib.js.gz
The file size is reduced from 31318 bytes to 8909 bytes, more than 3.5 times the compression effect.
You can also use the pipe method, combined with the cat command, to compress and save the file as any file:
$ ls -l nodejs-compressing-demo/README.md -rw-r--r-- 1 a a 13747 Feb 12 11:27 nodejs-compressing-demo/README.md $ cat nodejs-compressing-demo/README.md | gzip > README.md.gz $ ls -l README.md.gz -rw-r--r-- 1 a a 4903 Feb 12 11:50 README.md.gz
Node.js implements gzip
Of course , we will not really implement a gzip algorithm and tool from scratch. In the world of Node.js, these basic libraries have already been prepared for you, we just need to use them out of the box.
This article will use the compressing module to implement all compression and decompression codes.
Why choose compressing? Because it has sufficient code quality and unit testing guarantees, is in an active maintenance state, has a very friendly API, and also supports streaming interfaces.
Promise interface
const compressing = require('compressing');
// 选择 gzip 格式，然后调用 compressFile 方法
compressing.gzip.compressFile('nodejs-compressing-demo/lib/urllib.js', 'nodejs-compressing-demo/lib/urllib.js.gz')
.then(() => {
console.log('success');
})
.catch(err => {
console.error(err);
});
// 解压缩是反响过程，接口都统一为 uncompress
compressing.gzip.uncompress('nodejs-compressing-demo/lib/urllib.js.gz', 'nodejs-compressing-demo/lib/urllib.js2')
.then(() => {
console.log('success');
})
.catch(err => {
console.error(err);
});Combined with the async/await programming model, the code is written as a normal asynchronous io operation.
const compressing = require('compressing');
async function main() {
try {
await compressing.gzip.compressFile('nodejs-compressing-demo/lib/urllib.js',
'nodejs-compressing-demo/lib/urllib.js.gz');
console.log('success');
} catch (err) {
console.error(err);
}
// 解压缩
try {
await compressing.gzip.uncompress('nodejs-compressing-demo/lib/urllib.js.gz',
'nodejs-compressing-demo/lib/urllib.js2');
console.log('success');
} catch (err) {
console.error(err);
}
}
main();Stream interface
It is important to note that when programming in Stream mode, you need to handle the error event of each stream and manually destroy all streams.
fs.createReadStream('nodejs-compressing-demo/lib/urllib.js') .on('error', handleError) .pipe(new compressing.gzip.FileStream()) // It's a transform stream .on('error', handleError) .pipe(fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js.gz2')) .on('error', handleError); // 解压缩，就是 pipe 的方向倒转过来 fs.createReadStream('nodejs-compressing-demo/lib/urllib.js.gz2') .on('error', handleError) .pipe(new compressing.gzip.UncompressStream()) // It's a transform stream .on('error', handleError) .pipe(fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js3')) .on('error', handleError);
According to the official Backpressuring in Streams recommendation, we should use the pump module to cooperate with Stream mode programming, and let pump complete the cleaning work of these Streams.
const pump = require('pump');
const source = fs.createReadStream('nodejs-compressing-demo/lib/urllib.js');
const target = fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js.gz2');
pump(source, new compressing.gzip.FileStream(), target, err => {
if (err) {
console.error(err);
} else {
console.log('success');
}
});
// 解压缩
pump(fs.createReadStream('nodejs-compressing-demo/lib/urllib.js.gz2'),
new compressing.gzip.FileStream(),
fs.createWriteStream('nodejs-compressing-demo/lib/urllib.js3'),
err => {
if (err) {
console.error(err);
} else {
console.log('success');
}
});Advantages of the Stream interface
The Stream interface seems much more complicated than the Promise interface. Why is there such an application scenario?
In fact, in the field of HTTP services, the Stream model will have greater advantages, because the HTTP request itself is a Request Stream. If you want to return an uploaded file with gzip compression, you do not need to save the uploaded file using the Stream interface. to the local disk, but consume this file stream directly.
Using the sample code for egg file upload, we can achieve gzip compression and return with a slight modification.
const pump = require('pump');
class UploadFormController extends Controller {
// ... other codes
async upload() {
const stream = await this.ctx.getFileStream();
// 直接将压缩流赋值给 ctx.body，实现边压缩边返回的流式响应
this.ctx.body = pump(stream, new compressing.gzip.FileStream());
}
}tar | gzip > tgz
gzip Chapter You can know in advance that tar is responsible for packaging the folder.
For example, if you want to package the entire nodejs-compressing-dem o folder into a file and send it to others, you can use the tar command.
$ tar -c -f nodejs-compressing-demo.tar nodejs-compressing-demo/ $ ls -l nodejs-compressing-demo.tar -rw-r--r-- 1 a a 206336 Feb 12 14:01 nodejs-compressing-demo.tar
As you can see, the files packaged by tar are generally larger because they are uncompressed and the size is close to the actual total size of the folder. So we will all compress at the same time as packaging.
$ tar -c -z -f nodejs-compressing-demo.tgz nodejs-compressing-demo/ $ ls -l nodejs-compressing-demo.tgz -rw-r--r-- 1 a a 39808 Feb 12 14:07 nodejs-compressing-demo.tgz
The difference in size between tar and tgz is more than 5 times, which can greatly reduce network transmission bandwidth.
Node.js implements tgz
Promise interface
First use compressing.tar.compressDir(sourceDir, targetFile) to Pack a folder into a tar file, and then use the gzip compression method above to compress the tar file into a tgz file.
const compressing = require('compressing');
compressing.tar.compressDir('nodejs-compressing-demo', 'nodejs-compressing-demo.tar')
.then(() => {
return compressing.gzip.compressFile('nodejs-compressing-demo.tar',
'nodejs-compressing-demo.tgz');
});
.then(() => {
console.log('success');
})
.catch(err => {
console.error(err);
});
// 解压缩
compressing.gzip.uncompress('nodejs-compressing-demo.tgz', 'nodejs-compressing-demo.tar')
.then(() => {
return compressing.tar.uncompress('nodejs-compressing-demo.tar',
'nodejs-compressing-demo2');
});
.then(() => {
console.log('success');
})
.catch(err => {
console.error(err);
});Combined with the async/await programming model, the code will be easier to read:
const compressing = require('compressing');
async function main() {
try {
await compressing.tar.compressDir('nodejs-compressing-demo',
'nodejs-compressing-demo.tar');
await compressing.gzip.compressFile('nodejs-compressing-demo.tar',
'nodejs-compressing-demo.tgz');
console.log('success');
} catch (err) {
console.error(err);
}
// 解压缩
try {
await compressing.gzip.uncompress('nodejs-compressing-demo.tgz',
'nodejs-compressing-demo.tar');
await compressing.tar.uncompress('nodejs-compressing-demo.tar',
'nodejs-compressing-demo2');
console.log('success');
} catch (err) {
console.error(err);
}
}
main();Stream interface
Through compressing.tar.Stream Class, you can dynamically add any files and folders to a tar stream object, which is very flexible.
const tarStream = new compressing.tar.Stream();
// dir
tarStream.addEntry('dir/path/to/compress');
// file
tarStream.addEntry('file/path/to/compress');
// buffer
tarStream.addEntry(buffer);
// stream
tarStream.addEntry(stream);
const destStream = fs.createWriteStream('path/to/destination.tgz');
pump(tarStream, new compressing.gzip.FileStream(), destStream, err => {
if (err) {
console.error(err);
} else {
console.log('success');
}
});zip
zip can actually be regarded as a "commercial" combination of tar gzip. It allows users to not distinguish between compressed files and compressed folders. Anyway, just use my zip.
Example of using the zip command line tool to compress a folder:
$ zip -r nodejs-compressing-demo.zip nodejs-compressing-demo/ adding: nodejs-compressing-demo/ (stored 0%) adding: nodejs-compressing-demo/test/ (stored 0%) ... adding: nodejs-compressing-demo/.travis.yml (deflated 36%) $ ls -l nodejs-compressing-demo.* -rw-r--r-- 1 a a 206336 Feb 12 14:06 nodejs-compressing-demo.tar -rw-r--r-- 1 a a 39808 Feb 12 14:07 nodejs-compressing-demo.tgz -rw-r--r-- 1 a a 55484 Feb 12 14:34 nodejs-compressing-demo.zip
By comparing the file sizes of tgz and zip, it can be seen that under the default compression parameters, gzip will perform better than zip.
Node.js implements zip
The implementation code is similar to tar, except that it is compressed by default, and there is no need to add the gzip process.
const compressing = require('compressing');
compressing.zip.compressDir('nodejs-compressing-demo', 'nodejs-compressing-demo.zip')
.then(() => {
console.log('success');
})
.catch(err => {
console.error(err);
});
// 解压缩
compressing.zip.uncompress('nodejs-compressing-demo.zip', 'nodejs-compressing-demo3')
.then(() => {
console.log('success');
})
.catch(err => {
console.error(err);
});Summary
Is compression and decompression based on Node.js easier than imagined? Thanks to the giant npm, we can have the simple experience of programming with command line tools.
Whether it is the Promise interface or the Stream interface, it has its most suitable scenario. Will you choose it?
At this point, with the compression and decompression capabilities you have, what kind of services and functions can you do?
The above is what I compiled for everyone. I hope it will be helpful to everyone in the future.
Related articles:
Using tween.js to implement easing tween animation algorithm
Detailed explanation of refs in React (detailed tutorial )
How to achieve intermittent text cycle scrolling effect through JS
The above is the detailed content of Using Node.js to implement compression and decompression functions. For more information, please follow other related articles on the PHP Chinese website!