Detailed explanation of Node.js Buffer usage

This time I will bring you Node.js Detailed explanation of Buffer usage, what are the precautions when using Node.js Buffer, the following is a practical case, let’s take a look.

What is Buffer?

Buffer exists as a global object and can be used without introducing a module. You must not ignore it.

It can be understood that Buffer is an area opened in memory for storing binary data. Buffer opens up off-heap memory.

What are the application scenarios of Buffer?

Stream

How to understand stream? A stream is a collection of data (similar to data and string), but the data of the stream cannot be obtained at one time, and the data will not all be loaded into the memory. Therefore, the stream is very suitable for big data processing and intermittent returns. The external source of the chunk. The speed between stream producers and consumers is usually inconsistent, so buffers are needed to temporarily store some data. The buffer size is specified by the highWaterMark parameter, which is 16Kb by default.

Storing data that requires a large amount of memory

The memory space occupied by the Buffer object is not counted in the Node.js process memory space limit, so it can be used for storage Large objects, but the size of the object is still limited. Generally, a 32-bit system is about 1G, and a 64-bit system is about 2G.

How to create a Buffer

In addition to the stream automatically creating a Buffer implicitly, you can also create a Buffer manually as follows:

The data stored in Buffer has been determined

Buffer.from(obj) // The types supported by obj string, buffer, arrayBuffer, array, or array-like object

Note: Buffer.from does not support passing in numbers, as shown below:

Buffer.from(1234);
buffer.js:208
  throw new errors.TypeError(
  ^
TypeError [ERR_INVALID_ARG_TYPE]: The "value" argument must not be of type number. Received type number
  at Function.from (buffer.js:208:11)
  ...

If you want to pass in numbers, you can pass in an array:

const buf = Buffer.from([1, 2, 3, 4]);
console.log(buf); // <Buffer 01 02 03 04>

But there is a problem with this method The problem is that when different values ​​are stored, the binary data recorded in the buffer will be the same, as shown below:

const buf2 = Buffer.from([127, -1]);
console.log(buf2);   // <Buffer 7f ff>
const buf3 = Buffer.from([127, 255]);
console.log(buf3);  // <Buffer 7f ff>
console.log(buf3.equals(buf2)); // true

When a set of numbers to be recorded all fall between 0 and 255 (readUInt8 to read) this range, or if they all fall within the range of -128 to 127 (readInt8 to read), then there will be no problem. Otherwise, it is strongly not recommended to use Buffer.from to save a set of numbers. Because different methods should be called when reading different numbers.

Buffer storage data is not determined

Buffer.alloc, Buffer.allocUnsafe, Buffer.allocUnsafeSlow

Buffer.alloc will Fills the allocated memory with 0 values, so it is slower than the latter two, but it is also safer. Of course, you can also use the --zero-fill-buffers flag to make allocUnsafe and allocUnsafeSlow fill with zero values ​​after allocating memory.

node --zero-fill-buffers index.js

When the allocated space is less than 4KB, allocUnsafe will directly slice the space from the previously pre-allocated Buffer, so the speed is faster than allocUnsafeSlow. When it is greater than or equal to 4KB, the speed of the two is the same.

// 分配空间等于4KB
function createBuffer(fn, size) {
 console.time('buf-' + fn);
 for (var i = 0; i < 100000; i++) {
  Buffer[fn](size);
 }
 console.timeEnd(&#39;buf-&#39; + fn);
}
createBuffer(&#39;alloc&#39;, 4096);
createBuffer(&#39;allocUnsafe&#39;, 4096);
createBuffer(&#39;allocUnsafeSlow&#39;, 4096);
// 输出
buf-alloc:      294.002ms
buf-allocUnsafe:   224.072ms
buf-allocUnsafeSlow: 209.22ms

function createBuffer(fn, size) {
 console.time('buf-' + fn);
 for (var i = 0; i < 100000; i++) {
  Buffer[fn](size);
 }
 console.timeEnd(&#39;buf-&#39; + fn);
}
createBuffer(&#39;alloc&#39;, 4095);
createBuffer(&#39;allocUnsafe&#39;, 4095);
createBuffer(&#39;allocUnsafeSlow&#39;, 4095);
// 输出
buf-alloc:      296.965ms
buf-allocUnsafe:   135.877ms
buf-allocUnsafeSlow: 205.225ms

One thing to keep in mind: the new Buffer(xxxx) method is no longer recommended.

Buffer usage

buffer to string

const buf = Buffer.from('test');
console.log(buf.toString('utf8'));         // test
console.log(buf.toString('utf8', 0, 2));      // te

buffer to json

const buf = Buffer.from([0x1, 0x2, 0x3, 0x4, 0x5]);
console.log(buf.toJSON());  // { type: 'Buffer', data: [ 1, 2, 3, 4, 5 ] }

buffer clipping, the new buffer returned after clipping points to the same point as the original buffer A piece of memory

buf.slice([start[, end]])

1. start starting position

2. end ending position (not included)

Example:

var buf1 = Buffer.from('test');
var buf2 = buf1.slice(1, 3).fill('xx');
console.log("buf2 content: " + buf2.toString()); // xx
console.log("buf1 content: " + buf1.toString()); // txxt

buffer copy, buffer is different from array, the length of buffer will not change once determined, so when the copied source buffer is larger than the target buffer, only part of the value will be copied

buf.copy(target[, targetStart[, sourceStart[, sourceEnd]]])

Example:

var buf1 = Buffer.from('abcdefghijkl');
var buf2 = Buffer.from('ABCDEF');
buf1.copy(buf2, 1);
console.log(buf2.toString()); //Abcdef

Buffer equality judgment, comparing binary values

buf.equals(otherBuffer)

Example:

const buf1 = Buffer.from('ABC');
const buf2 = Buffer.from('414243', 'hex'); 
console.log(buf1.equals(buf2));  // true

In addition to equals, compare In fact, it can also be used to determine whether they are equal (when the result is 0, they are equal), but the main function of compare is to sort the buffer instances in the array.

Whether the buffer contains a specific value

buf.includes(value[, byteOffset][, encoding])
buf.indexOf(value[, byteOffset][, encoding])

示例：

const buf = Buffer.from('this is a buffer');
console.log(buf.includes('this')); // true
console.log(buf.indexOf('this')); // 0

写入读取数值

写入方法：

位数固定且超过1个字节的： write{Double| Float | Int16 | Int32| UInt16 | UInt32 }{BE|LE}(value, offset)

位数不固定的： write{Int | UInt}{BE | LE}(value, offset, bytelength) //此方法提供了更灵活的位数表示数据（比如3位、5位）

位数固定是1个字节的： write{Int8 | Unit8}(value, offset)

读取方法：

位数固定且超过1个字节的： read{Double| Float | Int16 | Int32 | UInt16 | UInt32 }{BE|LE}（offset)

位数不固定的： read{Int | UInt}{BE | LE}(offset, byteLength)

位数固定是1个字节的： read{Int8 | Unit8}(offset)

Double、Float、Int16、Int32、UInt16、UInt32既确定了表征数字的位数，也确定了是否包含负数，因此定义了不同的数据范围。同时由于表征数字的位数都超过8位，无法用一个字节来表示，因此就涉及到了计算机的字节序区分（大端字节序与小端字节序）

关于大端小端的区别可以这么理解：数值的高位在buffer的起始位置的是大端，数值的低位buffer的起始位置则是小端

const buf = Buffer.allocUnsafe(2);
buf.writeInt16BE(256, 0) 
console.log(buf);      // <Buffer 01 00> 
buf.writeInt16LE(256, 0)
console.log(buf);      // <Buffer 00 01>

http://tools.jb51.net/transcoding/hexconvert这里可以查看数值的不同进制之间的转换，如果是大端的话，则直接按顺序（0100）拼接16进制即可，如果是小端则需要调换一下顺序才是正确的表示方式。

buffer合并

Buffer.concat(list[, totalLength]) //totalLength不是必须的，如果不提供的话会为了计算totalLength会多一次遍历

const buf1 = Buffer.from('this is');
const buf2 = Buffer.from(' funny');
console.log(Buffer.concat([buf1, buf2], buf1.length + buf2.length));
// <Buffer 74 68 69 73 20 69 73 20 66 75 6e 6e 79>

清空buffer

清空buffer数据最快的办法是buffer.fill(0)

buffer模块与Buffer的关系

Buffer是全局global上的一个引用，指向的其实是buffer.Buffer

 const buffer = require('buffer');
 console.log(buffer.Buffer === Buffer); //true

buffer模块上还有其他一些属性和方法

const buffer = require('buffer');
console.log(buffer);
{ Buffer:
  { [Function: Buffer]
   poolSize: 8192,
   from: [Function: from],
   alloc: [Function: alloc],
   allocUnsafe: [Function: allocUnsafe],
   allocUnsafeSlow: [Function: allocUnsafeSlow],
   isBuffer: [Function: isBuffer],
   compare: [Function: compare],
   isEncoding: [Function: isEncoding],
   concat: [Function: concat],
   byteLength: [Function: byteLength],
   [Symbol(node.isEncoding)]: [Function: isEncoding] },
 SlowBuffer: [Function: SlowBuffer],
 transcode: [Function: transcode],
 INSPECT_MAX_BYTES: 50,
 kMaxLength: 2147483647,
 kStringMaxLength: 1073741799,
 constants: { MAX_LENGTH: 2147483647, MAX_STRING_LENGTH: 1073741799 } }

上面的kMaxLength与MAX_LENGTH代表了新建buffer时内存大小的最大值，当超过限制值后就会报错

32为机器上是(2^30)-1(~1GB)

64位机器上是(2^31)-1(~2GB)

Buffer释放

相信看了本文案例你已经掌握了方法，更多精彩请关注php中文网其它相关文章！

推荐阅读：

如何搭建React全家桶环境

怎样使用JS实现调用本地摄像头

The above is the detailed content of Detailed explanation of Node.js Buffer usage. For more information, please follow other related articles on the PHP Chinese website!