What is Buffer? Detailed explanation of how to use Buffer and specific usage scenario examples

JavaScript is very friendly to string processing. Whether it is a wide byte or a single-byte string, it is considered a string. Node needs to process network protocols, operate databases, process pictures, file uploads, etc. It also needs to process a large amount of binary data. The built-in strings are far from meeting these requirements, so Buffer came into being.

Buffer structure

Buffer is a typical module that combines Javascript and C++. The performance-related parts are implemented in C++, and the non-performance-related parts are implemented in javascript.

Node has already installed the Buffer into the memory when the process starts and puts it into the global object, so there is no need to require

Buffer object: similar to an array, its elements are hexadecimal Double digits.

Buffer memory allocation

The memory allocation of the Buffer object is not in the heap memory of V8, and the memory application is implemented at the C++ level of Node.

In order to efficiently use the requested memory, Node adopts the slab allocation mechanism. Slab is a dynamic memory management mechanism that applies various *nix operating systems. Slab has three states:

(1) full: fully allocated state

(2) partial: partially allocated state

(3) empty: not allocated state

Buffer conversion

Buffer objects can be converted to strings. The supported encoding types are as follows:

ASCII, UTF-8, UTF-16LE/ UCS-2, Base64, Binary, Hex

String to Buffer

new Buffer(str, [encoding]), default UTF-8
buf. write(string, [offset], [length], [encoding])

Buffer to string

buf.toString([encoding], [start], [end])

Encoding type not supported by Buffer

Determine whether it is supported by Buffer.isEncoding(encoding)

iconv-lite: pure JavaScript implementation, more lightweight, Better performance and no need to convert C++ to javascript

iconv: Call C++’s libiconv library to complete

Buffer splicing

Note "res.on( 'data', function(chunk) {})", the parameter chunk is a Buffer object. Directly using + splicing will automatically convert it into a string. For wide byte characters, garbled characters may be generated,

Solution:

(1) Through the setEncoding() method in the readable stream, this method allows the data event to be delivered no longer as a Buffer object, but as an encoded string. The StringEncoder module is used internally.

(2) Temporarily store the Buffer object into an array, and finally assemble it into a large Buffer and then encode it into a string for output.

Buffer is widely used in file I/O and network I/O. Its performance is very important, and its performance is much higher than that of ordinary strings.

In addition to the performance loss caused by the conversion of strings, the use of Buffer has a highWaterMark setting that is crucial to performance when reading files.

a. The highWaterMark setting has a certain impact on the allocation and use of Buffer memory.

b. If highWaterMark is set too small, it may cause too many system calls.

When to use buffer and when not to use it ------ Pure javascript supports unicode code but not binary. When dealing with TCP stream or file stream, there is a need to process the stream. Necessary, when we save non-utf-8 strings, binary and other formats, we must use "Buffer".

Instance introduction

 var buf = new Buffer("this is text concat test !") ,str = "this is text concat test !" ;
 console.time("buffer concat test !");
 var list = [] ;
 var len = 100000 * buf.length ;
 for(var i=0;i<100000;i++){
     list.push(buf) ;
     len += buf.length ;
 }
 var s1 = Buffer.concat(list, len).toString() ;
 console.timeEnd("buffer concat test !") ;
 console.time("string concat test !") ;
 var list = [] ;
 for (var i = 100000; i >= 0; i--) {
   list.push(str) ;
 }
 var s2 = list.join("") ;
 console.timeEnd("string concat test !") ;

The following are the running results:

The reading speed is definitely faster for string and buffer The toString() operation is also required. So when we save a string, we should still use string. Even if a large string is spliced ​​into a string, the speed of splicing it will not be slower than that of the buffer.

So when do we need to use buffer again? When there is no other way, when we save non-utf-8 strings, binary and other formats, we must use it.

The above is the detailed content of What is Buffer? Detailed explanation of how to use Buffer and specific usage scenario examples. For more information, please follow other related articles on the PHP Chinese website!