Python uses struct to handle binary (pack and unpack usage)

Sometimes you need to use python to process binary data, such as when accessing files and socket operations. At this time, you can use python's struct module to complete. You can use struct to process structures in the C language.

In the struct module The three most important functions are pack(), unpack(), calcsize()

pack(fmt, v1, v2, ...) according to the given format (fmt), encapsulate the data into a string (actually It is similar to the byte flow of the C structure)

UnPACK (FMT, String) based on the given format (FMT) to analyze the byte running string, and return the parsed Tuple

calcsize (FMT) calculation given (FMT). fmt) occupies how many bytes of memory

The supported formats in struct are as follows:

Format C Type Python Number of bytes

x pad byte no value 1

c char string of length 1 1

b signed char integer 1

B unsigned char integer 1

? _Bool bool 1

h short integer 2

H unsigned short integer 2

i int integer 4

I unsigned int integer or long 4

l long integer 4

L unsigned long long 4

q long long long 8

Q unsigned long long long 8

f float float 4

d double float 8

s char[] string 1

p char[] string 1

P void * long

Note 1.q and Q are only interesting when the machine supports 64-bit operation

Note 2. There can be a number before each format to indicate the number

Note 3.s format represents a certain length of characters String, 4s represents a string of length 4, but p represents a pascal string

Note 4. P is used to convert a pointer, its length is related to the machine word length

Note 5. The last one can be used to represent a pointer Type, occupies 4 bytes

In order to exchange data with the structure in c, it is also necessary to consider that some c or c++ compilers use byte alignment, usually 32-bit systems with 4 bytes as the unit. Therefore, the struct is converted according to the byte order of the local machine. The first character in the format can be used to change the alignment. The definition is as follows:

Character Byte order Size and alignment

@ native native Enough 4 bytes

= native standard                                                                                     use using byte using

– little-endian                                                                                                    .

The method of use is to put it in the first position of fmt, just like '@5s6sif'

Example 1:

The structure is as follows:

struct Header
{
    unsigned short id;
    char[4] tag;
    unsigned int version;
    unsigned int count;
}

The above structure data is received through socket.recv, and there is a string s, now you need to parse it out, you can use the unpack() function:

import struct
id, tag, version, count = struct.unpack("!H4s2I", s)

In the above format string, ! means that we want to use network byte order to parse, because our data is received from the network, When transmitted over the network, it is in network byte order. The following H represents an unsigned short id, 4s represents a 4-byte long string, and 2I represents two unsigned int type data.

Just pass one unpack, now our information has been saved in id, tag, version, and count.

Similarly, it is also very convenient to pack local data into struct format:

ss = struct.pack("!H4s2I", id, tag, version, count);

The pack function puts id, tag, version, count is converted into a structure Header according to the specified format. ss is now a string (actually a byte stream similar to a c structure). This string can be sent out through socket.send(ss).

Example 2:

import struct
a=12.34
#将a变为二进制
bytes=struct.pack('i',a)

At this time, bytes is a string string, and the string is the same as the binary storage content of a in bytes.

Then perform the reverse operation, and convert the existing binary data bytes (actually a string) into the python data type:

#Note that unpack returns a tuple!!

a,=struct.unpack('i',bytes)

If it is composed of multiple data, it can be like this:

a='hello'
b='world!'
c=2
d=45.123
bytes=struct.pack('5s6sif',a,b,c,d)

The bytes at this time are data in binary form, and can be written directly to a file such as binfile.write(bytes)Then, we can read it out when we need it , bytes=binfile.read()

and then decoded into python variables through struct.unpack():

a,b,c,d=struct.unpack('5s6sif',bytes)

'5s6sif' is called fmt, which is a formatted string, consisting of numbers and characters, 5s means 5 characters The string, 2i, represents 2 integers, etc. The following are the available characters and types. The ctype representation can correspond to the types in python one-to-one.

Note: Problems encountered when processing binary files

When we process binary files, we need to use the following method:

binfile=open(filepath,'rb')    
#读二进制文件
binfile=open(filepath,'wb')   
#写二进制文件

So what is the difference between the result of binfile=open(filepath,'r')?

There are two differences:

First, if you encounter '0x1A' when using 'r', it will be regarded as the end of the file, which is EOF. Using 'rb' does not have this problem. That is, if you write in binary and read out in text, only part of the file will be read out if '0X1A' is present. When using 'rb', it will read to the end of the file.

Second, for the string x=’abcndef’, we can use len(x) to get its length to be 7. We call n the newline character, which is actually ‘0X0A’. When we write in 'w', which is text mode, '0X0A' will be automatically changed into two characters '0X0D', '0X0A' on the Windows platform, that is, the file length actually becomes 8. When reading in 'r' text mode, it is automatically converted to the original newline character. If you change to 'wb' binary mode to write, one character will remain unchanged, and it will be read as it is when reading. So if you write in text mode and read in binary mode, you have to consider this extra byte. '0X0D' is also called the carriage return character. It will not change under Linux. Because linux only uses '0X0A' to represent line breaks.