Tutorial on how to operate byte streams/binary streams using the struct module in Python

Recently, I have been learning python network programming. When writing simple socket communication code, I encountered the use of the struct module. At the time, I was not sure what it did and what it does. Later, I checked the relevant information and roughly understood it. This article The article mainly introduces the operation of byte stream/binary stream by the struct module in Python. Friends in need can refer to it.

Preface

Recently using Python to parse the MNIST data set in IDX file format, I need to read the binary file, in which I use is the struct module. I checked a lot of tutorials on the Internet and they were all very good, but not very friendly to novices, so I reorganized some notes to help you get started quickly.

Note: The following four nouns in the tutorial are synonymous: binary stream, binary array, byte stream, byte array

Get started quickly

In the struct module, convert an integer number, floating point number or character stream (character array) into a byte stream (word section array), you need to use the format string fmt to tell the struct module what type of object to be converted, for example, an integer number is 'i', a floating point number is 'f', and an ascii code character is 's'.

def demo1():
 # 使用bin_buf = struct.pack(fmt, buf)将buf为二进制数组bin_buf
 # 使用buf = struct.unpack(fmt, bin_buf)将bin_buf二进制数组反转换回buf

 # 整型数 -> 二进制流
 buf1 = 256
 bin_buf1 = struct.pack('i', buf1) # 'i'代表'integer'
 ret1 = struct.unpack('i', bin_buf1)
 print bin_buf1, ' <====> ', ret1

 # 浮点数 -> 二进制流
 buf2 = 3.1415
 bin_buf2 = struct.pack('d', buf2) # 'd'代表'double'
 ret2 = struct.unpack('d', bin_buf2)
 print bin_buf2, ' <====> ', ret2

 # 字符串 -> 二进制流
 buf3 = 'Hello World'
 bin_buf3 = struct.pack('11s', buf3) # '11s'代表长度为11的'string'字符数组
 ret3 = struct.unpack('11s', bin_buf3)
 print bin_buf3, ' <====> ', ret3

 # 结构体 -> 二进制流
 # 假设有一个结构体
 # struct header {
 # int buf1;
 # double buf2;
 # char buf3[11];
 # }
 bin_buf_all = struct.pack('id11s', buf1, buf2, buf3)
 ret_all = struct.unpack('id11s', bin_buf_all)
 print bin_buf_all, ' <====> ', ret_all

The output results are as follows:

demo1 output results

Detailed explanation of struct module

Main functions

The three most important functions in the struct module arepack() , unpack() , calcsize()

# 按照给定的格式化字符串，把数据封装成字符串(实际上是类似于c结构体的字节流)
string = struct.pack(fmt, v1, v2, ...)

# 按照给定的格式(fmt)解析字节流string，返回解析出来的tuple
tuple = unpack(fmt, string)

# 计算给定的格式(fmt)占用多少字节的内存
offset = calcsize(fmt)

Format string in struct

The supported formats in struct are as follows:

##FormatC Type PythonNumber of bytesxpad byteno value 1ccharstring of length 11bsigned charinteger1Bunsigned charinteger1?_Boolbool1hshortinteger2Hunsigned short integer2iintinteger4Iunsigned intinteger or lon4llonginteger4Lunsigned longlong4qlong longlong8Qunsigned long longlong8##fds##pchar[]string1Pvoid *long　 Note 1: q and Q are only interesting when the machine supports 64-bit operation

float	float	4
double	float	8
char[]	string	1

Note 2: There can be a number before each format to indicate the individual Number

Note 3: The s format represents a string of a certain length, 4s represents a string of length 4, but p represents a pascal string

Note 4 :P is used to convert a pointer, its length is related to the machine word length

Note 5: The last one can be used to represent the pointer type, occupying 4 bytes

In order to exchange data with the structure in c, it is also necessary to consider that some c or c++ compilers use byte alignment, usually 32-bit systems with 4 bytes, so the struct is converted according to the local machine byte order .You can use the first character in the format to change the alignment. It is defined as follows:

##CharacterByte order@nativenativelittle-endianbig-endian##!network (= big-endian)standard According to the original number of bytesFor more tutorials on the operation of byte stream/binary stream by the struct module in Python, please pay attention to PHP Chinese net!

Size and alignment
native Make up enough 4 bytes		=
standard according to the original number of bytes
standard According to the original number of bytes		>
standard According to the original number of bytes Number
		The method of use is to put it in the first position of fmt, just like '@5s6sif'