Problem with Python parsing abnormal bytes of Socket data stream (details)

The content of this article is about the problem of Python parsing abnormal bytes of Socket data flow (details). It has certain reference value. Friends in need can refer to it. I hope it will be helpful to you.

When python sends data through socket, English characters are escaped into their original characters, occupying one byte (for example: s is transferred to s after being transferred), while Chinese characters require two characters after being escaped. section to identify a Chinese character (for example: bell is \x92\x9f after escape). There is no problem on the sending end, but the problem mainly occurs on the socket client. Because when the client accepts the data stream, there is a byte limit for each acceptance, and a Chinese character will be accepted twice, causing a UnicodeDecodeError to be reported each time the accepted stream is escaped into a character.

Currently there is data flow:

bmsg = b'\xe5\x88\x86\n\xe9\x92\x9f' # 分\n钟

Normal acceptance and escape to:

smsg = str(bmsg, 'utf-8')   #方式一 或 
smsg = bmsg.decode()  # 方式二 第一参数默认utf8，第二参数默认strict，还有 ignore （忽略）、 replace （替代=?）

If the data stream received now is

bmsg = b'\xe5\x88\x86\n\xe9\x92'    # \x9f 作为下次接受

How to avoid the abnormal exit of the program, there are two ways to deal with it:
1) Data loss processing
At this time, if the bytes stream is escaped using the normal acceptance method, a UnicodeDecodeError exception will occur. In order not to report the exception, we use method 2 to specify the second one when escaping. The parameter is ignore, as follows:

smsg = bmsg.decode('utf-8', 'ignore') # 输出： 分\n  ，如果为 replace 则 \n 后为 ?

2) Split the data stream and then process it
Generally, when the socket server sends data, each data stream It is complete and will be sent together with the end of a specific character (such as: \n). Based on this current situation, every time we receive data, we split it according to this specific character into a list that saves a single data stream. The first stream in the list may be incomplete. Merge it with the last accepted stream into a complete data stream and escape it; the middle stream is a data stream that can be escaped normally; the last stream is also incomplete, so save it. , used to splice the stream accepted next time and repeat the above operation. This ensures that the program will not exit abnormally and the integrity of the data is ensured. The approximate code is as follows:
Data acceptance:

The first time msg1 = b'\xe5\x88\x86\n\xe9' is accepted; # The clock is split into \xe9 and \x92\x9f
Accept msg2 = b'\x92\x9f_stone\n'

init_msg = b'' # 初始化流
　　while True:
　　　　msg = soc.recv(128) # 接受数据
　　　　init_msg += msg
　　　　msg_arr = init_msg.split(b'\n') # 注意此处的 b'\n' ，因为被拆分的为bytes串，所以也要用bytes串来拆分
　　　　init_msg += msg_arr[-1]
　　　　msg_arr.remove(msg_arr[-1])
　　　　for i in range(len(msg_arr)):
　　　　　　string = msg_arr[i].decode('utf-8')
　　　　　　print(string

The above is the detailed content of Problem with Python parsing abnormal bytes of Socket data stream (details). For more information, please follow other related articles on the PHP Chinese website!