How to Solve Python's 'UnicodeDecodeError: 'ascii' codec can't decode byte' Error?

How to fix "UnicodeDecodeError: 'ascii' codec can't decode byte"

tl;dr / Quick Fix

• Avoid unnecessary decoding/encoding.
• Don't assume UTF-8 encoding for strings.
• Convert strings to Unicode strings as soon as possible in your code.
• Adjust your locale (see: How to solve UnicodeDecodeError in Python 3.6?).
• Resist the temptation for quick reload hacks.

Unicode Zen in Python 2.x

UnicodeDecodeError: 'ascii' codec can't decode byte usually occurs when you attempt to convert a Python 2.x str containing non-ASCII characters to a Unicode string without specifying the original string's encoding.

Unicode strings (also known as unicodes) are a separate string type in Python that holds Unicode point codes and can represent any Unicode point throughout the spectrum. In contrast, strings contain encoded text in various formats (e.g., UTF-8, UTF-16, ISO-8895-1).

The Markdown module developers likely use unicode() as a quality gate to ensure incoming strings are Unicode. Since they cannot determine the encoding of the incoming string, you must decode it before passing it to Markdown.

Unicode strings can be declared in your code with the "u" prefix:

Unicode strings can also arise from files, databases, or network modules, where you do not need to specify the encoding.

Gotchas

Unicode conversion can occur even without explicit unicode() calls:

Examples

In the following diagram, "café" is encoded differently in "UTF-8" and "Cp1252" depending on the terminal type. In both cases, "caf" is encoded in plain ASCII. While UTF-8 uses two bytes to represent "é," Cp1252 uses a single byte that also happens to match the Unicode point value. In this case, decode() is invoked with the correct encoding and a successful conversion to Unicode is performed:

[Diagram of a successful Unicode conversion with the correct encoding]

However, if decode() is called with "ascii", which is similar to calling unicode() without specifying an encoding, a UnicodeDecodeError will occur:

[Diagram of an unsuccessful Unicode conversion with the wrong encoding]

The Unicode Sandwich

It is best practice to create a "Unicode sandwich" in your code, where you:

1. Decode all incoming data to Unicode strings.
2. Perform operations on Unicode strings.
3. Encode to str on the way out.

This approach prevents you from having to worry about string encoding throughout your code.

Input / Decode

• For source code, use Unicode string literals (e.g., u'Zürich') and add an encoding header (e.g., # encoding: utf-8).

• For files, use the io module's TextWrapper with the appropriate encoding:

• For databases, configure the connection to return Unicode data and use Unicode strings for SQL queries.
• For HTTP, consider using the Python Requests library, which returns Unicode in response.text.
• For manual decoding, use my_string.decode(encoding), where encoding is the appropriate value.

Output

• stdout/printing: Python attempts to configure an encoder forstdout to encode Unicode strings to the console's encoding. If the console's encoding is incorrect, you may encounter errors.
• Files: io.open can encode Unicode to byte strings transparently.
• Databases: Proper configuration allows you to write Unicode data directly to the database.

The above is the detailed content of How to Solve Python's 'UnicodeDecodeError: 'ascii' codec can't decode byte' Error?. For more information, please follow other related articles on the PHP Chinese website!