Python built-in type str source code analysis

1 Unicode

The basic unit of computer storage is the byte, which is composed of 8 bits. Since English only consists of 26 letters plus a number of symbols, English characters can be stored directly in bytes. But other languages ​​(such as Chinese, Japanese, Korean, etc.) have to use multiple bytes for encoding due to the large number of characters.

With the spread of computer technology, non-Latin character encoding technology continues to develop, but there are still two major limitations:

• Does not support multiple languages: The encoding scheme of one language cannot be used for another language

• There is no unified standard: for example, Chinese has multiple encoding standards such as GBK, GB2312, GB18030

Because the encoding methods are not uniform, developers need to convert back and forth between different encodings, and many errors will inevitably occur. In order to solve this kind of inconsistency problem, the Unicode standard was proposed. Unicode organizes and encodes most of the writing systems in the world, allowing computers to process text in a unified way. Unicode currently contains more than 140,000 characters and naturally supports multiple languages. (Unicode’s uni is the root of “unification”)

2 Unicode in Python

2.1 Benefits of Unicode objects

After Python 3, Unicode is used internally in the str object Represents, and therefore becomes a Unicode object in the source code. The advantage of using Unicode representation is that the core logic of the program uses Unicode uniformly, and only needs to be decoded and encoded at the input and output layers, which can avoid various encoding problems to the greatest extent.

The diagram is as follows:

##2.2 Python’s optimization of Unicode

Problem: Since Unicode contains more than 140,000 characters, each A character requires at least 4 bytes to save (this is probably because 2 bytes are not enough, so 4 bytes are used, and 3 bytes are generally not used). The ASCII code for English characters requires only 1 byte. Using Unicode will quadruple the cost of frequently used English characters.

First of all, let’s take a look at the size difference of different forms of str objects in Python:

>>> sys.getsizeof('ab') - sys.getsizeof('a')
1
>>> sys.getsizeof('一二') - sys.getsizeof('一')
2
>>> sys.getsizeof('????????') - sys.getsizeof('????')
4

It can be seen that Python internally optimizes Unicode objects: according to the text content, the underlying storage unit is selected .

The underlying storage of Unicode objects is divided into three categories according to the Unicode code point range of text characters:

• PyUnicode_1BYTE_KIND: All character code points are between U 0000 and U 00FF

• PyUnicode_2BYTE_KIND: All character code points are between U 0000 and U FFFF, and at least one character has a code point greater than U 00FF

• PyUnicode_1BYTE_KIND: All character code points are between U 0000 and U 10FFFF, and at least one character has a code point greater than U FFFF

##The corresponding enumeration is as follows:

enum PyUnicode_Kind {
/* String contains only wstr byte characters.  This is only possible
   when the string was created with a legacy API and _PyUnicode_Ready()
   has not been called yet.  */
    PyUnicode_WCHAR_KIND = 0,
/* Return values of the PyUnicode_KIND() macro: */
    PyUnicode_1BYTE_KIND = 1,
    PyUnicode_2BYTE_KIND = 2,
    PyUnicode_4BYTE_KIND = 4
};

According to different Classification, select different storage units:

/* Py_UCS4 and Py_UCS2 are typedefs for the respective
   unicode representations. */
typedef uint32_t Py_UCS4;
typedef uint16_t Py_UCS2;
typedef uint8_t Py_UCS1;

The corresponding relationship is as follows:

Text typePyUnicode_1BYTE_KINDPyUnicode_2BYTE_KIND PyUnicode_4BYTE_KINDSince the Unicode internal storage structure varies depending on the text type, the type kind must be saved as a Unicode object public field. Python internally defines some flag bits as Unicode public fields: (Due to the author's limited level, all the fields here will not be introduced in the subsequent content. You can learn about it yourself later. Hold your fist~)

Character storage unit	Character storage unit size (bytes)
Py_UCS1	1
Py_UCS2	2
Py_UCS4	4

interned: Whether to maintain the interned mechanism
kind: type, used to distinguish the size of the underlying storage unit of characters
compact: memory allocation method, whether the object and the text buffer are separated
asscii: Whether the text is all pure ASCII
Through the PyUnicode_New function, according to the number of text characters size and the maximum character maxchar initializes the Unicode object. This function mainly selects the most compact character storage unit and underlying structure for Unicode objects based on maxchar: (The source code is relatively long, so it will not be listed here. You can understand it by yourself. It is shown in table form below)

##kindPyUnicode_1BYTE_KINDPyUnicode_1BYTE_KIND PyUnicode_2BYTE_KINDPyUnicode_4BYTE_KINDascii1000Character storage unit size (bytes) 1124Underlying structurePyASCIIObjectPyCompactUnicodeObjectPyCompactUnicodeObjectPyCompactUnicodeObject

3 Unicode对象的底层结构体

3.1 PyASCIIObject

C源码：

typedef struct {
    PyObject_HEAD
    Py_ssize_t length;          /* Number of code points in the string */
    Py_hash_t hash;             /* Hash value; -1 if not set */
    struct {
        unsigned int interned:2;
        unsigned int kind:3;
        unsigned int compact:1;
        unsigned int ascii:1;
        unsigned int ready:1;
        unsigned int :24;
    } state;
    wchar_t *wstr;              /* wchar_t representation (null-terminated) */
} PyASCIIObject;

源码分析：

length：文本长度

hash：文本哈希值

state：Unicode对象标志位

wstr：缓存C字符串的一个wchar_t指针，以“\0”结束（这里和我看的另一篇文章讲得不太一样，另一个描述是：ASCII文本紧接着位于PyASCIIObject结构体后面，我个人觉得现在的这种说法比较准确，毕竟源码结构体后面没有别的字段了）

图示如下：

（注意这里state字段后面有一个4字节大小的空洞，这是结构体字段内存对齐造成的现象，主要是为了优化内存访问效率）

ASCII文本由wstr指向，以’abc’和空字符串对象’'为例：

3.2 PyCompactUnicodeObject

如果文本不全是ASCII，Unicode对象底层便由PyCompactUnicodeObject结构体保存。C源码如下：

/* Non-ASCII strings allocated through PyUnicode_New use the
   PyCompactUnicodeObject structure. state.compact is set, and the data
   immediately follow the structure. */
typedef struct {
    PyASCIIObject _base;
    Py_ssize_t utf8_length;     /* Number of bytes in utf8, excluding the
                                 * terminating \0. */
    char *utf8;                 /* UTF-8 representation (null-terminated) */
    Py_ssize_t wstr_length;     /* Number of code points in wstr, possible
                                 * surrogates count as two code points. */
} PyCompactUnicodeObject;

PyCompactUnicodeObject在PyASCIIObject的基础上增加了3个字段：

utf8_length：文本UTF8编码长度

utf8：文本UTF8编码形式，缓存以避免重复编码运算

wstr_length：wstr的“长度”（这里所谓的长度没有找到很准确的说法，笔者也不太清楚怎么能打印出来，大家可以自行研究下）

注意到，PyASCIIObject中并没有保存UTF8编码形式，这是因为ASCII本身就是合法的UTF8，这也是ASCII文本底层由PyASCIIObject保存的原因。

结构图示：

3.3 PyUnicodeObject

PyUnicodeObject则是Python中str对象的具体实现。C源码如下：

/* Strings allocated through PyUnicode_FromUnicode(NULL, len) use the
   PyUnicodeObject structure. The actual string data is initially in the wstr
   block, and copied into the data block using _PyUnicode_Ready. */
typedef struct {
    PyCompactUnicodeObject _base;
    union {
        void *any;
        Py_UCS1 *latin1;
        Py_UCS2 *ucs2;
        Py_UCS4 *ucs4;
    } data;                     /* Canonical, smallest-form Unicode buffer */
} PyUnicodeObject;

3.4 示例

在日常开发时，要结合实际情况注意字符串拼接前后的内存大小差别：

>>> import sys
>>> text = 'a' * 1000
>>> sys.getsizeof(text)
1049
>>> text += '????'
>>> sys.getsizeof(text)
4080

4 interned机制

如果str对象的interned标志位为1，Python虚拟机将为其开启interned机制，

源码如下：（相关信息在网上可以看到很多说法和解释，这里笔者能力有限，暂时没有找到最确切的答案，之后补充。抱拳~但是我们通过分析源码应该是能看出一些门道的）

/* This dictionary holds all interned unicode strings.  Note that references
   to strings in this dictionary are *not* counted in the string's ob_refcnt.
   When the interned string reaches a refcnt of 0 the string deallocation
   function will delete the reference from this dictionary.
   Another way to look at this is that to say that the actual reference
   count of a string is:  s->ob_refcnt + (s->state ? 2 : 0)
*/
static PyObject *interned = NULL;
void
PyUnicode_InternInPlace(PyObject **p)
{
    PyObject *s = *p;
    PyObject *t;
#ifdef Py_DEBUG
    assert(s != NULL);
    assert(_PyUnicode_CHECK(s));
#else
    if (s == NULL || !PyUnicode_Check(s))
        return;
#endif
    /* If it's a subclass, we don't really know what putting
       it in the interned dict might do. */
    if (!PyUnicode_CheckExact(s))
        return;
    if (PyUnicode_CHECK_INTERNED(s))
        return;
    if (interned == NULL) {
        interned = PyDict_New();
        if (interned == NULL) {
            PyErr_Clear(); /* Don't leave an exception */
            return;
        }
    }
    Py_ALLOW_RECURSION
    t = PyDict_SetDefault(interned, s, s);
    Py_END_ALLOW_RECURSION
    if (t == NULL) {
        PyErr_Clear();
        return;
    }
    if (t != s) {
        Py_INCREF(t);
        Py_SETREF(*p, t);
        return;
    }
    /* The two references in interned are not counted by refcnt.
       The deallocator will take care of this */
    Py_REFCNT(s) -= 2;
    _PyUnicode_STATE(s).interned = SSTATE_INTERNED_MORTAL;
}

可以看到，源码前面还是做一些基本的检查。我们可以看一下37行和50行：将s添加到interned字典中时，其实s同时是key和value（这里我不太清楚为什么会这样做），所以s对应的引用计数是+2了的（具体可以看PyDict_SetDefault()的源码），所以在50行时会将计数-2，保证引用计数的正确。

考虑下面的场景：

>>> class User:
    def __init__(self, name, age):
        self.name = name
        self.age = age
>>> user = User('Tom', 21)
>>> user.__dict__
{'name': 'Tom', 'age': 21}

由于对象的属性由dict保存，这意味着每个User对象都要保存一个str对象‘name’，这会浪费大量的内存。而str是不可变对象，因此Python内部将有潜在重复可能的字符串都做成单例模式，这就是interned机制。Python具体做法就是在内部维护一个全局dict对象，所有开启interned机制的str对象均保存在这里，后续需要使用的时候，先创建，如果判断已经维护了相同的字符串，就会将新创建的这个对象回收掉。

示例：

由不同运算生成’abc’，最后都是同一个对象：

>>> a = 'abc'
>>> b = 'ab' + 'c'
>>> id(a), id(b), a is b
(2752416949872, 2752416949872, True)

maxchar	128	256	65536

The above is the detailed content of Python built-in type str source code analysis. For more information, please follow other related articles on the PHP Chinese website!