Details and usage notes of char and String in Java

Note on the use of char data type

In Java, the char data type is used to represent characters, but the char type cannot represent all characters.

Unicode character set

First of all, we need to know that we use the Unicode character set in Java. Before its appearance, there were many character sets, such as ANSI, GB2312, etc. Since there are many character sets with different standards, this leads to two problems:

• For any given code value, different characters may correspond to different characters in different character sets;

• Using large character sets Languages ​​may have different encoding lengths, such as single-byte encoding for common characters and multi-byte encoding for other characters.

The emergence of the Unicode character set is to unify encoding and eliminate the above problems. The so-called character set is a collection composed of many different characters. The Unicode character set assigns a unique code point to each character to identify the character itself. The so-called code point is a hexadecimal integer with the U+ prefix added. For example, the code point of the letter A is U+0041.

With the Unicode character set, what we have to consider is how to transmit and store these characters. This is how Unicode encoding is implemented, which we call Unicode Transformation Format (UTF). The familiar UTF-8, UTF-16, etc. are different Unicode encoding implementation methods.

At the beginning of the birth of the Unicode character set, the fixed-length encoding method UCS-2 (2-byte Universal Character Set) was used to encode the Unicode character set. This method uses a length of 16 bits for character encoding, so Up to 2^16 = 65536 characters can be encoded (encoding range from U+0000 ~ U+FFFF). Under the circumstances, the designers used less than half of the number to encode all characters, and believed that the remaining space was sufficient for encoding new characters in the future.

Unfortunately, with the continuous addition of Chinese, Japanese, Korean and other ideographic characters, the number of characters in the Unicode character set quickly exceeded the maximum number of characters that can be encoded by 16 bits, so designers made changes to the Unicode character set. New design.

The new design divides all characters in the character set into 17 code planes. Among them, the code point range U+0000 ~ U+FFFF is designated as the Basic Multilingual Plane (abbreviated as BMP). The remaining characters are divided into 16 auxiliary planes (Supplementary Plane). The code point range is U +10000 ~ U+10FFFF, these characters in the auxiliary plane are called supplementary characters.

After the characters in the Unicode character set are reclassified into different planes, you need to pay attention to the following two aspects:

• The characters in the BMP range are basically the same as the character encoding under UCS-2, but the U+ in BMP The D800 ~ U+DFFF part is left blank and is not assigned to any characters. It is used to encode characters in the auxiliary plane.

• Not every position in each plane is assigned to the specified character, the reason is:

• Special purposes, such as the U+D800 ~ U+DFFF part in BMP;

• As reserved space

• There are not enough characters

UTF-16

UTF-16 also uses 16 bit encoding to represent Unicode characters, that is to say UTF-16 code unit (code unit) is 16 bits. Code unit refers to the most basic unit of character encoding, that is, any character must be composed of n (n≥1) code units.

Under UTF-16, since the 16-bit length can only represent 65536 characters, all characters within the BMP range are mapped by default, so the U+D800 ~ U+DFFF part is left blank, then the auxiliary plane Characters can also be expressed with the help of this blank part. This is the ingenuity of UTF-16 design, which solves the encoding problem of all characters without wasting space.

So how to express the characters of the auxiliary plane? In fact, the code points of the auxiliary plane characters are encoded into a pair of 16-bit long code units, called a surrogate pair, and the surrogate pair must fall in the U+D800 ~ U+DFFF part of the BMP. This solves the problem of encoding the entire Unicode character set with 16-bit code units. It should be noted that the part U+D800 ~ U+DFFF can be called the agent area, among which the part U+D800 ~ U+DBFF is called the high-level agent area (leading agent area), and the part U+DC00 ~ U+DFFF It is called the low-level agent area (the rear agent area).

The following explains the encoding method of auxiliary plane characters through an example of UTF-16 encoding U+64321, a character in the auxiliary plane.

First subtract 0x10000 from the code point of this character to obtain a value with a length of 20 bits. The range of this value must be within 0x0000 ~ 0xFFFF.

Use the value of the high-order 10 bits of Vx as the operation base Vh of the high-order agent, and use the value of the low-order 10 bits as the operation base Vl of the low-order agent. The value range of these two 10-bit values ​​must be between 0x0000 ~ 0x3FF.

Perform a bitwise OR operation on Vh and Vl with the code points at the starting position of the high-order surrogate area and low-order surrogate area respectively. The result is the UTF-16 encoding of the character U+64321 in the auxiliary plane.

So in the end the character U+64321 is encoded into a surrogate pair consisting of a high-order surrogate and a low-order surrogate. We need to use both 0xD950 and 0xDF21 to represent this character.

Through the above example, we can see that any character in the auxiliary plane will be encoded as a surrogate pair consisting of two 16-bit surrogate codes under UTF-16. When representing this character in the program, you need It no longer takes up 16 bits of space, but 32 bits.

It is not recommended to use the char data type in Java programs

After the above explanation of the Unicode character set and UTF-16, we will now discuss why it is not recommended to use the char data type in Java programs.

Since Java uses a 16-bit Unicode character set, that is, UTF-16, the char data type in Java is fixed-length, and its length is always only 16 bits. The char data type can only represent code points in U+ Characters between 0000 ~ U+FFFF, that is, characters in BMP. If the code point exceeds this range, even if supplementary characters are used, the char data type will not be supported, because the supplementary characters require a length of 32 bits to store, and we can only use String to store this character.

The code written above uses the char data type to save the characters of the auxiliary plane, and the compiler will report an error of Invalid character constant.

With the continuous increase of Internet users and the continuous enrichment of Internet languages, users are increasingly using some special characters on the Internet to express rich semantics, and these characters are likely to be supplementary characters in the auxiliary plane. , so if we use the char type for processing, it is likely to reduce the robustness of our program.

String details

Get the string length

String is a data type that we use a lot in programming, it is used to represent a string. Looking at the source code of String, we can see that the underlying layer actually uses a char type array to store our characters.

We also know that calling its length() method can get the length of the string, that is, the number of characters in the string. Its implementation is to directly return the length of the underlying value array. The code is as follows:

Combined with our knowledge of character encoding above, we know that the length of char in Java is always 16 bits. If we use supplementary characters in the string, it means that we need 2 char type lengths to store. For the String bottom layer For the array value that stores characters, 2 array element positions are needed. So in the following program we will get an unexpected result:

According to our idea, there should be only 8 characters in the string tt, but the actual output is 9. We have already mentioned above that Java uses a 16-bit Unicode character set, so the length of a code unit in Java is also 16 bits. A supplementary character requires two code units to represent, so the tt character in the string