How Do I Remove Invalid UTF-8 Characters from a Go String?

Eliminating Invalid UTF-8 Characters in a String in Go

Encoding/decoding data using UTF-8 ensures compatibility across different systems and character sets. However, invalid UTF-8 characters can arise due to transmission errors, malicious attacks, or other factors. Removing these characters is essential for data integrity and proper JSON encoding.

Solution:

To address this issue in Go, there are several approaches available:

1. Go 1.13 :

   • In Go 1.13 and later versions, the strings.ToValidUTF8 function provides a convenient solution. It takes a string and a replacement character as arguments and returns a copy of the string with invalid UTF-8 characters replaced by the specified character.

   • Example:

     fixedString := strings.ToValidUTF8("a\xc5z", "")

     Copy after login

2. Go 1.11 :

   • In Go 1.11 and later versions, you can use the strings.Map function in conjunction with utf8.RuneError to remove invalid UTF-8 characters. The strings.Map function applies a mapping function to each character in a string, and utf8.RuneError is a constant representing an invalid UTF-8 rune.

   • Example:

     fixUtf := func(r rune) rune {
         if r == utf8.RuneError {
             return -1
         }
         return r
     }
     
     fixedString := strings.Map(fixUtf, "a\xc5z")

     Copy after login

The above is the detailed content of How Do I Remove Invalid UTF-8 Characters from a Go String?. For more information, please follow other related articles on the PHP Chinese website!