UTF-8: Difference between revisions

Latest revision as of 12:56, 20 November 2025

Template:Short description Template:Infobox character encoding

UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode Transformation FormatTemplate:Snd 8-bit.^[1] As of July 2025, almost every webpage is transmitted as UTF-8.^[2]

UTF-8 supports all 1,112,064^[3] valid Unicode code points using a variable-width encoding of one to four one-byte (8-bit) code units.

Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. It was designed for backward compatibility with ASCII: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that a UTF-8-encoded file using only those characters is identical to an ASCII file. Most software designed for any extended ASCII can read and write UTF-8, and this results in fewer internationalization issues than any alternative text encoding.^[4]^[5]

UTF-8 is dominant for all countries/languages on the internet, is used in most standards, often the only allowed encoding, and is supported by all modern operating systems and programming languages.

History

Script error: No such module "Labelled list hatnote".

The International Organization for Standardization (ISO) set out to compose a universal multi-byte character set in 1989. The draft ISO 10646 standard contained a non-required annex called UTF-1 that provided a byte stream encoding of its 32-bit code points. This encoding was not satisfactory on performance grounds, among other problems, and the biggest problem was probably that it did not have a clear separation between ASCII and non-ASCII: new UTF-1 tools would be backward compatible with ASCII-encoded text, but UTF-1-encoded text could confuse existing code expecting ASCII (or extended ASCII), because it could contain continuation bytes in the range Template:Mono–Template:Mono that meant something else in ASCII, e.g., Template:Mono for /, the Unix path directory separator.

In July 1992, the X/Open committee XoJIG was looking for a better encoding. Dave Prosser of Unix System Laboratories submitted a proposal for one that had faster implementation characteristics and introduced the improvement that 7-bit ASCII characters would only represent themselves; multi-byte sequences would only include bytes with the high bit set. The name File System Safe UCS Transformation Format (FSS-UTF)^[6] and most of the text of this proposal were later preserved in the final specification.^[7]^[8]^[9] In August 1992, this proposal was circulated by an IBM X/Open representative to interested parties.

A modification by Ken Thompson of the Plan 9 operating system group at Bell Labs made it self-synchronizing, letting a reader start anywhere and immediately detect character boundaries, at the cost of being somewhat less bit-efficient than the previous proposal. It also abandoned the use of biases that prevented overlong encodings.^[9]^[10] Thompson's design was outlined on September 2, 1992, on a placemat in a New Jersey diner with Rob Pike. In the following days, Pike and Thompson implemented it and updated Plan 9 to use it throughout,^[11] and then communicated their success back to X/Open, which accepted it as the specification for FSS-UTF.^[9] UTF-8 was first officially presented at the USENIX conference in San Diego, from January 25 to 29, 1993.^[12] The Internet Engineering Task Force adopted UTF-8 in its Policy on Character Sets and Languages in RFC 2277 (BCP 18) for future internet standards work in January 1998, replacing Single Byte Character Sets such as Latin-1 in older RFCs.^[13]

In November 2003, UTF-8 was restricted by Template:IETF RFC to match the constraints of the UTF-16 character encoding: explicitly prohibiting code points corresponding to the high and low surrogate characters removed more than 3% of the three-byte sequences, and ending at Template:Tt removed more than 48% of the four-byte sequences and all five- and six-byte sequences.^[14]

Description

UTF-8 encodes code points in one to four bytes, depending on the value of the code point. In the following table, the characters Template:Mono to Template:Mono are replaced by the bits of the code point, from the positions Template:Mono:

Code point ↔ UTF-8 conversion
First code point	Last code point	Byte 1	Byte 2	Byte 3	Byte 4
Template:Tt	Template:Tt	Template:Mono
Template:Tt	Template:Tt	Template:Mono	Template:Mono
Template:Tt	Template:Tt	Template:Mono	Template:Mono	Template:Mono
Template:Tt	Template:Tt	Template:Mono	Template:Mono	Template:Mono	Template:Mono

The first 128 code points (ASCII) need 1 byte. The next 1,920 code points need two bytes to encode, which covers the remainder of almost all Latin-script alphabets, and also IPA extensions, Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic, Syriac, Thaana and N'Ko alphabets, as well as Combining Diacritical Marks. Three bytes are needed for the remaining 61,440 codepoints of the Basic Multilingual Plane (BMP), including most Chinese, Japanese and Korean characters. Four bytes are needed for the 1,048,576 non-BMP code points, which include emoji, less common CJK characters, and other useful characters.^[15]

UTF-8 is a prefix code and it is unnecessary to read past the last byte of a code point to decode it. Unlike many earlier multi-byte text encodings such as Shift-JIS, it is self-synchronizing so searches for short strings or characters are possible; and the start of a code point can be found from a random position by backing up at most 3 bytes. The values chosen for the lead bytes means sorting a list of UTF-8 strings puts them in the same order as sorting UTF-32 strings.

Overlong encodings

Script error: No such module "anchor".

Using a row in the above table to encode a code point less than "First code point" (thus using more bytes than necessary) is termed an overlong encoding. These are a security problem because they allow character sequences such as malicious JavaScript and ../ to bypass security validations, which has been reported in numerous high-profile products such as Microsoft's IIS web server^[16] and Apache's Tomcat servlet container.^[17] Overlong encodings should therefore be considered an error and never decoded. Modified UTF-8 allows the two-byte overlong encoding Template:Mono, Template:Mono, for Template:Tt instead of just the single-byte Template:Mono.^[18]

Error handling

Not all sequences of bytes are valid UTF-8. A UTF-8 decoder should be prepared for:

A "continuation byte" (Template:Tt Template:Ndash Template:Tt) at the start of a character
A non-continuation byte (or the string ending) before the end of a character
An overlong encoding (Template:Tt, Template:Tt, Template:Tt followed by less than Template:Tt, or Template:Tt followed by less than Template:Tt)
A 4-byte sequence that decodes to a value greater than Template:Tt (Template:Tt followed by Template:Tt or greater, Template:Tt Template:Ndash Template:Tt)

Many of the first UTF-8 decoders would decode these, ignoring incorrect bits. Carefully crafted invalid UTF-8 could make them either skip or create ASCII characters such as Template:Mono, slash, or quotes, leading to security vulnerabilities. It is also common to throw an exception or truncate the string at an error^[19] but this turns what would otherwise be harmless errors (i.e. "file not found") into a denial of service, for instance early versions of Python 3.0 would exit immediately if the command line or environment variables contained invalid UTF-8.^[20]

Template:Nobr states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."^[21] The Unicode Standard requires decoders to: "... treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence." The standard now recommends replacing each error with the replacement character "Template:Nat" (Template:Tt) and continue decoding.

Some decoders consider the sequence Template:Mono (a truncated 3-byte code followed by a space) as a single error. This is not a good idea as a search for a space character would find the one hidden in the error. Since Unicode 6 (October 2010)^[1] the standard (chapter 3) has recommended a "best practice" where the error is either one continuation byte, or ends at the first byte that is disallowed, so Template:Mono is a two-byte error followed by a space. An error is no more than three bytes long, never contains the start of a valid character, and there are Template:Val different possible errors. Many decoders instead make each byte be an error, in which case Template:Mono is two errors followed by a space; there are now only 128 different errors which makes it practical to store the errors in the output string,^[22] or replace them with characters from a legacy encoding.

Only a small subset of possible byte strings are error-free UTF-8: several bytes cannot appear; a byte with the high bit set cannot be alone; and in a truly random string a byte with a high bit set has only a <templatestyles src="Fraction/styles.css" />1⁄15 chance of starting a valid UTF-8 character. This has the consequence of making it easy to detect if a legacy text encoding is accidentally used instead of UTF-8, making conversion of a system to UTF-8 easier and avoiding the need to require a Byte Order Mark or any other metadata.

Surrogates

Since RFC 3629 (November 2003), the high and low surrogates used by UTF-16 (Template:Tt through Template:Tt) are not legal Unicode values, and their UTF-8 encodings must be treated as an invalid byte sequence.^[21] These encodings all start with Template:Tt followed by Template:Tt or higher. This rule is often ignored as surrogates are allowed in Windows filenames and this means there must be a way to store them in a string.^[23] UTF-8 that allows these surrogate halves has been (informally) called Template:Visible anchor,^[24] while another variation that also encodes all non-BMP characters as two surrogates (6 bytes instead of 4) is called CESU-8.

Byte map

The chart below gives the detailed meaning of each byte in a stream encoded in UTF-8. Template:UTF-8 byte map

Byte-order mark

If the Unicode byte-order mark Template:Tt is at the start of a UTF-8 file, the first three bytes will be Template:Mono, Template:Mono, Template:Mono.

The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file trans-coded from another encoding.^[25] While ASCII text encoded using UTF-8 is backward compatible with ASCII, this is not true when Unicode Standard recommendations are ignored and a BOM is added. A BOM can confuse software that isn't prepared for it but can otherwise accept UTF-8, e.g. programming languages that permit non-ASCII bytes in string literals but not at the start of the file. Nevertheless, there was and still is software that always inserts a BOM when writing UTF-8, and refuses to correctly interpret UTF-8 unless the first character is a BOM (or the file only contains ASCII).^[26]

Comparison to UTF-16

Script error: No such module "Labelled list hatnote". Script error: No such module "Unsubst".

For a long time there was considerable argument as to whether it was better to process text in UTF-16 or in UTF-8. The primary advantage of UTF-16 is that the Windows API required it for access to all Unicode characters (UTF-8 was not fully supported in Windows until May 2019). This caused several libraries such as Qt to also use UTF-16 strings which propagates this requirement to non-Windows platforms. In the early days of Unicode there were no characters greater than Template:Tt and combining characters were rarely used, so the 16-bit encoding was effectively fixed-size. Some believed fixed-size encoding could make processing more efficient, but any such advantages were lost as soon as UTF-16 became variable width as well. The code points Template:Tt Template:Ndash Template:Tt take 3 bytes in UTF-8 but only 2 in UTF-16. This led to the idea that text in Chinese and other languages would take more space in UTF-8. However, text is only larger if there are more of these code points than 1-byte ASCII code points, and this rarely happens in real-world documents due to spaces, newlines, digits, punctuation, English words, and markup^[27]. UTF-8 has the advantages of being trivial to retrofit to any system that could handle an extended ASCII, not having byte-order problems, and taking about half the space for any language using mostly Latin letters.

Implementations and adoption

File:UTF-8 takes over.png

Declared character set for the 10 million most popular websites from 2010 to 2021

File:Utf8webgrowth.svg

Use of the main encodings on the web from 2001 to 2012 as recorded by Google,^[28] with UTF-8 overtaking all others in 2008 and over 60% of the web in 2012 (since then hitting 100% use). UTF-8 is the only encoding of Unicode (explicitly) listed there, and the rest only provide subsets of Unicode. The ASCII-only figure includes all web pages that only contain ASCII characters, regardless of the declared header.

Script error: No such module "Labelled list hatnote".

UTF-8 has been the most common encoding for the World Wide Web since 2008.^[29] Template:As of, UTF-8 is used by 98.8% of surveyed web sites.^[2] Although many pages only use ASCII characters to display content, very few websites now declare their encoding to only be ASCII instead of UTF-8.^[30] Virtually all countries and languages have 95% or more use of UTF-8 encodings on the web.

Many standards only support UTF-8, e.g. JSON exchange requires it (without a byte-order mark (BOM)).^[31] UTF-8 is also the recommendation from the WHATWG for HTML and DOM specifications, and stating "UTF-8 encoding is the most appropriate encoding for interchange of Unicode"^[5] and the Internet Mail Consortium recommends that all e‑mail programs be able to display and create mail using UTF-8.^[32]^[33] The World Wide Web Consortium recommends UTF-8 as the default encoding in XML and HTML (and not just using UTF-8, also declaring it in metadata), "even when all characters are in the ASCII range ... Using non-UTF-8 encodings can have unexpected results".^[34]

Many software programs have the ability to read/write UTF-8. It may require the user to change options from the normal settings, or may require a BOM (byte-order mark) as the first character to read the file. Examples of software supporting UTF-8 include Microsoft Word,^[35]^[36]^[37] Microsoft Excel (Office 2003 and later),^[38]^[39] Google Drive, LibreOffice,^[40] and most databases.

Software that "defaults" to UTF-8 (meaning it writes it without the user changing settings, and it reads it without a BOM) has become more common since 2010.^[41] Windows Notepad, in all currently supported versions of Windows, defaults to writing UTF-8 without a BOM (a change from Template:Nobr Notepad), bringing it into line with most other text editors.^[42] Some system files on Windows 11 require UTF-8^[43] with no requirement for a BOM, and almost all files on macOS and Linux are required to be UTF-8 without a BOM.Script error: No such module "Unsubst". Programming languages that default to UTF-8 for I/O include Ruby 3.0,^[44]^[45] R 4.2.2,^[46] Raku and Java 18.^[47] Although the current version of Python requires an option to open() to read/write UTF-8,^[48] plans exist to make UTF-8 I/O the default in Python 3.15.^[49] C++23 adopts UTF-8 as the only portable source code file format.^[50]

Backwards compatibility is a serious impediment to changing code and APIs using UTF-16 to use UTF-8, but this is happening. Template:As of, Microsoft added the capability for an application to set UTF-8 as the "code page" for the Windows API, removing the need to use UTF-16; and more recently has recommended programmers use UTF-8,^[51] and even states "UTF-16 [...] is a unique burden that Windows places on code that targets multiple platforms".^[4] The default string primitive in Go,^[52] Julia, Rust, Swift (since version 5),^[53] and PyPy^[54] uses UTF-8 internally in all cases. Python (since version 3.3) uses UTF-8 internally for Python C API extensions^[55]^[56] and sometimes for strings^[55]^[57] and a future version of Python is planned to store strings as UTF-8 by default.^[58]^[59] Modern versions of Microsoft Visual Studio use UTF-8 internally.^[60] Microsoft's SQL Server 2019 added support for UTF-8, and using it results in a 35% speed increase, and "nearly 50% reduction in storage requirements."^[61]

Script error: No such module "anchor".Java internally uses UTF-16 for the char data type and, consequentially, the Character, String, and the StringBuffer classes,^[62] but for I/O uses Modified UTF-8 (MUTF-8), in which the null character Template:Tt uses the two-byte overlong encoding Template:Tt, Template:Tt, instead of just Template:Tt.^[18] Modified UTF-8 strings never contain any actual null bytes but can contain all Unicode code points including Template:Tt,^[63] which allows such strings (with a null byte appended) to be processed by traditional null-terminated string functions. Java reads and writes normal UTF-8 to files and streams,^[64] but it uses Modified UTF-8 for object serialization,^[65]^[66] for the Java Native Interface,^[67] and for embedding constant strings in Java class files.^[63] The dex format defined by Dalvik also uses the same modified UTF-8 to represent string values.^[68] Tcl also uses the same modified UTF-8^[69] as Java for internal representation of Unicode data, but uses strict CESU-8 for external data. All known Modified UTF-8 implementations also treat the surrogate pairs as in CESU-8.

The Raku programming language (formerly Perl 6) uses utf-8 encoding by default for I/O (Perl 5 also supports it); though that choice in Raku also implies "normalization into Unicode NFC (normalization form canonical). In some cases the user will want to ensure no normalization is done; for this utf8-c8" can be used.^[70] That UTF-8 Clean-8 variant, implemented by Raku, is an encoder/decoder that preserves bytes as is (even illegal UTF-8 sequences) and allows for Normal Form Grapheme synthetics.^[71]

Version 3 of the Python programming language treats each byte of an invalid UTF-8 bytestream as an error (see also changes with new UTF-8 mode in Python 3.7^[72]); this gives 128 different possible errors. Extensions have been created to allow any byte sequence that is assumed to be UTF-8 to be losslessly transformed to UTF-16 or UTF-32, by translating the 128 possible error bytes to 128 reserved code points, and transforming those code points back to error bytes to output UTF-8. The most common approach is to translate the codes to Template:Tt...Template:Tt which are low (trailing) surrogate values and thus "invalid" UTF-16, as used by Python's PEP 383 (or "surrogateescape") approach.^[22] NumPy version 2.0, and its file formats, support UTF-8 (adding StringDType for it).^[73] Another encoding called MirBSD OPTU-8/16 converts them to Template:Tt...Template:Tt in a Private Use Area.^[74] In either approach, the byte value is encoded in the low eight bits of the output code point. These encodings are needed if invalid UTF-8 is to survive translation to and then back from the UTF-16 used internally by Python, and as Unix filenames can contain invalid UTF-8 it is necessary for this to work.^[75]

Linux and macOS filesystems support UTF-8: ext4^[76] and Apple's APFS.^[77] Apple's older HFS Plus uses UTF-16 for file names, but uses UTF-8 in symbolic links.^[78] Windows' filesystem, NTFS, uses UTF-16 for file names.

Standards

The official name for the encoding is UTF-8, the spelling used in all Unicode Consortium documents. The hyphen-minus is required and no spaces are allowed. Some other names used are:

Most standards are also case-insensitive and utf-8 is often used.Script error: No such module "Unsubst".
Web standards (which include CSS, HTML, XML, and HTTP headers) also allow utf8 and many other aliases.^[79]
The official Internet Assigned Numbers Authority lists csUTF8 as the only alias,^[80] which is rarely used.
In some locales UTF-8N means UTF-8 without a byte-order mark (BOM), and in this case UTF-8 may imply there is a BOM.^[81]^[82]
In Windows, UTF-8 is codepage 65001^[83] with the symbolic name CP_UTF8 in source code.
In MySQL, UTF-8 is called utf8mb4,^[84] while utf8 and utf8mb3 refer to the obsolete CESU-8 variant.^[85]
In Oracle Database, AL32UTF8 means UTF-8 (since version 9.0), while UTF8 means CESU-8 (since 8.0),^[86] and Oracle's UTF8 encoding should not be used since it's only a subset of the full (UTF8/AL32UTF8; missing support for Japanese and Chinese, only supporting Unicode 3.0, not 4-byte form; AL32UTFFSS is another incomplete subset for older Oracle databases).^[87]
In HP PCL, the Symbol-ID for UTF-8 is 18N.^[88]

There are several current definitions of UTF-8 in various standards documents:

Template:IETF RFC / STD 63 (2003), which establishes UTF-8 as a standard internet protocol element
Template:IETF RFC defines UTF-8 NFC for Network Interchange (2008)
ISO/IEC 10646:2020/Amd 1:2023^[89]
The Unicode Standard, Version 17.0.0 (2025)

They supersede the definitions given in the following obsolete works:

The Unicode Standard, Version 2.0, Appendix A (1996)
ISO/IEC 10646-1:1993 Amendment 2 / Annex R (1996)
Template:IETF RFC (1996)
Template:IETF RFC (1998)
The Unicode Standard, Version 3.0, §2.3 (2000) plus Corrigendum #1 : UTF-8 Shortest Form (2000)
Unicode Standard Annex #27: Unicode 3.1 (2001)^[90]
The Unicode Standard, Version 5.0 (2006)^[91]
The Unicode Standard, Version 6.0 (2010)^[1]

They are all the same in their general mechanics, with the main differences being on issues such as allowed range of code point values and safe handling of invalid input.

References

Template:Reflist

External links

Original UTF-8 paper (or pdf) for Plan 9 from Bell Labs
History of UTF-8 by Rob Pike
Template:Replace on YouTubeScript error: No such module "Check for unknown parameters".

Template:Unicode navigation Template:Character encoding Template:Rob Pike navbox Template:Ken Thompson navbox

↑ ^a ^b ^c Script error: No such module "citation/CS1".
↑ ^a ^b Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ ^a ^b Script error: No such module "citation/CS1".
↑ ^a ^b Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "Citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ ^a ^b ^c Script error: No such module "citation/CS1".
↑ At that time subtraction was slower than bit logic on many computers, and speed was considered necessary for acceptance.Script error: No such module "Unsubst".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Template:Cite IETF
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Template:Cite report
↑ Script error: No such module "citation/CS1".
↑ ^a ^b Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ ^a ^b Template:Cite IETF
↑ ^a ^b Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Template:Cite IETF
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Template:Cite report
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Template:Cite report
↑ Script error: No such module "citation/CS1".
↑ Template:Cite report
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ ^a ^b Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ ^a ^b Script error: No such module "citation/CS1".
↑ InputStreamReader and OutputStreamWriter
↑ Script error: No such module "citation/CS1".
↑ DataInput and DataOutput
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ Script error: No such module "citation/CS1".
↑ The Unicode Standard, Version 5.0 §3.9–§3.10 ch. 3, 2006.

[:1-1] Script error: No such module "citation/CS1".

[W3TechsWebEncoding-2] Script error: No such module "citation/CS1".

[3] Script error: No such module "citation/CS1".

[Microsoft_GDK-4] Script error: No such module "citation/CS1".

[:3-5] Script error: No such module "citation/CS1".

[6] Script error: No such module "citation/CS1".

[FSS-UTF-7] Script error: No such module "Citation/CS1".

[8] Script error: No such module "citation/CS1".

[pikeviacambridge-9] Script error: No such module "citation/CS1".

[10] At that time subtraction was slower than bit logic on many computers, and speed was considered necessary for acceptance.Script error: No such module "Unsubst".

[11] Script error: No such module "citation/CS1".

[12] Script error: No such module "citation/CS1".

[rfc2277-13] Template:Cite IETF

[14] Script error: No such module "citation/CS1".

[15] Script error: No such module "citation/CS1".

[MS00-078-16] Template:Cite report

[CVE-2008-2938-17] Script error: No such module "citation/CS1".

[:2-18] Script error: No such module "citation/CS1".

[19] Script error: No such module "citation/CS1".

[20] Script error: No such module "citation/CS1".

[rfc3629-21] Template:Cite IETF

[pep383-22] Script error: No such module "citation/CS1".

[23] Script error: No such module "citation/CS1".

[24] Script error: No such module "citation/CS1".

[25] Script error: No such module "citation/CS1".

[26] Script error: No such module "citation/CS1".

[27] Script error: No such module "citation/CS1".

[MarkDavis2012-28] Script error: No such module "citation/CS1".

[markdavis-29] Script error: No such module "citation/CS1".

[30] Script error: No such module "citation/CS1".

[rfc8259-31] Template:Cite IETF

[IMC-32] Script error: No such module "citation/CS1".

[33] Script error: No such module "citation/CS1".

[html5charset-34] Template:Cite report

[35] Script error: No such module "citation/CS1".

[36] Script error: No such module "citation/CS1".

[37] Script error: No such module "citation/CS1".

[38] Script error: No such module "citation/CS1".

[39] Script error: No such module "citation/CS1".

[40] Script error: No such module "citation/CS1".

[41] Script error: No such module "citation/CS1".

[42] Script error: No such module "citation/CS1".

[43] Script error: No such module "citation/CS1".

[44] Script error: No such module "citation/CS1".

[45] Script error: No such module "citation/CS1".

[46] Script error: No such module "citation/CS1".

[Java_UTF-8_and_UTF-16-47] Script error: No such module "citation/CS1".

[48] Script error: No such module "citation/CS1".

[49] Script error: No such module "citation/CS1".

[50] Template:Cite report

[Microsoft-UTF-8-51] Script error: No such module "citation/CS1".

[52] Template:Cite report

[53] Script error: No such module "citation/CS1".

[54] Script error: No such module "citation/CS1".

[PEP393-55] Script error: No such module "citation/CS1".

[56] Script error: No such module "citation/CS1".

[57] Script error: No such module "citation/CS1".

[58] Script error: No such module "citation/CS1".

[59] Script error: No such module "citation/CS1".

[60] Script error: No such module "citation/CS1".

[61] Script error: No such module "citation/CS1".

[62] Script error: No such module "citation/CS1".

[:0-63] Script error: No such module "citation/CS1".

[64] InputStreamReader and OutputStreamWriter

[65] Script error: No such module "citation/CS1".

[66] DataInput and DataOutput

[67] Script error: No such module "citation/CS1".

[68] Script error: No such module "citation/CS1".

[69] Script error: No such module "citation/CS1".

[70] Script error: No such module "citation/CS1".

[71] Script error: No such module "citation/CS1".

[72] Script error: No such module "citation/CS1".

[73] Script error: No such module "citation/CS1".

[74] Script error: No such module "citation/CS1".

[davis383-75] Script error: No such module "citation/CS1".

[76] Script error: No such module "citation/CS1".

[77] Script error: No such module "citation/CS1".

[78] Script error: No such module "citation/CS1".

[79] Script error: No such module "citation/CS1".

[IANA_2013_CS-80] Script error: No such module "citation/CS1".

[81] Script error: No such module "citation/CS1".

[82] Script error: No such module "citation/CS1".

[83] Script error: No such module "citation/CS1".

[84] Script error: No such module "citation/CS1".

[mysql3-utf8mb3-85] Script error: No such module "citation/CS1".

[86] Script error: No such module "citation/CS1".

[87] Script error: No such module "citation/CS1".

[88] Script error: No such module "citation/CS1".

[89] Script error: No such module "citation/CS1".

[90] Script error: No such module "citation/CS1".

[91] The Unicode Standard, Version 5.0 §3.9–§3.10 ch. 3, 2006.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

[41]

[42]

[43]

[44]

[45]

[46]

[47]

[48]

[49]

[50]

[51]

[52]

[53]

[54]

[55]

[56]

[57]

[58]

[59]

[60]

[61]

[62]

[63]

[64]

[65]

[66]

[67]

[68]

[69]

[70]

[71]

[72]

[73]

[74]

[75]

[76]

[77]

[78]

[79]

[80]

[81]

[82]

[83]

[84]

[85]

[86]

[87]

[88]

[89]

[90]

[91]

@@ Line 1: / Line 1: @@
 {{Short description|ASCII-compatible variable-width encoding of Unicode}}
 {{Infobox character encoding
-| name = UTF-8
+ | name           = UTF-8
-| mime =
+ | mime           =
-| alias =
+ | alias          =
-| image =
+ | image          =
-| caption =
+ | caption        =
-| standard = [https://www.unicode.org/versions/latest/ Unicode Standard]
+ | standard       = [https://www.unicode.org/versions/latest/ Unicode Standard]
-| status =
+ | status         =
-| classification = [[Unicode Transformation Format]], [[extended ASCII]], [[variable-width encoding|variable-length encoding]]
+ | classification = [[Unicode Transformation Format]], [[extended ASCII]], [[variable-width encoding|variable-length encoding]]
-| encodes = [[ISO/IEC 10646]] ([[Unicode]])
+ | encodes        = [[ISO/IEC 10646]] ([[Unicode]])
-| extends = [[ASCII]]
+ | extends        = [[ASCII]]
-| prev = [[UTF-1]]
+ | prev           = [[UTF-1]]
-| next =
+ | next           =
 }}
-'''UTF-8''' is a [[character encoding]] standard used for electronic communication. Defined by the [[Unicode]] Standard, the name is derived from ''Unicode Transformation Format{{snd}} 8-bit''.<ref>{{Cite book |title=The Unicode Standard |edition=6.0 |chapter=Chapter 2. General Structure |publisher=[[The Unicode Consortium]] |location=Mountain View, California, US |isbn=978-1-936213-01-6 |chapter-url=https://www.unicode.org/versions/Unicode6.0.0/}}</ref> Almost every webpage is stored in UTF-8.
+'''UTF-8''' is a [[character encoding]] standard used for electronic communication. Defined by the [[Unicode]] Standard, the name is derived from ''Unicode Transformation Format{{snd}} 8-bit''.<ref name=":1">{{Cite book
+ | title         = Unicode® 6.0.0: Released: 2010 October 11 (Announcement)
+ | url           = https://www.unicode.org/versions/Unicode6.0.0/
-UTF-8 supports all 1,112,064<ref>{{cite book |title=The Unicode Standard |publisher=[[The Unicode Consortium]] |isbn=978-1-936213-01-6 |edition=6.0 |location=Mountain View, California, US |at=3.9 Unicode Encoding Forms |chapter=Conformance |quote=Each encoding form maps the Unicode code points U+0000..U+D7FF and U+E000..U+10FFFF |chapter-url=https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G7404}}</ref> valid Unicode [[Code point#In_Unicode|code points]] using a [[variable-width encoding]] of one to four one-[[byte]] (8-bit) code units.
+ | edition       = 6.0.0
+ | publisher     = [[The Unicode Consortium]]
+ | isbn          = 978-1-936213-01-6
+ | location      = Mountain View, California, US
+ | access-date   = 2025-08-23
+ | url-status    = live
+ | archive-url   = https://web.archive.org/web/20250728233736/https://www.unicode.org/versions/Unicode6.0.0/
+ | archive-date  = 2025-07-28
+}}</ref> As of July 2025, almost every webpage is transmitted as UTF-8.<ref name="W3TechsWebEncoding" />
-Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. It was designed for [[backward compatibility]] with [[ASCII]]: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that a UTF-8-encoded file using only those characters is identical to an ASCII file. Most software designed for any [[extended ASCII]] can read and write UTF-8, and this results in fewer internationalization issues than any alternative text encoding.<ref name="Microsoft GDK" /><ref name="whatwg" />
+UTF-8 supports all 1,112,064<ref>{{cite book
+ | title         = Unicode 16.0.0: Core Spec / Chapter 3
+ | chapter-url   = https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G7404
+ | edition       = 6.0.0
+ | at            = 3.9 Unicode Encoding Forms
+ | chapter       = Conformance
+ | quote         = Each encoding form maps the Unicode code points U+0000..U+D7FF and U+E000..U+10FFFF
+ | publisher     = [[The Unicode Consortium]]
+ | isbn          = 978-1-936213-34-4
+ | location      = Mountain View, California, US
+ | url-status    = live
+ | archive-url   = https://web.archive.org/web/20250701110809/https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3/#G7404
+ | archive-date  = 2025-07-01
+ | access-date   = 2025-08-23
+}}</ref> valid Unicode [[Code point#In Unicode|code points]] using a [[variable-width encoding]] of one to four one-[[byte]] (8-bit) code units.
-UTF-8 is dominant for all countries/languages on the internet<!-- on the web, but more generally: e-mail, JSON, and likely e.g. XML too -->, with 99% <!-- rounded up --> global average use, is used in most standards, often the only allowed encoding, and is supported by all modern operating systems and programming languages.
+Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. It was designed for [[backward compatibility]] with [[ASCII]]: the first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single byte with the same binary value as ASCII, so that a UTF-8-encoded file using only those characters is identical to an ASCII file. Most software designed for any [[extended ASCII]] can read and write UTF-8, and this results in fewer internationalization issues than any alternative text encoding.<ref name="Microsoft GDK" /><ref name=":3">{{Cite web |title=Encoding Standard |url=https://encoding.spec.whatwg.org/#preface |access-date=2025-11-20 |website=encoding.spec.whatwg.org}}</ref>
+UTF-8 is dominant for all countries/languages on the internet, is used in most standards, often the only allowed encoding, and is supported by all modern operating systems and programming languages.
 == History ==
@@ Line 28: / Line 54: @@
 The [[International Organization for Standardization]] (ISO) set out to compose a universal multi-byte character set in 1989. The draft ISO 10646 standard contained a non-required [[Addendum|annex]] called [[UTF-1]] that provided a byte stream encoding of its [[32-bit computing|32-bit]] code points. This encoding was not satisfactory on performance grounds, among other problems, and the biggest problem was probably that it did not have a clear separation between ASCII and non-ASCII: new UTF-1 tools would be backward compatible with ASCII-encoded text, but UTF-1-encoded text could confuse existing code expecting ASCII (or [[extended ASCII]]), because it could contain continuation bytes in the range {{mono|0x21}}–{{mono|0x7E}} that meant something else in ASCII, e.g., {{mono|0x2F}} for <code>/</code>, the [[Unix]] [[Path (computing)|path]] directory separator.
-In July 1992, the [[X/Open]] committee XoJIG was looking for a better encoding. Dave Prosser of [[Unix System Laboratories]] submitted a proposal for one that had faster implementation characteristics and introduced the improvement that 7-bit ASCII characters would ''only'' represent themselves; multi-byte sequences would only include bytes with the high bit set. The name ''File System Safe UCS Transformation Format'' (''FSS-UTF'')<ref>{{cite web|url=https://www.unicode.org/L2/Historical/wg20-n193-fss-utf.pdf|title=File System Safe UCS&nbsp;— Transformation Format (FSS-UTF) - X/Open Preliminary Specification|website=unicode.org}}</ref> and most of the text of this proposal were later preserved in the final specification.<ref name="FSS-UTF">{{cite journal |title=Appendix F. FSS-UTF / File System Safe UCS Transformation format |journal=The Unicode Standard 1.1 |url=https://www.unicode.org/versions/Unicode1.1.0/appF.pdf |access-date=2016-06-07 |url-status=live |archive-url=https://web.archive.org/web/20160607215950/https://www.unicode.org/versions/Unicode1.1.0/appF.pdf |archive-date=2016-06-07}}</ref><ref name="Whistler_2001">{{cite web |title=FSS-UTF, UTF-2, UTF-8, and UTF-16 |author-first=Kenneth |author-last=Whistler |date=2001-06-12 |url=https://unicode.org/mail-arch/unicode-ml/y2001-m06/0318.html |access-date=2006-06-07 |url-status=live |archive-url=https://web.archive.org/web/20160607220249/https://unicode.org/mail-arch/unicode-ml/y2001-m06/0318.html |archive-date=2016-06-07 }}</ref><ref name="pikeviacambridge">{{cite web |url=https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt |title=UTF-8 history |author-first=Rob |author-last=Pike |author-link=Rob Pike |date=2003-04-30 |access-date=2012-09-07}}</ref> In August 1992, this proposal was circulated by an [[IBM]] X/Open representative to interested parties. A modification by [[Ken Thompson]] of the [[Plan 9 from Bell Labs|Plan 9 operating system]] group at [[Bell Labs]] made it [[Self-synchronizing code|self-synchronizing]], letting a reader start anywhere and immediately detect character boundaries, at the cost of being somewhat less bit-efficient than the previous proposal. It also abandoned the use of biases that prevented [[#overlong encodings|overlong encodings]].<ref name=pikeviacambridge/><ref>At that time subtraction was slower than bit logic on many computers, and speed was considered necessary for acceptance.{{citation needed|date=October 2024}}</ref> Thompson's design was outlined on September 2, 1992, on a [[placemat]] in a New Jersey diner with [[Rob Pike]]. In the following days, Pike and Thompson implemented it and updated [[Plan 9 from Bell Labs|Plan 9]] to use it throughout,<ref>{{cite book |chapter-url=https://www.cl.cam.ac.uk/~mgk25/ucs/UTF-8-Plan9-paper.pdf |chapter=Hello World or Καλημέρα κόσμε or こんにちは 世界 |title=Proceedings of the Winter 1993 USENIX Conference |first1=Rob |last1=Pike |first2=Ken |last2=Thompson |year=1993}}</ref> and then communicated their success back to X/Open, which accepted it as the specification for FSS-UTF.<ref name=pikeviacambridge/>
+In July 1992, the [[X/Open]] committee XoJIG was looking for a better encoding. Dave Prosser of [[Unix System Laboratories]] submitted a proposal for one that had faster implementation characteristics and introduced the improvement that 7-bit ASCII characters would ''only'' represent themselves; multi-byte sequences would only include bytes with the high bit set. The name ''File System Safe UCS Transformation Format'' (''FSS-UTF'')<ref>{{cite web|url=https://www.unicode.org/L2/Historical/wg20-n193-fss-utf.pdf|title=File System Safe UCS&nbsp;— Transformation Format (FSS-UTF) - X/Open Preliminary Specification|website=unicode.org}}</ref> and most of the text of this proposal were later preserved in the final specification.<ref name="FSS-UTF">{{cite journal |title=Appendix F. FSS-UTF / File System Safe UCS Transformation format |journal=The Unicode Standard 1.1 |url=https://www.unicode.org/versions/Unicode1.1.0/appF.pdf |access-date=2016-06-07 |url-status=live |archive-url=https://web.archive.org/web/20160607215950/https://www.unicode.org/versions/Unicode1.1.0/appF.pdf |archive-date=2016-06-07}}</ref><ref>{{Cite web |last=Whistler |first=Kenneth |date=2001-06-12 |title=Unicode Mail List Archive: FSS-UTF, UTF-2, UTF-8, and UTF-16 |url=https://unicode.org/mail-arch/unicode-ml/y2001-m06/0318.html |archive-url=https://web.archive.org/web/20160607220249/https://unicode.org/mail-arch/unicode-ml/y2001-m06/0318.html |archive-date=2016-06-07 |access-date=2025-11-20 |website=unicode.org}}</ref><ref name="pikeviacambridge">{{cite web |url=https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt |title=UTF-8 history |author-first=Rob |author-last=Pike |author-link=Rob Pike |date=2003-04-30 |access-date=2012-09-07}}</ref> In August 1992, this proposal was circulated by an [[IBM]] X/Open representative to interested parties.
-UTF-8 was first officially presented at the [[USENIX]] conference in [[San Diego]], from January 25 to 29, 1993.<ref>{{cite web|url=https://www.usenix.org/legacy/publications/library/proceedings/sd93/| title=USENIX Winter 1993 Conference Proceedings|website=usenix.org}}</ref> The [[Internet Engineering Task Force]] adopted UTF-8 in its Policy on Character Sets and Languages in RFC&nbsp;2277 ([[Request for Comments#Best Current Practice|<abbr title="Best Current Practice">BCP</abbr>]] 18) for future internet standards work in January 1998, replacing [[Single Byte Character Set]]s such as [[ISO/IEC 8859-1|Latin-1]] in older RFCs.<ref name="rfc2277">{{cite IETF |rfc=2277 |bcp=18 |title=IETF Policy on Character Sets and Languages |date=January 1998 |last1=Alvestrand |first1=Harald T. |author-link=Harald Alvestrand |publisher=[[Internet Engineering Task Force|IETF]]}}</ref>
+A modification by [[Ken Thompson]] of the [[Plan 9 from Bell Labs|Plan 9 operating system]] group at [[Bell Labs]] made it [[Self-synchronizing code|self-synchronizing]], letting a reader start anywhere and immediately detect character boundaries, at the cost of being somewhat less bit-efficient than the previous proposal. It also abandoned the use of biases that prevented [[#overlong encodings|overlong encodings]].<ref name=pikeviacambridge/><ref>At that time subtraction was slower than bit logic on many computers, and speed was considered necessary for acceptance.{{citation needed|date=October 2024}}</ref> Thompson's design was outlined on September 2, 1992, on a [[placemat]] in a New Jersey diner with [[Rob Pike]]. In the following days, Pike and Thompson implemented it and updated [[Plan 9 from Bell Labs|Plan 9]] to use it throughout,<ref>{{cite book |chapter-url=https://www.cl.cam.ac.uk/~mgk25/ucs/UTF-8-Plan9-paper.pdf |chapter=Hello World or Καλημέρα κόσμε or こんにちは 世界 |title=Proceedings of the Winter 1993 USENIX Conference |first1=Rob |last1=Pike |first2=Ken |last2=Thompson |year=1993}}</ref> and then communicated their success back to X/Open, which accepted it as the specification for [[FSS-UTF]].<ref name=pikeviacambridge/> UTF-8 was first officially presented at the [[USENIX]] conference in [[San Diego]], from January 25 to 29, 1993.<ref>{{Cite web |title=USENIX WINTER 1993 CONFERENCE PROCEEDINGS |url=https://www.usenix.org/legacy/publications/library/proceedings/sd93/ |access-date=2025-11-20 |website=www.usenix.org}}</ref> The [[Internet Engineering Task Force]] adopted UTF-8 in its Policy on Character Sets and Languages in RFC&nbsp;2277 ([[Request for Comments#Best Current Practice|<abbr title="Best Current Practice">BCP</abbr>]] 18) for future internet standards work in January 1998, replacing [[Single Byte Character Set]]s such as [[ISO/IEC 8859-1|Latin-1]] in older RFCs.<ref name="rfc2277">{{cite IETF |rfc=2277 |bcp=18 |title=IETF Policy on Character Sets and Languages |date=January 1998 |last1=Alvestrand |first1=Harald T. |author-link=Harald Alvestrand |publisher=[[Internet Engineering Task Force|IETF]]}}</ref>
-In November 2003, UTF-8 was restricted by {{IETF RFC|3629}} to match the constraints of the [[UTF-16]] character encoding: explicitly prohibiting code points corresponding to the high and low surrogate characters removed <!-- 2*2^10/(2^16-2^11) --> more than 3% of the three-byte sequences, and ending at {{tt|U+10FFFF}} removed <!-- (2^21-(2^16+2^20))/(2^21-2^16) --> more than 48% of the four-byte sequences and all five- and six-byte sequences.<ref>{{cite web |author-last=Pike |author-first=Rob |author-link=Rob Pike |date=2012-09-06 |title=UTF-8 turned 20 years old yesterday |url=https://plus.google.com/u/0/101960720994009339267/posts/Rz1udTvtiMg |url-status=dead |archive-url=https://commandcenter.blogspot.com/2020/01/utf-8-turned-20-years-old-in-2012.html |archive-date=2020-01-26 |access-date=2012-09-07}}</ref>
+In November 2003, UTF-8 was restricted by {{IETF RFC|3629}} to match the constraints of the [[UTF-16]] character encoding: explicitly prohibiting code points corresponding to the high and low surrogate characters removed <!-- 2*2^10/(2^16-2^11) --> more than 3% of the three-byte sequences, and ending at {{tt|U+10FFFF}} removed <!-- (2^21-(2^16+2^20))/(2^21-2^16) --> more than 48% of the four-byte sequences and all five- and six-byte sequences.<ref>{{cite web |author-last=Pike |author-first=Rob |author-link=Rob Pike |date=2012-09-06 |title=UTF-8 turned 20 years old yesterday |url=https://plus.google.com/u/0/101960720994009339267/posts/Rz1udTvtiMg |url-status=dead |archive-url=https://web.archive.org/web/20121130120145/https://plus.google.com/u/0/101960720994009339267/posts/Rz1udTvtiMg |archive-date=2012-11-30 |access-date=2012-09-07 }}</ref>
 == Description ==
@@ Line 73: / Line 99: @@
 |}
-The first 128&nbsp;code points (ASCII) need 1&nbsp;byte. The next 1,920&nbsp;code points need two bytes to encode, which covers the remainder of almost all [[Latin-script alphabet]]s, and also [[International Phonetic Alphabet|IPA extensions]], [[Greek alphabet|Greek]], [[Cyrillic script|Cyrillic]], [[Coptic alphabet|Coptic]], [[Armenian alphabet|Armenian]], [[Hebrew alphabet|Hebrew]], [[Arabic alphabet|Arabic]], [[Syriac alphabet|Syriac]], [[Thaana]] and [[N'Ko script|N'Ko]] alphabets, as well as [[Combining Diacritical Marks]]. Three bytes are needed for the remaining 61,440&nbsp;codepoints of the [[Basic Multilingual Plane]] (BMP), including most [[CJK characters|Chinese, Japanese and Korean characters]]. Four bytes are needed for the 1,048,576&nbsp;non-BMP code points, which include [[emoji]], less common [[CJK characters]], and other useful characters.<ref name="problems_of_only_BMP">{{Cite web |last=Lunde |first=Ken |date=2022-01-09 |title=2022 Top Ten List: Why Support Beyond-BMP Code Points? |url=https://ken-lunde.medium.com/2022-top-ten-list-why-support-beyond-bmp-code-points-6a946d7735f9 |website=Medium |language=en|access-date=2024-01-07}}</ref>
+The first 128&nbsp;code points (ASCII) need 1&nbsp;byte. The next 1,920&nbsp;code points need two bytes to encode, which covers the remainder of almost all [[Latin-script alphabet]]s, and also [[International Phonetic Alphabet|IPA extensions]], [[Greek alphabet|Greek]], [[Cyrillic script|Cyrillic]], [[Coptic alphabet|Coptic]], [[Armenian alphabet|Armenian]], [[Hebrew alphabet|Hebrew]], [[Arabic alphabet|Arabic]], [[Syriac alphabet|Syriac]], [[Thaana]] and [[N'Ko script|N'Ko]] alphabets, as well as [[Combining Diacritical Marks]]. Three bytes are needed for the remaining 61,440&nbsp;codepoints of the [[Basic Multilingual Plane]] (BMP), including most [[CJK characters|Chinese, Japanese and Korean characters]]. Four bytes are needed for the 1,048,576&nbsp;non-BMP code points, which include [[emoji]], less common [[CJK characters]], and other useful characters.<ref>{{Cite web |last=Lunde |first=Dr Ken |date=2022-01-09 |title=2022 Top Ten List: Why Support Beyond-BMP Code Points? |url=https://ken-lunde.medium.com/2022-top-ten-list-why-support-beyond-bmp-code-points-6a946d7735f9 |access-date=2025-11-20 |website=Medium |language=en}}</ref>
-UTF-8 is a ''[[prefix code]]'' and it is unnecessary to read past the last byte of a code point to decode it. Unlike many earlier multi-byte text encodings such as [[Shift-JIS]], it is ''[[Self-synchronizing code|self-synchronizing]]'' so searches for short strings or characters are possible and that the start of a code point can be found from a random position by backing up at most 3 bytes. The values chosen for the lead bytes means sorting a list of UTF-8 strings puts them in the same order as sorting [[UTF-32]] strings.
+UTF-8 is a ''[[prefix code]]'' and it is unnecessary to read past the last byte of a code point to decode it. Unlike many earlier multi-byte text encodings such as [[Shift-JIS]], it is ''[[Self-synchronizing code|self-synchronizing]]'' so searches for short strings or characters are possible; and the start of a code point can be found from a random position by backing up at most 3 bytes. The values chosen for the lead bytes means sorting a list of UTF-8 strings puts them in the same order as sorting [[UTF-32]] strings.
 === Overlong encodings ===
 {{anchor|overlong encodings}}
-Using a row in the above table to encode a code point less than "First code point" (thus using more bytes than necessary) is termed an ''overlong encoding''. These are a security problem because they allow character sequences such as malicious JavaScript and <code>[[directory traversal attack|../]]</code> to bypass security validations, which has been reported in numerous high-profile products such as Microsoft's [[Internet Information Services|IIS]] web server<ref name=MS00-078>{{ cite report | first = Marvin |last = Marin | date = 2000-10-17 | title = Windows NT UNICODE vulnerability analysis | department = Web server folder traversal | id = MS00-078 | series = Malware FAQ | website=SANS Institute | url=https://www.sans.org/resources/malwarefaq/wnt-unicode.php | url-status=dead | archive-url=https://web.archive.org/web/20140827001204/http://www.sans.org/security-resources/malwarefaq/wnt-unicode.php | archive-date=Aug 27, 2014 }}</ref> and Apache's Tomcat servlet container.<ref name=CVE-2008-2938>{{ cite web | title = CVE-2008-2938 | year = 2008 | website = National Vulnerability Database (nvd.nist.gov) | publisher = U.S. [[National Institute of Standards and Technology]] | url = https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2008-2938 }}</ref> Overlong encodings should therefore be considered an error and never decoded. [[#Modified UTF-8|Modified UTF-8]] allows an overlong encoding of {{tt|U+0000}}.
+Using a row in the above table to encode a code point less than "First code point" (thus using more bytes than necessary) is termed an ''overlong encoding''. These are a security problem because they allow character sequences such as malicious JavaScript and <code>[[directory traversal attack|../]]</code> to bypass security validations, which has been reported in numerous high-profile products such as Microsoft's [[Internet Information Services|IIS]] web server<ref name=MS00-078>{{ cite report | first = Marvin |last = Marin | date = 2000-10-17 | title = Windows NT UNICODE vulnerability analysis | department = Web server folder traversal | id = MS00-078 | series = Malware FAQ | website=SANS Institute | url=https://www.sans.org/resources/malwarefaq/wnt-unicode.php | url-status=dead | archive-url=https://web.archive.org/web/20140827001204/http://www.sans.org/security-resources/malwarefaq/wnt-unicode.php | archive-date=Aug 27, 2014 }}</ref> and Apache's Tomcat servlet container.<ref name="CVE-2008-2938">{{cite web |year=2008 |title=CVE-2008-2938 |url=https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2008-2938 |access-date=2025-11-20 |website=National Vulnerability Database (nvd.nist.gov) |publisher=U.S. [[National Institute of Standards and Technology]]}}</ref> Overlong encodings should therefore be considered an error and never decoded. [[#Modified UTF-8|Modified UTF-8]] allows the two-byte overlong encoding {{Mono|0xC0}}, {{Mono|0x80}}, for {{tt|U+0000}} instead of just the single-byte {{Mono|0x00}}.<ref name=":2" />
-=== Byte map ===
-The chart below gives the detailed meaning of each byte in a stream encoded in UTF-8.
-{{UTF-8 byte map}}
 === Error handling ===
 Not all sequences of bytes are valid UTF-8. A UTF-8 decoder should be prepared for:
-* Bytes that never appear in UTF-8: {{tt|0xC0}}, {{tt|0xC1}}, {{tt|0xF5}}{{ndash}}{{tt|0xFF}}
 * A "continuation byte" ({{tt|0x80}}{{ndash}}{{tt|0xBF}}) at the start of a character
 * A non-continuation byte (or the string ending) before the end of a character
-* An overlong encoding ({{tt|0xE0}} followed by less than {{tt|0xA0}}, or {{tt|0xF0}} followed by less than {{tt|0x90}})
+* An overlong encoding ({{tt|0xC0}}, {{tt|0xC1}}, {{tt|0xE0}} followed by less than {{tt|0xA0}}, or {{tt|0xF0}} followed by less than {{tt|0x90}})
-* A 4-byte sequence that decodes to a value greater than {{tt|U+10FFFF}} ({{tt|0xF4}} followed by {{tt|0x90}} or greater)
+* A 4-byte sequence that decodes to a value greater than {{tt|U+10FFFF}} ({{tt|0xF4}} followed by {{tt|0x90}} or greater, {{tt|0xF5}}{{ndash}}{{tt|0xFF}})
-Many of the first UTF-8 decoders would decode these, ignoring incorrect bits. Carefully crafted invalid UTF-8 could make them either skip or create ASCII characters such as {{mono|NUL}}, slash, or quotes, leading to security vulnerabilities. It is also common to throw an exception or truncate the string at an error<ref>{{ cite web | title = DataInput | series = Java Platform SE 8 | website = docs.oracle.com | url = https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html | access-date = 2021-03-24 }}</ref> but this turns what would otherwise be harmless errors (i.e. "file not found") into a [[denial of service]], for instance early versions of Python 3.0 would exit immediately if the command line or [[environment variable]]s contained invalid UTF-8.<ref name=PEP383>{{ cite web | title = Non-decodable bytes in system character interfaces | date = 2009-04-22 | website = python.org | url = https://www.python.org/dev/peps/pep-0383/ | access-date = 2014-08-13 }}</ref>
+Many of the first UTF-8 decoders would decode these, ignoring incorrect bits. Carefully crafted invalid UTF-8 could make them either skip or create ASCII characters such as {{mono|NUL}}, slash, or quotes, leading to security vulnerabilities. It is also common to throw an exception or truncate the string at an error<ref>{{Cite web |title=DataInput (Java Platform SE 8 ) |url=https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html |access-date=2025-11-20 |website=docs.oracle.com}}</ref> but this turns what would otherwise be harmless errors (i.e. "file not found") into a [[denial of service]], for instance early versions of Python 3.0 would exit immediately if the command line or [[environment variable]]s contained invalid UTF-8.<ref>{{Cite web |title=PEP 383 – Non-decodable Bytes in System Character Interfaces {{!}} peps.python.org |url=https://peps.python.org/pep-0383/ |access-date=2025-11-20 |website=Python Enhancement Proposals (PEPs) |language=en}}</ref>
-{{nobr|RFC 3629}} states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."<ref name="rfc3629">{{cite IETF |title=UTF-8, a transformation format of ISO 10646 |rfc=3629 |std=63 |last1=Yergeau |first1=F. |date=November 2003 |publisher=[[Internet Engineering Task Force|IETF]] |access-date=August 20, 2020}}</ref> ''The Unicode Standard'' requires decoders to: "...&nbsp;treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."<!-- anyone have a copy of ISO/IEC 10646-1:2000 annex D for comparison?  --> The standard now recommends replacing each error with the [[replacement character]] "�" ({{tt|U+FFFD}}) and continue decoding.
+{{nobr|RFC 3629}} states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."<ref name="rfc3629">{{cite IETF |title=UTF-8, a transformation format of ISO 10646 |rfc=3629 |std=63 |last1=Yergeau |first1=F. |date=November 2003 |publisher=[[Internet Engineering Task Force|IETF]] |access-date=August 20, 2020}}</ref> ''The Unicode Standard'' requires decoders to: "...&nbsp;treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."<!-- anyone have a copy of ISO/IEC 10646-1:2000 annex D for comparison?  --> The standard now recommends replacing each error with the [[replacement character]] "{{nat|&#xFFFD;}}" ({{tt|U+FFFD}}) and continue decoding.
-Some decoders consider the sequence {{mono|E1,A0,20}} (a truncated 3-byte code followed by a space) as a single error. This is not a good idea as a search for a space character would find the one hidden in the error. Since Unicode&nbsp;6 (October&nbsp;2010)<ref>{{ cite report |  title = Unicode 6.0.0 |  date = October 2010 |  website = unicode.org |  url = https://www.unicode.org/versions/Unicode6.0.0/ }}</ref> the standard (chapter&nbsp;3) has recommended a "best practice" where the error is either one continuation byte, or ends at the first byte that is disallowed, so {{mono|E1,A0,20}} is a two-byte error followed by a space. This means an error is no more than three bytes long and never contains the start of a valid character, and there are {{val|21952|fmt=commas}}&nbsp;different possible errors. Technically this makes UTF-8 no longer a [[prefix code]] (the decoder has to read one byte past some errors to figure out they are an error), but searching still works if the searched-for string does not contain any errors.
+Some decoders consider the sequence {{mono|E1,A0,20}} (a truncated 3-byte code followed by a space) as a single error. This is not a good idea as a search for a space character would find the one hidden in the error. Since Unicode&nbsp;6 (October&nbsp;2010)<ref name=":1" /> the standard (chapter&nbsp;3) has recommended a "best practice" where the error is either one continuation byte, or ends at the first byte that is disallowed, so {{mono|E1,A0,20}} is a two-byte error followed by a space. An error is no more than three bytes long, never contains the start of a valid character, and there are {{val|21952|fmt=commas}}&nbsp;different possible errors. Many decoders instead make ''each'' byte be an error, in which case {{mono|E1,A0,20}} is ''two'' errors followed by a space; there are now only 128 different errors which makes it practical to store the errors in the output string,<ref name="pep383"/> or replace them with characters from a legacy encoding.
-Making each byte be an error, in which case {{mono|E1,A0,20}} is ''two'' errors followed by a space, also still allows searching for a valid string. This means there are only 128 different errors which makes it practical to store the errors in the output string,<ref name="pep383"/> or replace them with characters from a legacy encoding.
+Only a small subset of possible byte strings are error-free UTF-8: several bytes cannot appear; a byte with the high bit set cannot be alone; and in a truly random string a byte with a high bit set has only a {{frac|1|15}} chance of starting a valid UTF-8 character. This has the consequence of making it easy to detect if a legacy text encoding is accidentally used instead of UTF-8, making conversion of a system to UTF-8 easier and avoiding the need to require a [[Byte Order Mark]] or any other metadata.
-Only a small subset of possible byte strings are error-free UTF-8: several bytes cannot appear; a byte with the high bit set cannot be alone; and in a truly random string a byte with a high bit set has only a {{frac|1|15}} chance of starting a valid UTF-8 character. This has the consequence of making it easy to detect if a legacy text encoding is accidentally used instead of UTF-8, making conversion of a system to UTF-8 easier and avoiding the need to require a Byte Order Mark or any other metadata.
+=== Surrogates ===
+Since RFC 3629 (November&nbsp;2003), the high and low surrogates used by [[UTF-16]] ({{tt|U+D800}} through {{tt|U+DFFF}}) are not legal Unicode values, and their UTF-8 encodings must be treated as an invalid byte sequence.<ref name="rfc3629"/> These encodings all start with {{tt|0xED}} followed by {{tt|0xA0}} or higher. This rule is often ignored as surrogates are allowed in Windows filenames and this means there must be a way to store them in a string.<ref>{{Cite web |title=PEP 529 – Change Windows filesystem encoding to UTF-8 {{!}} peps.python.org |url=https://peps.python.org/pep-0529/ |access-date=2025-11-20 |website=Python Enhancement Proposals (PEPs) |language=en}}</ref> UTF-8 that allows these surrogate halves has been (informally) called ''{{visible anchor|WTF-8}}'',<ref>{{Cite web |title=The WTF-8 encoding |url=https://simonsapin.github.io/wtf-8/ |access-date=2025-11-20 |website=simonsapin.github.io}}</ref> while another variation that also encodes all non-BMP characters as two surrogates (6&nbsp;bytes instead of 4) is called ''[[CESU-8]]''.
-=== Surrogates ===
+=== Byte map ===
-Since RFC 3629 (November&nbsp;2003), the high and low surrogates used by [[UTF-16]] ({{tt|U+D800}} through {{tt|U+DFFF}}) are not legal Unicode values, and their UTF-8 encodings must be treated as an invalid byte sequence.<ref name="rfc3629"/> These encodings all start with {{tt|0xED}} followed by {{tt|0xA0}} or higher. This rule is often ignored as surrogates are allowed in Windows filenames and this means there must be a way to store them in a string.<ref name="PEP 529">{{ cite web | title = Change Windows filesystem encoding to UTF-8 | id = PEP&nbsp;529 | website = Python.org   |language = en | url = https://www.python.org/dev/peps/pep-0529/ | access-date = 2022-05-10 }}</ref> UTF-8 that allows these surrogate halves has been (informally) called ''{{visible anchor|WTF-8}}'',<ref name="wtf-8">{{cite web | title = The WTF-8 encoding | url = https://simonsapin.github.io/wtf-8/}}</ref> while another variation that also encodes all non-BMP characters as two surrogates (6&nbsp;bytes instead of 4) is called ''[[CESU-8]]''.
+The chart below gives the detailed meaning of each byte in a stream encoded in UTF-8.
+{{UTF-8 byte map}}
 === Byte-order mark ===
 If the Unicode [[byte-order mark]] {{tt|U+FEFF}} is at the start of a UTF-8 file, the first three bytes will be {{mono|0xEF}}, {{mono|0xBB}}, {{mono|0xBF}}.
-The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file trans-coded from another encoding.<ref>{{citation | chapter-url = https://www.unicode.org/versions/Unicode15.0.0/ch02.pdf | title = The Unicode Standard&nbsp;— Version 15.0.0 | chapter = Chapter 2 | page = 39 }}</ref> While ASCII text encoded using UTF-8 is backward compatible with ASCII, this is not true when Unicode Standard recommendations are ignored and a BOM is added. A BOM can confuse software that isn't prepared for it but can otherwise accept UTF-8, e.g. programming languages that permit non-ASCII bytes in [[string literal]]s but not at the start of the file. Nevertheless, there was and still is software that always inserts a BOM when writing UTF-8, and refuses to correctly interpret UTF-8 unless the first character is a BOM (or the file only contains ASCII).<ref>{{Cite web |title=UTF-8 and Unicode FAQ for Unix/Linux |url=https://www.cl.cam.ac.uk/~mgk25/unicode.html}}</ref>
+The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file trans-coded from another encoding.<ref>{{citation | chapter-url = https://www.unicode.org/versions/Unicode15.0.0/ch02.pdf | title = The Unicode Standard&nbsp;— Version 15.0.0 | chapter = Chapter 2 | page = 39 }}</ref> While ASCII text encoded using UTF-8 is backward compatible with ASCII, this is not true when Unicode Standard recommendations are ignored and a BOM is added. A BOM can confuse software that isn't prepared for it but can otherwise accept UTF-8, e.g. programming languages that permit non-ASCII bytes in [[string literal]]s but not at the start of the file. Nevertheless, there was and still is software that always inserts a BOM when writing UTF-8, and refuses to correctly interpret UTF-8 unless the first character is a BOM (or the file only contains ASCII).<ref>{{Cite web |title=UTF-8 and Unicode FAQ |url=https://www.cl.cam.ac.uk/~mgk25/unicode.html |access-date=2025-11-20 |website=www.cl.cam.ac.uk}}</ref>
 == Comparison to UTF-16 ==
 {{See also|Comparison of Unicode encodings}}
-{{unsourced section|find=UTF-8|find2=comparison to UTF-16|date=December 2024}}
+{{unreferenced section|find=UTF-8|find2=comparison to UTF-16|date=December 2024}}
-For a long time there was considerable argument as to whether it was better to process text in [[UTF-16]] or in UTF-8.
-The primary advantage of UTF-16 is that the [[Unicode in Microsoft Windows|Windows API]] required it for access to all Unicode characters (UTF-8 was not fully supported in Windows until May 2019). This caused several libraries such as [[Qt (software)|Qt]] to also use UTF-16 strings which propagates this requirement to non-Windows platforms.
+For a long time there was considerable argument as to whether it was better to process text in [[UTF-16]] or in UTF-8. The primary advantage of UTF-16 is that the [[Unicode in Microsoft Windows|Windows API]] required it for access to all Unicode characters (UTF-8 was not fully supported in Windows until May 2019). This caused several libraries such as [[Qt (software)|Qt]] to also use UTF-16 strings which propagates this requirement to non-Windows platforms. In the early days of Unicode there were no characters greater than {{tt|U+FFFF}} and [[combining characters]] were rarely used, so the 16-bit encoding was effectively fixed-size. Some believed fixed-size encoding could make processing more efficient, but any such advantages were lost as soon as UTF-16 became variable width as well. The code points {{tt|U+0800}}{{ndash}}{{tt|U+FFFF}} take 3 bytes in UTF-8 but only 2 in UTF-16. This led to the idea that text in Chinese and other languages would take more space in UTF-8. However, text is only larger if there are more of these code points than 1-byte ASCII code points, and this rarely happens in real-world documents due to spaces, newlines, digits, punctuation, English words, and [[markup language|markup]]<ref>{{Cite web |date=2025-07-23 |title=Kí tự đặc biệt |url=https://kitugenz.com/ |access-date=2025-11-20 |website=Ki Tu GenZ |language=en}}</ref>. UTF-8 has the advantages of being trivial to retrofit to any system that could handle an [[extended ASCII]], not having byte-order problems, and taking about half the space for any language using mostly [[Latin letters]].
-In the early days of Unicode there were no characters greater than {{tt|U+FFFF}} and [[combining characters]] were rarely used, so the 16-bit encoding was effectively fixed-size. Some believed fixed-size encoding could make processing more efficient, but any such advantages were lost as soon as UTF-16 became variable width as well.
-The code points {{tt|U+0800}}{{ndash}}{{tt|U+FFFF}} take 3 bytes in UTF-8 but only 2 in UTF-16. This led to the idea that text in Chinese and other languages would take more space in UTF-8. However, text is only larger if there are more of these code points than 1-byte ASCII code points, and this rarely happens in the real-world documents due to spaces, newlines, digits, punctuation, English words, and (depending on document format) markup.
-UTF-8 has the advantages of being trivial to retrofit to any system that could handle an [[extended ASCII]], not having byte-order problems, and taking about half the space for any language using mostly Latin letters.
 == Implementations and adoption ==
-[[File:UTF-8 takes over.png|thumb|400px|Declared character set for the 10&nbsp;million most popular websites from 2010 to 2021.]]
+[[File:UTF-8 takes over.png|thumb|400px|Declared character set for the 10&nbsp;million most popular websites from 2010 to 2021]]
-[[File:Utf8webgrowth.svg|thumb|400px|Use of the main encodings on the web from 2001 to 2012 as recorded by Google,<ref name=MarkDavis2012>{{ cite web | author-last=Davis |author-first=Mark |author-link=Mark Davis (Unicode) | date=2012-02-03 | title=Unicode over 60&nbsp;percent of the web | website=Official Google blog | url=https://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html | url-status=live |access-date=2020-07-24 | archive-url=https://web.archive.org/web/20180809152828/https://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html | archive-date=2018-08-09 }}</ref> with UTF-8 overtaking all others in 2008 and over 60% of the web in 2012 (since then approaching 100%). UTF-8 is the only encoding of Unicode (explicitly) listed there, and the rest only provide subsets of Unicode. The ASCII-only figure includes all web pages that only contain ASCII characters, regardless of the declared header.]]
+[[File:Utf8webgrowth.svg|thumb|400px|Use of the main encodings on the web from 2001 to 2012 as recorded by Google,<ref name=MarkDavis2012>{{ cite web | author-last=Davis |author-first=Mark |author-link=Mark Davis (Unicode) | date=2012-02-03 | title=Unicode over 60&nbsp;percent of the web | website=Official Google blog | url=https://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html | url-status=live |access-date=2020-07-24 | archive-url=https://web.archive.org/web/20180809152828/https://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html | archive-date=2018-08-09 }}</ref> with UTF-8 overtaking all others in 2008 and over 60% of the web in 2012 (since then hitting 100% use). UTF-8 is the only encoding of Unicode (explicitly) listed there, and the rest only provide subsets of Unicode. The ASCII-only figure includes all web pages that only contain ASCII characters, regardless of the declared header.]]
 {{See also|Popularity of text encodings}}
-UTF-8 has been the most common encoding for the [[World Wide Web]] since 2008.<ref name=markdavis>{{cite web | first=Mark |last=Davis |author-link=Mark Davis (Unicode) | date=2008-05-05 | title=Moving to Unicode&nbsp;5.1 | website=Official Google blog |language=en | url=https://googleblog.blogspot.com/2008/05/moving-to-unicode-51.html | access-date=2023-03-13 }}</ref> {{As of|2025|01}}, UTF-8 is used by 98.5% of surveyed web sites.<ref name=W3TechsWebEncoding>{{Cite web|url=https://w3techs.com/technologies/cross/character_encoding/ranking|title=Usage Survey of Character Encodings broken down by Ranking |website=W3Techs |language = en | date = January 2025 |access-date=2025-01-07}}</ref> Although many pages only use ASCII characters to display content, very few websites now declare their encoding to only be ASCII instead of UTF-8.<ref>{{cite web |url=https://w3techs.com/technologies/details/en-usascii |title = Usage statistics and market share of ASCII for websites | date = January 2025 | website = W3Techs | access-date = 2025-01-07 }}</ref> Virtually all countries and <!-- over 97% all of the tracked --> languages have 95% or more use of UTF-8 encodings on the web. <!-- Over 61% of the languages tracked have <!- currently 61.4% have at least 99.5% UTF-8 support which rounds up to 100% (44.5% have "100.0%" which means 99.95+%) -> 100% UTF-8 use. -->
+UTF-8 has been the most common encoding for the [[World Wide Web]] since 2008.<ref name=markdavis>{{cite web | first=Mark |last=Davis |author-link=Mark Davis (Unicode) | date=2008-05-05 | title=Moving to Unicode&nbsp;5.1 | website=Official Google blog |language=en | url=https://googleblog.blogspot.com/2008/05/moving-to-unicode-51.html | access-date=2023-03-13 }}</ref> {{As of|2025|07}}, UTF-8 is used by 98.8% of surveyed web sites.<ref name=W3TechsWebEncoding>{{Cite web|url=https://w3techs.com/technologies/cross/character_encoding/ranking|title=Usage Survey of Character Encodings broken down by Ranking |website=W3Techs |language=en |date=August 2025 |access-date=2025-08-28}}</ref> Although many pages only use ASCII characters to display content, very few websites now declare their encoding to only be ASCII instead of UTF-8.<ref>{{cite web |url=https://w3techs.com/technologies/details/en-usascii |title = Usage statistics and market share of ASCII for websites | date = January 2025 | website = W3Techs | access-date = 2025-01-07 }}</ref> Virtually all countries and <!-- over 97% all of the tracked --> languages have 95% or more use of UTF-8 encodings on the web. <!-- Over 61% of the languages tracked have <!- currently 61.4% have at least 99.5% UTF-8 support which rounds up to 100% (44.5% have "100.0%" which means 99.95+%) -> 100% UTF-8 use. -->
-Many standards only support UTF-8, e.g. [[JSON]] exchange requires it (without a byte-order mark (BOM)).<ref name=rfc8259>{{ cite IETF | last = Bray | first = Tim | editor-last = Bray | editor-first = Tim | date = December 2017 | title = The JavaScript Object Notation (JSON) Data Interchange Format | publisher = IETF | doi = 10.17487/RFC8259 | access-date = 16 February 2018 | rfc = 8259 }}</ref> UTF-8 is also the recommendation from the [[WHATWG]] for HTML and [[Document Object Model|DOM]] specifications, and stating "UTF-8 encoding is the most appropriate encoding for interchange of [[Unicode]]"<ref name=whatwg>{{ cite web | title = Encoding Standard | website = encoding.spec.whatwg.org | url = https://encoding.spec.whatwg.org/#preface | access-date = 2020-04-15 }}</ref> and the [[Internet Mail Consortium]] recommends that all e‑mail programs be able to display and create mail using UTF-8.<ref name=IMC>{{ cite web | url = https://www.imc.org/mail-i18n.html | title = Using International Characters in Internet Mail | publisher = Internet Mail Consortium | date = 1998-08-01 | access-date = 2007-11-08 | url-status = dead | archive-url = https://web.archive.org/web/20071026103104/https://www.imc.org/mail-i18n.html | archive-date = 2007-10-26}}</ref><ref name=mandatory>{{ cite web | title = Encoding Standard | website = encoding.spec.whatwg.org |language = en | url = https://encoding.spec.whatwg.org/#security-background | access-date = 2018-11-15 }}</ref> The [[World Wide Web Consortium]] recommends UTF-8 as the default encoding in XML and HTML (and not just using UTF-8, also declaring it in metadata), "even when all characters are in the ASCII range ... Using non-UTF-8 encodings can have unexpected results".<ref name=html5charset>{{ cite report | section = Specifying the document's character encoding | title = HTML&nbsp;5.2 | date = 14 December 2017 | publisher = [[World Wide Web Consortium]] | url = https://www.w3.org/TR/html5/document-metadata.html | section-url = https://www.w3.org/TR/html5/document-metadata.html#charset | access-date = 2018-06-03 | mode = cs1 }}</ref>
+Many standards only support UTF-8, e.g. [[JSON]] exchange requires it (without a byte-order mark (BOM)).<ref name=rfc8259>{{ cite IETF | last = Bray | first = Tim | editor-last = Bray | editor-first = Tim | date = December 2017 | title = The JavaScript Object Notation (JSON) Data Interchange Format | publisher = IETF | doi = 10.17487/RFC8259 | access-date = 16 February 2018 | rfc = 8259 }}</ref> UTF-8 is also the recommendation from the [[WHATWG]] for HTML and [[Document Object Model|DOM]] specifications, and stating "UTF-8 encoding is the most appropriate encoding for interchange of [[Unicode]]"<ref name=":3" /> and the [[Internet Mail Consortium]] recommends that all e‑mail programs be able to display and create mail using UTF-8.<ref name=IMC>{{ cite web | url = https://www.imc.org/mail-i18n.html | title = Using International Characters in Internet Mail | publisher = Internet Mail Consortium | date = 1998-08-01 | access-date = 2007-11-08 | url-status = dead | archive-url = https://web.archive.org/web/20071026103104/https://www.imc.org/mail-i18n.html | archive-date = 2007-10-26}}</ref><ref>{{Cite web |title=Encoding Standard |url=https://encoding.spec.whatwg.org/#security-background |access-date=2025-11-20 |website=encoding.spec.whatwg.org}}</ref> The [[World Wide Web Consortium]] recommends UTF-8 as the default encoding in XML and HTML (and not just using UTF-8, also declaring it in metadata), "even when all characters are in the ASCII range ... Using non-UTF-8 encodings can have unexpected results".<ref name=html5charset>{{ cite report | section = Specifying the document's character encoding | title = HTML&nbsp;5.2 | date = 14 December 2017 | publisher = [[World Wide Web Consortium]] | url = https://www.w3.org/TR/html5/document-metadata.html | section-url = https://www.w3.org/TR/html5/document-metadata.html#charset | access-date = 2018-06-03 | mode = cs1 }}</ref>
-Many software programs have the ability to read/write UTF-8. It may require the user to change options from the normal settings, or may require a BOM (byte-order mark) as the first character to read the file. Examples of software supporting UTF-8 include [[Microsoft Word]],<!-- "Unicode (UTF-8)", "Unicode (Big-Endian)" and "Unicode (UTF-7)" --><ref>{{ cite web | title=Choose text encoding when you open and save files | website=Microsoft Support (support.microsoft.com) | url=https://support.microsoft.com/en-us/office/choose-text-encoding-when-you-open-and-save-files-60d59c21-88b5-4006-831c-d536d42fd861 | access-date=2021-11-01 }}</ref><ref>{{ cite web | title=UTF-8 - Character encoding of Microsoft ''Word'' <code>DOC</code> and <code>DOCX</code> files? | website=Stack Overflow | url=https://stackoverflow.com/questions/28172022/character-encoding-of-microsoft-word-doc-and-docx-files | access-date=2021-11-01 }}</ref><!-- <ref>{{ cite web | last=Gao |first=Ivy | title=How to fix corrupted character encoding (corrupted text) in Microsoft ''Word'' | website=TurboFuture | url=https://turbofuture.com/computers/3-Easy-Ways-To-Fix-Corrupted-Character-Encoding-In-Plain-Text-Documents | access-date=2021-11-01 | lang=en }}</ref> --><ref>{{ cite web | title = Exporting a UTF-8 <code>.txt</code> file from ''Word'' | website = support.3playmedia.com | date = 14 March 2023 | url = https://support.3playmedia.com/hc/en-us/articles/227730088-Exporting-a-UTF-8-txt-file-from-Word }}</ref> [[Microsoft Excel]] (2016 and later),<ref>{{ cite web | title = Are <code>XLSX</code> files UTF-8 encoded, by definition? | series = Excel | website = Stack Overflow | url = https://stackoverflow.com/questions/45194771/are-xlsx-files-utf-8-encoded-by-definition | access-date = 2021-11-01 }}</ref><ref>{{ cite web | author1 = Abhinav, Ankit | author2 = Xu, Jazlyn | date = April 13, 2020 | title = How to open UTF-8 <code>CSV</code> file in ''Excel'' without mis-conversion of characters in Japanese and Chinese language for both Mac and Windows? | website = Microsoft Support Community | language = en-US | url = https://answers.microsoft.com/en-us/msoffice/forum/all/how-to-open-utf-8-csv-file-in-excel-without-mis/1eb15700-d235-441e-8b99-db10fafff3c2 | access-date = 2021-11-01 }}</ref> [[Google Drive]], [[LibreOffice]],<ref>{{ cite web | title = Save a CSV file as UTF-8 | series = LibreOffice | website = RO CSVI | url = https://rolandd.com/documentation/ro-csvi/save-a-csv-file-as-utf-8 | access-date = 2025-05-20 }}</ref> and most databases.
+Many software programs have the ability to read/write UTF-8. It may require the user to change options from the normal settings, or may require a BOM (byte-order mark) as the first character to read the file. Examples of software supporting UTF-8 include [[Microsoft Word]],<!-- "Unicode (UTF-8)", "Unicode (Big-Endian)" and "Unicode (UTF-7)" --><ref>{{ cite web | title=Choose text encoding when you open and save files | website=Microsoft Support (support.microsoft.com) | url=https://support.microsoft.com/en-us/office/choose-text-encoding-when-you-open-and-save-files-60d59c21-88b5-4006-831c-d536d42fd861 | access-date=2021-11-01 }}</ref><ref>{{ cite web | title=UTF-8 - Character encoding of Microsoft ''Word'' <code>DOC</code> and <code>DOCX</code> files? | website=Stack Overflow | url=https://stackoverflow.com/questions/28172022/character-encoding-of-microsoft-word-doc-and-docx-files | access-date=2021-11-01 }}</ref><!-- <ref>{{ cite web | last=Gao |first=Ivy | title=How to fix corrupted character encoding (corrupted text) in Microsoft ''Word'' | website=TurboFuture | url=https://turbofuture.com/computers/3-Easy-Ways-To-Fix-Corrupted-Character-Encoding-In-Plain-Text-Documents | access-date=2021-11-01 | lang=en }}</ref> --><ref>{{ cite web | title = Exporting a UTF-8 <code>.txt</code> file from ''Word'' | website = support.3playmedia.com | date = 14 March 2023 | url = https://support.3playmedia.com/hc/en-us/articles/227730088-Exporting-a-UTF-8-txt-file-from-Word }}</ref> [[Microsoft Excel]] ([[Office 2003]] and later),<ref>{{ cite web | title = Are <code>XLSX</code> files UTF-8 encoded, by definition? | series = Excel | website = Stack Overflow | url = https://stackoverflow.com/questions/45194771/are-xlsx-files-utf-8-encoded-by-definition | access-date = 2021-11-01 }}</ref><ref>{{ cite web | author1 = Abhinav, Ankit | author2 = Xu, Jazlyn | date = April 13, 2020 | title = How to open UTF-8 <code>CSV</code> file in ''Excel'' without mis-conversion of characters in Japanese and Chinese language for both Mac and Windows? | website = Microsoft Support Community | language = en-US | url = https://answers.microsoft.com/en-us/msoffice/forum/all/how-to-open-utf-8-csv-file-in-excel-without-mis/1eb15700-d235-441e-8b99-db10fafff3c2 | access-date = 2021-11-01 }}</ref> [[Google Drive]], [[LibreOffice]],<ref>{{ cite web | title = Save a CSV file as UTF-8 | series = LibreOffice | website = RO CSVI | url = https://rolandd.com/documentation/ro-csvi/save-a-csv-file-as-utf-8 | access-date = 2025-05-20 }}</ref> and most databases.
 Software that "defaults" to UTF-8 (meaning it writes it without the user changing settings, and it reads it without a BOM) has become more common since 2010.<ref>{{ cite web | last=Galloway |first=Matt | date=October 2012 | title=Character encoding for iOS developers; or, UTF-8 what now? | website=www.galloway.me.uk | language=en-UK | url=https://www.galloway.me.uk/2012/10/character-encoding-for-ios-developers-utf8/ | access-date=2021-01-02 | quote = ...&nbsp;in reality, you usually just assume UTF-8 since that is by far the most common encoding. }}</ref> [[Windows Notepad]], in all currently supported versions of Windows, defaults to writing UTF-8 without a BOM (a change from {{nobr|[[Windows 7]]}} ''Notepad''), bringing it into line with most other text editors.<ref>{{ cite web | title=Windows&nbsp;10 Notepad is getting better UTF-8 encoding support | website=BleepingComputer | url=https://www.bleepingcomputer.com/news/microsoft/windows-10-notepad-is-getting-better-utf-8-encoding-support/ | access-date=2021-03-24 | quote=Microsoft is now defaulting to saving new text files as UTF-8 without BOM, as shown below. | language=en-us }}</ref> Some system files on [[Windows 11|Windows&nbsp;11]] require UTF-8<ref>{{ cite web | title = Customize the Windows&nbsp;11 ''Start'' menu | url=https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/customize-the-windows-11-start-menu | access-date=2021-06-29 | website=docs.microsoft.com | language=en-us | quote=Make sure your LayoutModification.json uses UTF-8 encoding. }}</ref> with no requirement for a BOM, and almost all files on macOS and Linux are required to be UTF-8 without a BOM.{{citation needed|date=June 2021}} Programming languages that default to UTF-8 for [[input/output|I/O]] include [[Ruby (programming language)|Ruby]]&nbsp;3.0,<ref>{{ cite web | title = Set default for Encoding.default_external to UTF-8 on Windows | series = Ruby master | id = Feature&nbsp;#16604 | website = Ruby Issue Tracking System (bugs.ruby-lang.org) | url = https://bugs.ruby-lang.org/issues/16604 | access-date = 2022-08-01 }}</ref><ref>{{ cite web | title = Feature #12650: Use UTF-8 encoding for ENV on Windows | series = Ruby master | website = Ruby Issue Tracking System (bugs.ruby-lang.org) | url = https://bugs.ruby-lang.org/issues/12650 | access-date = 2022-08-01 }}</ref> [[R (programming language)|R]]&nbsp;4.2.2,<ref>{{ cite web | title = New features in R&nbsp;4.2.0 | date = 2022-04-01 | website = R&nbsp;bloggers (r-bloggers.com) | series = The Jumping Rivers Blog | url = https://www.r-bloggers.com/2022/04/new-features-in-r-4-2-0/ | access-date = 2022-08-01 | language = en-US }}</ref> [[Raku (programming language)|Raku]] and [[Java (programming language)|Java]]&nbsp;18.<ref name=Java_UTF-8_and_UTF-16>{{ cite web | title = UTF-8 by default | id = JEP 400 | website = openjdk.java.net | url = https://openjdk.java.net/jeps/400 | access-date=2022-03-30 }}</ref> Although the current version of [[Python (programming language)|Python]] requires an option to <code>open()</code> to read/write UTF-8,<ref>{{ cite web | title = add a new UTF-8 mode | website = peps.python.org | id = PEP&nbsp;540 | url = https://peps.python.org/pep-0540/ | access-date = 2022-09-23 }}</ref> plans exist to make UTF-8 I/O the default in Python&nbsp;3.15.<ref>{{ cite web | title = Make UTF-8 mode default | website = peps.python.org | id = PEP&nbsp;686 | url = https://peps.python.org/pep-0686/ | access-date=2023-07-26 }}</ref> [[C++23]] adopts UTF-8 as the only portable source code file format.<ref>{{ cite report | title = Support for UTF-8 as a portable source file encoding | year = 2022 | id = p2295r6 | website = open-std.org | url = https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p2295r6.pdf }}</ref>
@@ Line 143: / Line 158: @@
 Backwards compatibility is a serious impediment to changing code and APIs using [[UTF-16]] to use UTF-8, but this is happening. {{As of|2019|05}}, Microsoft [[Unicode in Microsoft Windows#UTF-8|added the capability]] for an application to set UTF-8 as the "code page" for the Windows API, removing the need to use UTF-16; and more recently has recommended programmers use UTF-8,<ref name=Microsoft-UTF-8>{{ cite web | title=Use UTF-8 code pages in Windows apps | website=[[Microsoft Learn]] | date=20 August 2024 |language=en-us | url=https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page | access-date=2024-09-24}}</ref> and even states "UTF-16 [...] is a unique burden that Windows places on code that targets multiple platforms".<ref name="Microsoft GDK">{{ cite web | title=UTF-8 support in the Microsoft GDK | series = Microsoft Game Development Kit (GDK) | website = [[Microsoft Learn]] |language=en-us | url=https://learn.microsoft.com/en-us/gaming/gdk/_content/gc/system/overviews/utf-8 | access-date = 2023-03-05 }}</ref>
 The default string primitive in [[Go (programming language)|Go]],<ref>{{ cite report | section=Source code representation | title=The ''Go'' Programming Language Specification | website=golang.org | section-url=https://golang.org/ref/spec#Source_code_representation | access-date=2021-02-10 }}</ref>
-[[Julia (programming language)|Julia]], [[Rust (programming language)|Rust]], [[Swift (programming language)#String support|Swift]] (since version 5),<ref>{{ cite web | last=Tsai |first=Michael J. | date=21 March 2019 | title=UTF-8 string in Swift&nbsp;5 | type=blog post |language=en | url=https://mjtsai.com/blog/2019/03/21/utf-8-string-in-swift-5/ | access-date=2021-03-15 }}</ref> and [[PyPy]]<ref>{{ cite web | title=PyPy v7.1 released; now uses UTF-8 internally for Unicode strings | department=Mattip | date=2019-03-24 | website=PyPy status blog | url=https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-uses-utf-8.html | access-date=2020-11-21 }}</ref> uses UTF-8 internally in all cases. Python (since version 3.3) uses UTF-8 internally for Python C API extensions<ref name=PEP393>{{ cite web | title = Flexible String Representation | id = PEP&nbsp;393 | website = Python.org |language=en | url = https://peps.python.org/pep-0393 | access-date = 2022-05-18 }}</ref><ref>{{Cite web |title=Common Object Structures |url=https://docs.python.org/3/c-api/structures.html |access-date=2024-05-29 |website=Python documentation |language=en}}</ref> and sometimes for strings<ref name=PEP393/><ref>{{ cite web | title=Unicode objects and codecs | url=https://docs.python.org/3/c-api/unicode.html | access-date=2023-08-19 |website=Python documentation | quote=UTF-8 representation is created on demand and cached in the Unicode object.}}</ref> and a future version of Python is planned to store strings as UTF-8 by default.<ref>{{ cite web | title=PEP&nbsp;623&nbsp;– remove wstr from Unicode | website=Python.org |language=en | url=https://www.python.org/dev/peps/pep-0623/ | access-date=2020-11-21 }}</ref><ref>{{ cite web | last=Wouters |first=Thomas | date=2023-07-11 | title=Python 3.12.0 beta 4 released | website = Python Insider (pythoninsider.blogspot.com) | type = blog post | url=https://pythoninsider.blogspot.com/2023/07/pleased-to-announce-release-of-python-3.html | access-date=2023-07-26 | quote=The deprecated <code>wstr</code> and <code>wstr_length</code> members of the C implementation of unicode objects were removed, per PEP 623. }}</ref> Modern versions of [[Microsoft Visual Studio]] use UTF-8 internally.<ref>{{ cite web | title=validate-charset (validate for compatible characters) | website=docs.microsoft.com |language=en-us | url=https://docs.microsoft.com/en-us/cpp/build/reference/validate-charset-validate-for-compatible-characters | access-date=2021-07-19 | quote=Visual Studio uses UTF-8 as the internal character encoding during conversion between the source character set and the execution character set. }}</ref> Microsoft's SQL Server 2019 added support for UTF-8, and using it results in a 35% speed increase, and "nearly 50% reduction in storage requirements."<ref>{{ cite web | title = Introducing UTF-8 support for SQL Server | date = 2019-07-02 | website = techcommunity.microsoft.com | url = https://techcommunity.microsoft.com/t5/sql-server/introducing-utf-8-support-for-sql-server/ba-p/734928 | access-date = 2021-08-24 | language = en-US }}</ref>
+[[Julia (programming language)|Julia]], [[Rust (programming language)|Rust]], [[Swift (programming language)#String support|Swift]] (since version 5),<ref>{{ cite web | last=Tsai |first=Michael J. | date=21 March 2019 | title=UTF-8 string in Swift&nbsp;5 | type=blog post |language=en | url=https://mjtsai.com/blog/2019/03/21/utf-8-string-in-swift-5/ | access-date=2021-03-15 }}</ref> and [[PyPy]]<ref>{{Cite web |last=Mattip |date=2019-03-24 |title=PyPy Status Blog: PyPy v7.1 released; now uses utf-8 internally for unicode strings |url=https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-uses-utf-8.html |access-date=2025-11-20 |website=PyPy Status Blog}}</ref> uses UTF-8 internally in all cases. Python (since version 3.3) uses UTF-8 internally for Python C API extensions<ref name=PEP393>{{ cite web | title = Flexible String Representation | id = PEP&nbsp;393 | website = Python.org |language=en | url = https://peps.python.org/pep-0393 | access-date = 2022-05-18 }}</ref><ref>{{Cite web |title=Common Object Structures |url=https://docs.python.org/3/c-api/structures.html |access-date=2025-11-20 |website=Python documentation |language=en}}</ref> and sometimes for strings<ref name=PEP393/><ref>{{ cite web | title=Unicode objects and codecs | url=https://docs.python.org/3/c-api/unicode.html | access-date=2023-08-19 |website=Python documentation | quote=UTF-8 representation is created on demand and cached in the Unicode object.}}</ref> and a future version of Python is planned to store strings as UTF-8 by default.<ref>{{ cite web | title=PEP&nbsp;623&nbsp;– remove wstr from Unicode | website=Python.org |language=en | url=https://www.python.org/dev/peps/pep-0623/ | access-date=2020-11-21 }}</ref><ref>{{ cite web | last=Wouters |first=Thomas | date=2023-07-11 | title=Python 3.12.0 beta 4 released | website = Python Insider (pythoninsider.blogspot.com) | type = blog post | url=https://pythoninsider.blogspot.com/2023/07/pleased-to-announce-release-of-python-3.html | access-date=2023-07-26 | quote=The deprecated <code>wstr</code> and <code>wstr_length</code> members of the C implementation of unicode objects were removed, per PEP 623. }}</ref> Modern versions of [[Microsoft Visual Studio]] use UTF-8 internally.<ref>{{ cite web | title=validate-charset (validate for compatible characters) | website=docs.microsoft.com |language=en-us | url=https://docs.microsoft.com/en-us/cpp/build/reference/validate-charset-validate-for-compatible-characters | access-date=2021-07-19 | quote=Visual Studio uses UTF-8 as the internal character encoding during conversion between the source character set and the execution character set. }}</ref> Microsoft's SQL Server 2019 added support for UTF-8, and using it results in a 35% speed increase, and "nearly 50% reduction in storage requirements."<ref>{{ cite web | title = Introducing UTF-8 support for SQL Server | date = 2019-07-02 | website = techcommunity.microsoft.com | url = https://techcommunity.microsoft.com/t5/sql-server/introducing-utf-8-support-for-sql-server/ba-p/734928 | access-date = 2021-08-24 | language = en-US }}</ref>
+{{anchor|Modified UTF-8}}[[Java (programming language)|Java]] internally uses UTF-16 for the ''char'' data type and, consequentially, the ''Character'', ''String'', and the ''StringBuffer'' classes,<ref>{{cite web |title=Character (Java SE 24 & JDK 24) |url=https://docs.oracle.com/en/java/javase/24/docs/api/java.base/java/lang/Character.html#unicode |year=2025 |publisher=[[Oracle Corporation]] |access-date=2025-04-08}}</ref> but for I/O uses ''Modified UTF-8'' (MUTF-8), in which the [[null character]] {{tt|U+0000}} uses the two-byte overlong encoding {{tt|0xC0}},&nbsp;{{tt|0x80}}, instead of just {{tt|0x00}}.<ref name=":2">{{cite web |title=Java SE documentation for Interface java.io.DataInput, subsection on Modified UTF-8 |url=https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html#modified-utf-8 |year=2015 |publisher=[[Oracle Corporation]] |access-date=2015-10-16}}</ref> Modified UTF-8 strings never contain any actual null bytes but can contain all Unicode code points including {{tt|U+0000}},<ref name=":0">{{cite web |url=https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.4.7 |title=The Java Virtual Machine Specification, section 4.4.7: "The CONSTANT_Utf8_info Structure" |publisher=[[Oracle Corporation]] |year=2015 |access-date=2015-10-16}}</ref> which allows such strings (with a null byte appended) to be processed by traditional [[null-terminated string]] functions. Java reads and writes normal UTF-8 to files and streams,<ref>{{Javadoc:SE|java/io|InputStreamReader}} and {{Javadoc:SE|java/io|OutputStreamWriter}}</ref> but it uses Modified UTF-8 for object [[Java serialization|serialization]],<ref>{{cite web |title=Java Object Serialization Specification, chapter 6: Object Serialization Stream Protocol, section 2: Stream Elements |url=https://docs.oracle.com/javase/8/docs/platform/serialization/spec/protocol.html#a8299 |year=2010 |publisher=[[Oracle Corporation]] |access-date=2015-10-16}}</ref><ref>{{Javadoc:SE|java/io|DataInput}} and {{Javadoc:SE|java/io|DataOutput}}</ref> for the [[Java Native Interface]],<ref>{{cite web |url=https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/types.html#modified_utf_8_strings |title=Java Native Interface Specification, chapter 3: JNI Types and Data Structures, section: Modified UTF-8 Strings |publisher=[[Oracle Corporation]] |year=2015 |access-date=2015-10-16}}</ref> and for embedding constant strings in [[Java class file]]s.<ref name=":0" /> The dex format defined by [[Dalvik (software)|Dalvik]] also uses the same modified UTF-8 to represent string values.<ref>{{cite web |url=https://source.android.com/tech/dalvik/dex-format.html |title=ART and Dalvik |work=Android Open Source Project |access-date=2013-04-09 |url-status=dead |archive-url=https://web.archive.org/web/20130426010617/https://source.android.com/tech/dalvik/dex-format.html |archive-date=2013-04-26 }}</ref> [[Tcl]] also uses the same modified UTF-8<ref>{{cite web |title=UTF-8 bit by bit |date=2001-02-28 |url=https://wiki.tcl-lang.org/page/UTF-8+bit+by+bit |access-date=2022-09-03 |website=Tcler's Wiki}}</ref> as Java for internal representation of Unicode data, but uses strict CESU-8 for external data. All known Modified UTF-8 implementations also treat the surrogate pairs as in [[CESU-8]].
-{{anchor|Modified UTF-8}}
+The [[Raku (programming language)|Raku]] programming language (formerly Perl 6) uses <code>utf-8</code> encoding by default for I/O ([[Perl]] 5 also supports it<!-- "utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code" -->)<!-- "Raku applies normalization by default to all input and output except for file names, which are read and written as UTF8-C8" -->; though that choice in Raku also implies "normalization into Unicode [[Unicode equivalence#Normal forms|NFC (normalization form canonical)]]. In some cases the user will want to ensure no normalization is done; for this <code>utf8-c8</code>" can be used.<ref>{{Cite web |title=encoding {{!}} Raku Documentation |url=https://docs.raku.org/routine/encoding |access-date=2025-11-20 |website=docs.raku.org}}</ref> That ''UTF-8 Clean-8'' variant, implemented by Raku, is an encoder/decoder <!-- that primarily works as the UTF-8 one. However, upon encountering a byte sequence that will either not decode as valid UTF-8, or that would not round-trip due to normalization, it will use NFG synthetics to keep track of the original bytes involved. This means that encoding back to UTF-8 Clean-8 will be able to recreate the bytes as they originally existed. The synthetics contain four codepoints: ... --> that preserves bytes as is (even illegal UTF-8 sequences) and allows for Normal Form Grapheme synthetics.<ref>{{Cite web |title=Unicode {{!}} Raku Documentation |url=https://docs.raku.org/language/unicode#UTF8-C8 |access-date=2025-11-20 |website=docs.raku.org}}</ref>
-[[Java (programming language)|Java]] internally uses UTF-16 for the ''char'' data type and, consequentially, the ''Character'', ''String'', and the ''StringBuffer'' classes,<ref>{{cite web |title=Character (Java SE 24 & JDK 24) |url=https://docs.oracle.com/en/java/javase/24/docs/api/java.base/java/lang/Character.html#unicode |year=2025 |publisher=[[Oracle Corporation]] |access-date=2025-04-08}}</ref> but for I/O uses ''Modified UTF-8'' (MUTF-8), in which the [[null character]] {{tt|U+0000}} uses the two-byte overlong encoding {{tt|0xC0}},&nbsp;{{tt|0x80}}, instead of just {{tt|0x00}}.<ref>{{cite web |title=Java SE documentation for Interface java.io.DataInput, subsection on Modified UTF-8 |url=https://docs.oracle.com/javase/8/docs/api/java/io/DataInput.html#modified-utf-8 |year=2015 |publisher=[[Oracle Corporation]] |access-date=2015-10-16}}</ref> Modified UTF-8 strings never contain any actual null bytes but can contain all Unicode code points including {{tt|U+0000}},<ref>{{cite web |url=https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.4.7 |title=The Java Virtual Machine Specification, section 4.4.7: "The CONSTANT_Utf8_info Structure" |publisher=[[Oracle Corporation]] |year=2015 |access-date=2015-10-16}}</ref> which allows such strings (with a null byte appended) to be processed by traditional [[null-terminated string]] functions. Java reads and writes normal UTF-8 to files and streams,<ref>{{Javadoc:SE|java/io|InputStreamReader}} and {{Javadoc:SE|java/io|OutputStreamWriter}}</ref> but it uses Modified UTF-8 for object [[Java serialization|serialization]],<ref>{{cite web |title=Java Object Serialization Specification, chapter 6: Object Serialization Stream Protocol, section 2: Stream Elements |url=https://docs.oracle.com/javase/8/docs/platform/serialization/spec/protocol.html#a8299 |year=2010 |publisher=[[Oracle Corporation]] |access-date=2015-10-16}}</ref><ref>{{Javadoc:SE|java/io|DataInput}} and {{Javadoc:SE|java/io|DataOutput}}</ref> for the [[Java Native Interface]],<ref>{{cite web |url=https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/types.html#modified_utf_8_strings |title=Java Native Interface Specification, chapter 3: JNI Types and Data Structures, section: Modified UTF-8 Strings |publisher=[[Oracle Corporation]] |year=2015 |access-date=2015-10-16}}</ref> and for embedding constant strings in [[Class (file format)|class files]].<ref>{{cite web |title=The Java Virtual Machine Specification, section 4.4.7: "The CONSTANT_Utf8_info Structure" |url=https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-4.html#jvms-4.4.7 |publisher=[[Oracle Corporation]] |year=2015 |access-date=2015-10-16}}</ref> The dex format defined by [[Dalvik (software)|Dalvik]] also uses the same modified UTF-8 to represent string values.<ref>{{cite web |url=https://source.android.com/tech/dalvik/dex-format.html |title=ART and Dalvik |work=Android Open Source Project |access-date=2013-04-09 |url-status=dead |archive-url=https://web.archive.org/web/20130426010617/https://source.android.com/tech/dalvik/dex-format.html |archive-date=2013-04-26 }}</ref> [[Tcl]] also uses the same modified UTF-8<ref>{{cite web |title=UTF-8 bit by bit |date=2001-02-28 |url=https://wiki.tcl-lang.org/page/UTF-8+bit+by+bit |access-date=2022-09-03 |website=Tcler's Wiki}}</ref> as Java for internal representation of Unicode data, but uses strict CESU-8 for external data. All known Modified UTF-8 implementations also treat the surrogate pairs as in [[CESU-8]].
-The [[Raku (programming language)|Raku]] programming language (formerly Perl 6) uses <code>utf-8</code> encoding by default for I/O ([[Perl]] 5 also supports it<!-- "utf8 - Perl pragma to enable/disable UTF-8 (or UTF-EBCDIC) in source code" -->)<!-- "Raku applies normalization by default to all input and output except for file names, which are read and written as UTF8-C8" -->; though that choice in Raku also implies "normalization into Unicode [[Unicode equivalence#Normal forms|NFC (normalization form canonical)]]. In some cases the user will want to ensure no normalization is done; for this <code>utf8-c8</code>" can be used.<ref>{{Cite web |title=encoding {{!}} Raku Documentation |url=https://docs.raku.org/routine/encoding |access-date=2024-10-06 |website=docs.raku.org}}</ref> That ''UTF-8 Clean-8'' variant, implemented by Raku, is an encoder/decoder <!-- that primarily works as the UTF-8 one. However, upon encountering a byte sequence that will either not decode as valid UTF-8, or that would not round-trip due to normalization, it will use NFG synthetics to keep track of the original bytes involved. This means that encoding back to UTF-8 Clean-8 will be able to recreate the bytes as they originally existed. The synthetics contain four codepoints: ... --> that preserves bytes as is (even illegal UTF-8 sequences) and allows for Normal Form Grapheme synthetics.<ref>{{Cite web |title=Unicode {{!}} Raku Documentation |url=https://docs.raku.org/language/unicode#UTF8-C8 |access-date=2024-10-06 |website=docs.raku.org}}</ref>
+Version 3 of the [[Python (programming language)|Python]] programming language treats each byte of an invalid UTF-8 bytestream as an error (see also changes with new UTF-8 mode in Python 3.7<ref>{{Cite web |title=PEP 540 – Add a new UTF-8 Mode {{!}} peps.python.org |url=https://peps.python.org/pep-0540/ |access-date=2025-11-20 |website=Python Enhancement Proposals (PEPs) |language=en}}</ref>); this gives 128 different possible errors. Extensions have been created to allow any byte sequence that is assumed to be UTF-8 to be losslessly transformed to UTF-16 or UTF-32, by translating the 128 possible error bytes to 128 reserved code points, and transforming those code points back to error bytes to output UTF-8. The most common approach is to translate the codes to {{tt|U+DC80}}...{{tt|U+DCFF}} which are low (trailing) surrogate values and thus "invalid" UTF-16, as used by [[Python (programming language)|Python]]'s [[Python Enhancement Proposal|PEP]] 383 (or "surrogateescape") approach.<ref name="pep383">{{cite web |last=von Löwis |first=Martin |date=2009-04-22 |title=Non-decodable Bytes in System Character Interfaces |url=https://www.python.org/dev/peps/pep-0383 |access-date=2025-11-20 |publisher=[[Python Software Foundation]] |language=en |id=PEP 383}}</ref> [[NumPy]] version 2.0, and its file formats, support UTF-8 (adding StringDType for it<!--, in addition to fixed-with Unicode type. Asked AI: What's the difference between StringDType and numpy.str_ ?  StringDType (or np.dtype.StringDType in NumPy 2.0+    numpy.str_ (or numpy.unicode_ or U): This is the traditional NumPy fixed-width Unicode string dtype. It stores strings using a fixed-size UCS4 encoding -->).<ref>{{Cite web |title=NEP 55 — Add a UTF-8 variable-width string DType to NumPy — NumPy Enhancement Proposals |url=https://numpy.org/neps/nep-0055-string_dtype.html |access-date=2025-11-20 |website=numpy.org}}</ref> Another encoding called [[MirBSD]] OPTU-8/16 converts them to {{tt|U+EF80}}...{{tt|U+EFFF}} in a [[Private Use Area]].<ref>{{cite web |title=RTFM optu8to16(3), optu8to16vis(3) |url=https://www.mirbsd.org/htman/i386/man3/optu8to16.htm |access-date=2025-11-20 |website=www.mirbsd.org}}</ref> In either approach, the byte value is encoded in the low eight bits of the output code point. These encodings are needed if invalid UTF-8 is to survive translation to and then back from the UTF-16 used internally by Python, and as Unix filenames can contain invalid UTF-8 it is necessary for this to work.<ref name="davis383">{{cite web |last1=Davis |first1=Mark |author-link1=Mark Davis (Unicode) |last2=Suignard |first2=Michel |year=2014 |title=3.7 Enabling Lossless Conversion |url=https://www.unicode.org/reports/tr36/#EnablingLosslessConversion |access-date=2025-11-20 |work=Unicode Security Considerations |id=Unicode Technical Report #36}}</ref>
-Version 3 of the [[Python (programming language)|Python]] programming language treats each byte of an invalid UTF-8 bytestream as an error (see also changes with new UTF-8 mode in Python 3.7<ref>{{Cite web|title=PEP 540 -- Add a new UTF-8 Mode|url=https://www.python.org/dev/peps/pep-0540/|access-date=2021-03-24|website=Python.org|language=en}}</ref>); this gives 128 different possible errors. Extensions have been created to allow any byte sequence that is assumed to be UTF-8 to be losslessly transformed to UTF-16 or UTF-32, by translating the 128 possible error bytes to 128 reserved code points, and transforming those code points back to error bytes to output UTF-8. The most common approach is to translate the codes to {{tt|U+DC80}}...{{tt|U+DCFF}} which are low (trailing) surrogate values and thus "invalid" UTF-16, as used by [[Python (programming language)|Python]]'s [[Python Enhancement Proposal|PEP]] 383 (or "surrogateescape") approach.<ref name="pep383">{{cite web |id=PEP 383 |title=Non-decodable Bytes in System Character Interfaces |url=https://www.python.org/dev/peps/pep-0383 |publisher=[[Python Software Foundation]] |language=en |first=Martin |last=von Löwis |date=2009-04-22}}</ref> Another encoding called [[MirBSD]] OPTU-8/16 converts them to {{tt|U+EF80}}...{{tt|U+EFFF}} in a [[Private Use Area]].<ref>{{cite web |title=RTFM optu8to16(3), optu8to16vis(3) |url=https://www.mirbsd.org/htman/i386/man3/optu8to16.htm |website=www.mirbsd.org}}</ref> In either approach, the byte value is encoded in the low eight bits of the output code point. These encodings are needed if invalid UTF-8 is to survive translation to and then back from the UTF-16 used internally by Python, and as Unix filenames can contain invalid UTF-8 it is necessary for this to work.<ref name="davis383">{{cite web |url=https://www.unicode.org/reports/tr36/#EnablingLosslessConversion |last1=Davis |first1=Mark |author-link1=Mark Davis (Unicode) |first2=Michel |last2=Suignard |title=3.7 Enabling Lossless Conversion |work=Unicode Security Considerations |id=Unicode Technical Report #36 |year=2014}}</ref>
+Linux and macOS filesystems support UTF-8: [[ext4]]<ref>{{cite web |title=Ext4 General Information — the Linux Kernel documentation |url=https://www.kernel.org/doc/html/latest/admin-guide/ext4.html |access-date=2025-11-20}}</ref> and Apple's [[Apple File System|APFS]].<ref>{{cite web |title=Frequently Asked Questions |url=https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html |access-date=2025-11-20 |work=Apple File System Guide |publisher=[[Apple Inc.|Apple]]}}</ref> Apple's older [[HFS Plus]] uses [[UTF-16]] for file names, but uses UTF-8 in [[symbolic link]]s.<ref>{{cite web |title=Technical Note TN1150: HFS Plus Volume Format |url=https://developer.apple.com/library/archive/technotes/tn/tn1150.html |access-date=2025-11-20 |publisher=[[Apple Inc.|Apple]]}}</ref> Windows' filesystem, [[NTFS]], uses UTF-16 for file names.
 == Standards ==
 The official name for the encoding is {{code|UTF-8}}, the spelling used in all Unicode Consortium documents. The [[hyphen-minus]] is required and no spaces are allowed. Some other names used are:
-* Most standards are also case-insensitive and <code>utf-8</code> is often used.{{citation needed|date=March 2023}}
+* Most standards are also case-insensitive and {{code|utf-8}} is often used.{{citation needed|date=March 2023}}
 * Web standards (which include [[Cascading Style Sheets|CSS]], [[HTML]], [[XML]], and [[HTTP headers]]) also allow {{code|utf8}} and many other aliases<!-- e.g. "unicode20utf8" for UTF-8, likely not useful to list any or all, just stating "many"-->.<ref>{{cite web|url=https://encoding.spec.whatwg.org/#names-and-labels|title=Encoding Standard § 4.2. Names and labels|publisher=[[WHATWG]]|access-date=2018-04-29}}</ref>
 * The official [[Internet Assigned Numbers Authority]] lists {{code|csUTF8}} as the only alias,<ref name="IANA_2013_CS">{{cite web |publisher=[[Internet Assigned Numbers Authority]] |url=https://www.iana.org/assignments/character-sets |title=Character Sets |date=2013-01-23 |access-date=2013-02-08}}</ref> which is rarely used.
 * In some locales {{code|UTF-8N}} means UTF-8 ''without'' a [[byte order mark|byte-order mark]] (BOM), and in this case {{code|UTF-8}} ''may'' imply there ''is'' a BOM.<ref>{{cite web |url=https://suika.fam.cx/~wakaba/wiki/sw/n/BOM |title=BOM | work = suikawiki |archive-url=https://web.archive.org/web/20090117052232/https://suika.fam.cx/~wakaba/wiki/sw/n/BOM |archive-date=2009-01-17 |language=ja}}</ref><ref>{{cite web |author-last=Davis |author-first=Mark |author-link=Mark Davis (Unicode) |title=Forms of Unicode |publisher=[[IBM]] |url=https://www-128.ibm.com/developerworks/library/utfencodingforms/index.html |access-date=2013-09-18 |archive-url=https://web.archive.org/web/20050506211548/https://www-128.ibm.com/developerworks/library/utfencodingforms/index.html |archive-date=2005-05-06}}</ref>
-* In [[Windows]], UTF-8 is [[Windows code page|codepage]] <code>65001</code><ref>{{Cite web |url=https://www.dostips.com/forum/viewtopic.php?t=5357 |title=UTF-8 codepage 65001 in Windows 7 - part I |author=Liviu |quote=Previously under XP (and, unverified, but probably Vista, too) for loops simply did not work while codepage 65001 was active |language=en-gb |date=2014-02-07 |access-date=2018-01-30}}</ref> with the symbolic name <code>CP_UTF8</code> in source code.
+* In [[Windows]], UTF-8 is [[Windows code page|codepage]] {{code|65001}}<ref>{{Cite web |url=https://www.dostips.com/forum/viewtopic.php?t=5357 |title=UTF-8 codepage 65001 in Windows 7 - part I |author=Liviu |quote=Previously under XP (and, unverified, but probably Vista, too) for loops simply did not work while codepage 65001 was active |language=en-gb |date=2014-02-07 |access-date=2018-01-30}}</ref> with the symbolic name {{code|CP_UTF8}} in source code.
-* In [[MySQL]], UTF-8 is called <code>utf8mb4</code>,<ref>{{Cite web |title=MySQL :: MySQL 8.0 Reference Manual :: 10.9.1 The utf8mb4 Character Set (4-Byte UTF-8 Unicode Encoding) |url=https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb4.html |work=MySQL 8.0 Reference Manual |publisher=[[Oracle Corporation]] |access-date=2023-03-14}}</ref> while {{code|utf8}} and {{code|utf8mb3}} refer to the obsolete [[CESU-8]] variant.<ref name="mysql3-utf8mb3">{{Cite web |title=MySQL :: MySQL 8.0 Reference Manual :: 10.9.2 The utf8mb3 Character Set (3-Byte UTF-8 Unicode Encoding) |url=https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb3.html |work=MySQL 8.0 Reference Manual |publisher=[[Oracle Corporation]] |access-date=2023-02-24}}</ref>
+* In [[MySQL]], UTF-8 is called {{code|utf8mb4}},<ref>{{Cite web |title=MySQL :: MySQL 8.0 Reference Manual :: 10.9.1 The utf8mb4 Character Set (4-Byte UTF-8 Unicode Encoding) |url=https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb4.html |work=MySQL 8.0 Reference Manual |publisher=[[Oracle Corporation]] |access-date=2023-03-14}}</ref> while {{code|utf8}} and {{code|utf8mb3}} refer to the obsolete [[CESU-8]] variant.<ref name="mysql3-utf8mb3">{{Cite web |title=MySQL :: MySQL 8.0 Reference Manual :: 10.9.2 The utf8mb3 Character Set (3-Byte UTF-8 Unicode Encoding) |url=https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb3.html |work=MySQL 8.0 Reference Manual |publisher=[[Oracle Corporation]] |access-date=2023-02-24}}</ref>
-* In [[Oracle Database]] (since version 9.0), <code>AL32UTF8</code><ref>{{Cite web |title=Database Globalization Support Guide |url=https://docs.oracle.com/cd/E11882_01/server.112/e10729/ch6unicode.htm |access-date=2023-03-16 |website=docs.oracle.com |language=en}}</ref> means UTF-8, while {{code|UTF-8}} means CESU-8.
+* In [[Oracle Database]], {{code|AL32UTF8}} means UTF-8 (since version 9.0), while {{code|UTF8}} means CESU-8 (since 8.0),<ref>{{Cite web |title=Database Globalization Support Guide |url=https://docs.oracle.com/cd/E11882_01/server.112/e10729/ch6unicode.htm |access-date=2023-03-16 |website=docs.oracle.com |language=en}}</ref> and ''Oracle's UTF8'' encoding should not be used since it's only a subset of the full (UTF8/AL32UTF8; missing support for Japanese and Chinese, only supporting Unicode 3.0, not 4-byte form; {{code|AL32UTFFSS}} is another incomplete subset for older Oracle databases).<ref>{{Cite web |last=Hood |first=Doug |date=July 10, 2025 |title=Why the Database Character Set Matters |url=https://blogs.oracle.com/timesten/post/why-databasecharacterset-matters |access-date=2025-11-20 |website=blogs.oracle.com}}</ref>
-* In HP [[Printer Command Language|PCL]], the Symbol-ID for UTF-8 is <code>18N</code>.<ref>{{Cite web|url=https://pclhelp.com/pcl-symbol-sets/ |archive-url=https://web.archive.org/web/20150219212843/http://pclhelp.com/pcl-symbol-sets/|url-status=dead|archive-date=2015-02-19|title=HP PCL Symbol Sets {{!}} Printer Control Language (PCL & PXL) Support Blog|date=2015-02-19|access-date=2018-01-30}}</ref>
+* In HP [[Printer Command Language|PCL]], the Symbol-ID for UTF-8 is {{code|18N}}.<ref>{{Cite web|url=https://pclhelp.com/pcl-symbol-sets/ |archive-url=https://web.archive.org/web/20150219212843/http://pclhelp.com/pcl-symbol-sets/|url-status=dead|archive-date=2015-02-19|title=HP PCL Symbol Sets {{!}} Printer Control Language (PCL & PXL) Support Blog|date=2015-02-19|access-date=2018-01-30}}</ref>
 There are several current definitions of UTF-8 in various standards documents:
@@ Line 168: / Line 184: @@
 * {{IETF RFC|3629|link=no}} / STD 63 (2003), which establishes UTF-8 as a standard internet protocol element
 * {{IETF RFC|5198|link=no}} defines UTF-8 [[Unicode equivalence|NFC]] for Network Interchange (2008)
-* ISO/IEC 10646:2020/Amd 1:2023<!-- §9.1 (2023? or 2020)--><ref>[https://www.iso.org/standard/83362.html ISO/IEC 10646].</ref>
+* ISO/IEC 10646:2020/Amd 1:2023<!-- §9.1 (2023? or 2020)--><ref>{{Cite web |title=ISO/IEC 10646:2020/Amd 1:2023 |url=https://www.iso.org/standard/83362.html |access-date=2025-11-20 |website=ISO |language=en}}</ref>
-* ''The Unicode Standard, Version 16.0.0'' (2024)<ref>''[https://www.unicode.org/versions/Unicode16.0.0/ The Unicode Standard, Version 16.0]'' [https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G31703 §3.9 D92, §3.10 D95], 2021.</ref>
+* ''The Unicode Standard, Version 17.0.0'' (2025)<!-- <ref>''[https://www.unicode.org/versions/Unicode16.0.0/ The Unicode Standard, Version 16.0]'' [https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G31703 §3.9 D92, §3.10 D95], 2021.</ref> -->
 They supersede the definitions given in the following obsolete works:
@@ Line 178: / Line 194: @@
 * {{IETF RFC|2279|link=no}} (1998)
 * ''The Unicode Standard, Version 3.0'', §2.3 (2000) plus Corrigendum #1 : UTF-8 Shortest Form (2000)
-* ''Unicode Standard Annex #27: Unicode 3.1'' (2001)<ref>[https://www.unicode.org/reports/tr27/tr27-3.html ''Unicode Standard Annex #27: Unicode 3.1''], 2001.</ref>
+* ''Unicode Standard Annex #27: Unicode 3.1'' (2001)<ref>{{Cite web |title=UAX #27: Unicode 3.1 |url=https://www.unicode.org/reports/tr27/tr27-3.html |access-date=2025-11-20 |website=www.unicode.org}}</ref>
 * <!-- Is there a reason to single out 5.0 and 6.0, but not e.g. 15? Skip all after 3.0, since only then encoding of UTF-8 changed? -->''The Unicode Standard, Version 5.0'' (2006)<ref>[https://www.unicode.org/versions/Unicode5.0.0/ ''The Unicode Standard, Version 5.0''] [https://www.unicode.org/versions/Unicode5.0.0/ch03.pdf §3.9–§3.10 ch. 3], 2006.</ref>
-* ''The Unicode Standard, Version 6.0'' (2010)<ref>[https://www.unicode.org/versions/Unicode6.0.0/ ''The Unicode Standard, Version 6.0''] [https://www.unicode.org/versions/Unicode6.0.0/ch03.pdf §3.9 D92, §3.10 D95], 2010.</ref>
+* ''The Unicode Standard, Version 6.0'' (2010)<ref name=":1" />
 They are all the same in their general mechanics, with the main differences being on issues such as allowed range of code point values and safe handling of invalid input.

UTF-8: Difference between revisions

Latest revision as of 12:56, 20 November 2025

Contents

History

Description

Overlong encodings

Error handling

Surrogates

Byte map

Byte-order mark

Comparison to UTF-16

Implementations and adoption

Standards

See also

References

External links

Navigation menu

UTF-8: Difference between revisions

Latest revision as of 12:56, 20 November 2025

History

Description

Overlong encodings

Error handling

Surrogates

Byte map

Byte-order mark

Comparison to UTF-16

Implementations and adoption

Standards

See also

References

External links

Navigation menu

Search