View source for Byte order mark

{{Use dmy dates|date=April 2022}}
{{Short description|Unicode character}}
{{Redirect|FEFF}}

The '''byte-order mark''' ('''BOM''') is a particular usage of the special [[Unicode]] character code, {{unichar|FEFF|Zero Width No-Break Space}}, whose appearance as a [[Magic number (programming)#Magic numbers in files|magic number]] at the start of a text stream can signal several things to a [[computer program|program]] reading the text:<ref name="unicode FAQ">{{cite web|url=https://www.unicode.org/faq/utf_bom.html#BOM |title=FAQ - UTF-8, UTF-16, UTF-32 & BOM |website=Unicode.org |access-date=28 January 2017}}</ref>
* the byte order, or [[endianness]], of the text stream in the cases of 16-[[bit]] and 32-bit encodings;
* the fact that the text stream's encoding is Unicode, to a high level of confidence;
* which Unicode character encoding is used.

BOM use is optional. Its presence interferes with the use of [[UTF-8]] by software that does not expect non-[[ASCII]] bytes at the start of a file but that could otherwise handle the text stream.

Unicode can be encoded in units of 8-bit, 16-bit, or 32-bit integers. For the 16- and 32-bit representations, a computer receiving text from arbitrary sources needs to know which byte order the integers are encoded in. The BOM becomes a [[Universal Character Set characters#Noncharacters|{{Proper name|noncharacter}}]] Unicode code point if its bytes are swapped. Hence, the process accessing the text can examine these first few bytes to determine the endianness, without requiring some contract or [[metadata]] outside of the text stream itself. Generally the receiving computer will swap the bytes to its own endianness, if necessary, and would no longer need the BOM for processing.

The byte sequence of the BOM differs per Unicode encoding (including [[UTF-8]] and ones outside the Unicode standard such as [[UTF-7]], see [[#Byte_order_marks_by_encoding|table below]]), and none of the sequences is likely to appear at the start of text streams stored in other encodings. Therefore, placing an encoded BOM at the start of a text stream can indicate that the text is Unicode and identify the encoding scheme used. This use of the BOM is called a "Unicode signature".

==Usage==
The BOM is, simply, the Unicode codepoint {{unichar|FEFF|Zero Width No-Break Space}}, encoded in the current encoding. A text file beginning with the bytes <code>FE FF</code> suggests that the file is encoded in big-endian UTF-16.

The name ZWNBSP (from ''zero-width no-break space'') should be used if the BOM appears in the middle of a data stream. Unicode says it should be interpreted as a normal codepoint (namely a [[word joiner]]), not as a BOM. Since Unicode 3.2, this usage has been deprecated in favor of {{unichar|2060|WORD JOINER}}.<ref name="unicode FAQ"></ref>

The Unicode 1.0 name for this codepoint is also <code>BYTE ORDER MARK</code>.<!-- not in source there: --><ref name="fileformat">{{cite web | url=https://www.fileformat.info/info/unicode/char/feff/index.htm | title=Zero Width No-Break Space (U+Feff) }}</ref>

===UTF-8===
The [[UTF-8]] representation of the BOM is the ([[hexadecimal]]) byte sequence <code>EF&nbsp;BB&nbsp;BF</code>.

The Unicode Standard permits the BOM in [[UTF-8]],<ref>{{cite web|url=https://www.unicode.org/versions/Unicode5.0.0/ch02.pdf |title=The Unicode Standard 5.0, Chapter 2:General Structure |access-date=29 March 2009 |page=36 | quote=Table 2-4. The Seven Unicode Encoding Schemes}}</ref> but does not require or recommend its use.<ref>{{cite web|url=https://www.unicode.org/versions/Unicode5.0.0/ch02.pdf |title=The Unicode Standard 5.0, Chapter 2:General Structure |access-date=30 November 2008 |page=36 |quote=Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature}}</ref> UTF-8 always has the same byte order,<ref name="utf-8-bom">{{cite web|title = FAQ - UTF-8, UTF-16, UTF-32 & BOM: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If yes, then can I still assume the remaining UTF-8 bytes are in big-endian order?|url = http://unicode.org/faq/utf_bom.html#bom5|website=Unicode.org|access-date = 4 January 2009}}</ref> so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8, or that it was converted to UTF-8 from a stream that contained an optional BOM. The standard also does not recommend removing a BOM when it is there, so that round-tripping between encodings does not lose information, and so that code that relies on it continues to work.<ref>{{cite web|title=Re: pre-HTML5 and the BOM from Asmus Freytag on 2012-07-13 (Unicode Mail List Archive)|url=https://www.unicode.org/mail-arch/unicode-ml/y2012-m07/0268.html|website=Unicode.org|access-date = 14 July 2012}}</ref><ref>{{cite web|url=https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6378911 |title=Bug ID: JDK-6378911 UTF-8 decoder handling of byte-order mark has changed |website=Bugs.java.com |access-date=14 October 2021}}</ref> The IETF recommends that if a protocol either (a) always uses UTF-8, or (b) has some other way to indicate what encoding is being used, then it "SHOULD forbid use of U+FEFF as a signature."<ref name=rfc3629>{{cite IETF|title = UTF-8, a transformation format of ISO 10646|rfc = 3629|last1 = Yergeau|first1 = Francois|date=November 2003|publisher = [[Internet Engineering Task Force|IETF]]|access-date = 15 May 2014}}</ref> An example of not following this recommendation is the IETF [[Syslog]] protocol which requires text to be in UTF-8 and also requires the BOM.<ref>{{cite IETF |title=The Syslog Protocol |rfc=5424 |sectionname=MSG |section=6.4 |last= Gerhards |first= Rainer |date=March 2009 |publisher=[[Internet Engineering Task Force|IETF]] }}</ref>

Not using a BOM allows text to be backwards-compatible with software designed for [[extended ASCII]]. For instance many programming languages permit non-[[ASCII]] bytes in [[string literal]]s but not at the start of the file.

A BOM is unnecessary for detecting UTF-8 encoding.{{Citation needed|date=May 2024}} UTF-8 is a sparse encoding: a large fraction of possible byte combinations do not result in valid UTF-8 text. Binary data and text in any other encoding are likely to contain byte sequences that are invalid as UTF-8, so existence of such invalid sequences indicates the file is not UTF-8, while lack of invalid sequences is a very strong indication the text ''is'' UTF-8. Practically the only exception is text containing only ASCII-range bytes, as this may be a non-ASCII 7-bit encoding, but this is unlikely in any modern data and even then the difference from ASCII is minor (such as changing '\' to '¥').

[[Microsoft]] compilers<ref>{{cite web|url=http://alfps.wordpress.com/2011/11/22/unicode-part-1-windows-console-io-approaches/|title=Unicode part 1: Windows console i/o approaches|author=Alf P. Steinbach|access-date=24 March 2012|year=2011|quote=However, since the C++ source code was encoded as UTF-8 without BOM (as is usual in Linux), the Visual C++ compiler erroneously assumed that the source code was encoded as Windows ANSI.}}</ref> and interpreters, and many pieces of software on [[Microsoft Windows]] such as [[Notepad (Windows)|Notepad]] (prior to Windows 10 Build 1903<ref>{{Cite web |title=Windows 10 Notepad is Getting Better UTF-8 Encoding Support |url=https://www.bleepingcomputer.com/news/microsoft/windows-10-notepad-is-getting-better-utf-8-encoding-support/ |access-date=2023-03-07 |website=BleepingComputer |language=en-us}}</ref>) treat the BOM as a required [[Magic number (programming)#Magic numbers in files|magic number]] rather than use heuristics. These tools add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless the BOM is present or the file contains only ASCII. [[Windows PowerShell]] (up to 5.1) will add a BOM when it saves UTF-8 XML documents. However, PowerShell Core 6 has added a <code>-Encoding</code> switch on some cmdlets called utf8NoBOM so that document can be saved without BOM. [[Google Docs]] also adds a BOM when converting a document to a [[plain text]] file for download.

===UTF-16===
In [[UTF-16]], a BOM (<code>U+FEFF</code>) may be placed as the first bytes of a file or character stream to indicate the endianness (byte order) of all the 16-bit [[Character encoding#Terminology|code units]] of the file or stream. If an attempt is made to read this stream with the wrong endianness, the bytes will be swapped, thus delivering the character <code>U+FFFE</code>, which [[Specials (Unicode block)|is defined]] by Unicode as a "[[{{Proper name|noncharacter}}]]" that should never appear in the text.

* If the 16-bit units are represented in [[big-endian]] byte order ("UTF-16BE"), the BOM is the ([[hexadecimal]]) byte sequence <code>FE&nbsp;FF</code>
* If the 16-bit units use [[little-endian]] order ("UTF-16LE"), the BOM is the ([[hexadecimal]]) byte sequence <code>FF&nbsp;FE</code>

For the [[Internet Assigned Numbers Authority|IANA]] registered charsets UTF-16BE and UTF-16LE, a byte-order mark should not be used because the names of these character sets already determine the byte order.

Clause D98 of conformance (section 3.10) of the Unicode standard states, "The UTF-16 encoding scheme may or may not begin with a BOM. However, when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian." Whether or not a higher-level protocol is in force is open to interpretation. Files local to a computer for which the native byte ordering is little-endian, for example, might be argued to be encoded as UTF-16LE implicitly. Therefore, the presumption of big-endian is widely ignored. The [[W3C]]/[[Comparison of web browser engines (HTML support)|WHATWG]] encoding standard used in HTML5 specifies that content labelled either "utf-16" or "utf-16le" are to be interpreted as little-endian "to deal with deployed content".<ref>{{cite web | url=https://encoding.spec.whatwg.org/#utf-16le | title=UTF-16LE | publisher=WHATWG | work=Encoding Standard}}</ref> However, if a byte-order mark is present, then that BOM is to be treated as "more authoritative than anything else".<ref>{{cite web | url=https://encoding.spec.whatwg.org/#decode | title=Decode | publisher=WHATWG | work=Encoding Standard}}</ref>

Without a BOM, it is still fairly reliable to detect if text is UTF-16 and what byte order it is in if the text is sufficiently long. The characters 1-255, such as line-endings and spaces which are much used also in non-latin scripts, have a NUL high byte. If NUL bytes are much more often at even offsets in the file then it is likely to be big-endian UTF-16, and for odd offsets little-endian.

===UTF-32===
Although a BOM could be used with [[UTF-32]], this encoding is rarely used for transmission. Otherwise the same rules as for [[UTF-16]] are applicable.

The BOM for little-endian UTF-32 is the same pattern as a little-endian UTF-16 BOM followed by a UTF-16 NUL character, an unusual example of the BOM being the same pattern in two different encodings. Programmers using the BOM to identify the encoding will have to decide whether UTF-32 or UTF-16 with a NUL first character is more likely. UTF-32 is easily detected without a BOM because every 4th byte is NUL.

==Byte-order marks by encoding==
This table illustrates how the BOM is represented as a byte sequence in various encodings and how those sequences might appear in a text editor that is [[mojibake|interpreting each byte as a legacy encoding]] ([[Windows-1252]] and [[caret notation]] for the [[C0 and C1 control codes|C0 controls]]):

{| class="wikitable"
|-
! Encoding
! Representation ([[hexadecimal]])
! Representation ([[decimal]])
! Bytes interpreted as Windows-1252
|-
| [[UTF-8]]{{refn|This is not literally a "byte order" mark, since a code unit in these encodings is one byte and therefore cannot have bytes in a "wrong" order. Nevertheless, the BOM can be used to indicate the encoding of the text that follows it.<ref name="utf-8-bom"/><ref>{{cite journal|url=http://tools.ietf.org/html/rfc3629#section-6 |title=RFC 3629 - UTF-8, a transformation format of ISO 10646 |newspaper=Ietf Datatracker |date=8 November 2003 |access-date=28 January 2017|last1=Yergeau |first1=François }}</ref>|group=lower-alpha|name=b}}
| {{tt|EF BB BF}}
| {{tt|239 187 191}}
| {{tt|ï»¿}}
|-
| [[UTF-16]] ([[Big Endian|BE]])
| {{tt|FE FF}}
| {{tt|254 255}}
| {{tt|þÿ}}
|-
| [[UTF-16]] ([[Little Endian|LE]])
| {{tt|FF FE}}
| {{tt|255 254}}
| {{tt|ÿþ}}
|-
| [[UTF-32]] (BE)
| {{tt|00 00 FE FF}}
| {{tt|0   0   254 255}}
| {{tt|^@^@þÿ}} ({{tt|^@}} is the [[null character]])
|-
| [[UTF-32]] (LE)
| {{tt|FF FE 00 00}}
| {{tt|255 254 0 0}}
| {{tt|ÿþ^@^@}} ({{tt|^@}} is the null character)
|-
| [[UTF-7]]<ref name=b group=lower-alpha/>
| {{tt|2B 2F 76}}<ref group=lower-alpha>Followed by {{code|38}}, {{code|39}}, {{code|2B}}, or {{code|2F}} (ASCII {{code|8}}, {{code|9}}, {{code|+}} or {{code|/}}), depending on what the next character is.</ref><ref>{{Cite web |last=Honermann |first=Tom |date=January 2, 2021 |title=Clarify guidance for use of a BOM as a UTF-8 encoding signature |url=https://unicode.org/L2/L2021/21038-bom-guidance.pdf |website=Unicode}}</ref><ref>{{Cite web|url=https://docs.sdl.com/791187/581899/sdl-contenta-5-7/representations-of-boms-by-encoding|title=SDL Documentation}}</ref>
| {{tt|43 47 118}}
| {{tt|+/v}}
|-
| [[UTF-1]]<ref name=b group=lower-alpha/>
| {{tt|F7 64 4C}}
| {{tt|247 100 76}}
| {{tt|÷dL}}
|-
| [[UTF-EBCDIC]]<ref name=b group=lower-alpha/>
| {{tt|DD 73 66 73}}
| {{tt|221 115 102 115}}
| {{tt|Ýsfs}}
|-
| [[Standard Compression Scheme for Unicode|SCSU]]<ref name=b group=lower-alpha/>
| {{tt|0E FE FF}}{{refn|SCSU allows other encodings of U+FEFF, the shown form is the signature recommended in UTR #6.<ref>{{cite web|author=Markus Scherer |url=https://www.unicode.org/reports/tr6/#Signature |title=UTS #6: Compression Scheme for Unicode |website=Unicode.org |access-date=28 January 2017}}</ref>|group=lower-alpha}}
| {{tt|14 254 255}}
| {{tt|^Nþÿ}} ({{tt|^N}} is the [[Shift Out and Shift In characters|"shift out" character]])
|-
| [[Binary Ordered Compression for Unicode|BOCU-1]]<ref name=b group=lower-alpha/>
| {{tt|FB EE 28}}
| {{tt|251 238 40}}
| {{tt|ûî(}}
|-
| [[GB 18030|GB18030]]<ref name=b group=lower-alpha/>
| {{tt|84 31 95 33}}
| {{tt|132 49 149 51}}
| {{tt|„1•3}}
|}
{{Reflist|group=lower-alpha}}

==See also==
* [[Arabic Presentation Forms-B]], block to which code point <code>U+FEFF</code> belongs
* [[Left-to-right mark]]
* [[Mojibake]], garbled text that results from incorrect decoding

==References==
{{Reflist|30em}}

==External links==
* [https://www.unicode.org/faq/utf_bom.html Unicode FAQ: ''UTF-8, UTF-16, UTF-32 & BOM'']
* [https://web.archive.org/web/20210422192604/http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf#G9354 The Unicode Standard, chapter 2.6 ''Encoding Schemes'']
* [https://web.archive.org/web/20210422192604/http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf#G9354 The Unicode Standard, chapter 2.13 ''Special Characters and Noncharacters'', section ''Byte Order Mark (BOM)'']
* [https://web.archive.org/web/20210506134650/http://www.unicode.org/versions/Unicode5.0.0/ch16.pdf#G25817 The Unicode Standard, chapter 16.8 ''Specials'', section ''Byte Order Mark (BOM): U+FEFF'']
* [https://www.w3.org/International/questions/qa-byte-order-mark The byte-order mark (BOM) in HTML]

{{Unicode_navigation}}

{{DEFAULTSORT:Byte Order Mark}}
[[Category:Unicode special code points]]