Unicode vs UTF-8: your file is never "Unicode"

A bug report lands on your desk: "the file is Unicode, but the page shows garbage." You open the config and find the culprit, a line from 2004 that says charset=unicode. Both statements are wrong in the same way. There is no such thing as a file that "is Unicode," and no byte layout called "unicode." The file was actually UTF-16LE, the reader assumed UTF-8, and the mismatch shredded every non-ASCII character. The confusion is so common that browsers still have to special-case it.

Unicode and UTF-8 are not two competing encodings. They are two different layers of the same system, and conflating them is like saying a house "is a street address" when you mean the building. Unicode is the numbering. UTF-8 is one way to write the numbers down.

One character, three byte sequences

Take the character 中. Paste it into the Unicode converter and three panels answer with three different representations of the same thing:

code point panel:   U+4E2D
UTF-8 panel:        E4 B8 AD
UTF-16 panel:       4E2D

The code point U+4E2D is the character's number in Unicode's table. Its official name is CJK UNIFIED IDEOGRAPH-4E2D, it sits in the CJK Unified Ideographs block (U+4E00..U+9FFF), and the number itself is pure abstraction. It is not stored anywhere. What gets stored is the output of an encoding. UTF-8 turns U+4E2D into three bytes, E4 B8 AD. UTF-16 turns it into one 16-bit code unit, 4E2D, which serializes to the bytes 4E 2D in big-endian order or 2D 4E in little-endian order. UTF-32 pads the number into a 32-bit code unit, 00 00 4E 2D big-endian.

char  code point  UTF-8         UTF-16BE      UTF-32BE
A     U+0041      41            00 41         00 00 00 41
é     U+00E9      C3 A9         00 E9         00 00 00 E9
中     U+4E2D      E4 B8 AD      4E 2D         00 00 4E 2D
😀    U+1F600     F0 9F 98 80   D8 3D DE 00   00 01 F6 00

Note the units before reading on. The UTF-16BE column serializes bytes, so the emoji shows four: D8 3D DE 00. The pair the paragraph below names, D83D and DE00, are the two UTF-16 code units that produce those bytes.

Read the last row carefully. The grinning face U+1F600 lies above U+FFFF, so UTF-16 cannot fit it in one code unit. It uses a surrogate pair instead: D83D, the lead surrogate (also called high surrogate), followed by DE00, the trail surrogate (also called low surrogate). Those two values are UTF-16 code units, not code points. Calling them "the Unicode of the emoji" is exactly the category error this article is about.

The standard's own words: set versus encoding

The Unicode glossary draws the line precisely. A coded character set is a character set in which each character is assigned a numeric code point. That is what Unicode is. A character encoding form is a mapping from the character set to actual code units: 8-bit units for UTF-8, 16-bit units for UTF-16, 32-bit units for UTF-32. A character encoding scheme adds byte serialization on top, which is where endianness enters. The standard defines exactly seven encoding schemes: UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE and UTF-32LE. The bare names UTF-16 and UTF-32 may carry a byte order mark to signal their endianness; the BE and LE variants are explicit.

UTF-8's layout is defined in RFC 3629, also known as STD 63. Every code point from U+0800 to U+FFFF takes three bytes of the form 1110xxxx 10xxxxxx 10xxxxxx, with the code point's sixteen bits distributed into the x slots. For 中, 0x4E2D in binary fills those slots to produce E4 B8 AD, matching the converter's output. ASCII survives untouched as a single byte, which is why UTF-8 won the web: the WHATWG Encoding standard now mandates UTF-8 for everything and calls it "the most appropriate encoding for interchange of Unicode, the universal coded character set."

Unicode is the numbering. UTF-8, UTF-16 and UTF-32 are three different ways to write the numbers down. A file is never "Unicode"; it is UTF-8, UTF-16LE, or something else.

How "Unicode" came to mean UTF-16

So why does half the industry keep saying "Unicode" when it means an encoding? Blame Windows. Windows NT committed to 16-bit characters in the early 1990s, when Unicode was still a 16-bit codespace and UTF-16 barely existed as a distinct concept. The operating system called its 16-bit text "Unicode," and the name stuck everywhere users could see it. Open an old version of Notepad and the Save As dialog offers four encodings: ANSI, Unicode, Unicode big endian, and UTF-8. "Unicode" means UTF-16LE with a byte order mark. "Unicode big endian" means UTF-16BE. The menu never says UTF-16 at all.

The byte order mark is the tell. It is the code point U+FEFF written at the start of the file, and its bytes identify the scheme:

encoding    first bytes    Notepad's name
UTF-8       EF BB BF       UTF-8
UTF-16LE    FF FE          Unicode
UTF-16BE    FE FF          Unicode big endian

The web inherited the same shortcut. Early HTML authors wrote <meta charset="unicode"> because that is the vocabulary Windows taught them, and browsers learned to honor it. To this day the WHATWG Encoding standard maps the label unicode to UTF-16LE, and the label unicodefffe to UTF-16BE, right alongside the proper labels utf-16le and utf-16be. Every time a browser parses that meta tag, it is decoding a mistake from 1995.

Where "charset" went wrong

The word charset itself is the other half of the problem. MIME introduced the charset parameter in RFC 2045 and RFC 2046, and the definition is explicit: in MIME, "character set" means a method of converting a sequence of octets into a sequence of characters. That is an encoding, not a coded character set. RFC 2045 even admits the collision in a note, observing that other communities use "character encoding" for what MIME calls a character set, and reserve "coded character set" for the abstract mapping from integers to characters.

So the parameter in Content-Type: text/html; charset=utf-8 is correctly used: it names an encoding. But the name trained a generation of developers to think that a "charset" value identifies the character repertoire, when it actually identifies the byte layout. Writing charset=unicode stacks the error twice: it uses an encoding slot to name a coded character set, and that set has no single byte layout to begin with.

What to do instead

The fix costs nothing and pays off the first time you debug mojibake:

• Never say a file or a string "is Unicode." Name the encoding: UTF-8, UTF-16LE, UTF-16BE. If you do not know, say you do not know.
• Default to UTF-8 for storage and for the wire, and declare it: charset=utf-8 in the Content-Type header or the meta tag. The WHATWG standard treats UTF-8 as the only sane choice for new formats.
• When text breaks, look at the bytes before blaming the characters. Paste the text into the converter and compare the UTF-8 and UTF-16 panels: E4 B8 AD versus 2D 4E tells you instantly which side of the connection mislabeled its output.
• Keep the vocabulary straight in code reviews: code point for the number, code unit for the encoding's atoms, Unicode scalar value for any code point that is not a surrogate.

Once the two layers are separate in your head, the follow-up questions get easier: how UTF-8 packs bits into its bytes, and when you would pick one encoding over another in the UTF-8 vs UTF-16 vs UTF-32 trade-off.