What is UTF-16? Code units and surrogate pairs explained

Type 😀 into the Unicode converter and read the UTF-16 code units panel. It answers D83D DE00: one emoji, two four-digit numbers. The Unicode panel says the same emoji is the single code point U+1F600 (GRINNING FACE). UTF-16 is the encoding that turns that one code point into two 16-bit values, and the mechanics behind it explain some of the most stubborn bugs in JavaScript, Java and Windows programming.

Code point, code unit, scalar value

Three terms get mixed up constantly, so pin them down first. A code point is any value in the Unicode codespace, U+0000 through U+10FFFF. That range includes values that can never be characters: U+D800 through U+DFFF are surrogate code points, reserved for UTF-16's internal use. A Unicode scalar value is any code point except those 2,048 surrogates. Every character you can actually type maps to a scalar value.

A code unit is something else: the minimal unit of an encoding form, 8 bits for UTF-8, 16 bits for UTF-16, 32 bits for UTF-32. The UTF-16 encoding form maps each scalar value in U+0000..U+D7FF and U+E000..U+FFFF to a single 16-bit code unit with the same numeric value, and each scalar value in U+10000..U+10FFFF to a pair of code units called a surrogate pair. So é is one code point and one UTF-16 code unit (00E9), while 😀 is one code point and two UTF-16 code units. "One character, one number" is true at the code point level; it stops being true the moment you count code units.

The surrogate formula, step by step

The supplementary range U+10000..U+10FFFF holds exactly 2^20 values, which need 20 bits. UTF-16's cells are 16 bits wide, so the standard splits the 20 bits into two 10-bit halves and stores each half in a reserved container where no BMP character can live: the first half goes into a lead surrogate (U+D800..U+DBFF, 1,024 values), the second into a trail surrogate (U+DC00..U+DFFF, another 1,024). 1,024 times 1,024 is exactly 2^20, so the fit is exact with no wasted space.

The procedure for any scalar value at or above U+10000:

• Subtract 0x10000, leaving a 20-bit value.
• Take the top 10 bits and add 0xD800. That is the lead surrogate.
• Take the bottom 10 bits and add 0xDC00. That is the trail surrogate.

This is Table 3-5 of The Unicode Standard in disguise: the bit pattern 000uuuuuxxxxxxxxxxxxxxxx becomes 110110wwwwxxxxxx 110111xxxxxxxxxx, with wwww = uuuuu - 1 doing the same job as the subtraction. Here is the full arithmetic for the emoji:

U+1F600  GRINNING FACE
0x1F600 - 0x10000   = 0xF600    (offset into the supplementary range)
0xF600 >> 10        = 0x3D      (top 10 bits)
0xD800 + 0x3D       = 0xD83D    (lead surrogate)
0xF600 & 0x3FF      = 0x200     (bottom 10 bits)
0xDC00 + 0x200      = 0xDE00    (trail surrogate)
UTF-16 code units: <D83D DE00>

Decoding runs the formula backwards: subtract 0xD800 from the lead, subtract 0xDC00 from the trail, then compute lead * 0x400 + trail + 0x10000. For D83D DE00 that is 0x3D * 0x400 + 0x200 + 0x10000 = 0x1F600. ECMA-262 specifies exactly this formula for interpreting a surrogate pair.

A note on names. The formal definitions in The Unicode Standard 16.0 (D71 through D74) call the two halves high-surrogate and low-surrogate code units, and the Unicode glossary records lead and trail as the official synonyms. Newer specifications have flipped which term leads: ECMA-262 writes "leading surrogate" and "trailing surrogate" and treats high/low as the more formal alias. Both pairs of names mean the same ranges, and this article uses lead and trail.

You can check all of this against the tool: paste D83D DE00 into the UTF-16 code units panel of the converter, hit Convert, and the Characters panel hands you back 😀.

Lone surrogates are ill-formed

A lead surrogate only means anything when a trail surrogate follows it. The standard is blunt: isolated UTF-16 code units in the range D800..DFFF are ill-formed, and no conformant process may emit them. Surrogate code points are not scalar values, so they can never represent a character on their own, in any encoding form.

A surrogate pair is one character wearing two code units. A lone surrogate is no character at all; it is a broken half.

JavaScript tolerates the broken halves anyway. "\uD83D".length is 1 and the string is perfectly legal to hold, but "\uD83D".isWellFormed() returns false. Feed it to encodeURIComponent and you get a URIError; feed it to TextEncoder and the lone surrogate silently becomes EF BF BD, the UTF-8 bytes of U+FFFD REPLACEMENT CHARACTER. The ES2024 method toWellFormed() makes that replacement explicit, swapping every unmatched surrogate for U+FFFD. Lone surrogates in real data usually mean someone sliced a string in the middle of a pair: in JavaScript, "a😀".slice(1, 2) returns the lead surrogate alone.

Where UTF-16 lives

UTF-16 was born as a rescue. In the early 1990s, when Unicode still fit inside 16 bits, Java, Windows NT and JavaScript all built their text handling on fixed-width 16-bit units (UCS-2). When Unicode grew past U+FFFF, those platforms could not widen their types without breaking everything, so UTF-16 retrofitted the surrogate mechanism onto the 16-bit cell.

The descendants of that decision are everywhere:

• JavaScript: ECMA-262 defines the String type as a sequence of 16-bit unsigned integers treated as UTF-16 code unit values, and length as the number of elements in that sequence.
• Java: the language spec states that text is represented as sequences of 16-bit code units using UTF-16, and the String class documentation says index values refer to char code units, so a supplementary character occupies two indices.
• Windows: the native wide-character APIs (the W in CreateFileW) take UTF-16LE strings, and WCHAR is a 16-bit code unit. .NET strings work the same way.

string.length counts code units, not characters

The consequence every JavaScript developer meets eventually:

"😀".length           // 2: two UTF-16 code units
"é".length            // 1
"😀".charCodeAt(0)    // 55357 = 0xD83D, the lead surrogate
"😀".charCodeAt(1)    // 56832 = 0xDE00, the trail surrogate
"😀".codePointAt(0)   // 128512 = 0x1F600, the actual code point
[..."😀"].length      // 1: iteration walks code points

charCodeAt exposes raw code units, surrogates and all. codePointAt, String.fromCodePoint, the \u{1F600} escape syntax, and string iteration (spread, for...of) all work at the code point level and treat a surrogate pair as one item. Any code that iterates by index or reports length to a user is counting code units, and will overcount every emoji, CJK Extension B ideograph and historic-script character in the string.

Putting it to work

• To see the UTF-16 form of any text, paste it into the converter: the UTF-16 code units panel shows each BMP character as one value and each supplementary character as a pair, and the panel accepts the same spaced hex as input.
• In JavaScript, count perceived characters with Intl.Segmenter or code points with [...s].length; never ship s.length as a character count.
• When a hex dump shows a value in D800..DFFF standing alone, the data is ill-formed or someone cut a pair in half.

If the code point side of this story is still fuzzy, start with What is Unicode?, then compare UTF-16 against the other two encoding forms in UTF-8 vs UTF-16 vs UTF-32.