Unicode in JavaScript, Python, Java, Rust, and Go

Take the string é😀 and ask five languages a simple question: how long is it? You get three different answers. JavaScript says 3. Java says 3. Python says 2. Rust says 6. Go says 6. Every answer is correct, and none of them counts what a user would call characters. The difference is in what each language defines a string to be.

The test string is two code points: é, U+00E9 LATIN SMALL LETTER E WITH ACUTE, and 😀, U+1F600 GRINNING FACE. Paste it into the Unicode converter and the U+ panel answers U+00E9U+1F600. The UTF-8 panel shows six bytes, C3 A9 F0 9F 98 80, and the UTF-16 panel shows three code units, 00E9 D83D DE00. Those three spellings of the same text, 2 code points, 3 UTF-16 code units, 6 UTF-8 bytes, are exactly what the five languages disagree about.

JavaScript: a sequence of UTF-16 code units

ECMA-262 defines the String type as an ordered sequence of 16-bit unsigned integers, each treated as a UTF-16 code unit. The spec is explicit that length is the number of these elements, not the number of code points. A character above U+FFFF, like our emoji, occupies two positions: a lead surrogate (0xD800 to 0xDBFF, also called high surrogate) followed by a trail surrogate (0xDC00 to 0xDFFF, also low surrogate).

const s = "é😀"
s.length                 // 3, UTF-16 code units
s.charCodeAt(1)          // 55357 (0xD83D), the lead surrogate
s.codePointAt(1)         // 128512 (0x1F600)
[...s].length            // 2, code points
String.fromCodePoint(0xE9, 0x1F600)  // "é😀"

The code point APIs arrived in ES2015: codePointAt and String.fromCodePoint work with full code points, and a for...of loop iterates code points, so it yields é then 😀, two turns. But length, charAt, charCodeAt and bracket indexing still speak code units, and they always will. When you slice a JavaScript string, count in code units or expect to split surrogate pairs. The encoding behind this is covered in what UTF-16 is.

Python 3: a sequence of code points

Python's documentation states it in one sentence: strings are immutable sequences of Unicode code points. There is no code unit layer exposed at all. len counts code points, indexing returns a code point, and iteration walks code points.

s = "é😀"
len(s)                      # 2
s[0]                        # 'é'
[hex(ord(c)) for c in s]    # ['0xe9', '0x1f600']
len(s.encode("utf-8"))      # 6, bytes appear only after encoding

This is not UTF-32 in a trench coat. Since Python 3.3, PEP 393 gives each string a flexible internal representation: 1 byte per code point if every value fits in Latin-1, 2 bytes if everything fits in the Basic Multilingual Plane, 4 bytes otherwise. Our test string stores 4 bytes per code point because of the emoji; a pure ASCII string stores 1. Indexing stays O(1) and no surrogate ever leaks into your code, but the encoding only exists at the boundary: str.encode() produces bytes, bytes.decode() produces a string, and mixing the two types is a TypeError.

Java: UTF-16 with two generations of APIs

Java's String documentation says the class represents a string in UTF-16 format, that index values refer to char code units, and that a supplementary character uses two positions. char itself is a 16-bit code unit, too narrow for anything above U+FFFF. The class carries two API generations: the original char-based methods, and codePointAt, codePointCount and codePoints(), which work with int values wide enough to hold any code point.

String s = "é😀";
s.length()                       // 3, char code units
s.codePointCount(0, s.length())  // 2, code points
s.charAt(1)                      // '\uD83D', a lead surrogate
s.codePointAt(1)                 // 128512 (0x1F600)
s.codePoints().count()           // 2, as an IntStream

The trap is the same one JavaScript has, with sharper edges: length() answers in code units, substring cuts by code unit index, and a loop over charAt visits surrogate halves one at a time. Any code that measures or truncates user text should use the code point APIs, or better, BreakIterator if the goal is what users perceive as characters.

Rust: UTF-8, and indexing is a compile error

Rust makes a different bet. A String and its borrowed form &str are UTF-8 bytes with a validity invariant: the standard library states that string slices are always valid UTF-8, and anything else is undefined behavior. len() returns bytes. To see characters you decode explicitly with .chars(), which yields char values. A Rust char is a 4-byte Unicode scalar value, meaning any code point except the surrogate range U+D800 to U+DFFF, which UTF-8 never encodes.

let s = "é😀";
s.len();            // 6, bytes
s.chars().count();  // 2
// s[0];           // does not compile: str is not indexable by position

The refusal to index by integer is deliberate: a position in a UTF-8 string is a byte offset, not a character number, and allowing s[0] would invite code that silently slices mid-character. Even range slicing like &s[0..2] panics if the offset is not a character boundary. The compiler forces the question the other languages let you skip: bytes or characters? For the byte format itself, see what UTF-8 is.

Go: bytes by default, runes on request

Go is the most permissive of the five. A string is, in effect, a read-only slice of bytes. It is not required to hold UTF-8 at all, although source code literals do. len counts bytes, indexing yields bytes, and the type never pretends otherwise. The Unicode machinery is opt-in: the word rune is an alias for int32 and means exactly one code point, and a for...range loop over a string decodes UTF-8 one rune at a time.

s := "é😀"
len(s)                     // 6, bytes
utf8.RuneCountInString(s)  // 2
for i, r := range s {
    fmt.Printf("%d %U\n", i, r)
}
// 0 U+00E9
// 2 U+1F600   (the index is a byte offset)

Two details follow from the byte model. The loop index is a byte position, so 😀 starts at 2, not 1. And because strings may hold invalid UTF-8, the decoder substitutes U+FFFD REPLACEMENT CHARACTER for each bad byte; the unicode/utf8 package exposes that constant as utf8.RuneError along with validators and decoders for when the loop is not enough.

Every "how long is this string" bug is a vocabulary bug. Code points, code units and bytes are three different counts, and each language picks one as the default meaning of length.

What to check before you count

• In JavaScript and Java, length counts UTF-16 code units. Emoji and other supplementary characters count as 2. Use [...s] or codePointCount for code points.
• In Python 3, len already counts code points. Bytes exist only after an explicit encode().
• In Rust and Go, len counts bytes. Use .chars().count() or utf8.RuneCountInString for code points, and expect byte offsets everywhere.
• None of the five counts grapheme clusters. é written as e plus U+0301 COMBINING ACUTE ACCENT is two code points in every one of them. User-perceived characters need a segmentation library or normalization first.
• When debugging, paste the text into the converter: the U+ panel shows the code points, the UTF-8 and UTF-16 panels show the exact units each language is counting, and Inspect names every character.