Unicode to HTML entities: NCRs and named references

These three lines of HTML produce exactly the same glyph in a browser:

<p>&#xE9;</p>
<p>&#233;</p>
<p>&eacute;</p>

Each one renders as é, the letter U+00E9 LATIN SMALL LETTER E WITH ACUTE. The first spelling is a hexadecimal numeric character reference (NCR), the second a decimal NCR, the third a named character reference. Together they are HTML's character reference mechanism: a way to write any character using only ASCII bytes. This article covers when you must use them, when you can skip them, and why the numeric forms will never let you down while the named ones sometimes do.

What the HTML standard requires

The HTML standard defines exactly three kinds of character reference, and all of them begin with U+0026 AMPERSAND. A named reference is an ampersand followed by a case-sensitive name from the named character references table, like é. A decimal NCR is an ampersand, a number sign, and the code point in base ten: é. A hexadecimal NCR inserts an x (or X) before the hex digits: é. All three must end with a semicolon, and the rule has teeth, as we will see below.

The standard is also specific about what must be escaped in the first place. Text in HTML must not contain a raw U+003C LESS-THAN SIGN, because it starts a tag, and it must not contain what the standard calls an ambiguous ampersand: an ampersand followed by alphanumerics and a semicolon that matches no known name. A double-quoted attribute value must not contain a raw U+0022 QUOTATION MARK. That is the whole mandatory list. The familiar four escapes &, <, > and " cover it, with the greater-than sign included mostly by convention and symmetry.

When entities are mandatory and when they are not

If your text says Fish & Chips or compares values with a less-than sign, the ampersand and the angle bracket are syntax, so they must be escaped:

input:  5 < 6 & "7" > 2
output: 5 &lt; 6 &amp; &quot;7&quot; &gt; 2

Everything else is optional. An é in your page can travel as two raw UTF-8 bytes (C3 A9) provided the document is encoded in UTF-8 and says so with <meta charset="utf-8">. That is the default on the modern web, and it is smaller and more readable than é. Entities earned their keep in the era of 7-bit transport and ASCII-only source files, and they still earn it in specific spots: invisible characters like NO-BREAK SPACE or RIGHT-TO-LEFT MARK that you want to see in the markup, code samples that must display markup without triggering it, and authoring environments where the encoding is out of your control.

Why numbers scale and names do not

An NCR is computed, not looked up. Take the code point, write it in decimal or hex, wrap it in &# and ;. The mechanism therefore covers the entire codespace, from A for A to 😀 for 😀. The standard restricts conformance slightly: numeric references may not target carriage return, noncharacters, or control characters other than ASCII whitespace. But as a mechanism, it reaches every assigned character in Unicode, all 154,000-plus of them.

A named reference is a lookup in a fixed table. The HTML standard's table has 2,231 entries, frozen when published so that existing documents never change meaning. That is roughly one name for every seventy Unicode characters, heavily weighted toward Latin letters, math symbols and Greek. If your character is not in the table, no name exists, full stop. XML is even stricter: it predefines only five entities (&, <, >, ", '), so é in a raw XML document is an error unless a DTD declares it. Numeric references work everywhere, which is why generators and converters emit them by default.

Named references are a fixed list of 2,231 favorites. Numeric character references are arithmetic. Only one of those covers all of Unicode.

The semicolon is not optional

The standard requires the trailing semicolon on every character reference, and omitting it is a parse error with the gloriously literal name missing-semicolon-after-character-reference. Browsers still recover for 106 legacy names left over from early HTML: &amp and &copy without the semicolon do decode. But there is a trap. Inside an attribute, if a semicolon-less name is immediately followed by an equals sign or an alphanumeric, the parser refuses to expand it, for historical reasons. The URL <a href="?page=1&copy=2"> keeps the text &copy=2 verbatim, which is what you want, while the same string in paragraph text would lose the &copy to a ©. Relying on error recovery gives you different results depending on context. Write the semicolon every time and the rule never applies to you.

Converting with the tool

The Unicode converter works in both directions. Paste é, é or é into the main input and all three decode to é (the named table covers about 250 classic names; an unknown name like &madeupname; passes through untouched, mirroring how a browser treats text that matches no reference). Going the other way, two panels produce NCRs:

input:       café
Hex NCRs:    café
Decimal NCRs: café

The Hex panel pads to four hex digits, which keeps BMP references aligned and readable. Both panels have a Keep ASCII switch, on by default, that leaves ASCII text alone and converts only the non-ASCII characters. Uncheck it and every character becomes a reference:

Hex NCRs:     café
Decimal NCRs: café

Supplementary-plane characters stay a single reference per code point: 😀 converts to 😀 or 😀, with no surrogate pair involved, because character references are defined on code points, not on UTF-16 code units. If what you need is the four markup escapes rather than full-text conversion, the XML/HTML entities panel emits exactly those. And if the target is source code rather than markup, the same idea exists as language-specific Unicode escapes.

The short version

• Always escape < and & in text, and the quote character inside a quoted attribute value. These are the only mandatory escapes.
• For everything else, declare <meta charset="utf-8"> and ship raw UTF-8. Reach for NCRs when a character is invisible, confusable, or your toolchain cannot carry non-ASCII bytes.
• End every character reference with a semicolon. The parser's error recovery is context-dependent and not a feature to rely on.
• Prefer numeric references when generating markup programmatically. Names are for humans, and only 2,231 of them exist.