ASCII vs Latin-1 vs Unicode: where mojibake comes from

Paste café into the Unicode converter and read the U+ notation panel. It answers cafU+00C3U+00A9. Those last two code points were never typed by anyone. They are the UTF-8 bytes of é, decoded one byte per character by something that believed it was reading Latin-1. That single line of output contains the whole story of this article: where ASCII stopped, what Latin-1 added, and why the two still collide with UTF-8 forty years later.

ASCII: seven bits and 128 characters

ASCII was first published in 1963 as ASA standard X3.4 and settled into its familiar form with the 1967 revision. It is a 7-bit code: 128 positions, 0x00 to 0x7F. Thirty-three of them are control codes (0x00 through 0x1F, plus 0x7F for DEL), and the rest hold the English alphabet, digits and punctuation. There are no accented letters at all. The eighth bit of each byte was left over, often burned as a parity bit on serial lines, so the range 0x80 to 0xFF simply did not exist as far as ASCII was concerned.

Unicode adopted ASCII unchanged. Code points U+0000 through U+007F are the Basic Latin block, and A is simultaneously ASCII 0x41 and U+0041. Every ASCII file is already valid Unicode text, and every ASCII byte string is already valid UTF-8. That decision is why ASCII refuses to die.

Latin-1 spends the eighth bit

ISO-8859-1, published in 1987 and known as Latin-1, put the spare bit to work. It is an 8-bit code with 256 positions. The bottom half is ASCII, byte for byte. The range 0x80 to 0x9F is reserved for the C1 control codes, which almost no real system used. The top half, 0xA0 to 0xFF, finally gives Western Europe its letters: é, ñ, ü, plus symbols like © and £. The letter é is byte 0xE9.

When Unicode arrived, it made a deliberate compatibility move. The Unicode Standard states that the first 256 code points follow precisely the arrangement of ISO/IEC 8859-1. So é is byte 0xE9 in Latin-1 and U+00E9 LATIN SMALL LETTER E WITH ACUTE in Unicode, and U+0080 through U+00FF is literally named the Latin-1 Supplement block. Converting Latin-1 to Unicode needs no table: each byte value becomes a code point with the same number. This 1:1 mapping is the single most important fact for understanding mojibake, so hold on to it.

Windows-1252: the Latin-1 everyone actually used

In the early 1990s Microsoft shipped code page 1252, which keeps the Latin-1 layout but fills the idle 0x80 to 0x9F range with printable characters: the euro sign at 0x80, the typographic quotation marks at 0x91 to 0x94, the trademark sign at 0x99, dashes at 0x96 and 0x97. Documents tagged as ISO-8859-1 overwhelmingly contained these bytes, because they were produced on Windows.

The WHATWG Encoding Standard eventually ratified reality. The labels iso-8859-1, latin1, ascii and us-ascii are all defined as aliases for windows-1252. Ask a browser for the Latin-1 decoding of byte 0x80 and you get U+20AC EURO SIGN, not a control code. On the web, Latin-1 effectively means Windows-1252, and the distinction matters when you decode damaged text.

How café becomes café

UTF-8 (RFC 3629) encodes code points U+0080 through U+07FF as two bytes. For é (U+00E9) those bytes are C3 and A9, so the word café sits on disk as five bytes:

text:    c    a    f    é
bytes:   63   61   66   C3 A9

Now feed those bytes to a decoder that assumes Latin-1 or Windows-1252. Thanks to the 1:1 mapping, each byte becomes one code point with the same value: 63 is c, 66 is f, C3 is U+00C3 LATIN CAPITAL LETTER A WITH TILDE (Ã), and A9 is U+00A9 COPYRIGHT SIGN (©). The result is café. Nothing crashed, nothing was lost, and that is exactly why the bug spreads so quietly.

Repair runs the same mapping in reverse. Read the code points of the mojibake as byte values, then decode those bytes as UTF-8. In the converter: paste café, note that the U+ panel shows C3 and A9, then type 63 61 66 C3 A9 into the UTF-8 panel input and hit Convert. The Characters panel returns café.

The damage compounds if nobody notices. Save café as UTF-8 and the converter shows the bytes 63 61 66 C3 83 C2 A9: Ã and © have each been encoded again. Every round trip adds a layer, which is how a single é becomes four or eight garbage characters in old databases.

Windows-1252 leaves its own fingerprint. The UTF-8 bytes of U+2019 RIGHT SINGLE QUOTATION MARK are E2 80 99. Decoded as Windows-1252, E2 is â, 80 is € and 99 is ™, which is why apostrophes rot into it’s. Under true ISO-8859-1 the bytes 80 and 99 would be invisible C1 controls, so a visible €™ proves a Windows-1252 decoder was involved.

Mojibake is never random. The garbage characters are the original bytes, seen through a one-byte-per-character lens, and their code points tell you exactly which bytes were on the wire.

Headers versus meta: a generation of broken pages

A web page can declare its encoding in two places: the HTTP headerContent-Type: text/html; charset=utf-8 and the in-page <meta charset> tag. Browsers resolve conflicts in a fixed order: a byte order mark wins outright, then the HTTP header, then the meta tag, which is only honored if it appears within the first 1024 bytes of the document.

The classic disaster was a site saved as UTF-8 while the server (an Apache default_charset directive, a PHP default, a misconfigured proxy) sent charset=ISO-8859-1. The header wins, so every accented letter on the site decoded as two characters, and editing the meta tag changed nothing because the browser never reached it. The mirror image happened too: a UTF-8 header over Latin-1 bytes turns byte E9 into an invalid sequence, and browsers substitute U+FFFD, the black diamond with a question mark.

The durable fixes are boring. Serve charset=utf-8 in the header and keep the meta tag in agreement. If you cannot control the server, numeric character references such as é or é survive any 8-bit misconfiguration because they are pure ASCII; the converter's entity panels generate them for you.

Diagnosing mojibake in practice

When a bug report says the text is broken, the converter turns guesswork into reading:

• Paste the damaged text and read the U+ panel. Values that look like byte pairs, C3 A9 or E2 80 99, are the original UTF-8 wearing Latin-1 glasses.
• A Ã followed by another high character signals UTF-8 read as Latin-1. A â followed by € or ™ signals UTF-8 read as Windows-1252.
• To repair, feed the code point values back as hex bytes through the UTF-8 panel input: 63 61 66 C3 A9 becomes café.
• U+FFFD anywhere in the output means bytes were decoded as UTF-8 that never were UTF-8, so look for a legacy 8-bit source instead.

If the distinction between code points and the bytes that carry them still feels slippery, What is Unicode? covers the numbering side, and the encodings get their own treatment in Unicode vs UTF-8. The one habit worth keeping from this article: when text looks wrong, stop staring at the glyphs and read the numbers. They never lie about which encoding touched the data last.