Character Encoding Explained: ASCII, Unicode and UTF-8
Every developer eventually meets text that has gone wrong — an apostrophe rendered as ’, a name reduced to question marks, a CSV that opens fine in one program and as gibberish in another. These are all the same problem wearing different clothes: bytes being read with a different encoding than the one used to write them. This guide explains what is actually happening.
Characters are not bytes
Computers store bytes. Text is characters. An encoding is the agreed mapping between the two, and the crucial point is that bytes carry no record of which mapping produced them. A file is just a sequence of numbers; the encoding is an assumption the reader brings.
That is why mojibake happens. When text written as UTF-8 is read as Windows-1252, every byte still has a value and the reader still produces characters — just the wrong ones. Nothing errors, because nothing is technically invalid. The output is simply wrong, which is far harder to notice in an automated pipeline than a crash would be.
ASCII and the code page era
ASCII, standardised in 1963, defined 128 characters in seven bits: English letters, digits, punctuation and control codes. It was sufficient for American English and nothing else, which left one spare bit and 128 unused slots that everyone proceeded to fill differently.
The result was code pages — dozens of incompatible mappings for that upper range. Byte 233 was é in Latin-1, щ in Cyrillic, and something else again in Greek. Text was only readable if you knew which code page produced it, and that information travelled separately from the file, when it travelled at all. Most legacy encoding bugs still trace back to this period.
What Unicode actually is
Unicode assigns every character in every writing system a unique number called a code point, written like U+00E9 for é. It covers modern and historical scripts, mathematical notation, and emoji, with room for over a million code points.
The common misconception is that Unicode is an encoding. It is not — it is a catalogue. It says which number means which character, but says nothing about how to store that number as bytes. That is a separate decision, and it is where UTF-8, UTF-16 and UTF-32 come in.
Why UTF-8 won
UTF-8 encodes each code point as one to four bytes, and its decisive property is that the first 128 code points encode as a single byte identical to ASCII. Any ASCII file is already a valid UTF-8 file, which meant decades of existing text and software kept working with no migration.
It is also self-synchronising — you can tell from any byte whether it starts a character or continues one, so a corrupted stream recovers instead of cascading. And it is endianness-free, unlike UTF-16, which needs a byte-order mark to disambiguate. UTF-8 now accounts for the overwhelming majority of the web, and for new work there is rarely a good reason to choose anything else.
Why a character is not always one character
Once you leave ASCII, counting characters stops being simple. é can be a single code point or two — an e followed by a combining accent — and both render identically while comparing as different strings. Unicode normalisation exists to reconcile these forms, which is why user input should usually be normalised before it is compared or stored.
Emoji make it starker. A family emoji can be several emoji joined by zero-width joiners, and a flag is two regional indicator letters. One visible symbol can be a dozen code points and more bytes again, which is why character limits behave unexpectedly and why naive string truncation can slice an emoji in half.
Diagnosing mojibake from its shape
Corrupted text is not random. The pattern of what you see identifies which pair of encodings collided, which in turn tells you where in the pipeline to look.
| What you see | What went wrong |
|---|---|
| é è ü ’ “ | UTF-8 bytes read as Windows-1252 or Latin-1 |
| ? in place of accented letters | Converted to ASCII, or a code page with no such character |
| Black diamonds with a question mark | Invalid UTF-8 byte sequences — genuinely broken data |
| Empty rectangles (tofu) | Encoding is fine; the font has no glyph for those characters |
| Text reads right to left in a filename | A bidi override character — worth treating as suspicious |
| A stray before the first character | A byte-order mark being read as content |
The distinction in rows three and four matters most. Black diamonds mean the bytes cannot be decoded and data is genuinely damaged. Empty rectangles mean the decoding worked perfectly and only the font is missing — the text is intact, and installing a font or changing the typeface fixes it. They look equally broken and have completely different remedies.
Recovering text that has already gone wrong
Whether corrupted text can be repaired depends on whether the damage was reversible, and the first mojibake row above usually is. When UTF-8 was misread as Windows-1252, every original byte is still present — it was merely displayed as the wrong character. Encoding it back to Windows-1252 bytes and decoding those as UTF-8 restores the original exactly.
The lossy cases cannot be recovered. Once a character has been replaced by a question mark or the Unicode replacement character, the information identifying it is gone. There is nothing left to reverse, and the only fix is to go back to the source.
Practical rules that avoid most problems
Use UTF-8 everywhere and declare it explicitly — in your HTML meta charset, your HTTP Content-Type header, your database column collation and your editor's default. Most encoding bugs come from one link in that chain disagreeing with the others.
Be specific about the two places that most often go wrong. CSV files opened in Excel need a byte-order mark or Excel guesses the local code page and mangles accents. And URLs must be percent-encoded, since they may only contain ASCII — which is why a non-English filename in a link needs encoding before it will work.