What Is UTF-8? How Unicode's Default Encoding Actually Works

UTF-8 is a character encoding that turns Unicode text into bytes. It is the dominant encoding on the web, in source code, and in most file formats, and understanding it removes a whole category of mysterious "garbage character" bugs.

Characters, code points, and encodings

To see what UTF-8 does, separate two ideas that are easy to conflate. A character set assigns each character a number. Unicode is the universal character set: it maps every letter, digit, symbol, ideograph, and emoji to a unique number called a code point, written as U+ followed by hexadecimal, such as U+0041 for "A" or U+1F600 for a grinning face.

An encoding is the separate question of how those code-point numbers get stored as actual bytes on disk or sent over a network. Unicode defines several encodings (UTF-8, UTF-16, UTF-32). UTF-8 is one specific way to serialize Unicode code points into bytes. The character set says which number; the encoding says which bytes.

How UTF-8 encodes a code point

UTF-8 is a variable-width encoding: a single code point uses one to four bytes depending on its value. The scheme is designed so the leading bits of each byte announce its role.

• 1 byte (0xxxxxxx) covers U+0000 to U+007F — exactly the ASCII range.
• 2 bytes (110xxxxx 10xxxxxx) covers U+0080 to U+07FF — Latin accents, Greek, Cyrillic, Hebrew, Arabic.
• 3 bytes (1110xxxx 10xxxxxx 10xxxxxx) covers U+0800 to U+FFFF — most CJK characters and many symbols.
• 4 bytes (11110xxx 10xxxxxx 10xxxxxx 10xxxxxx) covers U+10000 to U+10FFFF — emoji, rare scripts, and historic characters.

A leading byte starting with 0 is a standalone ASCII character. A byte starting with 11 begins a multi-byte sequence, and the number of leading 1s tells you how many bytes follow. Every continuation byte starts with 10, so it can never be mistaken for the start of a character. The remaining x bits hold the code point, packed in big-endian order.

For example, "é" (U+00E9) encodes as the two bytes 0xC3 0xA9. You can take any text apart byte by byte with a Hex to ASCII converter or examine code points directly with a Unicode inspector; viewing the raw binary with a binary to text converter makes the 10xxxxxx continuation pattern visible.

Why UTF-8 became the default

Several properties made UTF-8 win over the alternatives.

ASCII compatibility

The first 128 code points encode as a single byte identical to their ASCII value. Any valid ASCII file is already valid UTF-8 with no changes. Decades of tools, protocols, and config formats that assumed ASCII kept working, which made adoption nearly free.

No byte-order problem

UTF-16 and UTF-32 store multi-byte units, so they need to agree on byte order (endianness), often via a byte-order mark. UTF-8 is defined as a stream of single bytes in a fixed order, so endianness never applies and no BOM is required.

Self-synchronizing and compact

Because continuation bytes are unmistakable, a decoder that starts mid-stream can find the next character boundary by skipping bytes until one does not begin with 10. UTF-8 is also space-efficient for Latin-script and markup-heavy text, since ASCII characters — the bulk of HTML, JSON, and code — cost only one byte each.

Where you encounter UTF-8

UTF-8 is the recommended encoding for HTML and the default for many formats. Web pages declare it with <meta charset="utf-8">, and servers announce it in the Content-Type header as charset=utf-8 — closely tied to the document's declared MIME type. JSON is defined to use UTF-8, and most programming languages now default to it for source files and string handling.

It is worth distinguishing UTF-8 from related transformations. Percent-encoding (URL encoding) escapes bytes that are unsafe in a URL, and on modern systems those bytes are the UTF-8 bytes of the character — so the two work together, as covered in the URL encoding guide and the URL encoder. Base64 is a binary-to-text scheme layered on top of bytes, not a character encoding; encoding text with the Base64 tool first treats it as UTF-8 bytes, then re-expresses those bytes in a 64-character alphabet.

Common pitfalls

Mojibake from a charset mismatch

The classic symptom is text like é appearing where "é" should be. This happens when bytes encoded as UTF-8 are decoded as a single-byte legacy encoding such as Windows-1252. Nothing is corrupted — the bytes are simply being interpreted under the wrong rules. The fix is to make the producer and consumer agree on UTF-8 explicitly.

The BOM where it is not wanted

A UTF-8 byte-order mark (0xEF 0xBB 0xBF) is optional and carries no ordering information. Some editors prepend it, which can break shell scripts, break JSON parsers, or show as a stray character at the start of a file. Save as "UTF-8 without BOM" unless a specific tool requires one.

Counting characters by byte length

Because characters vary in width, the byte length of a UTF-8 string is not its character count. Slicing a string at an arbitrary byte offset can split a multi-byte character and produce invalid data. Operate on decoded code points, and remember that a single user-perceived character (an emoji with a skin-tone modifier, for instance) can be several code points.

Assuming all text is UTF-8

Older files and external feeds may use Latin-1, Shift-JIS, or other encodings. Detection is heuristic and imperfect, so prefer an explicit, documented encoding for any data you control rather than guessing at read time.

How UTF-8 relates to ASCII and UTF-16

ASCII is a 7-bit set of 128 characters; UTF-8 is a strict superset, so the ASCII table describes exactly the one-byte range of UTF-8. UTF-16 encodes most characters in two bytes and uses surrogate pairs for code points above U+FFFF; it is common inside language runtimes (such as JavaScript and Java strings) but rare as a file or wire format. UTF-32 uses a fixed four bytes per code point, which simplifies indexing at the cost of size. For storage and transmission, UTF-8 is almost always the right default.

Frequently Asked Questions