Text entropy, explained: what predictability actually measures

Some writing feels alive and some feels like it could have written itself. The first kind keeps surprising you — the next word is rarely the one you'd have guessed. The second kind runs on rails: clichés, filler, and stock phrases that your brain completes before your eyes arrive. Text entropy is the attempt to put a number on that difference.

The idea comes from Claude Shannon, who in 1948 founded information theory by asking a deceptively simple question: how much information does a symbol actually carry? His answer — entropy, measured in bits — is now the backbone of every compression algorithm, every error-correcting code, and, more recently, a popular-but-shaky way to ask whether a passage was written by a person or a model.

What a "bit per symbol" really means

Entropy measures average surprise. If a symbol is highly predictable, learning it tells you almost nothing, so it carries little information. If a symbol could have been many things with roughly equal odds, learning which one it was tells you a lot.

The unit is bits. One bit is the information in a single fair coin flip — two equally likely outcomes. Two bits cover four equally likely outcomes, three bits cover eight, and so on: n bits distinguish 2ⁿ equally likely possibilities. So when the tool reports an entropy of, say, 4.1 bits per word, it's saying each word in your text carries about as much surprise as choosing fairly among roughly 17 options. The formula behind it weights every symbol by how often it appears: common symbols contribute little, rare ones contribute a lot, and the total is the average across the whole text.

Character, word, and bigram modes measure different things

The text entropy analyzer can treat your text as a stream of three different kinds of symbol, and they answer genuinely different questions.

• Character mode looks at letters and spaces. English characters have low entropy — usually under 5 bits even though there are dozens of possible characters — because the letter distribution is so lopsided ("e" and space are everywhere, "q" and "z" almost never). This mode mostly reflects the language itself, not your style.
• Word mode treats each distinct word as a symbol. This is the one most people mean by "how repetitive is my writing." A vocabulary that keeps reaching for the same handful of words scores low; varied, specific diction scores high.
• Bigram mode looks at consecutive word pairs. It catches a subtler kind of predictability: phrasing. Two pieces of writing can use the same vocabulary while one strings it into fresh combinations and the other recycles "in order to," "at the end of the day," and "it is important to note." Bigram entropy is where stock phrasing shows up.

Why predictability is relative, not absolute

Alongside raw entropy the tool reports a predictability score from 0 to 1, and this is where people misread the output. Predictability is computed against the maximum possible entropy for your text — the entropy you'd get if every distinct symbol you used appeared equally often. A text with 200 unique words has a higher ceiling than one with 50, so the two can't be compared on raw bits alone.

Predictability normalizes for that. A score near 0 means your symbols are spread about as evenly as they could be (high entropy, hard to predict); a score near 1 means a few symbols dominate (low entropy, very predictable). That's why the analyzer flags results in bands — high entropy, moderate, and very predictable — rather than asking you to interpret a raw bit count. It's measuring how concentrated your usage is relative to your own vocabulary, not against some universal scale.

One consequence worth internalizing: short samples are noisy. A 30-word paragraph hasn't given any symbol the chance to repeat, so it will look artificially high-entropy. Like every statistical text measure — the same caveat applies to the lexical density and word frequency tools — entropy wants a few hundred words before the number stabilizes.

The compression connection

Entropy and file compression are the same idea wearing different clothes. Shannon's theorem says entropy is the theoretical floor on how small you can losslessly squeeze a message: text averaging 4 bits per character can't be compressed below 4 bits per character, full stop. That's why the analyzer also shows a compression estimate — predictable text has lots of redundancy to squeeze out, so it estimates a higher compressible fraction. If you've ever noticed that a repetitive log file zips down to almost nothing while a dense, varied document barely shrinks, you've watched entropy in action. Low entropy, high compressibility; high entropy, stubborn file size.

Entropy as an AI signal — handle with care

The headline use right now is AI detection, and entropy is genuinely part of the picture. Language models are trained to output likely continuations, which nudges machine text toward flatter, more predictable phrasing — lower perplexity, the model's own cousin of entropy. So AI-generated prose often clusters in a narrower, more predictable band than the spiky, idiosyncratic distribution of an engaged human writer.

But entropy alone cannot decide authorship, and treating it as a verdict will burn you. Plenty of human writing is low-entropy: instructions, legal boilerplate, a tired first draft, a non-native writer leaning on safe constructions. Plenty of AI writing is high-entropy once it's been prompted for variety or edited by a person. Modern models can also be told to raise their "temperature" and produce deliberately surprising text. Entropy is a weak prior, useful as one input among several — alongside the patterns the AI text pattern detector looks for — never a standalone judge. Anyone selling you a single number as proof of machine authorship is overselling it.

How to actually use it

For your own writing, run word and bigram mode and watch for a predictability score creeping toward the "very predictable" band — that's the quantitative version of an editor telling you the prose feels samey. The fix isn't to chase a higher number for its own sake; it's to notice which symbols are dominating and decide whether the repetition is doing work (deliberate emphasis, a refrain) or just inertia (the same crutch verb, the same connective phrase). Pair it with a repeated words finder to see exactly which words are pulling the entropy down.

When you want the numbers without doing the bit-counting by hand, the text entropy & predictability analyzer computes Shannon entropy at the character, word, and bigram level, normalizes it into a predictability score, and estimates compressibility — so you can see, concretely, how much your writing keeps surprising the reader and how much of it could have written itself.