Lexical diversity: what TTR measures, and why you need MATTR

Two writers describe the same scene. One reaches for a fresh word almost every line; the other circles the same dozen terms over and over. We sense the difference instantly — one feels rich, the other thin — but how do you put a number on it? The classic answer is lexical diversity, and the classic measure is the type-token ratio. It's a beautifully simple idea with one famous trap, and understanding the trap is the whole point of using the tool well.

Types, tokens, and the ratio between them

Start with the vocabulary. In text analysis, a token is any word occurrence, and a type is a distinct word. The sentence "the cat sat on the mat" has six tokens but five types, because "the" appears twice. The type-token ratio (TTR) is simply types divided by tokens — here, 5 ÷ 6, or about 0.83.

The intuition is direct: a TTR near 1.0 means almost every word is different (high diversity); a low TTR means heavy repetition. The lexical diversity calculator computes this by lower-casing each word, stripping surrounding punctuation, counting the unique forms, and dividing by the total word count. For short passages it's a perfectly good snapshot of how much your vocabulary repeats.

The flaw that breaks TTR

Here's the problem that every serious user of TTR runs into: it falls as the text gets longer. Not because long texts are less varied, but for an unavoidable mathematical reason. Tokens can keep climbing forever — you can always add another word — but types cannot. There are only so many words a writer uses, and the common ones ("the", "and", "is") get reused constantly. So as a document grows, the denominator races ahead of the numerator and the ratio sinks toward zero.

The practical consequence is brutal: TTR is only comparable between texts of the same length. A 200-word abstract will always post a higher TTR than a 5,000-word essay by the same author, even if the essay is far richer. Comparing their raw TTRs tells you which is shorter, not which is more varied. Any tool that reports a bare TTR without warning you about length is inviting a wrong conclusion — which is why this calculator notes that "TTR drops with longer text" right next to the number.

How MATTR fixes it

The fix is to stop measuring the whole text at once. MATTR — the Moving-Average Type-Token Ratio — slides a fixed-size window across the text and computes the TTR within each window, then averages all those local ratios. This calculator uses a 50-word window: it takes words 1–50 and finds their TTR, then words 2–51, then 3–52, and so on to the end, averaging the lot.

Because every window is the same size, the length bias disappears. A 200-word text and a 50,000-word book are both measured in 50-word chunks, so their MATTR scores are directly comparable. MATTR captures something real and stable: the local richness of the prose, sampled everywhere and averaged. It's the number to use whenever you want to compare diversity across documents of unequal length, and it's why the tool reports MATTR alongside the raw TTR rather than instead of it — the two together tell you both the headline figure and the length-robust one.

Hapax legomena and the long tail

The tool surfaces one more figure worth knowing: the count of hapax legomena — a wonderful old term for words that appear exactly once in the text. A high proportion of one-off words signals a wide, exploratory vocabulary; a low proportion means you're recycling the same terms. The list of hapax words is also a quick way to spot rare or unusual terms you might want to reconsider for a general audience, or to confirm that your specialist vocabulary is doing real work rather than showing off.

Reading the scores sensibly

A few guidelines for interpretation. For short texts, a TTR above roughly 50% is healthy; for the length-independent measure, a MATTR above about 70% indicates genuinely varied prose. But high diversity is not automatically "better" — technical and instructional writing should repeat key terms consistently, because swapping in synonyms for a defined concept confuses the reader. So a low score on a manual or a legal document is often correct, while a low score on a personal essay might mean the writing is monotonous.

Lexical diversity is one of those measures that rewards knowing what it can't do. Used naively — comparing raw TTRs across texts of different lengths — it produces nonsense. Used well — leaning on MATTR for comparison, reading the score in light of the genre — it's a sharp lens on a quality of writing that's otherwise hard to pin down. Paste your text into the lexical diversity calculator, look at the MATTR rather than the bare TTR when you want to compare, and let the hapax count tell you how far your vocabulary really ranges.