A language model writes one word at a time, and each word needs a full pass through a very large network. That is why long answers feel slow, and why the bill grows with the length of the reply.
Speculative decoding attacks that delay with an old trick: guess first, check after.
How it works
A small, fast model drafts the next several words. The large model then reads all of those words in a single pass and confirms how many it would have written itself. Every correct guess is a word you got without waiting for a separate pass. At the first wrong guess, the large model takes over and the cycle starts again.
The final text is what the large model would have produced anyway. That is the point: quality does not drop, because the large model has the last word on every word.
When it pays off
It helps most when the small model guesses well. That happens with predictable text: boilerplate replies, code, structured output, answers that quote the question or a document. It helps less with open-ended creative writing, where the next word is hard to guess.
It also depends on how busy your servers are. When a GPU is already fully occupied serving many users at once, there is less spare capacity to spend on checking guesses, and the gain shrinks.
How to decide
Do not take a benchmark number from someone else’s workload. Test on a sample of your own traffic and record three things before and after: time to the first word, total time for the answer, and cost per request. Keep it only if the gain is real at your volume.
The takeaway
Speculative decoding is a good tool, not a magic one. It earns its place on workloads with predictable output and spare capacity, and it should be switched on by measurement, not by fashion.
If model speed or cost is holding back a product, inference engineering is where we start, and our LLM cost calculator gives a first estimate.
Work with us on this
- Inference Engineering and Optimisation: Serving stacks tuned for speed and cost: quantisation, batching, caching, speculative decoding and GPU planning, measured on your own traffic.