Lesson 48 · 2 min
Latency and streaming: why answers type themselves out
Chatbots show words as they are produced, called streaming, because a full answer takes seconds and seeing it start right away feels much faster.
Latency is the wait between asking and getting a response. For a chatbot, it has two parts: the time before the first word appears, and the time to finish the whole answer.
A model writes one token at a time, and each one takes a moment. A long answer can take many seconds. If the app waited for the very end, you would stare at a blank screen.
Streaming sends each token to you as soon as it exists. That is why answers seem to type themselves out. The total time is about the same, but reading starts almost at once.
Things that add wait: longer answers, bigger models, long chats or large files to read, busy times of day, and "thinking" modes that reason before replying. When speed matters, ask for a short answer or use a faster model.
Do it now
Feel the difference length makes
- 1. Ask a chatbot for a one-sentence answer, and notice how fast it finishes.
- 2. Ask the prompt below, which demands a long answer, and time it.
- 3. Next time you need something quick, say so in your prompt.
Explain how a bicycle works in 600 words, covering the pedals, chain, gears and brakes.