Temperature is a number you pass to an LLM API that controls how random the output is. Temperature 0 means the model always picks the highest-probability next token, producing deterministic output. Temperature 1.0 allows meaningful randomness, producing varied output. Temperature above 1.5 produces increasingly incoherent output. Understanding temperature, along with the related sampling parameters top-p and top-k, gives you direct control over the reliability and creativity of LLM responses.
How Token Selection Actually Works
Before explaining temperature, you need to understand what the model is doing at each step.
At every position in the output, the model produces a probability distribution over its entire vocabulary (roughly 100,000 tokens for GPT models). Entry for each token is a probability: the model's estimate of how likely that token is to come next given everything before it.
In a simple factual context, the distribution might be very peaked: "The capital of France is" produces a 97% probability for "Paris," a 1% probability for " the," and tiny probabilities for everything else. In a creative context, the distribution might be flatter: "She opened the door and saw" might assign 15% probability to "a," 12% to "her," 10% to "nothing," and so on.
Temperature applies a transformation to this distribution before sampling. A temperature of 1.0 uses the distribution as-is. A temperature below 1.0 sharpens the distribution (makes high-probability tokens even more likely). A temperature above 1.0 flattens it (makes low-probability tokens more competitive).
Concrete Examples of Temperature in Practice
Consider the prompt: "The best programming language for beginners is"
Temperature 0.0: "Python" - the model's highest-probability completion, repeated every time. Deterministic.
Temperature 0.5: "Python" most of the time, occasionally "JavaScript" or "Scratch." Some variation but still mostly the highest-probability options.
Temperature 1.0: "Python," "JavaScript," "Ruby," "Scratch," "Java" - meaningful variety across multiple runs. All plausible answers.
Temperature 1.5: "Python," "Lua," "an interesting question," "BASIC," "debated" - variety extends to less likely tokens. Some completions are odd.
Temperature 2.0: Increasingly incoherent. The model might complete with tokens that have no semantic connection to the prompt, because flattening the distribution enough makes rare tokens competitive with common ones.
The practical rule: never go above 1.0 for production applications. The gains in variety are outweighed by the degradation in coherence.
