Which hyperparameter, when set closer to zero, makes LLM outputs more deterministic?

The hyperparameter that, when set closer to zero, makes the outputs of a large language model (LLM) more deterministic is temperature. In the context of generative models, the temperature parameter controls the level of randomness in the output distributions. A higher temperature encourages more diversity and creativity in the generated text by allowing a wider range of possible outputs, thus introducing more randomness into the generation process. Conversely, when the temperature is set closer to zero, the model becomes more deterministic, favoring outputs that have higher probabilities. This results in the model consistently selecting the most likely options, leading to repeated and predictable responses.

Adjusting the temperature impacts how confident the model is in its predictions – a lower temperature focuses on the highest probability choices, reducing variability and resulting in outputs that are more consistent across generations. This property is crucial when reliability and uniformity in responses are prioritized over creativity or novelty in text generation.

Other hyperparameters, such as learning rate, batch size, and dropout rate, control aspects of the training process and model optimization but do not directly influence the randomness or determinism of the output generation in the same way that temperature does.