What does Inference Latency measure in LLMs?

Inference Latency specifically measures the duration from the moment an input is provided to the moment the output is fully generated by a language model. This metric is crucial in understanding how quickly an LLM can respond to queries or tasks, impacting user experience, especially in real-time applications. High inference latency can lead to delays in receiving information, making it less effective in interactive settings.

In the context of this question, other choices refer to different aspects of model development or operation. Training time pertains to how long it takes to develop the initial model, which is separate from inference. Data pre-processing refers to the time taken to prepare the input data for the model, a step that occurs before inference begins. Model evaluation encompasses the processes used to assess the model’s performance after training, which does not impact the speed of generating responses to user inputs. Thus, the focus on the time from input to output completion accurately defines inference latency in this context.