What metric measures how well a probability model predicts a sample?

Perplexity is a metric specifically designed to evaluate the performance of probability models, particularly in the context of language models and generative AI. It quantifies how well a probability model predicts a sample by measuring the likelihood of the predicted sequences compared to the actual sequences. A lower perplexity indicates that the model is better at predicting the sample, signifying that it assigns higher probabilities to the observed data. This is particularly important in generative tasks, where the goal is to generate text that closely resembles human language.

In contrast, accuracy, precision, and recall are performance metrics typically used in classification tasks rather than in the evaluation of probability distributions. Accuracy measures the proportion of correct predictions, precision focuses on the quality of positive predictions, and recall assesses how well all relevant instances are captured, but these metrics do not inherently provide insight into the probabilistic interpretation of predictions like perplexity does. Thus, perplexity is the appropriate measure for evaluating how a probability model predicts a sample.