Definition
A benchmark measures a system’s performance with a fixed dataset and a uniform scoring rule. In AI, it is typically a set of tasks plus automatic scoring, used to grade the capability of models such as large language models.
How it works
Every system under test runs the same tasks under the same conditions, so scores can be compared across models. Beyond model capability, the computing industry also runs benchmarks for hardware and service performance, such as the MLPerf program for inference throughput.
Usage
Model launches, procurement decisions, and research comparisons all lean on benchmarks; an evaluation effort usually picks a set of them to answer “is this model any good?” Scores only reflect what the benchmark covers and do not equal real-world usefulness.
Related and confusable terms
A benchmark is one standardized tool within evaluation, which is broader and also includes human review. Performance numbers like tokens per second measure speed, while capability benchmarks measure answer quality — the two kinds of figures should not be conflated.