AI Data Lab Wiki

AI Data Lab Wiki › Glossary › LLM evaluation

What Is LLM Evaluation?

By AI Data Lab Wiki editors · checked November 15, 2026

LLM evaluation is the practice of testing a large language model's outputs against criteria such as accuracy, safety and usefulness, using benchmarks, human raters or other models as judges. OpenAI's API documentation says evaluations, often called evals, test model outputs to ensure they meet the style and content criteria a developer specifies. 1 The same word covers academic benchmarks and the checks a company runs on its own application.

What Is LLM Evaluation

LLM evaluation is how developers measure what a language model can do and where it fails. OpenAI calls writing evals an essential part of building reliable applications, especially when upgrading or trying new models. 1

Evaluation differs from training. Training changes the model, while evaluation scores it, and the scores can then guide further training data, such as the human feedback described under RLHF.

How LLM Evaluation Works

LLM evaluation works by running a model on a fixed set of test inputs and scoring each answer with a grader. OpenAI's guide describes graders that decide whether an output is correct, with one example that compares model output to a human label. 1

  1. Choose scenarios and the qualities to measure, such as accuracy or toxicity.
  2. Run the model on the test inputs and collect its answers.
  3. Score the answers with string matching, human raters or another model acting as judge.
  4. Report results by metric, and repeat the run when the model changes.

The HELM project argues for measuring more than accuracy. It reports 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) so that trade-offs show up. 2

LLM Evaluation Example

A standard LLM evaluation example is MMLU, a test of 57 tasks that includes elementary mathematics, US history, computer science and law. When it was submitted on September 7, 2020, its authors found the largest GPT-3 model beat random chance by almost 20 percentage points on average. 3

HELM, submitted on November 16, 2022, evaluated 30 models on 42 scenarios. Its authors say models had on average been tested on just 17.9% of its core scenarios beforehand, and that the project raised this to 96.0%. 2

Who Does the Human Work in LLM Evaluation

Human raters and expert reviewers do the human work in LLM evaluation by scoring, comparing and voting on model answers. In the InstructGPT paper, OpenAI says it mainly evaluated models by having its labelers rate outputs on prompts from held-out customers. 4

Public leaderboards use crowds. The MT-Bench paper introduced Chatbot Arena, a crowdsourced battle platform, and released 3K expert votes and 30K conversations with human preferences. 5

Vendors sell rating as a service. Scale AI says on its evaluation page that expert human raters provide evaluations backed by quality assurance. 6 The people who take this work are described on the AI trainer page and in AI training jobs.

LLM Evaluation Tools

LLM evaluation tools include open benchmark suites, built-in graders in model APIs and vendor platforms. HELM released its raw model prompts and completions publicly along with a modular toolkit, and OpenAI's API includes evals with graders. 2

Scale AI lists risk categories its evaluation platform tests for, including misinformation, bias, privacy and cyberattacks. 6

Models can also grade other models. The MT-Bench authors report that GPT-4 as a judge reached over 80% agreement with human preferences, the same level as agreement between humans, and they also document position, verbosity and self-enhancement biases in such judges.

LLM Evaluation vs AI Red Teaming

LLM evaluation measures performance against a fixed test, while AI red teaming hunts for failures with open-ended adversarial testing. NIST's July 2024 Generative AI Profile lists internal and external evaluations and AI red-teaming as separate actions for measuring generative AI risk. 7

Both can involve a human in the loop, and NIST suggests documenting the instructions given to data annotators.

Questions About LLM evaluation

These are the questions about LLM evaluation that people search most.

What are LLM evals?

LLM evals are tests that check a language model's outputs against criteria the developer sets. OpenAI's documentation uses the word for tests of style and content, and the field also uses it for public benchmarks such as MMLU and HELM.

Which LLM evaluation tools exist?

LLM evaluation tools range from open benchmarks to commercial platforms. HELM publishes its prompts, completions and a toolkit, OpenAI's API has evals with graders, and Scale AI offers an evaluation platform that uses expert human raters.

Can an LLM evaluate another LLM?

An LLM can evaluate another LLM. The MT-Bench paper found GPT-4 as judge matched human preferences with over 80% agreement, but it also documented position, verbosity and self-enhancement biases, so judge models are checked against human ratings.

Who does LLM evaluation?

LLM evaluation is done by model developers, academic groups and vendors, with human raters scoring answers. OpenAI used about 40 contractors as labelers for InstructGPT, and Chatbot Arena collects crowdsourced votes from the public.

More AI Data Terms

More AI data terms explained in this wiki:

Sources for LLM evaluation

The sources for LLM evaluation are listed in the order the page cites them, read November 15, 2026.

  1. platform.openai.com/docs/guides/evals
  2. arxiv.org/abs/2211.09110
  3. arxiv.org/abs/2009.03300
  4. arxiv.org/pdf/2203.02155
  5. arxiv.org/abs/2306.05685
  6. scale.com/evaluation/model-developers
  7. nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf