AI Data Lab Wiki

AI Data Lab Wiki › Glossary › RLHF

What Is RLHF?

By AI Data Lab Wiki editors · checked November 15, 2026

RLHF, short for reinforcement learning from human feedback, is a method that trains an AI model using people's judgments of its outputs as the reward signal. Labelers compare or rank answers, a reward model learns their preferences, and the main model is then tuned to score well against that reward model. OpenAI described this recipe for language models in its InstructGPT paper, posted to arXiv on March 4, 2022. 1

What Is RLHF

RLHF is an extra training stage that replaces a fixed objective with human preference. OpenAI's post of January 27, 2022 explains why it matters: GPT-3 was trained to predict the next word on a large dataset of Internet text, not to do what a user wants. 2

In the original 2017 paper by Paul Christiano and five co-authors, goals were defined through human preferences between pairs of trajectory segments, so no one had to write a reward function by hand.

How RLHF Works

RLHF works in stages, and the InstructGPT paper lists them. 4

  1. Labelers write demonstrations of the desired output, and the model is fine-tuned on them with supervised learning.
  2. Labelers rank several outputs from the model for the same prompt, from best to worst.
  3. A reward model is trained on those rankings to predict which output a labeler would prefer.
  4. The model is tuned with the PPO reinforcement learning algorithm to maximize the reward model's score.

To speed up collection, each prompt came with between 4 and 9 responses to rank. The reward model is trained to predict which output the labelers would prefer, and it then serves as the reward function.

Reinforcement Learning from Human Feedback Example

InstructGPT is the best-known example of reinforcement learning from human feedback applied to a language model. In OpenAI's human evaluations, outputs from the 1.3 billion parameter InstructGPT were preferred to outputs from the 175 billion parameter GPT-3, which has 100x more parameters. 1

An earlier example used a simulated robot. OpenAI's June 13, 2017 post says its agent learned to backflip from around 900 bits of feedback, under an hour of one evaluator's time. 5

Other labs published the same pattern. A 2020 paper trained a summarization model on human comparisons between summaries, and Anthropic's April 12, 2022 paper applied RLHF to a helpful and harmless assistant, updating its models weekly with fresh human feedback. 6 7

Who Does the Human Work in RLHF

Contractors and subject experts do the human work in RLHF: they write demonstrations, compare outputs and rank them. For InstructGPT, OpenAI says in its paper that it hired about 40 contractors on Upwork and through Scale AI, chosen by a screening test. 4

The paper says these labelers were mostly English-speaking people living in the United States or Southeast Asia, and that they agreed with each other on about 73% of examples. It adds that the method aligns a model to the stated preferences of a specific group, mostly the labelers and researchers.

Companies that supply this labor are profiled here. Scale AI lists Human Feedback and Preference Ranking on its RLHF page, and Surge AI says in a blog case study that Anthropic began using its platform for RLHF feedback. 8 9 The people who take this work are described on the AI trainer page.

RLHF vs Direct Preference Optimization

Direct preference optimization (DPO) is an alternative to RLHF that skips the separate reinforcement learning stage. Its authors, in a paper submitted May 29, 2023, call RLHF a complex and often unstable procedure and say they solve the same problem with only a simple classification loss. 10

DPO still learns from human preference data. The difference is in how the model is trained on it, not in who supplies the comparisons.

RLHF vs RLAIF

RLAIF, or RL from AI Feedback, replaces some human judgments in RLHF with judgments made by a model. Anthropic's Constitutional AI paper, submitted December 15, 2022, trained a harmless assistant with no human labels identifying harmful outputs, using only a list of rules as human oversight. 11

That approach produces its own training labels, which links it to synthetic data. RLHF is also one form of human in the loop training.

RLHF Limitations

Limitations of RLHF reported by its developers include evaluator error and narrow labeler groups. OpenAI's 2017 post says the system is only as good as the evaluator's intuition, and that a robot instead placed its manipulator between the camera and the object so it only appeared to be grasping. 5

OpenAI's 2022 post says InstructGPT follows instructions in English and is biased toward the cultural values of English-speaking people. The company said it was researching disagreements between labelers' preferences. 2

Questions About RLHF

These are the questions about RLHF that people search most.

What does RLHF stand for?

RLHF stands for reinforcement learning from human feedback. The name covers training methods in which human judgments, such as rankings of model answers, supply the reward that guides reinforcement learning. The idea appeared in a 2017 paper on human preferences, and later work applied it to language models.

Is RLHF the same as fine-tuning?

RLHF is one kind of fine-tuning, so it is not the same as all fine-tuning. In the InstructGPT recipe, supervised fine-tuning on labeler demonstrations comes first, and the reinforcement learning stage that uses a reward model follows it.

Who are the RLHF labelers?

The RLHF labelers for InstructGPT were about 40 contractors hired on Upwork and through Scale AI, selected with a screening test. The paper says they were mostly English-speaking people in the United States or Southeast Asia. Other projects may use different workforces.

Does RLHF always need humans?

RLHF does not always need humans for every label. Anthropic's Constitutional AI work swapped human harm labels for model-made preferences, which it calls RL from AI Feedback, while keeping a human-written list of rules.

More AI Data Terms

More AI data terms explained in this wiki:

Sources for RLHF

The sources for RLHF are listed in the order the page cites them, read November 15, 2026.

  1. arxiv.org/abs/2203.02155
  2. openai.com/index/instruction-following
  3. arxiv.org/abs/1706.03741
  4. arxiv.org/pdf/2203.02155
  5. openai.com/index/learning-from-human-preferences
  6. arxiv.org/abs/2009.01325
  7. arxiv.org/abs/2204.05862
  8. scale.com/rlhf
  9. surgehq.ai/blog/anthropic-surge-ai-rlhf-platform-train-llm-assistant-human-feedback
  10. arxiv.org/abs/2305.18290
  11. arxiv.org/abs/2212.08073