AI Data Lab Wiki

AI Data Lab Wiki › Glossary › Human in the loop

What Is Human in the Loop?

By AI Data Lab Wiki editors · checked November 15, 2026

Human in the loop (HITL) is a way of building or running an AI system so that people take part in its training, checking or operation instead of leaving every step to the machine. IBM defines it as a system or process in which a human actively participates in the operation, supervision or decision-making of an automated system. 1 The idea appears in AI training, in model evaluation and, as human oversight, in rules for high-risk systems.

What Is Human in the Loop

Human in the loop is an approach in which a person supplies judgment at some point in an AI workflow. Google Cloud describes HITL machine learning as people providing guidance, feedback and annotations during the training, evaluation or operation of a model. 2

IBM adds that models can meet edge cases their training did not prepare them for, and that HITL lets humans correct inputs so the model can improve over time.

How Human in the Loop AI Works

Human in the loop AI works by putting people at specific points in the model's cycle, and Google Cloud names two of them. 2

  1. People label training data, either by hand or with tools, so a supervised model has examples to learn from.
  2. In active learning, the model selects the data it wants a person to label.
  3. People review outputs, fix mistakes and send the corrections back as new training data.

Costs follow the headcount. IBM says human annotation can be slow and expensive and can become a bottleneck as data volume grows, and Amazon Web Services says HITL does not scale linearly because workforce cost and coordination grow with each added reviewer. 3

Human in the Loop Example

A well-documented human in the loop example is the 2017 paper by Paul Christiano and co-authors, where people chose between pairs of trajectory segments of an agent's behavior. The authors report giving feedback on less than one percent of the agent's interactions and training new behaviors with about an hour of human time. 4

A language-model example is InstructGPT, where labelers wrote demonstrations and ranked outputs. This pattern is explained on the RLHF page.

Who Does the Human Work in HITL

Labelers, annotators and expert reviewers do the human work in HITL, and they may be employees, contractors or a vendor's workforce. Amazon SageMaker Ground Truth, for example, lets a customer build labeled datasets using workers from a vendor it chooses or from an internal private workforce. 5

For InstructGPT, OpenAI hired about 40 contractors on Upwork and through Scale AI to write demonstrations and rank outputs. 6 Task-level work is covered in data labeling, and the people who take it are described on the AI trainer page and in data annotation jobs.

Human in the Loop vs Human on the Loop

Human in the loop means a person actively reviews a task before any action occurs, while human on the loop means people monitor the system and step in only when needed. AWS draws this line and says in-the-loop review is typically used in regulated environments, while on-the-loop monitoring suits lower-risk or higher-volume work such as chatbots. 3

HITL also differs from AI red teaming, where people try to break a model, and from LLM evaluation, where people or models score it. Both can use humans in the loop.

Human in the Loop Rules

Rules on human in the loop oversight appear in a US framework and in EU law. NIST's AI Risk Management Framework, published in January 2023, says human-AI configurations range from fully autonomous to fully manual and that some systems, such as models used to improve video compression, may not need human oversight. 7

Article 14 of the EU AI Act, titled Human Oversight, requires high-risk AI systems to be designed so natural persons can effectively oversee them while in use. It also says overseers must stay aware of the tendency to over-rely on the system's output. 8

Questions About Human in the loop

These are the questions about Human in the loop that people search most.

What is human in the loop in AI?

Human in the loop in AI is the practice of involving people in training, evaluating or operating a model. Google Cloud lists labeling training data and active learning, where the model asks a person to label data, as typical ways humans take part.

Is HITL the same as RLHF?

HITL is not the same as RLHF, but RLHF is one form of it. HITL covers any point where a person helps an AI system. RLHF is a specific method in which human rankings of model outputs train a reward model.

Does human in the loop slow AI systems down?

Human in the loop can slow a system because reviewers take time. IBM says relying on humans can become a bottleneck as volume grows, and AWS says HITL does not scale linearly since workforce cost rises with each added reviewer.

Which rules require human oversight of AI?

The rules that require human oversight of AI include Article 14 of the EU AI Act, which applies to high-risk systems. The US NIST AI Risk Management Framework, which says its use is voluntary, calls for human oversight processes to be defined, assessed and documented.

More AI Data Terms

More AI data terms explained in this wiki:

Sources for Human in the loop

The sources for Human in the loop are listed in the order the page cites them, read November 15, 2026.

  1. ibm.com/think/topics/human-in-the-loop
  2. cloud.google.com/discover/human-in-the-loop
  3. aws.amazon.com/what-is/human-in-the-loop
  4. arxiv.org/abs/1706.03741
  5. docs.aws.amazon.com/sagemaker/latest/dg/sms.html
  6. arxiv.org/pdf/2203.02155
  7. nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
  8. artificialintelligenceact.eu/article/14