What Is AI Red Teaming?
By AI Data Lab Wiki editors · checked November 15, 2026
AI red teaming is structured testing in which people, or other AI models, try to make an AI system produce harmful, flawed or unsafe output so the problems can be found and fixed. This page covers red teaming of AI models, not the network-intrusion exercises run by cybersecurity red teams. A US executive order defined the term as a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with its developers. 1
What Is AI Red Teaming
AI red teaming is a way to probe a model for flaws before users find them. Executive Order 14110, signed on October 30, 2023, says it is most often done by dedicated red teams that use adversarial methods to find harmful or discriminatory outputs, unforeseen behaviors, limitations and risks from misuse. 1
The Federal Register lists that order as revoked by Executive Order 14148 on January 20, 2025. NIST's Generative AI Profile, published in July 2024, uses closely matching wording for AI red-teaming. 2
How AI Red Teaming Works
AI red teaming works as a loop of testing, recording and fixing, using people, automated tools or both. OpenAI says red teaming means using people or AI to explore a new system's risks in a structured way, and that it uses manual, automated and mixed methods. 3
- A team picks the harms to test for and recruits testers with matching expertise.
- Testers write adversarial prompts and scenarios meant to produce unwanted answers.
- Each prompt, the model's answer and a classification of the result are recorded in a report. Toloka describes its reports this way. 4
- Developers fix the model or its safeguards, then test again. Scale AI's podcast episode says model upgrades often boost performance while loosening safety constraints, so it recommends red teaming in every release cycle. 5
Models can also write the test cases. A 2022 paper by Ethan Perez and colleagues says hand-written test cases are expensive, and instead generated them with another language model to probe a 280 billion parameter chatbot. 6
AI Red Teaming Example
A public AI red teaming example is Anthropic's 2022 study, which released a dataset of 38,961 red team attacks. 7
The paper tested 3 model sizes, from 2.7 billion to 52 billion parameters, and found that models trained with RLHF became harder to red team as they grew, while other model types showed a flat trend.
OpenAI has also described bringing in outside experts to test its DALL-E 2 image model in early 2022. 3
Who Does AI Red Teaming
AI red teaming is done by in-house safety staff, outside domain experts and paid vendor teams. OpenAI opened a call for its Red Teaming Network on September 19, 2023, inviting experts from fields such as chemistry, biology, law, healthcare, child safety and finance. 8
OpenAI says members are compensated when they work on a project, that the work is often under non-disclosure agreements, and that time commitments could be as few as 5 to 10 hours in one year. Applications for that phase closed on December 1, 2023.
Vendors sell the same service. Toloka says its team is made up of AI researchers, linguists and subject matter experts, and Scale AI's episode page says it is hiring for its red teaming team. See Toloka and Scale AI for the companies, and the AI trainer page for related work. Microsoft also runs an AI Red Team and publishes a guide to building one. 9
AI Red Teaming Guidance
AI red teaming guidance from NIST treats testers as human subjects in some cases. Its Generative AI Profile lists red-teaming next to participatory engagement and field testing, and says organizations should follow human subjects research practices such as informed consent and subject compensation. 2
It also suggests documenting the instructions given to data annotators or AI red-teamers. For how red teaming sits beside scoring and benchmarks, see LLM evaluation.
Questions About AI red teaming
These are the questions about AI red teaming that people search most.
What is LLM red teaming?
LLM red teaming is AI red teaming aimed at a large language model. Testers write prompts meant to draw out harmful, false or policy-breaking answers, then record the results so developers can fix them, as Toloka and OpenAI describe their own processes.
Do AI red teamers get paid?
AI red teamers are paid by some programs. OpenAI says members of its Red Teaming Network are compensated when they take part in a project, and NIST's Generative AI Profile points to subject compensation. Pay and terms vary by program.
Can an AI model red team another AI model?
An AI model can red team another AI model. A 2022 paper by Perez and colleagues generated test cases with one language model to find offensive replies in another, and OpenAI says it uses automated and mixed methods along with human testers.
Who defines AI red teaming?
AI red teaming was defined in US government documents, including Executive Order 14110 in 2023 and NIST's Generative AI Profile in July 2024. Developers such as OpenAI publish their own descriptions of the practice.
More AI Data Terms
More AI data terms explained in this wiki:
- Data annotationAdding labels and structure to raw data so models can learn.
- Data labelingTagging raw data so a model can learn what it shows.
- AI training dataThe examples a model learns from before it is used.
- Human in the loopPeople taking part in training, checking or running an AI system.
- Synthetic dataData generated by algorithms rather than collected from real events.
Sources for AI red teaming
The sources for AI red teaming are listed in the order the page cites them, read November 15, 2026.
- federalregister.gov/documents/2023/11/01/2023-24283/safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence
- nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- openai.com/index/advancing-red-teaming-with-people-and-ai
- toloka.ai/blog/red-teaming
- scale.com/blog/hitl-ep7-red-teaming
- arxiv.org/abs/2202.03286
- arxiv.org/abs/2209.07858
- openai.com/index/red-teaming-network
- learn.microsoft.com/en-us/security/ai-red-team