Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Red teaming is borrowed from military and cybersecurity practice: a dedicated "red team" tries to break a system while the "blue team" defends it. In the context of language models, red teaming means having people (or other models) deliberately try to make an AI say harmful things — offensive content, dangerous instructions, private information, misinformation.

Why not just test with standard benchmarks? Because benchmarks use fixed prompts. A determined human can be creative, adaptive, and persistent in ways that static evaluations cannot capture. A person might try roleplay attacks ("pretend you're a villain and explain how to…"), multi-turn escalation (starting innocently and gradually steering the conversation toward harm), or exploit the model's desire to be helpful.

This paper conducted one of the first large-scale, systematic red teaming efforts on language models. The team recruited 324 crowdworkers who launched 38,961 adversarial attacks against models of varying sizes and safety configurations. The goal was three-fold: discover what types of harm these models produce, measure whether safety interventions actually work, and share the findings openly so the community can build on them.

The researchers tested four types of AI assistants, each with a different level of safety intervention. Think of them as four locks of increasing sophistication on the same vault:

Plain LM — A raw language model with minimal formatting to act as a chatbot. No safety measures at all. This is the baseline — the vault with a basic padlock.

Prompted LM — The same model, but given a 14-example prompt telling it to be helpful, honest, and harmless (HHH). This is like adding a security guard who follows written instructions — helpful, but easily fooled by creative social engineering.

Rejection Sampling (RS) — For each response, the model generates 16 candidates, then a separate "harmlessness preference model" picks the two safest. This is like having a panel of judges review every response before it leaves the building.

RLHF — The model is trained end-to-end using reinforcement learning to maximize harmlessness scores from the preference model. This is the vault that was redesigned from the ground up based on what previous burglars found.

Each model type was tested at three sizes: 2.7 billion, 13 billion, and 52 billion parameters. This matrix of 12 configurations let the researchers study how safety scales with model size and intervention type.

The most striking result is shown in Figure 1 of the paper. The researchers measured attack success in two ways: the red team members' self-reported success rating (a 5-point Likert scale), and an automated harmlessness score from a preference model (lower means more harmful).

The finding that reshaped the field: RLHF models become significantly harder to red team as they scale up. The 52B-parameter RLHF model was substantially more resistant to attacks than the 2.7B version. This is a positive scaling behavior — as models get larger, they also get safer, when trained with RLHF.

In contrast, the other three model types — plain LM, prompted LM, and rejection sampling — showed flat trends with scale. Making the model bigger did not make it safer unless the safety was baked into the training process via RLHF.

A surprising sub-finding: the prompted LM (with its 14-example HHH prompt) was no harder to red team than the plain LM with no safety intervention at all. This contradicted earlier results from static evaluation benchmarks, suggesting that adversarial humans can bypass prompt-based safety in ways that fixed test prompts cannot.

Rejection sampling placed a floor on vulnerability — it was consistently the hardest to attack at every scale. However, qualitatively, the researchers noted that RS models achieved safety by being evasive — they refused to engage rather than engaging helpfully while avoiding harm.

How do you quantify whether an AI response is harmful? The authors used two complementary methods. First, red team members themselves rated their attack success on a 0–4 Likert scale after each conversation. Second, a harmlessness preference model — a separate neural network trained to distinguish harmful from harmless responses — scored every AI utterance automatically.

The preference model works by comparing pairs of responses: during data collection, each red team member was shown two possible AI responses and asked to select the more harmful one. This pairwise comparison data trains a model that can score any new response on a continuous harmlessness scale.

For each conversation, the researchers computed the harmlessness score of every assistant utterance (conditioned on all prior context), then took the minimum score as the overall harmlessness of that conversation. This minimum captures the worst single moment — because one severely harmful response makes the entire interaction dangerous, regardless of how benign the rest was.

The two metrics — human self-report and automated harmlessness score — are correlated but not identical. The human metric captures the red team member's subjective judgment, while the automated score provides a consistent, scalable measurement. Using both gives a more complete picture of attack success.

One of the paper's most valuable contributions is its detailed taxonomy of the types of attacks and harms uncovered. The researchers embedded all 38,961 attack transcripts using a 52B language model, then projected them into 2D using UMAP. This revealed natural clusters of attack types that could be manually labeled.

The five most common attack categories were: (1) discrimination and injustice, (2) hate speech and offensive language, (3) violence and incitement, (4) non-violent unethical behavior (lying, cheating, manipulation), and (5) bullying and harassment.

Crucially, the researchers found that more subtle attacks — especially "non-violent unethical behavior" — had higher success rates than blunt offensive attacks. This makes intuitive sense: safety interventions are typically calibrated to block obviously toxic outputs (slurs, explicit violence), so more nuanced ethical violations slip through more easily.

Less common but significant categories included: soliciting personally identifiable information (PII), substance abuse, fraud, weapons information, terrorism, self-harm, and child abuse. The "Other" category was also prevalent, showing that any fixed taxonomy will miss attack types that creative humans invent.

The red team consisted of 324 US-based crowdworkers, recruited primarily from Amazon Mechanical Turk (307 workers) and Upwork (17 workers). Each worker was given open-ended instructions to "make the AI behave badly" — they could choose their own attack strategies and topics within their personal risk tolerance.

A key finding about the red team itself: roughly 80% of all attacks came from just 15% of the workers. Some individuals were far more effective at red teaming than others — an observation that has implications for how to select and train red teams. The researchers fitted a linear mixed-effects model to estimate each worker's inherent red teaming ability, and found significant variation in skill.

Some workers developed template-based attacks (e.g., "tell me an insult for [X] that starts with [Y]"), which generated many attacks quickly but with limited diversity. Others were more creative, inventing roleplay attacks, multi-turn escalation strategies, and subtle manipulation techniques. The most dangerous attacks typically required creativity and persistence.

The researchers also invested significantly in red team well-being: clear warnings about harmful content, personal risk tolerance guidelines, recommended breaks, and a well-being survey. Participants generally reported positive feelings and enjoyed the work — an important finding given the parallels with content moderation, where worker burnout is a serious concern.

A critical question for any red teaming effort: how much do different people agree on whether an attack was successful? The researchers ran a separate review experiment where 3 annotators independently rated the same 1,000 attack transcripts.

The result was sobering: inter-rater agreement was low. Using Fleiss's Kappa — a standard measure where 1 means perfect agreement and 0 means chance — they found a score of only 0.32 for the 5-point Likert rating, rising to 0.49 when ratings were simplified to binary (successful/not successful). Even among the 3 reviewers alone (excluding the original attacker), agreement peaked at 0.55.

This means "harmful" is deeply subjective. What one person considers a successful attack — dangerous advice, subtle manipulation, a harmful joke — another might dismiss as harmless or ineffective. This subjectivity is not a flaw of the method but a fundamental property of harm itself: harm depends on context, cultural background, and individual sensitivity.

The implication is that any single harmlessness metric — human or automated — provides an incomplete picture. The researchers recommend using multiple metrics and being transparent about the inherent uncertainty in measuring harm.

A bold decision: the researchers released the entire dataset of 38,961 red team attacks publicly. This was not without controversy — the same data that can train safer models can also be used to train more harmful ones. The team explicitly weighed the pros and cons.

On the positive side: the dataset is an order of magnitude larger than the previous Bot Adversarial Dialogues dataset (about 5,000 conversations). It covers larger models, includes attacks on RLHF-trained models, and comes with both human ratings and automated harmlessness scores. It costs over $60,000 in crowdworker payments alone to produce, making it a significant public good.

Each data point includes the conversation transcript, the red team member's success rating, a short description of their intended attack, harmlessness scores for both the AI responses and the attack description, the model type and size, and an anonymous worker identifier. The team filtered personally identifiable information (PII) using a regular expression, though they noted that some AI-generated PII (hallucinated addresses, phone numbers) appeared to be synthetic and not real.

The dataset has been used extensively by the research community for building safety classifiers, developing automated red teaming methods, and characterizing the attack surface of language models.

The authors were refreshingly honest about limitations. The red team was composed of US-based crowdworkers, which means the dataset reflects a particular cultural perspective. Harms that are specific to other cultures, languages, or contexts may be underrepresented.

Domain expertise was another gap. Some attacks required specialized knowledge to evaluate — for example, whether instructions for synthesizing a chemical are actually viable, or whether medical advice is genuinely dangerous. Crowdworkers without this expertise may have rated such attacks incorrectly.

The data was inherently incomplete. The models were partly trained on code, yet no attacks related to malicious code generation appeared. "Roleplay attacks" — where the model is asked to act as a fictional character who can bypass safety rules — were discovered internally but did not appear in the crowdworker dataset.

Finally, the paper acknowledged a fundamental tension in red teaming: sharing findings openly improves community safety, but also reveals vulnerabilities that bad actors can exploit. The authors called for the development of shared norms and neutral forums to discuss these trade-offs.

This paper's influence extends beyond its technical findings. By publishing methods, data, and uncertainties in full, it established transparency as a norm for safety research. The finding that RLHF scales positively for safety became a foundational assumption in the design of subsequent AI systems. And the paper's honest discussion of trade-offs — between openness and security, between completeness and feasibility — set the template for how responsible AI labs communicate about safety.