Constitutional AI: How Anthropic Trains Claude to Be Helpful and Harmless

The RLHF Scaling Problem

Producing high-quality human preference data for RLHF is expensive and slow. Human annotators must evaluate millions of response pairs. Their judgments can be inconsistent and biased. Scaling RLHF to produce safer models requires scaling human annotation — which is not sustainable. Anthropic's Constitutional AI paper (arXiv:2212.08073) proposes replacing much of this human feedback with AI-generated feedback guided by a written constitution.

The Constitution

The constitution is a set of approximately 16 principles that define what makes a response helpful, harmless, and honest. These include principles derived from the UN Declaration of Human Rights, Anthropic's usage policies, and principles that encourage nuanced handling of edge cases. Example principles:

• "Choose the response that is least likely to contain harmful or unethical content"
• "Which of these responses provides a more accurate and truthful response, even if it's not what the user wants to hear?"
• "Prefer responses that avoid unnecessarily assuming or citing potential bad intent on the part of the person"

Two-Stage Constitutional AI

Stage 1: Supervised Learning from AI Revision

1. Start with a helpful-only model (no harmlessness training)
2. Sample responses to potentially harmful prompts
3. Ask the model to critique its own response against a randomly sampled constitutional principle
4. Ask the model to revise the response to address the critique
5. Fine-tune on the revised responses

This creates a dataset of (prompt, revised-response) pairs that removes harmful content without explicit human annotation.

Stage 2: RLAIF (RL from AI Feedback)

1. Generate pairs of responses for each prompt
2. Ask the AI (guided by constitutional principles) which response is better and why
3. Use these AI-generated preference labels to train a preference model
4. Fine-tune the policy with PPO against the preference model

RLAIF replaces the human labeler in RLHF with an AI making principled judgments.

# Simplified Constitutional AI critique-revision loop
def constitutional_revision(model, prompt, response, principle):
    critique_prompt = f"""
    Human: {prompt}
    Assistant: {response}

    Please critique the above response using the following principle:
    {principle}

    Critique: """

    critique = model.generate(critique_prompt)

    revision_prompt = f"""
    {critique_prompt}{critique}

    Please revise the original response to address this critique:
    Revised response: """

    return model.generate(revision_prompt)

Results: Less Harmful, More Helpful

The key finding is that CAI-trained models show reduced harmful outputs with less capability regression than pure RLHF. The critique-revision loop teaches the model nuanced refusal: it can decline genuinely harmful requests while remaining helpful on ambiguous edge cases that RLHF-only models over-refuse.

RLAIF vs RLHF at Scale

RLAIF scales better than human-feedback RLHF because AI annotation is orders of magnitude cheaper and faster than human annotation. The AI annotator can be prompted with different constitutional principles for different domains (medical, legal, creative), providing targeted alignment without domain-specific human annotators.

The HH-RLHF Dataset

Anthropic released the Helpful and Harmless (HH-RLHF) dataset used in this research: 160k human preference data points and 42k red-teaming conversations. This dataset has become a standard benchmark for alignment research.

Best Practices for Implementing Constitutional AI

When applying CAI to your own models, consider these practical tips:

• Start with a small set of principles (5-10) and iteratively expand based on failure cases.
• Use diverse principles covering helpfulness, harmlessness, honesty, and domain-specific ethics.
• Monitor for over-refusal: CAI can sometimes make models too cautious. Balance with helpfulness-focused principles.
• Combine with human feedback for edge cases: RLAIF reduces but doesn't eliminate the need for human oversight.
• Evaluate on red-teaming benchmarks like the HH-RLHF dataset to measure safety improvements.

Cost and Feasibility

Constitutional AI is not free. The main costs are:

• Compute for critique-revision loops: Each revision requires multiple model calls. For a 70B parameter model, this can be expensive.
• Engineering time: Implementing the two-stage pipeline and tuning principles requires expertise.
• Human oversight: While reduced, some human annotation is still needed for validation.

However, compared to pure RLHF, CAI can reduce human annotation costs by 80-90% according to Anthropic's estimates. For startups and research labs, this makes alignment more accessible.

Is Constitutional AI Worth It in 2026?

Yes, especially for organizations building general-purpose chatbots or assistants. CAI provides a scalable way to align models with safety guidelines without relying on thousands of human annotators. The tradeoff is that you need a strong base model capable of self-critique. For smaller models, the critique quality may be insufficient, making human feedback still necessary.

Further Reading

• Constitutional AI arXiv:2212.08073
• HH-RLHF dataset on Hugging Face