RL from Human Feedback (RLHF) Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Training language models and agents from human preferences instead of hand-designed reward functions.

RL from Human Feedback (RLHF): 10 key papers

1. Deep reinforcement learning from human preferences
   Christiano et al. arXiv 2017.
2. Learning to summarize from human feedback
   Stiennon et al. arXiv 2020.
3. Fine-Tuning Language Models from Human Preferences
   Ziegler et al. arXiv 2019.
4. Constitutional AI: Harmlessness from AI Feedback
   Bai et al. arXiv 2022.
5. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
   Bai et al. arXiv 2022.
6. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
   Casper et al. arXiv 2023.
7. Secrets of RLHF in Large Language Models Part I: PPO
   Zheng et al. arXiv 2023.
8. Scaling Laws for Reward Model Overoptimization
   Gao et al. arXiv 2022.
9. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
   Lee et al. arXiv 2023.
10. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
    Wu et al. arXiv 2023.

Related reading lists

• Reward Learning & Reward Modeling
• Preference Learning & Learning from Comparisons
• Instruction Following & Finetuning
• Scaling Laws and Emergent Capabilities
• Inverse Reinforcement Learning
• Code Generation & Program Synthesis with RL
• Human-AI Collaboration & Teaming
• Reasoning in LLMs