Preference Learning & Learning from Comparisons Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Essential papers on learning from human preferences and comparisons — from Christiano's foundational RLHF work to Direct Preference Optimization and modern alignment methods.

Preference Learning & Learning from Comparisons: 10 key papers

1. Direct Preference Optimization: Your Language Model is Secretly a Reward Model
   Rafailov et al. arXiv 2023.
2. A General Theoretical Paradigm to Understand Learning from Human Preferences
   Azar et al. arXiv 2023.
3. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm
   Fürnkranz et al. Machine Learning 2012.
4. B-Pref: Benchmarking Preference-Based Reinforcement Learning
   Lee et al. arXiv 2021.
5. PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
   Lee et al. arXiv 2021.
6. SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
   Park et al. arXiv 2022.
7. Models of human preference for learning reward functions
   Knox et al. arXiv 2022.
8. Statistical Rejection Sampling Improves Preference Optimization
   Liu et al. arXiv 2023.
9. KTO: Model Alignment as Prospect Theoretic Optimization
   Ethayarajh et al. arXiv 2024.
10. Nash Learning from Human Feedback
    Munos et al. arXiv 2023.

Related reading lists

• Reward Learning & Reward Modeling
• RL from Human Feedback (RLHF)
• Inverse Reinforcement Learning
• Human-AI Collaboration & Teaming
• Instruction Following & Finetuning
• Active Learning & Adaptive Sampling
• AI Alignment & Safety
• Hindsight Experience Replay & Relabeling