Human preference data has become crucial to the success of Machine Learning (ML) systems in many application domains, from personalization to post-training of language models. As ML systems are more and more widely deployed, understanding models, methods, and algorithms for learning from preference data becomes important for both scientists and practitioners. This course covers learning from preferences in supervised, active, and reinforcement/assistance settings, and covers aspects specific to preference data, such as preference heterogeneity and aggregation, interpretation of human feedback, and privacy. In coding tasks, students implement supervised reward modeling and assistance games. Prerequisites: Recommended CS 221 and CS 229.
If you are a CS PhD student at Stanford, this course is counted toward the breath requirement for "Learning and Modeling" or "Human and Society".
We will announce computing resources soon.
This class will be discussion-based, and requires attendance. After the first two weeks of class (10/6) until the end of substantive classes (11/17),
The current class schedule is below (subject to change).
| Date | Topic | Materials | Dates |
|---|---|---|---|
| 09/22 | Intro to Preference Modeling, and logistics | slides_00 📕Introduction | |
| 09/24 | Models of Preferences: BT, PL, Luce | slides_01 📕Chapter 1 | |
| 09/29 | Investigating Assumptions of the Models: Noise and Heterogeneity | slides_02 📕Chapter 1 | |
| 10/01 | Supervised Learning of Preferences via MLE: Direct Preference Optimization | slides_03 📕Chapter 2 | |
| 10/06 | Reinforcement Learning from Human Feedback: PPO and GRPO | slides_04 📕Chapter 2 | The attendance policy starts Coding 1 released |
| 10/08 | Representing Uncertainty in Learning: Gaussian Processes | slides_05 📕Chapter 2 | |
| 10/13 | Active Learning of Preferences | slides_06 📕Chapter 3 | quiz 1 |
| 10/15 | Performance Metric Elicitation and inverse reinforcement learning | slides_07 📕Chapter 3 | |
| 10/20 | Inverse reinforcement learning for robotics | slides_08 📕Chapter 3 | coding 1 due coding 2 released |
| 10/22 | Assistance Games, learning about the user | slides_09 📕Chapter 4 | |
| 10/27 | No Regret and Dueling Bandits | slides_10 📕Chapter 4 | quiz 2 pre-analysis plan due |
| 10/29 | Preferential Bayesian Optimization | slides_11 📕Chapter 4 | |
| 11/03 | Simulators, simulacra, and AI-as-a-judge | slides_12 📕Chapter 4 | coding 2 due coding 3 released |
| 11/05 | Social choice, median voters, and community notes | slides_13 📕Chapter 5 | |
| 11/10 | The inversion problem and privacy | slides_14 📕Chapter 6 | quiz 3 |
| 11/12 | New Preference Modalities, and reviewing your toolkit | slides_15 📕Conclusion | |
| 11/17 | Guest Lecture 1: Alignment | gslides_1 | |
| 11/19 | Guest Lecture 2: Model Behavior | gslides_2 | Attendance policy ends coding 3 due |
| 11/25 | Thanksgiving | ||
| 11/28 | Thanksgiving | ||
| 12/01 | Neurips | ||
| 12/03 | Neurips | project due |
