Stanford University · Autumn 2026

CS329H Machine Learning from Human Preferences

Mechanisms for optimizing systems that are used by humans. The technical and conceptual underpinnings sit across psychometrics, discrete choice, reinforcement learning, information retrieval and normative theory; this course puts them in one place.

Lectures: Mon & Wed, 3:00–4:20 PM PT
Room: Bishop Auditorium
Level: Graduate
First session: Wed 23 September 2026
Project due: Thu 3 December 2026

What the course covers Schedule Textbook

The course

One subject, assembled from six fields that rarely talk.

Machine learning from human preferences (MLHP) has been central to modern language modelling, but its foundations are scattered. This is a careful, practical introduction to its main methods and models.

MLHP provides mechanisms for optimizing systems that are used by humans, with applications in personalization, finetuning and assistance. The course covers choice modelling and its axiomatic approaches; imitation, and static and active learning of choice; reinforcement learning from human feedback; assistance games, bandits and recommendation; behavioural biases and style control; and aggregation and fairness.

This is a graduate-level course. By the end of it, you should be able to build, analyse and optimize systems that are used by humans.

Prerequisites

Machine learning — e.g. CS 221, CS 229, CS 230, CS 224N. CS 221 and CS 229 recommended.
Probability & statistics — e.g. CS 109 or equivalent.
Linear algebra & calculus — e.g. MATH 51, CME 100.
Proficiency in Python.

Teaching team

Who is teaching and how to reach us.

For all contact with the teaching team, open a thread on Ed. It reaches everyone at once and gets you an answer fastest.

Instructor

Sanmi Koyejo

Course staff

Sang T. Truong

Jenn J. Wang Justin Hartenstein

Guest lectures

Andy Haupt Erdem Bıyık Modibo Camara Alex Nam

Logistics

Where, when, and where to ask.

Lectures

Monday and Wednesday, 3:00–4:20 PM in Bishop Auditorium. All times on this page are Pacific Time.

Office hours

On Zoom. Times and the meeting link will be announced on Ed.

Contact

Open a thread on Ed for everything — questions about material, logistics, regrades, absences and accommodations. The Ed link will be posted before the first session.

Submissions

Gradescope, for the pre-analysis plan, the final manuscript, the code, and weekly attendance tickets.

Feedback

We are grateful for feedback at any point in the quarter. A form will be linked here and posted on Ed.

Assessment

Three ways to demonstrate mastery.

50%

Project

A pre-analysis plan due 4 November, then a final manuscript and code due 3 December. Individually, in pairs, or in triples.

30%

Written quizzes

Three 15-minute closed-book quizzes, 10% each. About eight multiple-choice and three short-response questions. Bring a pen. Anything covered in class may appear.

20%

Oral exams

Two 12-minute oral exams, 10% each. Scheduling is announced on Ed. Don't be more than three minutes late. Anything covered in class may appear.

Grades can also be lowered. Non-attendance, non-completion of the weekly AI interviews, and omitting any of the three required project disclosures all carry deductions — see Policies.

Schedule

Eighteen sessions, five units.

Subject to change. Links to slides and other materials will be added as the quarter runs. An AI interview is due every week.

#	Date	Session	Milestone
01	Wed 23 Sep	Course introduction & motivation
Foundation
02	Mon 28 Sep	Choice data, generalization, and Bradley–Terry
03	Wed 30 Sep	Random utility models and axiomatizations
04	Mon 5 Oct	Factor models	Absence registration due
Learning
05	Wed 7 Oct	Learning to predict choice; latent-variable estimation with MLE
06	Mon 12 Oct	Bayesian estimation and finetuning	Written quiz 1Attendance policy begins
07	Wed 14 Oct	Fisher information & optimal experimental design
08	Mon 19 Oct	Reinforcement learning from human feedback
09	Wed 21 Oct	Direct preference optimization
10	Mon 26 Oct	Active elicitation for LLMs, and case studies
Action
11	Wed 28 Oct	Assistance games; assisted vs. contextual bandits
12	Mon 2 Nov	Acquisition functions & Thompson sampling	Written quiz 2
13	Wed 4 Nov	Dueling bandits & preferential Bayesian optimization	Pre-analysis plan due
Inversion
14	Mon 9 Nov	Style control, surrogates
15	Wed 11 Nov	Bounded rationality; computationally tractable choice
Aggregation
16	Mon 16 Nov	Impossibilities: Gibbard–Satterthwaite and Arrow
17	Wed 18 Nov	Fairness and information aggregation	Written quiz 3
	23–25 Nov	Thanksgiving — no class
18	Mon 30 Nov	Summary and practical tips	Last day of attendance policy
	Wed 2 Dec	No class — NeurIPS	Project due 3 Dec

Project

Make it yours. Preferences are contextual, and you have a lot of context.

Half the grade. Work individually, in pairs, or in triples — triples submit a contribution statement naming what each member did. Your project should show understanding of course concepts applied to a question you actually care about.

01 · 30% of project

Pre-analysis plan

Due 4 November

Everything except the experiments, so you get feedback before the expensive work.

Overview & motivation (6%)
Literature review, ≥ 3 papers (6%)
Proposed methods & analysis (6%)
Timeline & responsibilities (2%)
Formatting (10%) — NeurIPS style, 3–4 pages excluding references

02 · 60% of project

Final manuscript

Due 3 December

The paper. Honest about what worked and what didn't.

Introduction & background (10%)
Methods (15%)
Results & discussion (20%)
Conclusion & future work (5%)
Formatting (10%) — NeurIPS style, max 8 pages excluding references

03 · 10% of project

Code

Due 3 December

Research-quality, not production-quality — but someone else has to be able to run it.

Documentation (5%) — README, docstrings, pinned dependencies, fixed seeds
Functionality (5%) — reproduces every figure and table in the paper

Possible directions

Novel applications of preference learning to new domains.
Comparative studies of different choice or preference learning methods.
Investigations of preference heterogeneity or aggregation.
Active learning strategies for preference data.
RLHF variations or improvements.
Assistance games in specific contexts.
Preference modelling with limited data.
Inversion problems and bias corrections.

How we evaluate the code

Can we set up the environment from your instructions? Can we reproduce your main results within a reasonable tolerance? Is the code organized and understandable? Are the methods clearly implemented and documented? Mark anything adapted from an external source, include its licence, and explain your modifications.

Timeline

Weeks 1–2 — abstract brainstorming exercises, to iterate through ideas quickly.
Throughout — office hours for discussing topics and getting feedback.
4 November — pre-analysis plan due.
November – 3 December — run the experiments and finalize the analysis.
3 December — final manuscript and code due.

Advice

Start early, especially on data collection or generation. Use course concepts explicitly — it is how you demonstrate learning. Treat the pre-analysis plan as your chance to get feedback before heavy experimental work. Make the code reproducible from day one. Don't overscope: one thing done well beats three done poorly.

Submit via Gradescope as a PDF, with a .zip of the code attached and a link to your repository. Exceptions are possible for sensitive data and code — discuss it with course staff on Ed.

Resources & policies

Policies

Attendance

This class is discussion-based and requires attendance. Attendance is required at every session between 12 October and 30 November. It does not apply to CGOE/HCP students.

From the third unexcused absence, a half-grade is deducted from the final grade.
From the fifth, a full grade.
From the eighth, two full grades.
From the tenth, the class is graded as failed.

Excused absences are granted for religious observance, health reasons, family emergencies, and conferences at which you are presenting. Register absences for work travel by end of day 5 October; the teaching team confirms excusals by 12 October and posts them on Ed. After that, only illness, disability-related absence, family or personal emergency, and religious observance are excusable — write to course staff on Ed as soon as you can.

Attendance tickets

Students who attend a lecture receive a ticket with a unique code.
Submit the code to Gradescope in a file named ticket.txt to receive credit.
Tickets are graded automatically.
If two students submit the same code, neither receives credit — even if Gradescope initially assigns it.
You have until the end of the current week to submit that week's tickets.

AI interviews

To record your learning, you complete a weekly 15-minute AI exam. You are given a class identifier on the first day. All AI interviews are due Sunday, end of day, and there are no late days.

From the third non-completion, a half-grade is deducted from the final grade.
From the fifth, a full grade.

Use of AI

If you are struggling and feeling too much pressure in this course, please don't resort to chatbots as a shortcut to completing the project. Many Stanford students feel stressed and pressured; it is completely natural, as this is a challenging course and the university is a high-pressure environment. There are a lot of support resources available to you, and we believe you can succeed here. Contact the instructors any time and let's talk about it — we're open to extending due dates or adjusting assignments to fit your situation.

We will not use generative AI in grading, but might use it to give you feedback on submissions.

For the project you may use any generative AI tool, but you take responsibility for meeting scientific standards. Three disclosures are required with your submission:

Integrity statement (150 words) — how you identified and addressed plagiarism (of text and of ideas), bias, and inaccuracies.
Reflection (150 words) — how and why you used AI tools, their impact on your learning, and how you or we might use them in future.
Impact statement (150 words) — potential social, ethical or environmental impacts; how the methods, data or findings might be misused and what mitigates that; how you ensured responsible attribution, data handling and compliance; and, if the work involves human participants, sensitive data or societal applications, the ethical considerations that guided your approach.

Omitting any of the three disclosures costs 10% of the course grade.

Collaboration

Study groups are allowed, but each student submits their own work and lists their group members.

Submitting coursework

Submit the pre-analysis plan and the project via Gradescope by the deadline. You have 3 late days in total across the two. If you think we made a grading mistake, submit a regrade request on Gradescope within 3 days of receiving the grade.

Academic integrity

Produce your own work and give proper credit when referencing the ideas, words, data or code of others. Plagiarism, fabrication, unauthorized collaboration, or any other violation of the Honor Code is treated as a serious offence and reported accordingly. Follow standard scholarly conventions for citation. See the Stanford Honor Code and the guidance on what counts as plagiarism. Questions go on Ed.

Access and accommodations

Stanford is committed to providing equal educational opportunities for disabled students. Disabled students are a valued and essential part of the Stanford community, and you are welcome in our class. If you experience disability, please register with the Office of Accessible Education; professional staff will evaluate your needs, support appropriate and reasonable accommodations, and prepare an Academic Accommodation Letter for faculty. To get started or re-initiate services, visit oae.stanford.edu. If you already have a letter, share it with us at the earliest opportunity so we can partner with you and the OAE to identify any barriers to access in this course.

Recording

Video cameras at the back of the room capture the instructor presentations; recordings are available through the course Canvas site. These recordings might be reused in other Stanford courses, viewed by other Stanford students, faculty or staff, or used for other education and research purposes. The cameras are positioned to record only the instructor, but part of your image or voice might be incidentally captured. Questions go to a member of the teaching team.

Your wellbeing

Your wellbeing is a vital foundation for your learning and success in this course. If you are experiencing personal, academic or health challenges, reach out early — we can help connect you with resources.

Confidential mental health support: Counseling and Psychological Services (CAPS).
Primary care, medical services and wellness programs: Vaden Health Center.

If you are struggling, you are not alone. Please prioritize your wellbeing and don't hesitate to ask for help.

CGOE / HCP students

Different for you: the mandatory attendance policy does not apply. Written quizzes are administered by your registered CGOE/HCP exam monitor; you have a 24-hour window from the start of the on-campus quiz in which to complete it using the allotted exam time. You may also sit the exam at the on-campus seating — reach out to class staff on Ed if so, and note that per CGOE/HCP guidelines you still need a registered exam monitor even then. Graded exams are returned electronically via Gradescope.

Identical for you: the project and the pre-analysis plan, the oral (Zoom) and AI (internet-based) exams, and office hours.

Auditors

We admit auditors depending on capacity. You need the instructor's approval to participate in classes and to be granted access to class materials. Auditors do not participate in the attendance system.