Anupam Datta
John Mitchell
Ankur Taly


Michelle Bao
Ayush Singla


What is this course about?

Large Language Models (LLMs) and applications powered by them have recently received tremendous attention, especially since ChatGPT was released in November 2022. This course will provide an introduction to state-of-the-art methods and tools to make LLMs – models and applications – more trustworthy. The course will be organized into three modules: Part I will provide background on the emerging stack for LLMOps. Students will get a quick introduction to building LLM apps with LlamaIndex and work on a hands-on homework on evaluating a Retrieval-Augmented Generation question-answering app built with an LLM and a vector database. Part II will cover key application areas of LLMs, in particular, healthcare, education, and security. We will interleave presentations with brainstorming about project directions. Part III will cover state-of-the-art LLM (app) evaluation methods and tools. We will cover a sample of topics from relevance, groundedness, confidence, calibration, uncertainty, explainability, privacy, fairness, toxicity, adversarial attacks, and related topics. Students will gain understanding of a set of methods and tools for evaluating LLM applications. Students will complete one homework assignment to gain the necessary background. The bulk of the effort will be on a quarter long course project.


Students are expected to have the following background:

Honor Code

Permissive but strict. If unsure, please ask the course staff!

Audit policy

We’re generally open to auditing requests by all Stanford affiliates and external requests.

You will be able to attend all the lectures, but we won't be able to grade your homework or give advice on final projects because of limited resources.

Even if you’re not auditing, you can still access all the slides, notes, assignments, and final repot instructions. These are posted on the Syllabus page.

To audit the class, please send the TAs an email with the subject title "CS329T: Audit Request" with a few sentences introducing yourself and your relevant background.

Reference Text

There's no textbook. The course relies on lecture slides and accompanying readings.


Is this the first time the course is offered?
No. The course was also offered in Spring 2021 and Spring 2022. This year we are allocating more time to the term project since students last year enjoyed it.
What is the grading policy for the course?
Your grade will be based on homework (15%), project (75%) and class participation (10%). Homework and the project will involve significant programming. There are no exams.
Does the course count towards CS degrees?
Yes, this course can satisfy Area D: Computing and Society breadth requirement (for MSCS)
Will the videos be made available publicly?
No. We hope to make public course on this material but during this quarter, because the course is not an SCPD course, the videos will not be available.
Is attendance mandatory?
A portion of your grade will be based on your contributions to the class in these sessions.
What is the format of the class?
It will be lectures and guest speakers. We will often have industry experts to give us tutorials on fairness, ML explainability and privacy.
Do I need to know Python for the course?
Since Python has become the most popular language for machine learning, we expect most assignments will be in Python. Python fluency isn't required, but will make your life so much easier during the course.
Can I follow along from the outside?
We'd be happy if you join us! All the slides and lecture notes will be posted on this website.
Can I work in groups for the assignment?
No, the assignment has to be done and submitted individually. You can however discuss with your study group, although abiding by the Honor Code.
I have a question about the class. What is the best way to reach the course staff?
Please post your question in the Ed course forum so that other students can benefit from your questions. If you have a personal matter or emergencies, please email the TAs directly.