Autumn 2026 ✦ Stanford University
MS&E 319: Efficient Generative Language Models
Given a modeling goal and a limited computational budget, how should we choose the training objective, model architecture, and inference algorithm?
- Meets Tuesdays and Thursdays, 3:00 – 4:20 PM;
Building 200, Room 2New location: STLC 111, Sapp Center for Science Teaching and Learning, 376 Lomita Drive. - Prerequisites CS 224N, CS 324, or equivalent preparation is recommended. We will assume familiarity with deep learning, gradient-based optimization, the analysis of algorithms, and basic mathematical proofs. Mathematical and algorithmic maturity at the level of CS 161 or CS 261 is helpful.
- Instructors Amin Karbasi, Anay Mehrotra, Amin Saberi, and Grigoris Velegkas
- Note May be repeated for credit.
Overview
Modern language models owe their capabilities to scale and to careful design. This course is about the second part: given a limited computational budget, how should we choose the training objective, the architecture, and the inference method?
We study algorithmic innovations that make generative language models more efficient, such as mixtures of experts, sparse attention, quantization, and speculative decoding. We will examine how these ideas work and what they cost. The emphasis will be on understanding them deeply and identifying the research questions they open.
Who Is This For?
This course is intended for students interested in research on efficient generative modeling. Students should be comfortable with mathematical and algorithmic reasoning, and should want to understand why these models are built the way they are, and not only how to use them.
Tentative Lecture Outline
Topics and their order may change as the course develops. Assignments may explore additional ideas that build on and extend the lectures.
Course Overview
Overview of the Course and LLMs
Innovations in Pre-training
A Review of LLM Architecture and Training
Lecture contents
- Review of LLM architectures: then and now
- Review of training objectives, causal masking, and optimizers
- Scaling laws
Relevant readings
- Attention architecture
- Vaswani et al., Attention Is All You Need (NeurIPS 2017).
- Scaling laws
- Kaplan et al., Scaling Laws for Neural Language Models (arXiv, 2020).
- Hoffmann et al., Training Compute-Optimal Large Language Models (NeurIPS 2022).
- Porian et al., Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024).
Making Compute Go Further
Lecture contents
- Multi-token prediction
- Matrix-aware optimization methods (e.g., Muon)
Relevant readings
- Multi-token prediction
- Gloeckle et al., Better & Faster Large Language Models via Multi-token Prediction (ICML 2024).
- Optional background: Caruana, Multitask Learning (Machine Learning, 1997).
- Muon
- Jordan et al., Muon: An optimizer for hidden layers in neural networks (blog post, 2024).
Innovations in Model Architecture
Mixtures of Experts
Lecture contents
- Sparse expert routing: classic MoE, GShard, and Switch Transformers
- Expert specialization: shared and fine-grained experts
- Load balancing: auxiliary losses and auxiliary-loss-free methods
Relevant readings
- MoE foundations and routing
- Jacobs et al., Adaptive Mixtures of Local Experts (Neural Computation, 1991).
- Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (ICLR 2017).
- Fedus, Zoph, and Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (Journal of Machine Learning Research, 2022).
- Expert specialization (advanced)
- Dai et al., DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (ACL 2024).
Efficient Attention: Sparse, Linear, and IO-Aware Methods
Lecture contents
- Sparse attention: choosing what to attend to
- Linear attention: summarizing the past through recurrent updates
- Faster exact attention: FlashAttention
Compressing the KV Cache
Lecture contents
- Multi-head attention and the KV-cache bottleneck
- Multi-head latent attention: architectural design for KV-cache compression
Innovations in Efficient Inference
Efficiency via Quantization
Lecture contents
- Quantizing pretrained models
- Continued pre-training of quantized models
- KV-cache quantization
Speculative Decoding and Learned KV-Cache Compression
Lecture contents
- Speculative decoding and learned draft models
- Learned KV-cache compression (e.g., Cartridges)
Innovations in Post-training
Post-training I
Lecture contents
- The post-training pipeline and supervised fine-tuning
- Parameter-efficient fine-tuning: low-rank adaptation (LoRA)
Post-training II
Lecture contents
- RLHF: learning a reward model from pairwise preferences
- DPO: learning from preferences without a separate reward model
Distillation and Recursive Self-Improvement
Lecture contents
- Distillation and self-distillation
- Recursive self-improvement: possibilities and limits
Risks and Vulnerabilities
Risks and Vulnerabilities of Generative AI
Lecture contents
- Jailbreaking models: GCG, PAIR, TAP, Adversarial Reasoning
- Stealing part of a production language model through API access
- Text watermarking
Looking Ahead
Where Do We Go from Here?
Evaluation
Please read this section carefully and note the assignment and examination dates.
| Assessment | Due Date / Exam Time | Week | Weight |
|---|---|---|---|
| Assignment 1 (Written) | Friday, October 2, 2026 | Week 2 | 10% |
| Assignment 2 (Written) | Friday, October 23, 2026 | Week 5 | 10% |
| Midterm Examination | Thursday, November 5, 2026, 3:00 – 4:20 PM | Week 7 | 70% |
| Assignment 3 (Programming) | Friday, December 4, 2026 | Week 10 | 10% |
Assignments
Assignment 1 and Assignment 2 will focus on the mathematical and algorithmic ideas developed in class. They may ask students to analyze existing techniques, prove theoretical claims, or develop new variants of algorithms discussed in the lectures.
Assignment 3 will ask students to implement and investigate one or more methods for making language models efficient. Students should expect to explain their design choices, evaluate the method carefully, and interpret their results.
Assignment submission instructions and policies concerning late work will be posted on Canvas.
Midterm Examination
There will be one midterm examination during the regular class period on Thursday, November 5, 2026, from 3:00 to 4:20 PM.
The midterm will be closed book, completed on paper, and taken without the use of computers. It will assess conceptual understanding, mathematical reasoning, and the ability to analyze and compare the tradeoffs behind different methods. Students will be responsible for material from lectures, assigned readings, Assignment 1, and Assignment 2.
Logistics
Course Materials and Communication
Course materials, readings, lecture notes, assignments, and announcements will be posted on Canvas. Ed Discussion will be used for questions and class discussion.
All questions for the instructors must be posted on Ed Discussion. Please do not email the instructors individually. If your question is personal or concerns information you do not wish to share with the class, make the Ed post private so that only the instructors can see it.
There is no required textbook. The course will rely primarily on lecture notes and recent research papers.
Attendance and Participation
Regular attendance is strongly encouraged. The course emphasizes the reasoning behind recent algorithmic ideas, and lectures will provide context, derivations, and critical discussion that may not appear in the assigned readings. Students are expected to arrive prepared to engage with the material and participate in class discussions.
Office Hours
Instructor office hours are held on Thursdays from 4:30 to 5:30 PM in CoDA E446. Course assistant office hours will be announced.
Policies
Collaboration Policy
Students may discuss general ideas and problem-solving strategies with classmates, but every student must work through the arguments independently and write their own solutions. Written solutions should reflect the student’s own understanding and should not be copied, jointly composed, or adapted from another person’s work.
The collaboration rules for Assignment 3 will be stated in the assignment. Unless group work is explicitly authorized, submitted code, experiments, analysis, and written explanations must be the student’s own.
Students must acknowledge collaborators and any external resources that materially influenced their work.
Use of Generative AI
Generative AI tools may be useful for exploring ideas, working through mathematics, and writing code, but students remain responsible for everything they submit. Any substantive use of such tools must be disclosed.
A generated argument is not a substitute for understanding it. Students must verify all generated arguments and calculations themselves, and may be asked to explain or defend any part of their submitted work. Generative AI tools may not be used during the midterm. More restrictive rules may be announced for individual assignments.
OAE Accommodations
Stanford is committed to providing equal educational opportunities for students with disabilities. Students who may need academic accommodation should register with the Office of Accessible Education and obtain an accommodation letter. Please contact the teaching team privately and as early as possible so that appropriate arrangements can be made.
Course Staff
Instructors
Course Assistant
Senem Işık
Ph.D. Student in Management Science and Engineering, Stanford University