CS229T/STATS231: Statistical Learning Theory
Stanford / Autumn 2018-2019
- 9/8: Welcome to CS229T/STATS231! Previous years' home pages are here and here for reference. (Currently this page is still under construction.)
- Yu Bai (head CA)
- Tum Chaturapruek
- Jim Zhiyuan Li
- Luigi Nardi
- Colin Wei
Office hours TBD
Please use Piazza
for questions and discussions.
When do machine learning algorithms work and why? How do we formalize what it means for an algorithm to learn from data? How do we use mathematical thinking to design better machine learning methods?
This course focuses on developing a theoretical
understanding of the statistical properties of learning algorithms.
- Uniform convergence (VC dimension, Rademacher complexity, etc)
- Implicit/algorithmic regularization, generalization theory for neural networks
- Kernel methods
- Online learning and bandits problems
- Unsupervised learning: exponential family, method of moments, statistical theory of GANs
- A solid background in
and general ability to do mathematical proofs
- Machine learning (CS229) or statistics (STATS315A)
- Convex optimization (EE364A) is recommended
- Homeworks (40%): there will be three homeworks (plus a warmup which does not count towards your grade),
centered around proving properties of statistical procedures.
Each homework must be submitted through Gradescope. Sign up for the course using entry code M4V34N.
You are encouraged to use LaTeX to typeset your homeworks;
we've provided a template for your convenience.
- Exam (25%): open-book, open-notes.
Problems will be like the homeworks, but simpler.
You can use laptops as long as you turn off the wireless.
Date: Wed Nov 14, 6-10 PM (subject to change)
- Paper review (30%):
you will write a 2-4 page review of papers. The goal is to learn to
read technically demanding papers critically, and hopefully in the process, generate
novel research ideas. Your review should not only summarize the main result of the paper,
but critique it, instantiate it on examples, discuss its overall significance,
and suggest possible future directions.
See this Google doc
for detailed guidelines and a list of papers.
Each review should be submitted electronically by 11pm.
Instead of doing the paper review, with approval from the course
staff on the project topic, you can do a final project.
The project can be done in pairs.
- Scribe notes (5%):
Because there is no textbook or set of readings that perfectly fits this course, you will be asked to scribe a note for a lecture in LaTeX. The course staff will select one note for each lecture and share it with other students. See this Google doc for the detailed guidelines. The scribe notes are due 2 days after the lecture (11pm Wed for Mon lecture, and Fri 11pm for Wed lecture). Please sign up here before Sept 29th and plan the time ahead. Extra credits will be given to the notes that are selected for posting.
The scribe notes can be done in pairs.
By default, no late work is accepted. Under extentuating circumstances,
you may request an extension by contacting the course staff.
we encourage you to form study groups and
discuss homeworks. However, you must write up all homeworks from scratch independently without referring to any notes from the joint session.
Please follow the honor code
There is no required text for the course. A number of useful references:
(subject to change)
- Mon 09/24: Lecture 1: overview, formulation of prediction problems, error decomposition
- Wed 09/26: Lecture 2: asymptotics of maximum likelihood estimators (MLE)
- Mon 10/01: Lecture 3: uniform convergence, overview
- Mon 10/01: Homework 0 (warmup) due
- Mon 10/01: Homework 1 out
- Wed 10/03: Lecture 4: uniform convergence, finite hypothesis class
- Mon 10/08: Lecture 5
- Wed 10/11: Lecture 6
- Wed 10/11: Homework 1 due
- Wed 10/11: Homework 2 out
- Mon 10/15: Lecture 7
- Wed 10/17: Lecture 8
- Mon 10/22: Lecture 9
- Wed 10/24: Lecture 10
- Mon 10/29: Lecture 11
- Wed 10/31: Lecture 12
- Wed 10/31: Homework 2 (uniform convergence) due
- Wed 10/31: Homework 3 out
- Mon 11/05: Lecture 13
- Wed 11/07: Lecture 14
- Mon 11/12: Lecture 15
- Wed 11/14: Lecture 16
- Wed 11/14: Exam (6-10pm, location TBD)
- Mon 11/26: Lecture 17
- Wed 11/28: Lecture 18
- Wed 11/28: Homework 3 due
- Mon 12/03: Lecture 19
- Wed 12/05: Lecture 20
- Sun 12/09: Paper review due
- Sun 12/09: Final project due (if you didn't do the paper review)