Introduction to Optimization Theory
MS&E213 / CS269O - Fall 2020
Aaron Sidford (sidford@stanford.edu)
Welcome
This page has informatoin and lecture notes from the course "Introduction to Optimization Theory" (MS&E213 / CS 269O) which I taught in Fall 2020. I hope you enjoy the content as much as I enjoyed teaching the class and if you have questions or feedback on the note, feel free to email me.
Course Overview
This class will introduce the theoretical foundations of continuous optimization. Starting from first principles we show how to design and analyze simple iterative methods for efficiently solving broad classes of optimization problems. The focus of the course will be on achieving provable convergence rates for solving large-scale problems.
Email: sidford@stanford.edu
Lecture Notes
Here are the links for the course lecture notes. Feedback is welcome and if there is anything that is unclear or you would like added, please feel free to contact me. My apologies for missing citations and references; these may be added over time in future course offerings
- Chapter 1: Introduction: the notes for this chapter are here.
- Chapter 2: Smoothness: the notes for this chapter are here.
- Chapter 3: Convexity: the notes for this chapter are here.
- Chapter 4: Acceleration: the notes for this chapter are here.
- Chapter 5: Smooth Extensions: the notes for this chapter are here.
- Chapter 6: Non-smooth Convex Functions: the notes for this chapter are here.
- Chapter 7: Subgradient / Mirror Descent: the notes for this chapter are here.
- Bonus Chapter 1: Cutting Plane Methods: the notes for this chapter are here.
- Bonus Chapter 2: Interior Point Methods: the notes for this chapter are here.
- Appendix A: Norms: last year's notes for this chapter are here.
Lecture Schedule
Here is the schedule of material for the course.
Week 1
- Lecture #1 (Tu 9/15): intro: course overview: oracles, efficiency, and optimization impossibility (slides)
- Lecture #2 (Th 9/17): intro: example problem / algorithm: critical points by gradient descent (slides)
- Reading: Start Chapter 1 and Chapter 2
Week 2
- Lecture #3 (Tu 9/22): intro: Lipschitz functions and global minimization (slides)
- Lecture #4 (Th 9/24): intro: smooth functions and critical points revisited (slides)
- Problem Set #1: due 5PM Friday 9/24
- Reading: finish Chapter 1 and Chapter 2
Week 3
- Lecture #5 (Tu 9/29): smooth convex functions – gradient descent (slides)
- Lecture #6 (Th 10/1): smooth convex functions– geometry and acceleration (slides)
- Problem Set #2: due 5PM Friday 10/1
- Reading: finish Chapter 3 and start Chapter 4/li>
Week 4
- Lecture #7 (Tu 10/6): smooth convex functions – acceleration and momentum (slides)
- Lecture #8 (Th 10/8): applications and extensions – arbitrary norms (slides)
- Problem Set #3: due 5PM Friday 10/9
- Reading: finish Chapter 4 and start Chapter 5
Week 5
- Lecture #9 (Tu 10/13): applications and extensions - composite functions (slides)
- Lecture #10 (Tu 10/15): applications and extensions - coordinate descent (slides)
- Reading: finish Chapter 5
Week 6
- Lecture #11 (Tu 10/20): non-smooth functions – convex sets, and oracle definitions (slides)
- Take-home midterm: due 10/21
- Lecture #12 (Th 10/22): non-smooth functions – separating hyperplane theorem (slides)
- Reading: finish Chapter 6
Week 7
- Lecture #13 (Tu 10/27): non-smooth functions – oracle existence and subgradient descent (slides)
- Lecture #14 (Th 10/29): non-smooth functions – online linear optimization (slides)
- Problem Set #4: due 5PM Friday 10/30
- Reading: start Chapter 7
Week 8
- Lecture #1-- (Tu 11/3): class cancelled
- Lecture #15 (Th 11/5): non-smooth functions – experts, subgradients, and mirror descent (slides)
- Reading: continue Chapter 7
Week 9
- Lecture #16 (Tu 11/10): non-smooth functions – SGD, variance reduction, applications (slides)
- Problem Set #5: due 5PM Wed 11/11
- Lecture #17 (Th 11/12): finish SVRG and review session (slides)
- Reading: finish Chapter 7
Week 10
- Lecture #18 (Tu 11/17): cutting plane methods (slides)
- Lecture #19 (Th 11/19): interior point methods (slides)
- Take-home final: due 5PM Thursday 11/19
- Optional reading: optional chapters on cutting plane methods and interior point methods
Additional References
The material in the lecture notes is based primarily on my own experience with optimization and the following two texts:
Additional resources that may be helpful include the following: