Aaron Sidford (sidford@stanford.edu)

Welcome

This page has informatoin and lecture notes from the course "Introduction to Optimization Theory" (MS&E213 / CS 269O) which I taught in Fall 2020. I hope you enjoy the content as much as I enjoyed teaching the class and if you have questions or feedback on the note, feel free to email me.

Course Overview

This class will introduce the theoretical foundations of continuous optimization. Starting from first principles we show how to design and analyze simple iterative methods for efficiently solving broad classes of optimization problems. The focus of the course will be on achieving provable convergence rates for solving large-scale problems.

Email: sidford@stanford.edu

Here are the links for the course lecture notes. Feedback is welcome and if there is anything that is unclear or you would like added, please feel free to contact me. My apologies for missing citations and references; these may be added over time in future course offerings

**Chapter 1: Introduction**: the notes for this chapter are here.**Chapter 2: Smoothness**: the notes for this chapter are here.**Chapter 3: Convexity**: the notes for this chapter are here.**Chapter 4: Acceleration**: the notes for this chapter are here.**Chapter 5: Smooth Extensions**: the notes for this chapter are here.**Chapter 6: Non-smooth Convex Functions**: the notes for this chapter are here.**Chapter 7**:**Subgradient / Mirror Descent**: the notes for this chapter are here.**Bonus Chapter 1: Cutting Plane Methods**: the notes for this chapter are here.**Bonus Chapter 2: Interior Point Methods**: the notes for this chapter are here.**Appendix A: Norms**: last year's notes for this chapter are here.

Here is the schedule of material for the course.

__Week 1__

**Lecture #1 (Tu 9/15)**: intro: course overview: oracles, efficiency, and optimization impossibility (slides)**Lecture #2 (Th 9/17)**: intro: example problem / algorithm: critical points by gradient descent (slides)**Reading**: Start Chapter 1 and Chapter 2

__Week 2__

**Lecture #3 (Tu 9/22)**: intro: Lipschitz functions and global minimization (slides)**Lecture #4 (Th 9/24)**: intro: smooth functions and critical points revisited (slides)**Problem Set #1**: due 5PM Friday 9/24**Reading**: finish Chapter 1 and Chapter 2

__Week 3__

**Lecture #5 (Tu 9/29)**: smooth convex functions – gradient descent (slides)**Lecture #6 (Th 10/1)**: smooth convex functions– geometry and acceleration (slides)**Problem Set #2**: due 5PM Friday 10/1**Reading**: finish Chapter 3 and start Chapter 4/li>

__Week 4__

**Lecture #7 (Tu 10/6)**: smooth convex functions – acceleration and momentum (slides)**Lecture #8 (Th 10/8)**: applications and extensions – arbitrary norms (slides)**Problem Set #3**: due 5PM Friday 10/9**Reading**: finish Chapter 4 and start Chapter 5

__Week 5__

**Lecture #9 (Tu 10/13)**: applications and extensions - composite functions (slides)**Lecture #10 (Tu 10/15)**: applications and extensions - coordinate descent (slides)**Reading**: finish Chapter 5

__Week 6__

**Lecture #11 (Tu 10/20)**: non-smooth functions – convex sets, and oracle definitions (slides)**Take-home midterm**: due 10/21**Lecture #12 (Th 10/22)**: non-smooth functions – separating hyperplane theorem (slides)**Reading**: finish Chapter 6

__Week 7__

**Lecture #13 (Tu 10/27)**: non-smooth functions – oracle existence and subgradient descent (slides)**Lecture #14 (Th 10/29)**: non-smooth functions – online linear optimization (slides)**Problem Set #4**: due 5PM Friday 10/30**Reading**: start Chapter 7

__Week 8__

**Lecture #1-- (Tu 11/3)**:*class cancelled***Lecture #15 (Th 11/5)**: non-smooth functions – experts, subgradients, and mirror descent (slides)**Reading**: continue Chapter 7

__Week 9__

**Lecture #16 (Tu 11/10)**: non-smooth functions – SGD, variance reduction, applications (slides)**Problem Set #5**: due 5PM Wed 11/11**Lecture #17 (Th 11/12)**: finish SVRG and review session (slides)**Reading**: finish Chapter 7

__Week 10__

**Lecture #18 (Tu 11/17)**: cutting plane methods (slides)**Lecture #19 (Th 11/19)**: interior point methods (slides)**Take-home final**: due 5PM Thursday 11/19**Optional reading**: optional chapters on cutting plane methods and interior point methods

The material in the lecture notes is based primarily on my own experience with optimization and the following two texts:

- "Introductory Lectures on Convex Programming Volume I: Basic Course" by Yurii Nesterov.
- Convex Optimization: Algorithms and Complexity by Sébastien Bubeck.

Additional resources that may be helpful include the following:

- Convex Optimization by Stephen Boyd and Lieven Vandenberghe.
- CSE 599: Interplay between Convex Optimization and Geometry a course by Yin Tat Lee.