Schedule And Syllabus

Meeting Times And Location

Unless otherwise specified the course lectures and meeting times are:

Monday, Wednesday 11:30 AM - 12:50 PM
Location: NVIDIA Auditorium

Schedule And Course Materials

The preliminary schedule is given below and is subject to change. We will also be posting suggested readings in this section a few days before each lecture. The lecture slides will be posted after each class. Please check back regularly for updates !

The lecture videos are recorded. You can watch them here.

In addition, lecture notes for each class (upto midterm) will be posted within a few days of each lecture. See more information about them here.

A practice midterm is now available to help the students prepare for the upcoming midterm. Note that Question 3 and 5b have not been covered in this year's class. [CS234 Midterm 2017][Solutions].
EventDateDescriptionCourse Materials
Lecture Jan 8 Introduction to Reinforcement Learning
  1. For a high level introduction: SB (Sutton and Barton) Chp 1
  2. [Linear Algebra Review]
  3. [Probability Review]
  4. [python tutorial]
[slides]
Lecture Jan 10 How to act given know how the world works. Tabular setting. Markov processes. Policy search. Policy iteration. Value iteration
  1. SB (Sutton and Barton) Chp 3, 4.1-4.4
[slides]
A1 Assignment 1 released [Assignment 1] [Solution]
Lecture Jan 15 No Class
Lecture Jan 17 Learning to evaluate a policy when don't know how the world works.
  1. SB (Sutton and Barton) Chp 5.1, 5.5, 6.1-6.3
[slides,, slides(annotated)]
Lecture Jan 22 Model-free learning to make good decisions. Q-learning. SARSA.
  1. SB (Sutton and Barton) Chp 5.2, 5.4, 6.4-6.5, 6.7
[slides, slides(annotated)]
A1 Jan 24 Assignment 1 due
Lecture Jan 24 Scaling up: value function approximation. Deep Q Learning
  1. SB (Sutton and Barton) Chp 9.3, 9.6-9.7, 10.1, 11.1, 11.2, 11.3
  2. [Human-level control through deep reinforcement learning]
[slides, slides(annotated)]
A2 Assignment 2 released [Assignment 2 ] [Solution]
Lecture Jan 29 Deep reinforcement learning continued
  1. SB (Sutton and Barton) 9.7
  2. [Human-level control through deep reinforcement learning]
  3. [Introduction to Tensorflow(from CS224N)]
[slides, slides(annotated)]
Lecture Jan 31 Imitation Learning
  1. [Maximum Entropy Inverse Reinforcement Learning]
  2. [Apprenticeship Learning via Inverse Reinforcement Learning]
[slides(annotated)]
Project Feb 5 Project proposal due
Lecture Feb 5 Policy search
  1. Sutton and Barto Chp 13
[slides], [slides(annotated)]
Lecture Feb 7 Policy search
  1. Sutton and Barto Chp 13
[slides (with some typos fixed post lecture)], [slides(annotated)]
A2 Feb 10 Assignment 2 due
Lecture Feb 12 Midterm review [draft review slides,annotated review slides]
A3 Assignment 3 released [Assignment 3] [Solution]
Exam Feb 14 In-class Midterm [Solution]
Lecture Feb 19 No Class
Lecture Feb 21 Fast reinforcement learning (Exploration/Exploitation) Part I
  1. Sutton and Barto Sections 2.1-2.7
[draft slides, annotated slides]
A3 Feb 23 Assignment 3 due
Lecture Feb 26 Fast reinforcement learning (Exploration/Exploitation) Part II
  1. Sutton and Barto Sections 2.1-2.7
[draft slides,annoated slides]
Lecture Feb 28 Batch Reinforcement Learning [draft slides,annotated slides]
Project Feb 28 Project milestone due
Lecture Mar 5 Monte Carlo Tree Search Suggested Readings:
  1. Gelly and Silver 2011[link]
  2. (AlphaGo Zero) Silver et al. Nature 2017[link]
[draft slides,annoated slides]
Lecture Mar 7 Human in the loop RL with a focus on transfer learnign [draft slides, annotated slides]
Exam Mar 12 In-class Quiz
Project Mar 14 Poster Session 11:50-2:50pm. Location TBA
Project Mar 19 Project final paper due

Lecture Notes

This section contains the CS234 course notes being created during the Winter 2018 offering of the course. These notes should be considered as additional resources for students, but they are also very much a work in progress. The course staff are working hard to create these lecture notes and cover as much of the material covered in the class as possible, and in some places provide further background information. However, we are aware that we may omit some things and there may be unintended typos. Of course, these notes should not be considered as an alternative to attending classes.

Git repositories for lecture notes

There are two versions of git repositories for the lecture notes, which are hosted on AFS at the following links:
Cloning the repositories

Type the commands shown below to clone each repository using git:
Adding a new lecture

To add a new lecture note please follow the instructions below:

  1. Copy the file template.tex in the repository and rename appropriately (e.g. lectureX.tex).
  2. Add packages when needed in the preamble section.
  3. Fill out the information in the config section of the file.
  4. Type out the lecture content.
  5. If you are making notes for lecture X, put any images needed in the directory images/lectureX.
  6. Possibly add frequently needed packages to template.tex.

Staff contributions

The course staff will be producing lecture notes for the first 10 lectures, all of the lectures prior to the midterm exam. They will be pushed to both repositories.

Student contributions

Students can contribute to the editable repository, which will be monitored by the course staff to assure that the updates are correct, and when approved they will be copied into the stable repository. We welcome anyone to correct any typos, add additional sections which they feel may be missing, add figures or any other additions that you think will improve the lecture notes. Extra credit may be awarded for contributions made to the lecture notes. To be given credit for contributions, make sure that your SUNet id appears in your commit before pushing (i.e. when you type git log).