Schedule

Meeting Times And Location

Unless otherwise specified the course lectures and meeting times are:

Monday, Wednesday 11:30 AM - 12:50 PM
Location: Gates B1

Schedule And Course Materials

The preliminary schedule is given below and is subject to change. We will also be posting suggested readings in this section before each lecture. The lecture slides will be posted after each class. Please check back regularly for updates !

The lecture videos are recorded. You can watch them here.

In addition, draft lecture notes for the class material up to the midterm are available. See more information about them here.
Practice midterm is now available to help the students prepare for the upcoming midterm:
EventDate        DescriptionCourse Materials
Lecture Jan 7 Introduction to Reinforcement Learning
  1. [Slides, Draft lecture notes]
  2. Additional Materials:
Lecture Jan 9 How to act given know how the world works.
  • Tabular setting
  • Markov processes
  • Policy search
  • Policy iteration
  • Value iteration
  1. [Slides, Draft lecture notes]
  2. Additional Materials:
    • SB (Sutton and Barton) Chp 3, 4.1-4.4
A1 Jan 9 Assignment 1 released Assignment 1
Lecture Jan 14 Learning to evaluate a policy when don't know how the world works.
  1. [Slides, Class slides with annotations, Draft lecture notes]
  2. Additional Materials:
    • SB (Sutton and Barton) Chp 5.1, 5.5, 6.1-6.3
    • David Silver's Lecture 4 [link]
Lecture Jan 16 Model-free learning to make good decisions.
  • Q-learning
  • SARSA
  1. [Slides, Class slides with annotations, Draft lecture notes]
  2. Additional Materials:
    • SB (Sutton and Barton) Chp 5.2, 5.4, 6.4-6.5, 6.7
    • Week 2 Session: [video], [ slides]
Jan 21 No Class
Lecture Jan 23 Scaling up: RL with function approximation
  1. [Slides, Class slides with annotations, Draft lecture notes]
  2. Additional Materials:
A1 Jan 23 Assignment 1 due, 11:59pm
A2 Jan 23 Assignment 2 released Assignment 2
Lecture Jan 28 RL with function approximation.
  1. [Slides, Class slides with annotations, Draft lecture notes]
  2. Additional Materials:
Lecture Jan 30 Imitation learning in large spaces
  1. [Draft slides, Class slides with annotations, Draft lecture notes]
  2. Additional Materials:
Lecture Feb 4 Policy search
  1. [Draft slides, Class slides with annotations, Draft lecture notes]
  2. Sutton and Barto Chp 13
Project Feb 4 Project proposal due, 11:59pm
Lecture Feb 6 Policy search
  1. [Draft slides, Class slides, Draft lecture notes]
  2. Additional Materials
    • Sutton and Barto Chp 13
    • Week 5 Session: [video], [ slides]
Project Feb 6 Assignment 2 due, 11:59pm
Lecture Feb 11 Midterm review
  1. [Slides, Draft lecture notes]
  2. [Midterm Review]
  3. Week 6 Session: [video], [ slides]
Exam Feb 13 In-class Midterm
A3 Feb 13 Assignment 3 released
Lecture Feb 18 No Class: President's Day Holiday
Lecture Feb 20 Exploration/Exploitation
  1. [Class slides with annotations, Draft lecture notes]
  2. Additional Materials:
Lecture Feb 25 Exploration / Exploitation
  1. [Class slides with annotations, Draft lecture notes, Sutton and Barto Sections 2.1-2.7]
  2. Additional Materials:
A3 Feb 25 Project Milestone 3 due, 11:59pm
Lecture Feb 27 Exploration / Exploitation
  1. [Class slides with annotations, Draft lecture notes]
  2. Supplementary Materials:
Project Feb 27 Assignment 3 due, 11:59pm
Lecture Mar 4 Meta-Learning (Chelsea Finn guest lecture)
Lecture Mar 6 Batch Reinforcement Learning
  1. [Draft Slides, Class slides with annotations, Draft lecture notes]
Exam Mar 11 In-class Quiz
Lecture Mar 13 Monte Carlo Tree Search
  1. [Draft Slides, Class slides with annotations]
Project Mar 20 Project final paper due, 11:59pm
Project Mar 22 Poster Session 8:30 - 11:30am ACSR Basketball court 1 and 2

Lecture Notes

This section contains the CS234 course notes being created during the Winter 2019 offering of the course. These notes should be considered as additional resources for students, but they are also very much a work in progress. The course staff are working hard to create these lecture notes and cover as much of the material covered in the class as possible, and in some places provide further background information. However, we are aware that we may omit some things and there may be unintended typos. Of course, these notes should not be considered as an alternative to attending classes.

Git repositories for lecture notes

There are two versions of git repositories for the lecture notes, which are hosted on AFS at the following links:
Cloning the repositories

Type the commands shown below to clone each repository using git:
Adding a new lecture

To add a new lecture note please follow the instructions below:

  1. Copy the file template.tex in the repository and rename appropriately (e.g. lectureX.tex).
  2. Add packages when needed in the preamble section.
  3. Fill out the information in the config section of the file.
  4. Type out the lecture content.
  5. If you are making notes for lecture X, put any images needed in the directory images/lectureX.
  6. Possibly add frequently needed packages to template.tex.

Staff contributions

The course staff will be producing lecture notes for the first 10 lectures, all of the lectures prior to the midterm exam. They will be pushed to both repositories.

Student contributions

Students can contribute to the editable repository, which will be monitored by the course staff to assure that the updates are correct, and when approved they will be copied into the stable repository. We welcome anyone to correct any typos, add additional sections which they feel may be missing, add figures or any other additions that you think will improve the lecture notes. Extra credit may be awarded for contributions made to the lecture notes. To be given credit for contributions, make sure that your SUNet id appears in your commit before pushing (i.e. when you type git log).