CS234: Reinforcement Learning

Course Description

To realize the dreams and impact of AI requires autonomous systems that learn to make good decisions. Reinforcement learning is one powerful paradigm for doing so, and it is relevant to an enormous range of tasks, including robotics, game playing, consumer modeling and healthcare. This class will provide a solid introduction to the field of reinforcement learning and students will learn about the core challenges and approaches, including generalization and exploration. Through a combination of lectures, and written and coding assignments, students will become well versed in key ideas and techniques for RL. Assignments will include the basics of reinforcement learning as well as deep reinforcement learning-- an extremely promising new area that combines deep learning techniques with reinforcemetn learning. In addition, students will advance their understanding and the field of RL through an open ended project.

Learning Objectives

By the end of the class students should be able to

Define the key features of RL that distinguish it from AI and non-interactive machine learning
Define MDP, POMDP, bandit, batch offline RL, online RL
Describe the exploration vs exploitation challenge and compare and contrast at least two approaches for addressing this challenge
Given an application problem (e.g. from computer vision, robotics, etc) decide if it should be formulated as a RL problem, if yes how to formulate, what algorithm (from class) is best suited to addressing, and justify answer.
Implement common RL algorithms including a deep RL algorithm
Describe multiple criteria for analyzing RL algorithms and evaluate algorithms on these metrics: e.g. regret, sample complexity, computational complexity, convergence, etc.
List at least two open challenges or hot topics in RL

Course Instructors

Emma Brunskill

Teaching Assistants

Class Time and Location

Spring quarter (April - June, 2017)
Lecture: Monday, Wednesday 1:30-2:50
Location: Cubberley Auditorium

Office Hours

See calendar

Grading

See the Grading page.

Final Project Details

See the Project Page for more details on the course project.

Useful Reference Texts

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning.[link]
Stuart J. Russell and Peter Norvig. Artificial Intelligence. A Modern Approach.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. [link]

Prerequisites

Proficiency in Python
All class assignments will be in Python (using numpy and Tensorflow and optionally Keras). There is a tutorial here for those who aren't as familiar with Python. If you have a lot of programming experience but in a different language (e.g. C/C++/Matlab/Javascript) you will probably be fine.
College Calculus, Linear Algebra (e.g. MATH 51, CME 100)
You should be comfortable taking derivatives and understanding matrix vector operations and notation.
Basic Probability and Statistics (e.g. CS 109 or other stats course)
You should know basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
Foundations of Machine Learning
We will be formulating cost functions, taking derivatives and performing optimization with gradient descent. Either cs221 or cs229 cover this background. Some optimization tricks will be more intuitive with some knowledge of convex optimization.