CS 224R: Deep Reinforcement Learning

MDPs, policies, value functions, state-visitation distributions, and the family of deep RL algorithms

Behavior cloning, expressive policy distributions, action chunking, DAgger, and demonstration collection

REINFORCE, the log-derivative trick, reward-to-go, baselines, off-policy importance sampling, and practical considerations

Policy evaluation, advantage estimates, basic and off-policy actor-critic, PPO, and SAC

From actor-critic to critic-only methods, Q-learning, target networks, double Q-learning, n-step returns, and the DQN recipe

Why offline RL differs from off-policy RL, filtered BC, advantage-weighted regression, IQL, and conservative Q-learning

Goal classifiers, preference-based reward models, RLHF for language models, and RLAIF

Learning dynamics models, MBPO, planning with learned models, MPC, CEM, and the tradeoffs of model-based RL

Universal policies and value functions, multi-task imitation and RL, hindsight relabeling, and HER

Meta-RL problem setup, task distributions, black-box meta-RL, exploration-exploitation, posterior sampling, and task-inference objectives

Long-horizon tasks, high-level and low-level policies, subgoal representations, hierarchical imitation, hierarchical RL, and skill discovery

Physics simulators, sim2real gaps, domain randomization, adaptation, Real2Sim, human data, retargeting, and RL algorithms for robot policy learning

Formula sheet, data regimes, method comparison table, bias/variance/distribution shift, meta-RL, hierarchy, sim2real, common failure modes, and exam traps

LLM training pipeline, instruction tuning, RLHF, preference modeling, and Direct Preference Optimization

Inference-time scaling, lessons from poker and Go, RL for reasoning, and serial vs. parallel test-time compute