Policy Gradient Methods Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

The core ideas behind gradient-based policy optimization in RL.

Policy Gradient Methods: 10 key papers

1. Trust Region Policy Optimization
   Schulman et al. arXiv 2015.
2. Proximal Policy Optimization Algorithms
   Schulman et al. arXiv 2017.
3. High-Dimensional Continuous Control Using Generalized Advantage Estimation
   Schulman et al. arXiv 2015.
4. Asynchronous Methods for Deep Reinforcement Learning
   Mnih et al. arXiv 2016.
5. Continuous control with deep reinforcement learning
   Lillicrap et al. arXiv 2015.
6. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
   Haarnoja et al. arXiv 2018.
7. Addressing Function Approximation Error in Actor-Critic Methods
   Fujimoto et al. arXiv 2018.
8. Sample Efficient Actor-Critic with Experience Replay
   Wang et al. arXiv 2016.
9. Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator
   Fazel et al. arXiv 2018.
10. On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
    Agarwal et al. arXiv 2019.

Related reading lists

• Linear MDPs & Linear Function Approximation
• Optimal Control & LQR/LQG
• Hindsight Experience Replay & Relabeling
• Bayesian Optimization & Global Optimization
• Multi-Agent Reinforcement Learning
• Model-Based Reinforcement Learning
• Offline-to-Online RL & Fine-Tuning
• Hyperparameter Optimization & AutoML