Offline Reinforcement Learning Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Learning policies from fixed datasets without environment interaction. The data-driven paradigm for RL.

Offline Reinforcement Learning: 10 key papers

1. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
   Levine et al. arXiv 2020.
2. Conservative Q-Learning for Offline Reinforcement Learning
   Kumar et al. arXiv 2020.
3. Off-Policy Deep Reinforcement Learning without Exploration
   Fujimoto et al. arXiv 2018.
4. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
   Kumar et al. arXiv 2019.
5. Offline Reinforcement Learning with Implicit Q-Learning
   Kostrikov et al. arXiv 2021.
6. A Minimalist Approach to Offline Reinforcement Learning
   Fujimoto and Gu. arXiv 2021.
7. D4RL: Datasets for Deep Data-Driven Reinforcement Learning
   Fu et al. arXiv 2020.
8. MOReL : Model-Based Offline Reinforcement Learning
   Kidambi et al. arXiv 2020.
9. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
   An et al. arXiv 2021.
10. RvS: What is Essential for Offline RL via Supervised Learning?
    Emmons et al. arXiv 2021.

Related reading lists

• Offline Imitation Learning
• Off-Policy Evaluation & Offline Evaluation
• Offline-to-Online RL & Fine-Tuning
• Transformers for Sequential Decision Making
• Behavioral Cloning & Offline Imitation
• Sample Complexity in Reinforcement Learning
• Model-Based Reinforcement Learning
• Robust & Risk-Sensitive RL