Off-Policy Evaluation & Offline Evaluation Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Before deploying a policy you need to know how good it is from logged data alone. This list covers importance sampling, doubly robust, and minimax approaches to off-policy evaluation without new environment interaction.

Off-Policy Evaluation & Offline Evaluation: 10 key papers

1. Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
   Jiang and Li. arXiv 2015.
2. Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning
   Thomas and Brunskill. arXiv 2016.
3. Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
   Liu et al. arXiv 2018.
4. Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
   Kallus and Uehara. arXiv 2019.
5. Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
   Voloshin et al. arXiv 2019.
6. Minimax Weight and Q-Function Learning for Off-Policy Evaluation
   Uehara et al. arXiv 2019.
7. Benchmarks for Deep Off-Policy Evaluation
   Fu et al. arXiv 2021.
8. Batch Policy Learning under Constraints
   Le et al. arXiv 2019.
9. Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
   Kallus and Zhou. arXiv 2020.
10. Off-Policy Evaluation via Off-Policy Classification
    Irpan et al. arXiv 2019.

Related reading lists

• Offline Reinforcement Learning
• Linear MDPs & Linear Function Approximation
• Causal Inference & RL
• Offline Imitation Learning
• Diffusion Models for Decision Making
• Robust & Risk-Sensitive RL
• Sample Complexity in Reinforcement Learning
• Generalisation in RL