Causal Inference & RL Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Correlation is not decision. These papers connect causal inference and reinforcement learning through counterfactual reasoning, confounded MDPs, and data fusion, giving agents tools to ask what would have happened.

Causal Inference & RL: 10 key papers

1. Causal Bandits: Learning Good Interventions via Causal Inference
   Lattimore et al. arXiv 2016.
2. Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
   Buesing et al. arXiv 2018.
3. Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models
   Oberst and Sontag. arXiv 2019.
4. Causal Reasoning from Meta-reinforcement Learning
   Dasgupta et al. arXiv 2019.
5. Explainable Reinforcement Learning Through a Causal Lens
   Madumal et al. arXiv 2019.
6. Causal Reinforcement Learning: A Survey
   Deng et al. arXiv 2023.
7. Deconfounding Reinforcement Learning in Observational Settings
   Lu et al. arXiv 2018.
8. Causal Confusion in Imitation Learning
   de Haan et al. arXiv 2019.
9. Learning Neural Causal Models from Unknown Interventions
   Ke et al. arXiv 2019.
10. Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
    Ke et al. arXiv 2021.

Related reading lists

• Causal Discovery & Structure Learning
• Off-Policy Evaluation & Offline Evaluation
• Block MDPs & Latent State Discovery
• Meta-Reinforcement Learning
• LLM Agents & Tool-Augmented Language Models
• Imitation Learning and Inverse RL
• Offline Imitation Learning
• Reasoning in LLMs