Hindsight Experience Replay & Relabeling Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Core readings on Hindsight Experience Replay and hindsight relabeling — the simple yet powerful idea of learning from failed trajectories by pretending they succeeded at what they achieved.

Hindsight Experience Replay & Relabeling: 10 key papers

1. Hindsight Experience Replay
   Andrychowicz et al. arXiv 2017.
2. Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
   Eysenbach et al. arXiv 2020.
3. Competitive Experience Replay
   Liu et al. arXiv 2019.
4. Prioritized Experience Replay
   Schaul et al. arXiv 2015.
5. Energy-Based Hindsight Experience Prioritization
   Zhao and Tresp. arXiv 2018.
6. Hindsight Credit Assignment
   Harutyunyan et al. arXiv 2019.
7. Soft Hindsight Experience Replay
   He et al. arXiv 2020.
8. Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
   Zhao et al. arXiv 2019.
9. Bias-Reduced Hindsight Experience Replay with Virtual Goal Prioritization
   Manela and Biess. arXiv 2019.
10. Generalized Hindsight for Reinforcement Learning
    Li et al. arXiv 2020.

Related reading lists

• Policy Gradient Methods
• Hierarchical RL & Options Framework
• Continual & Lifelong Learning
• Multi-Agent Reinforcement Learning
• Preference Learning & Learning from Comparisons
• Out-of-Distribution & Robustness
• Inverse Reinforcement Learning
• Unsupervised Environment Design