Offline-to-Online RL & Fine-Tuning Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Bridging offline pretraining and online adaptation is central to sample-efficient reinforcement learning. This list brings together methods for conservative pretraining, calibrated fine-tuning, and expansion strategies that keep early online performance stable while unlocking rapid improvement.

Offline-to-Online RL & Fine-Tuning: 10 key papers

1. Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
   Lee et al. arXiv 2021.
2. Efficient Online Reinforcement Learning with Offline Data
   Ball et al. arXiv 2023.
3. Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
   Nakamoto et al. arXiv 2023.
4. Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
   Zheng et al. arXiv 2023.
5. Policy Expansion for Bridging Offline-to-Online Reinforcement Learning
   Zhang et al. arXiv 2023.
6. Jump-Start Reinforcement Learning
   Uchendu et al. arXiv 2022.
7. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
   Nair et al. arXiv 2020.
8. Launchpad: Learning to Schedule Using Offline and Online RL Methods
   Venkataswamy et al. arXiv 2022.
9. Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
   Lei et al. arXiv 2023.
10. Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gap
    Wang et al. arXiv 2023.

Related reading lists

• Offline Reinforcement Learning
• Policy Gradient Methods
• Goal-Conditioned Reinforcement Learning
• Robust & Risk-Sensitive RL
• Offline Imitation Learning
• Linear MDPs & Linear Function Approximation
• Transformers for Sequential Decision Making
• Zero-Shot RL & Generalization