Robust QA System with Task-Adaptive Pretraining, Data Augmentation, and Hyperparameter Tuning

Despite their significant success, transformer-based models trained on massive amounts of text still lack robustness to out-of-distribution data. In this project, we aim to build a robust question answering system by improving the DistilBERT model. To accomplish this goal, we implement task-adaptive pretraining (TAPT), model tuning such as transformer block re-initialization and increasing the number of training epochs, and ensemble methods. We also use data augmentation techniques to enable the model to generalize well even with limited data in the domains of interest.