Exploring First Order Gradient Approximation Meta Learning for Robust QA Systems

Reptile is a meta learning approach that searches for initial model parameters to allow a model to be fine tuned with a small dataset. However when fine tuning a language model on a small set of tasks and low learning rate, Reptile may still over-fit on training batches. RandEPTILE adds additional noise to initial model parameters to efficiently search for areas of lower validation loss in the parameter domain. This project explored the effects of RandEPTILE with a distilBERT pre-trained model for question answering using small fine-tuning datasets. While the improvement on final test accuracy was inconclusive, adding additional noise to model parameters could be worth exploring in future meta learning techniques.