Our goal is to build a question answering system that can adapt to unseen domains with only a few training samples from the domain.. We experimented with several approaches, including mixture of experts approach and various techniques to fine tune the pre-trained model better. Although we are able to to outperform the baseline, we found that model architecture is less important when it comes to improving performance. Relevant training data is by far the most important factor. Various fine tune techniques also help to some extend
