Improving Out-of-Domain Question Answering with Mixture of Experts

Question answering (QA) is an important problem with numerous applications in real life. Sometimes, the resource of certain QA tasks is limited. Our work aims to build a robust QA system that can generalize to novel QA tasks with few examples and gradient steps. We propose a Mixture-of-Experts (MoE) style training framework, where we learn a gating network to construct the embeddings by performing a weighted sum of the base "expert" models with fixed parameters. We find that using the mixture of expert models improves generalization performance and reduces overfitting, especially when using "expert" models trained with data augmentation. We use meta-learning methods, specifically the MAML algorithm, to train the gating network for domain adaptation. Training the gating network with the MAML algorithm and finetuning on out-of-domain tasks improved out-of-domain QA performance of baseline models on all metrics. We also discovered a correlation between expert-model performance and the weight the MoE framework puts on each of them. Our approach achieves a F-1 score of 60.8 and EM score of 42.2 on the out-of-domain QA testing leaderboard.