Question answering system has always been an active field in the Natural Language Processing (NLP) researches. In the past few years, the most successful models are primarily based on Recurrent Neural Networks (RNNs) with attention. Though a lot of progress has been made, due to its sequential nature, RNN's operations are unparallelizable, which makes both training and inference slow. In addition, with linear interaction distance, RNNs have difficulty in learning long dependencies. This is a severe problem in QA system, since the context are usually long paragraphs.
Based on these problems, in this project, we implemented a QA model based on Transformer, hoping to achieve both accurate and fast reading comprehension. We focused on reading comprehension among all QA problems, which is to select a part of text from the given context to answer some certain question. Instead of LSTM, this model used convolution layers and self-attention to form encoders. Given a paragraph of context and a question, it will output the probability of each context word being the start or end of the answer. However, against our expectation, this model did not perform very well. The speed is low due to its large amount of parameters, and the accuracy cannot match that of BiDAF because of overfitting.
