CheXGB: Combining Graph Neural Networks and BERT for automated radiology report labeling

Healthcare systems wish to utilize the large quantities of unlabeled free-text radiology reports for training medical image models. Automated labelers allow healthcare systems to annotate tens of thousands of reports without expensive labor from doctors which would enable many hospitals around the world to train AI systems on their data. We propose CheXGB, an automated labeler that combines global information encoded by a heterogeneous graph of the free text reports and their associated words from a large chest X-ray data set (MIMIC-CXR) with local context information encoded by BERT. The input to CheXGB is a heterogeneous graph consisting of reports (both labeled and unalabeled) and words. First, all heterogeneous graph nodes are fed through TextGCN while only labeled reports are passed to BERT. Second, attention is performed on the output of BERT and the nodes corresponding to the labeled reports. Finally, the output of the attention layer is passed through a linear layer for multi-label class prediction. Using explicit global relations encoded by a graph neural network allows for inputs that purely NLP models are not trained to provide which is particularly useful in the data sparse regime we study. We find that variants of CheXGB outperforms CheXbert -- the current state of the art in radiology report labeling -- in 13 out of 14 classes and improve the average kappa across tasks from 0.830 to 0.843.