Enhancing a Text Summarization System with ELMo

Text summarization has gained a considerable amount of research interest due to deep learning based techniques. We lever- age recent results in transfer learning for Natural Language Processing (NLP) using pre-trained deep contextualized word embeddings in a sequence-to-sequence architecture based on pointer-generator networks. We evaluate our approach on the two largest summarization datasets: CNN/Daily Mail and the recent Newsroom dataset. We show how using pre-trained contextualized embeddings on Newsroom improves significantly the state-of-the-art ROUGE-1 measure and obtains comparable scores on the other ROUGE values.