Knowledge-augmented Methods for Natural Language Processing

Over the last few years, natural language processing has seen remarkable progress due to the emergence of larger-scale models, better training techniques, and greater availability of data. Examples of these advancements include GPT-4, ChatGPT, and other pre-trained language models. These models are capable of characterizing linguistic patterns and generating context-aware representations, resulting in high-quality output. However, these models rely solely on input-output pairs during training and, therefore, struggle to incorporate external world knowledge, such as named entities, their relations, common sense, and domain-specific content. Incorporating knowledge into the training and inference of language models is critical to their ability to represent language accurately. Additionally, knowledge is essential in achieving higher levels of intelligence that cannot be attained through statistical learning of input text patterns alone. In this book, we will review recent developmentsin the field of natural language processing, specifically focusing on the role of knowledge in language representation. We will examine how pre-trained language models like GPT-4 and ChatGPT are limited in their ability to capture external world knowledge and explore various approaches to incorporate knowledge into language models. Additionally, we will discuss the significance of knowledge in enabling higher levels of intelligence that go beyond statistical learning on input text patterns. Overall, this survey aims to provide insights into the importance of knowledge in natural language processing and highlight recent advances in this field.

About the Author

Dr. Meng Jiang is currently an assistant professor at the Department of Computer Science and Engineering in the University of Notre Dame. He obtained his B.E. and Ph.D. from Tsinghua University. He spent two years in UIUC as a postdoc and joined ND in 2017. His research interests include data mining, machine learning, and natural language processing. He has published more than 100 peer-reviewed papers of these topics. He is the recipient of the Notre Dame International Faculty Research Award. The honors and awards he received include Best Paper Finalist in KDD 2014, Best Paper Award in KDD-DLG 2020, and ACM SIGSOFT Distinguished Paper Award in ICSE 2021. He received NSF CRII Award in 2019 and CAREER Award in 2022.

Bill Yuchen Lin is a postdoctoral young investigator at Allen Institute for AI (AI2), advised by Prof. Yejin Choi. He received his PhD from University of Southern California in 2022, advised by Prof. Xiang Ren. His research goal is to teach machines to think, talk, and act with commonsense knowledge and commonsense reasoning ability as humans do. Towards this ultimate goal, he has been developing knowledge-augmented reasoning methods (e.g., KagNet, MHGRN, DrFact) and constructing benchmark datasets (e.g., CommonGen, RiddleSense, X-CSR) that require commonsense knowledge and complex reasoning for both NLU and NLG. He initiated an online compendium of commonsense reasoning research, which serves as a portal for the community.

Dr. Shuohang Wang is a senior researcher in the Knowledge and Language Team of Cognitive Service Research Group. His research mainly focuses on question answering, multilingual NLU, summarization with deep learning, reinforcement learning, and few-shot learning. He served as area chair or senior PC member for ACL, EMNLP, and AAAI. He co-organized AAAI’23 workshop on Knowledge Augmented Methods for NLP.

Dr. Yichong Xu is a senior researcher in the Knowledge and Language Team of Cognitive Service Research Group. His research focuses on the combination of knowledge and NLP, with applications to question answering, summarization, and multimodal learning. He led the effort to achieve the human parity on the CommonsenseQA benchmark. He has held tutorials on knowledge-augmented NLP methods in ACL and WSDM. Prior to joining Microsoft, Dr. Xu got his Ph.D. in machine learning from Carnegie Mellon University.

Wenhao Yu is a Ph.D. candidate in the Department of Computer Science and Engineering at the University of Notre Dame. His research lies in language model + knowledge for solving knowledge-intensive applications, such as open-domain question answering and commonsense reasoning. He has published over 15 conference papers and presented 3 tutorials in machine learning and natural language processing conferences, including ICLR, ICML, ACL, and EMNLP. He was the recipient of Bloomberg Ph.D. Fellowship in 2022 and won the Best Paper Award at SoCal NLP in 2022.He was a research intern in Microsoft Research and Allen Institute for AI.

Dr. Chenguang Zhu is a principal research manager in Microsoft Cognitive Services Research Group, where he leads the Knowledge and Language Team. His research covers knowledge-enhanced language model, text summarization, and prompt learning. Dr. Zhu has led teams to achieve human parity in CommonsenseQA, HellaSwag, and CoQA, and first places in CommonGen, FEVER, ARC, and SQuAD v1.0. He holds a Ph.D. degree in Computer Science from Stanford University. Dr. Zhu has published over 100 papers on NLP and knowledge-augmented methods. He has held tutorials and workshops in knowledge-augmented NLP in conferences like ACL, AAAI, and WSDM. He has published the book Machine Reading Comprehension: Algorithm and Practice published in Elsevier.