Since deep learning became a key player in natural language processing (NLP), many deep
learning models have been showing remarkable performances in a variety of NLP tasks …

The pre-training models such as BERT have achieved great results in various natural
language processing problems. However, a large number of parameters need significant …

In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit
designed for natural language processing. It works with different neural network models and …

This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural
Network (MT-DNN)(Liu et al., 2019) for learning text representations across multiple natural …

Pre-trained language models learn informative word representations on a large-scale text
corpus through self-supervised learning, which has achieved promising performance in …

In natural language processing (NLP) tasks, slow inference speed and huge footprints in
GPU usage remain the bottleneck of applying pre-trained deep models in production. As a …

Fine-tuning transformer models after unsupervised pre-training reaches a very high
performance on many different natural language processing tasks. Unfortunately …

Recent breakthroughs of pretrained language models have shown the effectiveness of self-
supervised learning for a wide range of natural language processing (NLP) tasks. In addition …

This paper investigates the effectiveness of knowledge distillation in the context of
multilingual models. We show that with knowledge distillation, Long Short-Term Memory …

Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to
compress Large Language Models (LLMs). Contrary to prior assertions that reverse …