Pre-trained language models have been applied to various NLP tasks with considerable
performance gains. However, the large model sizes, together with the long inference time …

Pre-trained language models (PLMs) achieve great success in NLP. However, their huge
model sizes hinder their applications in many practical systems. Knowledge distillation is a …

The billions, and sometimes even trillions, of parameters involved in pre-trained language
models significantly hamper their deployment in resource-constrained devices and real-time …

Layer-wise distillation is a powerful tool to compress large models (ie teacher models) into
small ones (ie, student models). The student distills knowledge from the teacher by …

Existing language model compression methods mostly use a simple L2 loss to distill
knowledge in the intermediate representations of a large BERT model to a smaller one …

Large-scale language models have recently demonstrated impressive empirical
performance. Nevertheless, the improved results are attained at the price of bigger models …

Pre-trained deep neural network language models such as ELMo, GPT, BERT and XLNet
have recently achieved state-of-the-art performance on a variety of language understanding …

This study proposes a knowledge distillation algorithm based on large language models
and feature alignment, aiming to effectively transfer the knowledge of large pre-trained …

Pre-trained language models such as BERT have proven to be highly effective for natural
language processing (NLP) tasks. However, the high demand for computing resources in …

Pre-trained Language Models (PLMs) have been successful for a wide range of natural
language processing (NLP) tasks. The state-of-the-art of PLMs, however, are extremely …