RAIL-KD: RAndom intermediate layer mapping for knowledge distillation
MA Haidar, N Anchuri, M Rezagholizadeh… - arXiv preprint arXiv …, 2021 - arxiv.org
Intermediate layer knowledge distillation (KD) can improve the standard KD technique
(which only targets the output of teacher and student models) especially over large pre …
(which only targets the output of teacher and student models) especially over large pre …
[HTML][HTML] AdaDS: Adaptive data selection for accelerating pre-trained language model knowledge distillation
Abstract Knowledge distillation (KD) is a widely used method for transferring knowledge
from large teacher models to computationally efficient student models. Unfortunately, the …
from large teacher models to computationally efficient student models. Unfortunately, the …
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection
Knowledge distillation addresses the problem of transferring knowledge from a teacher
model to a student model. In this process, we typically have multiple types of knowledge …
model to a student model. In this process, we typically have multiple types of knowledge …
Reaugkd: Retrieval-augmented knowledge distillation for pre-trained language models
Abstract Knowledge Distillation (KD) is one of the most effective approaches to deploying
large-scale pre-trained language models in low-latency environments by transferring the …
large-scale pre-trained language models in low-latency environments by transferring the …
Gradient knowledge distillation for pre-trained language models
Knowledge distillation (KD) is an effective framework to transfer knowledge from a large-
scale teacher to a compact yet well-performing student. Previous KD practices for pre …
scale teacher to a compact yet well-performing student. Previous KD practices for pre …
Cost-effective distillation of large language models
Abstract Knowledge distillation (KD) involves training a small “student” model to replicate the
strong performance of a high-capacity “teacher” model, enabling efficient deployment in …
strong performance of a high-capacity “teacher” model, enabling efficient deployment in …
f-Divergence Minimization for Sequence-Level Knowledge Distillation
Y Wen, Z Li, W Du, L Mou - arXiv preprint arXiv:2307.15190, 2023 - arxiv.org
Knowledge distillation (KD) is the process of transferring knowledge from a large model to a
small one. It has gained increasing attention in the natural language processing community …
small one. It has gained increasing attention in the natural language processing community …
Ddk: Distilling domain knowledge for efficient large language models
Despite the advanced intelligence abilities of large language models (LLMs) in various
applications, they still face significant computational and storage demands. Knowledge …
applications, they still face significant computational and storage demands. Knowledge …
MiniPLM: Knowledge Distillation for Pre-Training Language Models
Knowledge distillation (KD) is widely used to train small, high-performing student language
models (LMs) using large teacher LMs. While effective in fine-tuning, KD during pre-training …
models (LMs) using large teacher LMs. While effective in fine-tuning, KD during pre-training …
Dynamic knowledge distillation for pre-trained language models
Knowledge distillation~(KD) has been proved effective for compressing large-scale pre-
trained language models. However, existing methods conduct KD statically, eg, the student …
trained language models. However, existing methods conduct KD statically, eg, the student …