MiniPLM: Knowledge Distillation for Pre-Training Language Models
Knowledge distillation (KD) is widely used to train small, high-performing student language
models (LMs) using large teacher LMs. While effective in fine-tuning, KD during pre-training …
models (LMs) using large teacher LMs. While effective in fine-tuning, KD during pre-training …
Knowledge distillation of large language models
Knowledge Distillation (KD) is a promising technique for reducing the high computational
demand of large language models (LLMs). However, previous KD methods are primarily …
demand of large language models (LLMs). However, previous KD methods are primarily …
Ddk: Distilling domain knowledge for efficient large language models
Despite the advanced intelligence abilities of large language models (LLMs) in various
applications, they still face significant computational and storage demands. Knowledge …
applications, they still face significant computational and storage demands. Knowledge …
PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
Recent advancements in large language models (LLMs) have raised concerns about
inference costs, increasing the need for research into model compression. While knowledge …
inference costs, increasing the need for research into model compression. While knowledge …
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models
J Koo, Y Hwang, Y Kim, T Kang, H Bae… - arXiv preprint arXiv …, 2024 - arxiv.org
Despite the success of Large Language Models (LLMs), they still face challenges related to
high inference costs and memory requirements. To address these issues, Knowledge …
high inference costs and memory requirements. To address these issues, Knowledge …
MiniLLM: Knowledge distillation of large language models
Knowledge Distillation (KD) is a promising technique for reducing the high computational
demand of large language models (LLMs). However, previous KD methods are primarily …
demand of large language models (LLMs). However, previous KD methods are primarily …
Dynamic knowledge distillation for pre-trained language models
Knowledge distillation~(KD) has been proved effective for compressing large-scale pre-
trained language models. However, existing methods conduct KD statically, eg, the student …
trained language models. However, existing methods conduct KD statically, eg, the student …
Revisiting knowledge distillation for autoregressive language models
Knowledge distillation (KD) is a common approach to compress a teacher model to reduce
its inference cost and memory footprint, by training a smaller student model. However, in the …
its inference cost and memory footprint, by training a smaller student model. However, in the …
[HTML][HTML] AdaDS: Adaptive data selection for accelerating pre-trained language model knowledge distillation
Abstract Knowledge distillation (KD) is a widely used method for transferring knowledge
from large teacher models to computationally efficient student models. Unfortunately, the …
from large teacher models to computationally efficient student models. Unfortunately, the …
Pre-training Distillation for Large Language Models: A Design Space Exploration
Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a
smaller student model. Previous work applying KD in the field of large language models …
smaller student model. Previous work applying KD in the field of large language models …