SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models
J Koo, Y Hwang, Y Kim, T Kang, H Bae… - arXiv preprint arXiv …, 2024 - arxiv.org
Despite the success of Large Language Models (LLMs), they still face challenges related to
high inference costs and memory requirements. To address these issues, Knowledge …
high inference costs and memory requirements. To address these issues, Knowledge …
Revisiting knowledge distillation for autoregressive language models
Knowledge distillation (KD) is a common approach to compress a teacher model to reduce
its inference cost and memory footprint, by training a smaller student model. However, in the …
its inference cost and memory footprint, by training a smaller student model. However, in the …
Mixkd: Towards efficient distillation of large-scale language models
Large-scale language models have recently demonstrated impressive empirical
performance. Nevertheless, the improved results are attained at the price of bigger models …
performance. Nevertheless, the improved results are attained at the price of bigger models …
On-policy distillation of language models: Learning from self-generated mistakes
Knowledge distillation (KD) is widely used for compressing a teacher model to reduce its
inference cost and memory footprint, by training a smaller student model. However, current …
inference cost and memory footprint, by training a smaller student model. However, current …
Distillm: Towards streamlined distillation for large language models
Knowledge distillation (KD) is widely used for compressing a teacher model to a smaller
student model, reducing its inference cost and memory footprint while preserving model …
student model, reducing its inference cost and memory footprint while preserving model …
MiniPLM: Knowledge Distillation for Pre-Training Language Models
Knowledge distillation (KD) is widely used to train small, high-performing student language
models (LMs) using large teacher LMs. While effective in fine-tuning, KD during pre-training …
models (LMs) using large teacher LMs. While effective in fine-tuning, KD during pre-training …
Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
Knowledge distillation (KD) is an effective model compression method that can transfer the
internal capabilities of large language models (LLMs) to smaller ones. However, the multi …
internal capabilities of large language models (LLMs) to smaller ones. However, the multi …
PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
Recent advancements in large language models (LLMs) have raised concerns about
inference costs, increasing the need for research into model compression. While knowledge …
inference costs, increasing the need for research into model compression. While knowledge …
Knowledge distillation of large language models
Knowledge Distillation (KD) is a promising technique for reducing the high computational
demand of large language models (LLMs). However, previous KD methods are primarily …
demand of large language models (LLMs). However, previous KD methods are primarily …
Dual-Space Knowledge Distillation for Large Language Models
Knowledge distillation (KD) is known as a promising solution to compress large language
models (LLMs) via transferring their knowledge to smaller models. During this process, white …
models (LLMs) via transferring their knowledge to smaller models. During this process, white …