Xai-driven knowledge distillation of large language models for efficient deployment on low-resource devices
Abstract Large Language Models (LLMs) are characterized by their inherent memory
inefficiency and compute-intensive nature, making them impractical to run on low-resource …
inefficiency and compute-intensive nature, making them impractical to run on low-resource …
Empowering Compact Language Models with Knowledge Distillation
AR Junaid - Authorea Preprints, 2025 - techrxiv.org
Large language models (LLMs) have revolutionized the field of artificial intelligence,
achieving unprecedented performance in tasks such as text generation, translation, and …
achieving unprecedented performance in tasks such as text generation, translation, and …
Ddk: Distilling domain knowledge for efficient large language models
Despite the advanced intelligence abilities of large language models (LLMs) in various
applications, they still face significant computational and storage demands. Knowledge …
applications, they still face significant computational and storage demands. Knowledge …
Survey on knowledge distillation for large language models: methods, evaluation, and application
Large Language Models (LLMs) have showcased exceptional capabilities in various
domains, attracting significant interest from both academia and industry. Despite their …
domains, attracting significant interest from both academia and industry. Despite their …
Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
X Zhu, J Li, C Ma, W Wang - arXiv preprint arXiv:2411.14698, 2024 - arxiv.org
Large Language Models (LLMs) demonstrate exceptional reasoning capabilities, often
achieving state-of-the-art performance in various tasks. However, their substantial …
achieving state-of-the-art performance in various tasks. However, their substantial …
A survey on knowledge distillation of large language models
This survey presents an in-depth exploration of knowledge distillation (KD) techniques
within the realm of Large Language Models (LLMs), spotlighting the pivotal role of KD in …
within the realm of Large Language Models (LLMs), spotlighting the pivotal role of KD in …
WisdomBot: Tuning Large Language Models with Artificial Intelligence Knowledge
Large language models (LLMs) have emerged as powerful tools in natural language
processing (NLP), showing a promising future of artificial generated intelligence (AGI) …
processing (NLP), showing a promising future of artificial generated intelligence (AGI) …
Sparse Mixture of Experts Language Models Excel in Knowledge Distillation
H Xu, H Liu, W Gong, X Deng, H Wang - CCF International Conference on …, 2024 - Springer
Abstract Knowledge distillation is an effective method for reducing the computational
overhead of large language models. However, recent optimization efforts in distilling large …
overhead of large language models. However, recent optimization efforts in distilling large …
Mixed distillation helps smaller language model better reasoning
While large language models (LLMs) have demonstrated exceptional performance in recent
natural language processing (NLP) tasks, their deployment poses substantial challenges …
natural language processing (NLP) tasks, their deployment poses substantial challenges …
Gkd: A general knowledge distillation framework for large-scale pre-trained language model
Currently, the reduction in the parameter scale of large-scale pre-trained language models
(PLMs) through knowledge distillation has greatly facilitated their widespread deployment …
(PLMs) through knowledge distillation has greatly facilitated their widespread deployment …