Abstract Low-Rank Adaptation (LoRA), which updates the dense neural network layers with
pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning …

Estimating the uncertainty of responses of Large Language Models~(LLMs) remains a
critical challenge. While recent Bayesian methods have demonstrated effectiveness in …

Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their
ability to avoid excessive inference costs. However, LoRA still encounters the following …

To adapt a well-trained large model to downstream tasks, we propose constraining learning
within its original latent space by leveraging linear combinations of its basis vectors. This …

Fine-tuning pre-trained models is crucial for adapting large models to downstream tasks,
often delivering state-of-the-art performance. However, fine-tuning all model parameters is …

Recent advances in Large Language Models (LLMs) have led to significant improvements in
natural language processing tasks, but their ability to generate human-quality text raises …

Self-adaptive large language models (LLMs) aim to solve the challenges posed by
traditional fine-tuning methods, which are often computationally intensive and static in their …

Large language models (LLMs) exhibit remarkable capabilities in natural language
processing but face catastrophic forgetting when learning new tasks, where adaptation to a …

Low-rank adaptation (LoRA) is a powerful parameter-efficient fine-tuning method that utilizes
low-rank projectors $ A $ and $ B $ to learn weight updates $\Delta W $ for adaptation …

Previous research has shown that the principal singular vectors of a pre-trained model's
weight matrices capture critical knowledge. In contrast, those associated with small singular …