Few-shot in-context learning (ICL) enables pre-trained language models to perform a
previously-unseen task without any gradient-based training by feeding a small number of …

Recent work in natural language processing (NLP) has yielded appealing results from
scaling model parameters and training data; however, using only scale to improve …

Large pretrained language models are widely used in downstream NLP tasks via task-
specific fine-tuning, but such procedures can be costly. Recently, Parameter-Efficient Fine …

Soft prompt tuning is a widely studied parameter-efficient fine-tuning method. However, it
has a clear drawback: many soft tokens must be inserted into the input sequences to …

Recently, a series of works have looked into further improving the adapter-based tuning by
manually designing better adapter architectures. Understandably, these manually designed …

Adapter tuning, which updates only a few parameters, has become a mainstream method for
fine-tuning pretrained language models to downstream tasks. However, it often yields …

Adapting large pre-trained models (PTMs) through fine-tuning imposes prohibitive
computational and storage burdens. Recent studies of delta tuning (DT), ie, parameter …

Parameter-efficient fine-tuning (PEFT) is widely studied for its effectiveness and efficiency in
the era of large language models. Low-rank adaptation (LoRA) has demonstrated …

Large pretrained multilingual models, trained on dozens of languages, have delivered
promising results due to cross-lingual learning capabilities on variety of language tasks …

The advent of transformers with attention mechanisms and associated pre-trained models
have revolutionized the field of Natural Language Processing (NLP). However, such models …