The softmax operator is a crucial component of large language models (LLMs), which have
played a transformative role in computer research. Due to the centrality of the softmax …

Pre-trained large language models (LLMs) require fine-tuning to improve their
responsiveness to natural language instructions. Federated learning (FL) offers a way to …

Large language models (LLMs) have brought about significant transformations in human
society. Among the crucial computations in LLMs, the softmax unit holds great importance …

The widespread practice of fine-tuning pretrained large language models (LLMs) on domain-
specific data faces two major challenges in memory and privacy. First, as the size of LLMs …

When personalized federated learning (FL) meets large foundation models, new challenges
arise from various limitations in resources. In addition to typical limitations such as data …

Since the invention of GPT2--1.5 B in 2019, large language models (LLMs) have
transitioned from specialized models to versatile foundation models. The LLMs exhibit …

Large Language Models (LLMs) have become indispensable in numerous real-world
applications. Unfortunately, fine-tuning these models at scale, especially in federated …

We introduce CYBER-0, the first zero-order optimization algorithm for memory-and-
communication efficient Federated Learning, resilient to Byzantine faults. We show through …

Large foundation models, such as large language models, have performed exceptionally
well in various application scenarios. Building or fully fine-tuning such large models is …

Fine-tuning large language models (LLM) can be costly. Parameter-efficient fine-tuning
(PEFT) addresses the problems by training a fraction of the parameters, whose success …