Language model training in distributed settings is limited by the communication cost of
gradient exchanges. In this short note, we extend recent work from Malladi et al.(2023) …

The fundamental success of large language models hinges upon the efficacious
implementation of large-scale distributed training techniques. Nevertheless, building a vast …

Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is
an approach to distributed optimization where each device performs more than one SGD …

Large language models (LLM) have become a critical component in many applications of
machine learning. However, standard approaches to training LLM require a large number of …

Pre-trained large language models (LLMs) require fine-tuning to improve their
responsiveness to natural language instructions. Federated learning (FL) offers a way to …

Transfer learning via fine-tuning pre-trained transformer models has gained significant
success in delivering state-of-the-art results across various NLP tasks. In the absence of …

Large language models (LLMs) are useful in many NLP tasks and become more capable
with size, with the best open-source models having over 50 billion parameters. However …

Large language models (LLMs) power a new generation of interactive AI applications
exemplified by ChatGPT. The interactive nature of these applications demands low latency …

Modern large-scale machine learning applications require stochastic optimization
algorithms to be implemented on distributed compute systems. A key bottleneck of such …

The confluence of Federated Learning (FL) and Large Language Models (LLMs) is ushering
in a new era in privacy-preserving natural language processing. However, the intensive …