Decentralized optimization is emerging as a viable alternative for scalable distributed
machine learning, but also introduces new challenges in terms of synchronization costs. To …

Communication-reduction techniques are a popular way to improve scalability in data-
parallel training of deep neural networks (DNNs). The recent emergence of large language …

Network consensus optimization has received increasing attention in recent years and has
found important applications in many scientific and engineering fields. To solve network …

We propose and analyze a new type of stochastic first order method: gradient descent with
compressed iterates (GDCI). GDCI in each iteration first compresses the current iterate using …

This article develops a communication-efficient algorithm to solve the stochastic optimization
problem defined over a distributed network, aiming at reducing the burdensome …

In this paper, we develop a quantized and communication-censored alternating direction
method of multipliers (ADMM) to solve a dynamic optimization problem defined over a …

The scalability of concurrent data structures and distributed algorithms strongly depends on
reducing the contention for shared resources and the costs of synchronization and …

The ability to scale distributed optimization to large node counts has been one of the main
enablers of recent progress in machine learning. To this end, several techniques have been …

Distributed optimization has become one of the standard ways of speeding up machine
learning training, and most of the research in the area focuses on distributed first-order …

Federated learning is an emerging machine learning setting, which can train a shared
model on large amounts of decentralized data while protecting data privacy. However, the …