Recent work by Power et al.(2022) highlighted a surprising" grokking" phenomenon in
learning arithmetic tasks: a neural net first" memorizes" the training set, resulting in perfect …

Existing theory predicts that data heterogeneity will degrade the performance of the
Federated Averaging (FedAvg) algorithm in federated learning. However, in practice, the …

Abstract We prove the Fast Equilibrium Conjecture proposed by Li et al.,(2020), ie,
stochastic gradient descent (SGD) on a scale-invariant loss (eg, using networks with various …

Decentralized stochastic gradient descent (D-SGD) allows collaborative learning on
massive devices simultaneously without the control of a central server. However, existing …

Large language models (LLM) have become a critical component in many applications of
machine learning. However, standard approaches to training LLM require a large number of …

Momentum is known to accelerate the convergence of gradient descent in strongly convex
settings without stochastic gradient noise. In stochastic optimization, such as training neural …

Bilevel optimization has various applications such as hyper-parameter optimization and
meta-learning. Designing theoretically efficient algorithms for bilevel optimization is more …

We investigate the generalization error of statistical learning models in a Federated
Learning (FL) setting. Specifically, we study the evolution of the generalization error with the …

Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is
an approach to distributed optimization where each device performs more than one SGD …

The federated learning paradigm has motivated the development of methods for
aggregating multiple client updates into a global server model, without sharing client data …