Convergence analysis of sequential federated learning on heterogeneous data
Y Li, X Lyu - Advances in Neural Information Processing …, 2024 - proceedings.neurips.cc
There are two categories of methods in Federated Learning (FL) for joint training across
multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) …
multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) …
On the convergence of federated averaging with cyclic client participation
Abstract Federated Averaging (FedAvg) and its variants are the most popular optimization
algorithms in federated learning (FL). Previous convergence analyses of FedAvg either …
algorithms in federated learning (FL). Previous convergence analyses of FedAvg either …
Tighter lower bounds for shuffling SGD: Random permutations and beyond
We study convergence lower bounds of without-replacement stochastic gradient descent
(SGD) for solving smooth (strongly-) convex finite-sum minimization problems. Unlike most …
(SGD) for solving smooth (strongly-) convex finite-sum minimization problems. Unlike most …
Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond
In distributed learning, local SGD (also known as federated averaging) and its simple
baseline minibatch SGD are widely studied optimization methods. Most existing analyses of …
baseline minibatch SGD are widely studied optimization methods. Most existing analyses of …
Federated optimization algorithms with random reshuffling and gradient compression
Gradient compression is a popular technique for improving communication complexity of
stochastic first-order methods in distributed training of machine learning models. However …
stochastic first-order methods in distributed training of machine learning models. However …
Benign underfitting of stochastic gradient descent
We study to what extent may stochastic gradient descent (SGD) be understood as
a``conventional''learning rule that achieves generalization performance by obtaining a good …
a``conventional''learning rule that achieves generalization performance by obtaining a good …
A general analysis of example-selection for stochastic gradient descent
Training example order in SGD has long been known to affect convergence rate. Recent
results show that accelerated rates are possible in a variety of cases for permutation-based …
results show that accelerated rates are possible in a variety of cases for permutation-based …
Convergence Analysis of Sequential Split Learning on Heterogeneous Data
Y Li, X Lyu - arXiv preprint arXiv:2302.01633, 2023 - arxiv.org
Federated Learning (FL) and Split Learning (SL) are two popular paradigms of distributed
machine learning. By offloading the computation-intensive portions to the server, SL is …
machine learning. By offloading the computation-intensive portions to the server, SL is …
Permutation-Based SGD: Is Random Optimal?
A recent line of ground-breaking results for permutation-based SGD has corroborated a
widely observed phenomenon: random permutations offer faster convergence than with …
widely observed phenomenon: random permutations offer faster convergence than with …
Mini-Batch Optimization of Contrastive Loss
Contrastive learning has gained significant attention as a method for self-supervised
learning. The contrastive loss function ensures that embeddings of positive sample pairs …
learning. The contrastive loss function ensures that embeddings of positive sample pairs …