Enhancing sharpness-aware optimization through variance suppression

B Li, G Giannakis - Advances in Neural Information …, 2024 - proceedings.neurips.cc
Sharpness-aware minimization (SAM) has well documented merits in enhancing
generalization of deep neural networks, even without sizable data augmentation. Embracing …

Adaptive stochastic variance reduction for non-convex finite-sum minimization

A Kavis, S Skoulakis… - Advances in …, 2022 - proceedings.neurips.cc
We propose an adaptive variance-reduction method, called AdaSpider, for minimization of $
L $-smooth, non-convex functions with a finite-sum structure. In essence, AdaSpider …

Decentralized TD tracking with linear function approximation and its finite-time analysis

G Wang, S Lu, G Giannakis… - Advances in neural …, 2020 - proceedings.neurips.cc
The present contribution deals with decentralized policy evaluation in multi-agent Markov
decision processes using temporal-difference (TD) methods with linear function …

A momentum-guided Frank-Wolfe algorithm

B Li, M Coutino, GB Giannakis… - IEEE Transactions on …, 2021 - ieeexplore.ieee.org
With the well-documented popularity of Frank Wolfe (FW) algorithms in machine learning
tasks, the present paper establishes links between FW subproblems and the notion of …

SVRG meets adagrad: Painless variance reduction

B Dubois-Taine, S Vaswani, R Babanezhad… - Machine Learning, 2022 - Springer
Variance reduction (VR) methods for finite-sum minimization typically require the knowledge
of problem-dependent constants that are often unknown and difficult to estimate. To address …

Adaptive accelerated (extra-) gradient methods with variance reduction

Z Liu, TD Nguyen, A Ene… - … Conference on Machine …, 2022 - proceedings.mlr.press
In this paper, we study the finite-sum convex optimization problem focusing on the general
convex case. Recently, the study of variance reduced (VR) methods and their accelerated …

On the convergence of SARAH and beyond

B Li, M Ma, GB Giannakis - International Conference on …, 2020 - proceedings.mlr.press
The main theme of this work is a unifying algorithm,\textbf {L} oop\textbf {L} ess\textbf {S}
ARAH (L2S) for problems formulated as summation of $ n $ individual loss functions. L2S …

Finite-sum optimization: Adaptivity to smoothness and loopless variance reduction

B Batardière, J Kwon - arXiv preprint arXiv:2307.12615, 2023 - arxiv.org
For finite-sum optimization, variance-reduced gradient methods (VR) compute at each
iteration the gradient of a single function (or of a mini-batch), and yet achieve faster …

Communication-efficient robust federated learning over heterogeneous datasets

Y Dong, GB Giannakis, T Chen, J Cheng… - arXiv preprint arXiv …, 2020 - arxiv.org
This work investigates fault-resilient federated learning when the data samples are non-
uniformly distributed across workers, and the number of faulty workers is unknown to the …

Enhancing parameter-free Frank Wolfe with an extra subproblem

B Li, L Wang, GB Giannakis, Z Zhao - Proceedings of the AAAI …, 2021 - ojs.aaai.org
Aiming at convex optimization under structural constraints, this work introduces and
analyzes a variant of the Frank Wolfe (FW) algorithm termed ExtraFW. The distinct feature of …