Abstract Sharpness-Aware Minimization (SAM) is a recent training method that relies on
worst-case weight perturbations which significantly improves generalization in various …

The recently proposed Sharpness-Aware Minimization (SAM) improves generalization by
minimizing a\textit {perturbed loss} defined as the maximum loss within a neighborhood in …

Federated learning methods typically learn a model by iteratively sampling updates from a
population of clients. In this work, we explore how the number of clients sampled at each …

Decentralized training of deep learning models is a key element for enabling data privacy
and on-device learning over networks. In realistic learning scenarios, the presence of …

We study the generalization properties of the popular stochastic optimization method known
as stochastic gradient descent (SGD) for optimizing general non-convex loss functions. Our …

Deep learning is becoming a standard tool in chemistry and materials science. Although
there are learning materials available for deep learning, none cover the applications in …

Continual lifelong learning is an machine learning framework inspired by human learning,
where learners are trained to continuously acquire new knowledge in a sequential manner …

Local SGD is a communication-efficient variant of SGD for large-scale training, where
multiple GPUs perform SGD independently and average the model parameters periodically …

A major obstacle to achieving global convergence in distributed and federated learning is
the misalignment of gradients across clients or mini-batches due to heterogeneity and …

We consider synchronous data-parallel neural network training with a fixed large batch size.
While the large batch size provides a high degree of parallelism, it degrades the …