Zeroth-order (ZO) optimization is a subset of gradient-free optimization that emerges in many
signal processing and machine learning (ML) applications. It is used for solving optimization …

Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but
as LMs grow in size, backpropagation requires a prohibitively large amount of memory …

The goal of a decision-based adversarial attack on a trained model is to generate
adversarial examples based solely on observing output labels returned by the targeted …

In many optimization problems arising from scientific, engineering and artificial intelligence
applications, objective and constraint functions are available only as the output of a black …

With the surge of large-scale pre-trained models (PTMs), fine-tuning these models to
numerous downstream tasks becomes a crucial problem. Consequently, parameter efficient …

Nonsmooth nonconvex optimization problems broadly emerge in machine learning and
business decision making, whereas two core challenges impede the development of …

Abstract Stochastic Gradient Descent (SGD) is arguably the most important single algorithm
in modern machine learning. Although SGD with unbiased gradient estimators has been …

In this paper, we analyze several methods for approximating gradients of noisy functions
using only function values. These methods include finite differences, linear interpolation …

Current transfer learning methods are mainly based on finetuning a pretrained model with
target-domain data. Motivated by the techniques from adversarial machine learning (ML) …

The adaptive momentum method (AdaMM), which uses past gradients to update descent
directions and learning rates simultaneously, has become one of the most popular first-order …