Momentum is known to accelerate the convergence of gradient descent in strongly convex
settings without stochastic gradient noise. In stochastic optimization, such as training neural …

Due to the broad usage and widespread popularity of drones, the demand for a more
accurate object detection algorithm for images captured by drone platforms has become …

Learning Rate Warmup is a popular heuristic for training neural networks, especially at
larger batch sizes, despite limited understanding of its benefits. Warmup decreases the …

While stochastic gradient descent (SGD) with momentum has fast convergence and
excellent generalizability, a theoretical explanation for this is lacking. In this paper, we show …

ABSTRACT∗ In fields such as groundwater extraction, geothermal energy, nuclear waste
storage, and geoenergy exploration, borehole imaging is crucial for understanding fluid …

Learning Rate Warmup is a popular heuristic for training neural networks, which downscales
early updates relative to later ones. This aids training, suggesting that the initial updates are …

Although gradient descent with Polyak's momentum is widely used in modern machine and
deep learning, a concrete understanding of its effects on the training trajectory remains …