Demystifying softmax gating function in Gaussian mixture of experts
Understanding the parameter estimation of softmax gating Gaussian mixture of experts has
remained a long-standing open problem in the literature. It is mainly due to three …
remained a long-standing open problem in the literature. It is mainly due to three …
Sharp global convergence guarantees for iterative nonconvex optimization with random data
KA Chandrasekher, A Pananjady… - The Annals of …, 2023 - projecteuclid.org
Sharp global convergence guarantees for iterative nonconvex optimization with random data
Page 1 The Annals of Statistics 2023, Vol. 51, No. 1, 179–210 https://doi.org/10.1214/22-AOS2246 …
Page 1 The Annals of Statistics 2023, Vol. 51, No. 1, 179–210 https://doi.org/10.1214/22-AOS2246 …
Statistical perspective of top-k sparse softmax gating mixture of experts
Top-K sparse softmax gating mixture of experts has been widely used for scaling up massive
deep-learning architectures without increasing the computational cost. Despite its popularity …
deep-learning architectures without increasing the computational cost. Despite its popularity …
Improving transformer with an admixture of attention heads
Transformers with multi-head self-attention have achieved remarkable success in sequence
modeling and beyond. However, they suffer from high computational and memory …
modeling and beyond. However, they suffer from high computational and memory …
Avoiding inferior clusterings with misspecified Gaussian mixture models
Clustering is a fundamental tool for exploratory data analysis, and is ubiquitous across
scientific disciplines. Gaussian Mixture Model (GMM) is a popular probabilistic and …
scientific disciplines. Gaussian Mixture Model (GMM) is a popular probabilistic and …
A doubly enhanced em algorithm for model-based tensor clustering
Modern scientific studies often collect datasets in the form of tensors. These datasets call for
innovative statistical analysis methods. In particular, there is a pressing need for tensor …
innovative statistical analysis methods. In particular, there is a pressing need for tensor …
On the computational and statistical complexity of over-parameterized matrix sensing
We consider solving the low-rank matrix sensing problem with the Factorized Gradient
Descent (FGD) method when the specified rank is larger than the true rank. We refer to this …
Descent (FGD) method when the specified rank is larger than the true rank. We refer to this …
Randomly initialized EM algorithm for two-component Gaussian mixture achieves near optimality in iterations
We analyze the classical EM algorithm for parameter estimation in the symmetric two-
component Gaussian mixtures in d dimensions. We show that, even in the absence of any …
component Gaussian mixtures in d dimensions. We show that, even in the absence of any …
On the minimax optimality of the EM algorithm for learning two-component mixed linear regression
We study the convergence rates of the EM algorithm for learning two-component mixed
linear regression under all regimes of signal-to-noise ratio (SNR). We resolve a long …
linear regression under all regimes of signal-to-noise ratio (SNR). We resolve a long …
Refined convergence rates for maximum likelihood estimation under finite mixture models
We revisit the classical problem of deriving convergence rates for the maximum likelihood
estimator (MLE) in finite mixture models. The Wasserstein distance has become a standard …
estimator (MLE) in finite mixture models. The Wasserstein distance has become a standard …