Modern deep neural networks, particularly recent large language models, come with
massive model sizes that require significant computational and storage resources. To …

Abstract Recently, Vision Transformer (ViT) has continuously established new milestones in
the computer vision field, while the high computation and memory cost makes its …

The deployment constraints in practical applications necessitate the pruning of large-scale
deep learning models, ie, promoting their weight sparsity. As illustrated by the Lottery Ticket …

Although deep learning has made great progress in recent years, the exploding economic
and environmental costs of training neural networks are becoming unsustainable. To …

Many recent works have shown trainability plays a central role in neural network pruning--
unattended broken trainability can lead to severe under-performance and unintentionally …

Sparse training is a popular technique to reduce the overhead of training large models.
Although previous work has shown promising results for nonstructured sparse models, it is …

As real-world graphs expand in size, larger GNN models with billions of parameters are
deployed. High parameter count in such models makes training and inference on graphs …

Sparse training has received an upsurging interest in machine learning due to its tantalizing
saving potential for both the entire training process as well as the inference. Dynamic sparse …

Lightweight speech recognition models have seen explosive demands owing to a growing
amount of speech-interactive features on mobile devices. Since designing such systems …

Recent studies demonstrate that deep networks, even robustified by the state-of-the-art
adversarial training (AT), still suffer from large robust generalization gaps, in addition to the …