Ammus: A survey of transformer-based pretrained models in natural language processing

KS Kalyan, A Rajasekharan, S Sangeetha - arXiv preprint arXiv …, 2021 - arxiv.org
Transformer-based pretrained language models (T-PTLMs) have achieved great success in
almost every NLP task. The evolution of these models started with GPT and BERT. These …

Structured pruning for deep convolutional neural networks: A survey

Y He, L Xiao - IEEE transactions on pattern analysis and …, 2023 - ieeexplore.ieee.org
The remarkable performance of deep Convolutional neural networks (CNNs) is generally
attributed to their deeper and wider architectures, which can come with significant …

A review of vision-based traffic semantic understanding in ITSs

J Chen, Q Wang, HH Cheng, W Peng… - IEEE Transactions on …, 2022 - ieeexplore.ieee.org
A semantic understanding of road traffic can help people understand road traffic flow
situations and emergencies more accurately and provide a more accurate basis for anomaly …

Emotion recognition in EEG signals using deep learning methods: A review

M Jafari, A Shoeibi, M Khodatars… - Computers in Biology …, 2023 - Elsevier
Emotions are a critical aspect of daily life and serve a crucial role in human decision-making,
planning, reasoning, and other mental states. As a result, they are considered a significant …

A survey of active learning for natural language processing

Z Zhang, E Strubell, E Hovy - arXiv preprint arXiv:2210.10109, 2022 - arxiv.org
In this work, we provide a survey of active learning (AL) for its applications in natural
language processing (NLP). In addition to a fine-grained categorization of query strategies …

No train no gain: Revisiting efficient training algorithms for transformer-based language models

J Kaddour, O Key, P Nawrot… - Advances in Neural …, 2024 - proceedings.neurips.cc
The computation necessary for training Transformer-based language models has
skyrocketed in recent years. This trend has motivated research on efficient training …

Spiking neural networks: A survey

JD Nunes, M Carvalho, D Carneiro, JS Cardoso - IEEE Access, 2022 - ieeexplore.ieee.org
The field of Deep Learning (DL) has seen a remarkable series of developments with
increasingly accurate and robust algorithms. However, the increase in performance has …

Deep learning: Systematic review, models, challenges, and research directions

T Talaei Khoei, H Ould Slimane… - Neural Computing and …, 2023 - Springer
The current development in deep learning is witnessing an exponential transition into
automation applications. This automation transition can provide a promising framework for …

The efficiency misnomer

M Dehghani, A Arnab, L Beyer, A Vaswani… - arXiv preprint arXiv …, 2021 - arxiv.org
Model efficiency is a critical aspect of developing and deploying machine learning models.
Inference time and latency directly affect the user experience, and some applications have …

Compute-efficient deep learning: Algorithmic trends and opportunities

BR Bartoldson, B Kailkhura, D Blalock - Journal of Machine Learning …, 2023 - jmlr.org
Although deep learning has made great progress in recent years, the exploding economic
and environmental costs of training neural networks are becoming unsustainable. To …