Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

V Kunc, J Kléma - arXiv preprint arXiv:2402.09092, 2024 - arxiv.org
Neural networks have proven to be a highly effective tool for solving complex problems in
many areas of life. Recently, their importance and practical usability have further been …

SMU: smooth activation function for deep networks using smoothing maximum technique

K Biswas, S Kumar, S Banerjee, AK Pandey - arXiv preprint arXiv …, 2021 - arxiv.org
Deep learning researchers have a keen interest in proposing two new novel activation
functions which can boost network performance. A good choice of activation function can …

[HTML][HTML] Deep learning model for regional solar radiation estimation using satellite images

EO Yuzer, A Bozkurt - Ain Shams Engineering Journal, 2023 - Elsevier
Solar energy is one of the most prominent renewable energy sources today and it can
contribute to the energy market with accurate solar radiation estimation. This study proposes …

Catalysis of neural activation functions: Adaptive feed-forward training for big data applications

S Sarkar, S Agrawal, T Baker, PKR Maddikunta… - Applied …, 2022 - Springer
Deep Learning in the field of Big Data has become essential for the analysis and perception
of trends. Activation functions play a crucial role in the outcome of these deep learning …

PolyLU: A simple and robust polynomial-based linear unit activation function for deep learning

HS Feng, CH Yang - IEEE Access, 2023 - ieeexplore.ieee.org
The activation function has a critical influence on whether a convolutional neural network in
deep learning can converge or not; a proper activation function not only makes the …

Adjustable memory-efficient image super-resolution via individual kernel sparsity

X Luo, M Dai, Y Zhang, Y Xie, D Liu, Y Qu… - Proceedings of the 30th …, 2022 - dl.acm.org
Though single image super-resolution (SR) has witnessed incredible progress, the
increasing model complexity impairs its applications in memory-limited devices. To solve …

Parametric RSigELU: a new trainable activation function for deep learning

S Kiliçarslan, M Celik - Neural Computing and Applications, 2024 - Springer
Activation functions are used to extract meaningful relationships from real-world problems
with the help of deep learning models. Thus, the development of activation functions which …

SAU: Smooth activation function using convolution with approximate identities

K Biswas, S Kumar, S Banerjee… - European Conference on …, 2022 - Springer
Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the
origin. Over the years, many smooth approximations of ReLU have been proposed using …

[HTML][HTML] Compressed Deep Learning Models for Wearable Atrial Fibrillation Detection through Attention

M Mäkynen, GA Ng, X Li, FS Schlindwein, TC Pearce - Sensors, 2024 - mdpi.com
Deep learning (DL) models have shown promise for the accurate detection of atrial
fibrillation (AF) from electrocardiogram/photoplethysmography (ECG/PPG) data, yet …

ErfReLU: adaptive activation function for deep neural network

A Rajanand, P Singh - Pattern Analysis and Applications, 2024 - Springer
Recent research has found that the activation function (AF) plays a significant role in
introducing non-linearity to enhance the performance of deep learning networks …