Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning

Hock, Hung Chieng and Wahid, Noorhaniza and Ong, Pauline and Perla, Sai Raj Kishore (2018) Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning. International Journal of Advances in Intelligent Informatics, 4 (2). pp. 76-86. ISSN 2442-6571

Text AJ 2020 (102).pdf Restricted to Registered users only Download (1MB) | Request a copy

Abstract

Activation functions are essential for deep learning methods to learn and perform complex tasks such as image classification. Rectified Linear Unit (ReLU) has been widely used and become the default activation function across the deep learning community since 2012. Although ReLU has been popular, however, the hard zero property of the ReLU has heavily hindering the negative values from propagating through the network. Consequently, the deep neural network has not been benefited from the negative representations. In this work, an activation function called Flatten-T Swish (FTS) that leverage the benefit of the negative values is proposed. To verify its performance, this study evaluates FTS with ReLU and several recent activation functions. Each activation function is trained using MNIST dataset on five different deep fully connected neural networks (DFNNs) with depth vary from five to eight layers. For a fair evaluation, all DFNNs are using the same configuration settings. Based on the experimental results, FTS with a threshold value, T=-0.20 has the best overall performance. As compared with ReLU, FTS (T=-0.20) improves MNIST classification accuracy by 0.13%, 0.70%, 0.67%, 1.07% and 1.15% on wider 5 layers, slimmer 5 layers, 6 layers, 7 layers and 8 layers DFNNs respectively. Apart from this, the study also noticed that FTS converges twice as fast as ReLU. Although there are other existing activation functions are also evaluated, this study elects ReLU as the baseline activation function.

Item Type:	Article
Uncontrolled Keywords:	Deep learning; Activation function; Flatten-T Swish; Fully connected neural networks
Subjects:	T Technology > T Technology (General) T Technology > TA Engineering (General). Civil engineering (General) > TA1501-1820 Applied optics. Photonics
Divisions:	Faculty of Computer Science and Information Technology > Department of Multimedia
Depositing User:	Miss Afiqah Faiqah Mohd Hafiz
Date Deposited:	06 Jan 2022 07:30
Last Modified:	06 Jan 2022 07:30
URI:	http://eprints.uthm.edu.my/id/eprint/5227

Actions (login required)

View Item