International Journal of Scientific & Technology Research

Home About Us Scope Editorial Board Blog/Latest News Contact Us
10th percentile
Powered by  Scopus
Scopus coverage:
Nov 2018 to May 2020


IJSTR >> Volume 9 - Issue 6, June 2020 Edition

International Journal of Scientific & Technology Research  
International Journal of Scientific & Technology Research

Website: http://www.ijstr.org

ISSN 2277-8616

Adaptive Activation Functions For Artificial Neural Networks

[Full Text]



Marakhimov A.R., Khudaybergenov K.K., Ohundadaev U.R.



artificial neural networks, classification, activation function, adaptive activation, convolution.



Activation functions are considered as main component in artificial neural networks. The current paper considers learning activation functions with combination of activation functions. We propose two approaches to use activation functions and construction of adaptive activation parameters to input data. Namely, to show effectiveness, we investigate linear form and non-linear form to combine activation functions, then introduce adaptive activation function. Numerical experiments show the proposed activation techniques overcome by performances and accuracy than standard rectified unit family functions.



[1] Guo Y., Liu Y., Oerlemans A., Lao S., Wu S., Lew M.S. Deep learning for visual understanding: a review, Neurocomputing, Vol. 187, 27–48 (2016).
[2] Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks, Advances In Neural Information Processing Systems, Vol. 25, 1106–1114 (2012).
[3] Li X., Cai C., Zhang R., Ju L., He J. Deep cascaded convolutional models for cattle pose estimation, Computers and Electronics in Agriculture, Vol. 164, 45-67 (2019).
[4] Gu G., Liu J., Li Z., Huo W., Zhao Y. Joint learning based deep supervised hashing for large-scale image retrieval, Neurocomputing, Vol. 385, 348-357 (2020).
[5] Yang J., Zhang D., Frangi A., Yang J. Two-dimensional PCA a new approach to appearance-based face representation and recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, Issue 1, 131-137 (2004).
[6] Taigman Y., Yang M., Ranzato M., Wolf L. Deepface: Closing the gap to human-level performance in face verification, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, U.S.A, 1701-1708 (2014).
[7] Li C., Chen Z., Wu Q. M., Liu C. Deep saliency detection via channel-wise hierarchical feature responses, Neurocomputing, Vol. 322, 80-92 (2018).
[8] Tuo Q., Zhao H., Hu Q. Hierarchical feature selection with subtree based graph regularization, Knowledge-Based Systems, Vol. 163, 996-1008 (2019).
[9] Wu G., Lu W., Gao G., Zhao C., Liu J. Regional deep learning model for visual tracking, Neurocomputing, Vol. 175, 310–323 (2016).
[10] An S., Boussaid F., Bennamoun M., Sohel F. Exploiting layerwise convexity of rectifier networks with sign constrained weights, Neural Networks, Vol. 105, 419-430 (2018).
[11] Apicella A., Isgro F., Prevete R. A simple and efficient architecture for trainable activation functions, Neurocomputing, Vol. 370, 1-15 (2019).
[12] He K., Zhang X., Ren S., Sun J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification, The IEEE International Conference on Computer Vision, 1026-1034 (2015).
[13] Li Y., Fan C., Li Y., Wu Q., Ming Y. Improving deep neural network with Multiple Parametric Exponential Linear Units, Neurocomputing, Vol. 301, 11-24 (2018).
[14] Glorot X., Bordes A., Bengio Y. Deep sparse rectifier neural networks. Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, FL, USA. Vol. 15, 315-323 (2011).
[15] Marakhimov A. R., Khudaybergenov K. K. A fuzzy MLP approach for identification of nonlinear systems, Contemporary problems in mathematics and physics, CMFD, Vol. 65, no. 1, Peoples' Friendship University of Russia, M., 44–53 (2019).
[16] Marakhimov A.R., Khudaybergenov K.K. Convergence analysis of feedforward neural networks with backpropagation, Bulletin of National University of Uzbekistan: Mathematics and Natural Sciences: Vol. 2, Issue 2, 77-93 (2019), Available at: https://uzjournals.edu.uz/mns_nuu/vol2/iss2/1
[17] Yusupbekov N. R., Marakhimov A. R., Igamberdiev H. Z., Umarov Sh. X. An Adaptive Fuzzy-Logic Traffic Control System in Conditions of Saturated Transport Stream, The Scientific World Journal Vol. 2016, 23-36 (2016).
[18] Yusupbekov N.R., Marakhimov A.R., Igamberdiev H.Z., Umarov Sh.X. Application of soft-computing technologies to the traffic control system design problems. 12th International Conference on Application of Fuzzy Systems and Soft Computing, ICAFS 2016, 29-30 August, Vienna, Austria (2016).
[19] Marakhimov A.R., Siddikov I.H., Nasridinov A., Byun J.Y. Structural Synthesis of Information Computer Networks of Automated Control Systems Based on Genetic Algorithms, Computer Science and its Applications, Vol. 330, 1055-1063 (2015).
[20] Nasridinov A., Marakhimov A. Park Y.H. A design of wireless sensor networks based on fuzzy modeling for comfortable human life, Asia Live Sciences, The Asian International Journal of Life Sciences, July 2015, Philippines, 265-277.
[21] Yusupbekov N.R., Marakhimov A.R. Synthesis of the intelligent traffic control systems in conditions saturated transport stream, International Journal of International Journal of Chemical Technology, Control and Management Jointly with The Journal of Korea Multimedia Society. Special Issue, South Korea, Seoul, 12-18 (2015).
[22] Jagtap D.A., Kawaguchi K., Karniadakis G.E. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks, Journal of Computational Physics, Vol. 404, 45-67 (2020).
[23] Konstantinidis D., Argyriou V., Stathaki T., Grammalidis N. A modular CNN-based building detector for remote sensing images, Computer Networks, Vol. 168, 93-121 (2020).
[24] Jiang W., Wu L., Liu S., Liu M. CNN-based two-stage cell segmentation improves plant cell tracking, Pattern Recognition Letters, Vol. 128, 311-317 (2019).
[25] Xu Z., Zhao J., Yu Y., Zeng H. Improved 1D-CNNs for behavior recognition using wearable sensor network, Computer Communications, Vol. 15, Issue 11, 165-171 (2020).
[26] Amin S. U., Alsulaiman M., Muhammad G., Mekhtiche M. A., Hossain M. S. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion, Future Generation Computer Systems, Vol. 101, 542-554 (2019).
[27] LeCun Y., Bottou L., Bengio Y., Haffner P. Gradient-based learning applied to document recognition, IEEE 86, Vol. 11, 2278–2324 (1998).
[28] Krizhevsky A., Learning multiple layers of features from tiny images. Technical report, University of Toronto, (2009).
[29] J. Deng, W. Dong, R. Socher, L. Li, K. Li, F. Li, Imagenet: a large-scale hierarchical image database, Conference: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, Florida, USA, 20-25 June 2009.
[30] MNST Dataset, available at: http://yann.lecun.com/exdb/mnist/
[31] CIFAR Dataset, available at: https://www.cs.toronto.edu/~kriz/cifar.html
[32] ImageNet Dataset, available at: http://image-net.org/download
[33] https://en.wikipedia.org/wiki/Activation_function.