IJSTR >> Volume 9 - Issue 5, May 2020 Edition

[Full Text]

AUTHOR(S)

El houssaine HSSAYNI , Mohamed ETTAOUIL

KEYWORDS

Convolutional Neural Networks, Deep Neural Networks, Generalization Ability, l^(1/2) Regularization, Pooling methods, Regularization methods.

ABSTRACT

As powerful visual models, deep learning models, in particular, deep convolutional neural networks(DCNNs) have demonstrated remarkable performance in various challenging artificial intelligence and machine learning tasks and attracted considerable interests in recent years. A pooling process plays a very important role in deep convolutional neural networks, which serves to reduce the dimensionality of processed data for decreasing computational cost as well as for avoiding overfitting and improving the generalization capability of the network. Although standard pooling techniques, such as the max pooling and the l^p pooling (where p≥1 ) are typically adopted in various studies, we alternatively propose, in this paper, a new pooling method named l^(1/2) pooling in order to improve the generalization capability of DCNNs. Experimental results on two image benchmarks indicate that l^(1/2) pooling outperforms the existing pooling techniques in classification performance as well as is efficient for enhancing the generalization capability of DCNNs. Moreover, we show that the l^(1/2)pooling combined with other regularization methods, such as dropout and batch normalization, is competitive with other existing strategies in classification performance.

REFERENCES

[1]. W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, “A survey of deep neural network architectures and their applications,” Neurocomputing, vol. 234, pp. 11–26, 2017.
[2]. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[3]. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[4]. G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” IEEE Transactions on audio, speech, and language processing, vol. 20, no. 1, pp. 30–42, 2011.
[5]. W. Chen and K. Shi, “A deep learning framework for time series classification using relative position matrix and convolutional neural network,” Neurocomputing, vol. 359, pp. 384–394, 2019.
[6]. D. Scherer, A. Muller, and S. Behnke, “Evaluation of pooling operations in convolutional architectures for object recognition,” in¨ International conference on artificial neural networks. Springer, 2010, pp. 92–101.
[7]. M. Sun, Z. Song, X. Jiang, J. Pan, and Y. Pang, “Learning pooling for convolutional neural network,” Neurocomputing, vol. 224, pp. 96–104, 2017.
[8]. M. Ranzato, Y.-L. Boureau, and Y. LeCun, “Sparse feature learning for deep belief networks,” in Advances in neural information processing systems, 2008, pp. 1185–1192.
[9]. M. D. Zeiler and R. Fergus, “Stochastic pooling for regularization of deep convolutional neural networks,” arXiv preprint arXiv:1301.3557, 2013.
[10]. Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D. Jackel, “Handwritten digit recognition with a backpropagation network,” in Advances in neural information processing systems, 1990, pp. 396–404.
[11]. Z. Tong and G. Tanaka, “Hybrid pooling for enhancement of generalization ability in deep convolutional neural networks,” Neurocomputing, vol. 333, pp. 76–85, 2019.
[12]. P. Sermanet, S. Chintala, and Y. LeCun, “Convolutional neural networks applied to house numbers digit classification,” in Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE, 2012, pp. 3288–3291.
[13]. T. Zhang, “Analysis of multi-stage convex relaxation for sparse regularization,” Journal of Machine Learning Research, vol. 11, no. Mar, pp. 1081–1107, 2010.
[14]. W. Wu, Q. Fan, J. M. Zurada, J. Wang, D. Yang, and Y. Liu, “Batch gradient method with smoothing l1/2 regularization for training of feedforward neural networks,” Neural Networks, vol. 50, pp. 72–78, 2014.
[15]. H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” arXiv preprint arXiv:1708.07747, 2017.
[16]. L. Wan, M. Zeiler, S. Zhang, Y. LeCun, and R. Fergus, “Regularization of neural networks using dropconnect,” in International conference on machine learning, 2013, pp. 1058–1066.
[17]. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015.
[18]. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research, vol. 15, no. 1, pp. 1929–1958, 2014.
[19]. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in Proceedings of the 22nd ACM international conference on Multimedia, 2014, pp. 675–678.