Convolutional Neural Network-Based Natural Image and MRI Classification Using Gaussian Activated Parametric (GAP) Layer

Bijen Khagi,Goo Rak Kwon

doi:10.1109/access.2021.3093455

Abstract

We propose a novel gaussian activated parametric (GAP) layer for deep neural networks specifically for CNN. This layer normalizes the feature vector using the Gaussian filter-based un-sharpening technique. The goal of the proposed method is to normalize and activate the initial and intermediate feature layers of a deep CNN so that a customized layer can make the feature more distinguish and layers are smoothly tuned for the target-domain classification. Our experiment shows the use of the proposed layer for normalization gives a result almost similar to that of batch normalization and in few cases, the result is slightly better at the cost of higher training time. To demonstrate the result of the proposed layer we are using 4 layers of an encoder-based network as a base architecture for classification in which the first two normalization layers are GAP layers or BN layers whereas the remaining are BN layers. The result of the experiment is better in classification with two GAP layer as normalization layer in a bulkier dataset like ADNI MRI (3D CNN accuracy: 93.58% vs. 91.89%) and CIFAR-10 (2D CNN accuracy: 75.21% vs. 75.11%), whereas the result is better with replacement of first BN layer with GAP in a smaller dataset like 5-animals dataset (2D CNN accuracy: 62.92% vs 58.48%). Therefore, we suggest the use of single GAP layer normalization for smaller datasets and two GAP normalization layers for bigger dataset training. Also, the proposed method produced better results than the cross-channel normalization-based AlexNet network under scratch training.

Highlights

Deep neural network (DNN) has been the dark horse in the field of machine learning (ML) and deep learning after the success of LeNet-5, an emerging Convolutional neural network (CNN), in 1989 for handwriting recognition [1], [2]
While the result is better with the replacement of the first batch normalization (BN) layer with gaussian activated parametric (GAP) in a smaller dataset like the 5-animals dataset (2D CNN accuracy: 62.92% vs. 58.48%), which supports the use of the proposed method for normalization and activation
To conclude we have experimented with our proposed idea of the GAP layer as a normalization layer in 2D and 3D architecture CNN

Summary

INTRODUCTION

Deep neural network (DNN) has been the dark horse in the field of machine learning (ML) and deep learning after the success of LeNet-5, an emerging Convolutional neural network (CNN), in 1989 for handwriting recognition [1], [2]. BN uses layer-wise whitening technique image i.e., mean zero and unit variance for normalization and decorrelation, with only two extra parameters per activation one for scaling and the other for shifting. It helps to reduce training time and preserves the interpretation capacity of the network [15]. Similar works done lately to design normalization layers without using minibatch mean in DNN includes filter response normalization (FRN) [38], group normalization (GN) [39], and layer normalization (LN) [40] All these methods do not operate in batch dimension to avoid minibatch dependency for calculating scaling factor and only use activation map channel statistics.

BACKGROUND

Findings

CONCLUSION