Trained Convolutional Neural Network Model Research Articles

Deep Learning-Based Prediction of Unsteady Reynolds-Averaged Navier-Stokes Solutions for Vertical-Axis Turbines

The following study investigates the effectiveness of a deep learning-based method for predicting the flow field and flow-driven rotation of a vertical-axis hydrokinetic turbine operating in previously unseen free-stream velocities. A Convolutional Neural Network (CNN) is trained and tested using the solutions of five two-dimensional (2-D), foil-resolved Unsteady Reynolds-Averaged Navier-Stokes (URANS) simulations, with free-stream velocities of 1.0, 1.5, 2.0, 2.5, and 3.0 m/s. Based on the boundary conditions of free-stream velocity and rotor position, the flow fields of x-velocity, y-velocity, pressure, and turbulent viscosity are inferred, in addition to the angular velocity of the rotor. Three trained CNN models are developed to evaluate the effects of (1) the dimensions of the training data, and (2) the number of simulations used as training cases. Reducing data dimensions was found to diminish mean relative error in predictions of velocity and turbulent viscosity, while increasing it in predictions of pressure and angular velocity. Increasing the number of training cases from two to three was found to reduce relative error for all predicted unknowns. With the best achieved CNN model, the variables of x-velocity, y-velocity, pressure, turbulent viscosity, and angular velocity were inferred with mean relative errors of 6.93%, 9.82%, 10.7%, 7.48%, and 0.817%, respectively.

Democratization of deep learning for segmenting cartilage from MRIs of human knees: Application to data from the osteoarthritis initiative.

In this study, we aimed to democratize access to convolutional neural networks (CNN) for segmenting cartilage volumes, generating state-of-the-art results for specialized, real-world applications in hospitals and research. Segmentation of cross-sectional and/or longitudinal magnetic resonance (MR) imagesof articular cartilage facilitates both clinical management of joint damage/disease and fundamental research. Manual delineation of such images is a time-consuming task susceptible to high intra- and interoperator variability and prone to errors. Thus, enabling reliable and efficient analyses of MRIs of cartilage requires automated segmentation of cartilage volumes. Two main limitations arise in the development of hospital- or population-specific deep learning (DL) models for image segmentation: specialized knowledge and specialized hardware. We present a relatively easy and accessible implementation of a DL model to automatically segment MRIs of human knees with state-of-the-art accuracy. In representative examples, we trained CNN models in 6-8 hand obtained results quantitatively comparable to state-of-the-art for every anatomical structure. We established and evaluated our methods using two publicly available MRI data sets originating from the Osteoarthritis Initiative, Stryker Imorphics, and Zuse Institute Berlin (ZIB), as representative test cases. We use Google Colabfor editing and adapting the Python codes and selecting the runtime environment leveraging high-performance graphical processing units. We designed our solution for novice users to apply to any data set with relatively few adaptations requiring only basic programming skills. To facilitate the adoption of our methods, we provide a complete guideline for using our methods and software, as well as the software tools themselves.Clinical significance:We establish and detail methods that clinical personal can apply to create their own DLmodels without specialized knowledge of DL nor specialized hardware/infrastructure and obtain results comparable with the state-of-the-art to facilitate both clinical management of joint damage/disease and fundamental research.

Quantitative Mass Spectrometry Imaging Using Multivariate Curve Resolution and Deep Learning: A Case Study

In the present contribution, a novel approach based on multivariate curve resolution and deep learning (DL) is proposed for quantitative mass spectrometry imaging (MSI) as a potent technique for identifying different compounds and creating their distribution maps in biological tissues without need for sample preparation. As a case study, chlordecone as a carcinogenic pesticide was quantitatively determined in mouse liver using matrix-assisted laser desorption ionization-MSI (MALDI-MSI). For this purpose, data from seven standard spots containing 0 to 20 picomoles of chlordecone and four unknown tissues from the mouse livers infected with chlordecone for 1, 5, and 10 days were analyzed using a convolutional neural network (CNN). To solve the lack of sufficient data for CNN model training, each pixel was considered as a sample, the designed CNN models were trained by pixels in training sets, and their corresponding amounts of chlordecone were obtained by multivariate curve resolution-alternating least-squares (MCR-ALS). The trained models were then externally evaluated using calibration pixels in test sets for 1, 5, and 10 days of exposure, respectively. Prediction R2 for all three data sets ranged from 0.93 to 0.96, which was superior to support vector machine (SVM) and partial least-squares (PLS). The trained CNN models were finally used to predict the amount of chlordecone in mouse liver tissues, and their results were compared with MALDI-MSI and GC-MS methods, which were comparable. Inspection of the results confirmed the validity of the proposed method.

An evidential combination method with multi-color spaces for remote sensing image scene classification

Remote sensing image scene classification aims to commit the semantic labels according to the content of images. Convolutional Neural Network (CNN) is often used here to extract deep discriminative feature of remote sensing images for classification. In practice, CNN is usually trained by images in the Red Green Blue (RGB) color space. Whereas, CNN also can be trained by images in some other color spaces, e.g., Hue Saturation Value. The CNN models trained by images in diverse color spaces will perform differently because different color spaces often emphasize diverse color information. Thus, we present an Evidential Combination method with Multi-color Spaces (ECMS) to integrate the complementary information of different color spaces for classification performance improvement. In ECMS, labeled remote sensing images in the RGB color space are first converted into other color spaces, and then they are used to train CNN models, respectively. The soft classification results (of query images) yielded by these CNN models are combined by evidence theory. During fusion, the reliabilities/weights of these outputs of different CNN models are usually different, so they should not be equally treated for combination. In our approach, the weights are learnt by minimizing the mean squared error between the combination results and ground truth on labeled images. By doing this, weighted evidence combination of soft classification results is employed to make scene class decision. We conducted experiments on several datasets to verify the effectiveness of ECMS, and the results show ECMS can significantly improve classification accuracy compared with many existing methods.

UNREADABLE OFFLINE HANDWRITING SIGNATURE VERIFICATION BASED ON GENERATIVE ADVERSARIAL NETWORK USING LIGHTWEIGHT DEEP LEARNING ARCHITECTURES

Today, it is known that there are great difficulties and problems in signature and signature examinations, which have a very important place in both our private life and business and commercial life. The major issue arises when the manuscript’s signature is so illegible and unclear that it is difficult, if not impossible, to authenticate it with the human eye. Researchers have proposed traditional deep learning techniques to solve or improve this challenge. However, the results are not satisfactory. In this study, a new use of Generative Adversarial Network (GAN) model is proposed as a high-quality data synthesis method to address the unreadable data problem on signature verification. A unique signature verification method based on Lightweight deep learning architecture is also proposed. The suggested data synthesizing approach is evaluated using three frequently used Convolutional Neural Network (CNN) methods: MobileNet, SqueezeNet, and ShuffleNet. In addition, in preprocessing phase, we added three different types of high-intensity noise, including Salt & Pepper (S&P), Gaussian, and Gaussian Blur, to the images to make the signature unreadable. We utilized Indic scripts dataset to train GAN and CNN models in our approach. The great quality of images generated by GAN model, as well as the signature verification of the generated images, point to the suggested model’s strong performance.

Automatic anatomical site recognition of laryngoscopic images using convolutional neural network

Objective:To explore the automatic recognition and classification of 20 anatomical sites in laryngoscopy by an artificial intelligence（AI） quality control system using convolutional neural network（CNN）. Methods: Laryngoscopic image data archived from laryngoscopy examinations at the Department of Endoscopy, Cancer Hospital, Chinese Academy of Medical Sciences from January to December 2018 were collected retrospectively, and a CNN model was constructed using Inception-ResNet-V2+SENet. Using 14000 electronic laryngoscope images as the training set, these images were classified into 20 specific anatomical sites including the whole head and neck, and their performance was tested by 2000 laryngoscope images and 10 laryngoscope videos. Results:The average time of the trained CNN model for recognition of each laryngoscopic image was（20.59 ± 1.55） ms, and the overall accuracy of recognition of 20 anatomical sites in laryngoscopic images was 97.75%（1955/2000）, with average sensitivity, specificity, positive predictive value, and negative predictive value of 100%, 99.88%, 97.76%, and 99.88%, respectively. The model had an accuracy of ≥ 99% for the identification of 20 anatomical sites in laryngoscopic videos. Conclusion:This study confirms that the CNN-based AI system can perform accurate and fast classification and identification of anatomical sites in laryngoscopic pictures and videos, which can be used for quality control of photo documentation in laryngoscopy and shows potential application in monitoring the performance of laryngoscopy.

Sound Recognition based on Convolutional Neural Network for Real-Time Cutting State Monitoring of Tube Cutting Machine

Machine sound monitoring is widely used in various applications of operational state and diagnostic monitoring as machine-emitted sound contains the operational and process information. In the metal cutting industry, it is not surprising that operators are easily able to recognize whether cutting is engaging by listening to the operational sounds based on their experiences even if the cutting parameters are changed. Inspired by the ability of recognizing human sound, we propose a real-time sound monitoring for cutting state based on the convolutional neural network (CNN). Three sound sensors were deployed to a tube cutting machine on different places to capture and analyze sound signals in real-time. To enable sound stream automatically, the MTConnect framework was employed. Four different sizes: 0.25, 0.5, 0.75, and 1-second, of the time frames, were selected and the effects of the time frames, as well as the sensors, were compared in terms of performances of predicting the cutting time and the productivity. The Log-Mel spectrum was adopted as a feature for convolutional neural network (CNN) model. In the CNN model training, a random search method was used to determine the hyperparameters of the CNN models. To verify the trained CNN models, the evaluation dataset which consisted of unknown and untrained part production was used. The accuracies of predicted cutting state on the evaluation dataset for all sensors and all sizes of the time frames ranged from 83.3 to 98.6%. The accuracies of the predicted productivity were also analyzed according to the sensors and the size of time frames. Furthermore, real-time monitoring response was evaluated to figure out the effects of the size of time frames on the prediction speed and the time delay. The sizes of the time frames play an important role in various aspects. As a result, it is suggested that the shorter size of the time frames is the better choice if the prediction accuracy of cutting time and the monitoring response time is critical whereas the longer size is the better if the prediction accuracy of productivity is more important. Among the internal sound sensors in various locations, one on the machine base shows the best performances in all sizes of the time frames with respect to not only the prediction of the cutting time but also the performance of predicting the productivity.

Hybrid opto-electronic deep neural network based orbital angular momentum mode recognition scheme in oceanic turbulence

Orbital angular momentum (OAM) has been widely used in underwater wireless optical communication (UWOC) systems due to the mutual orthogonality between modes. However, wavefront distortion caused by oceanic turbulence (OT) on the OAM mode seriously affects its mode recognition and communication quality. In this work, we propose a hybrid opto-electronic deep neural network (HOEDNN) based OAM mode recognition scheme. The HOEDNN model consists of a diffractive DNN (DDNN) and convolutional neural network (CNN), where the DDNN is trained to obtain the mapping between intensity patterns of a distorted OAM mode and intensity distributions without OT interference, and the CNN is used to recognize the output of the DDNN. The diffractive layers of the trained DDNN model are solidified, fabricated, and loaded into a spatial light modulator, and the results recorded by a charge-coupled device camera are processed and fed into the trained CNN model. The results show that the proposed scheme can overcome the interference of OT to OAM modes and recognize accurately azimuthal and radial indices. The OAM mode recognition scheme based on HOEDNN has potential application value in UWOC systems.

Road Pavement Damage Detection Based on Local Minimum of Grayscale and Feature Fusion

In this work, we propose a road pavement damage detection deep learning model based on feature points from a local minimum of grayscale. First, image blocks, consisting of the neighborhood of feature points, are cut from the image window to form an image block dataset. The image blocks are then input into a convolutional neural network (CNN) to train the model, extracting the image block features. In the testing process, the feature points as well as the image blocks are selected from a test image, and the trained CNN model can output the feature vectors for these feature image blocks. All the feature vectors will be combined to a composite feature vector as the feature descriptor of the test image. At last, the classifier of the model, constructed by a support vector machine (SVM), gives the classification as to whether the image window contains damaged areas or not. The experimental results suggest that the proposed pavement damage detection method based on feature-point image blocks and feature fusion is of high accuracy and efficiency. We believe that it has application potential in general road damage detection, and further investigation is desired in the future.

Convolutional neural network -based phantom image scoring for mammography quality control

BackgroundVisual evaluation of phantom images is an important, but time-consuming part of mammography quality control (QC). Consistent scoring of phantom images over the device’s lifetime is highly desirable. Recently, convolutional neural networks (CNNs) have been applied to a wide range of image classification problems, performing with a high accuracy. The purpose of this study was to automate mammography QC phantom scoring task by training CNN models to mimic a human reviewer.MethodsEight CNN variations consisting of three to ten convolutional layers were trained for detecting targets (fibres, microcalcifications and masses) in American College of Radiology (ACR) accreditation phantom images and the results were compared with human scoring. Regular and artificially degraded/improved QC phantom images from eight mammography devices were visually evaluated by one reviewer. These images were used in training the CNN models. A separate test set consisted of daily QC images from the eight devices and separately acquired images with varying dose levels. These were scored by four reviewers and considered the ground truth for CNN performance testing.ResultsAlthough hyper-parameter search space was limited, an optimal network depth after which additional layers resulted in decreased accuracy was identified. The highest scoring accuracy (95%) was achieved with the CNN consisting of six convolutional layers. The highest deviation between the CNN and the reviewers was found at lowest dose levels. No significant difference emerged between the visual reviews and CNN results except in case of smallest masses.ConclusionA CNN-based automatic mammography QC phantom scoring system can score phantom images in a good agreement with human reviewers, and can therefore be of benefit in mammography QC.

Extracting 3-D Radar Features to Improve Quantitative Precipitation Estimation in Complex Terrain based on Deep Learning Neural Networks

Abstract This paper proposes a new quantitative precipitation estimation (QPE) technique to provide accurate rainfall estimates in complex terrain, where conventional QPE has limitations. The operational radar QPE in Taiwan is mainly based on the simplified relationship between radar reflectivity Z and rain rate R [R(Z) relation] and only utilizes the single-point lowest available echo to estimate rain rates, leading to low accuracy in complex terrain. Here, we conduct QPE using deep learning that extracts features from 3-D radar reflectivities to address the above issues. Convolutional neural networks (CNN) are used to analyze contoured frequency by altitude diagrams (CFADs) to generate the QPE. CNN models are trained on existing rain gauges in northern and eastern Taiwan with the three-year data during 2015–17 and validated and tested using 2018 data. The weights of heavy rains (≧10 mm h-1) are increased in the model loss calculation to handle the unbalanced rainfall data and improve accuracy. Results show that the CNN outperforms the R(Z) relation based on the 2018 rain-gauge data. Furthermore, this research proposes methods to conduct 2-D gridded QPE at every pixel by blending estimates from various trained CNN models. Verification based on independent rain gauges shows that the CNN QPE solves the underestimation of the R(Z) relation in mountainous areas. Case studies are presented to visualize the results, showing that the CNN QPE generates better small-scale rainfall features and more accurate precipitation information. This deep learning QPE technique may be helpful for the disaster prevention of small-scale flash floods in complex terrain.

Prediction of Machining Condition Using Time Series Imaging and Deep Learning in Slot Milling of Titanium Alloy

Low surface quality, undesired geometrical and dimensional tolerances, and product damage due to tool wear and tool breakage lead to a dramatic increase in production cost. In this regard, monitoring tool conditions and the machining process are crucial to prevent unwanted events during the process and guarantee cost-effective and high-quality production. This study aims to predict critical machining conditions concerning surface roughness and tool breakage in slot milling of titanium alloy. Using the Siemens SINUMERIK Edge Box integrated into a CNC machine tool, signals were recorded from main spindle and different axes. Instead of extraction of features from signals, the Gramian angular field (GAF) was used to encode the whole signal into an image with no loss of information. Afterwards, the images obtained from different machining conditions were used for training a convolutional neural network (CNN) as a suitable and frequently applied deep learning method for images. The combination of GAF and trained CNN model indicates good performance in predicting critical machining conditions, particularly in the case of an imbalanced dataset. The trained classification CNN model resulted in recall, precision, and accuracy with 75%, 88%, and 94% values, respectively, for the prediction of workpiece surface quality and tool breakage.

Comparison of convolutional neural networks for classification of vocal fold nodules from high-speed video images.

Deep learning is in this study used through convolutional neural networks (CNN) to the determination of vocal fold nodules. Through high-speed video (HSV) images and computer-assisted tools, a comparison of convolutional neural network models and their accuracy will be presented. The data have been collected by an Ear Nose Throat (ENT) specialist with a 90° rigid scope in the years from 2007 to 2019, where 15.732 high-speed videos have been collected from 7909 patients. A total of 4000 images have been carefully selected, 2000 images were of normal vocal folds and 2000 images were of vocal folds with varying degrees of vocal fold nodules. These images were then split into training-, validation-, and testing-data set, for use with a CNN model with 5 layers (CNN5) and compared to other models: VGG19, MobileNetV2, and Inception-ResNetV2. To compare the neural network models, the following evaluation metrics have been calculated: accuracy, sensitivity, specificity, precision, and negative predictive values. All the trained CNN models have shown high accuracy when applied to the test set. The accuracy is 97.75%, 83.5%, 91.5%, and 89.75%, for CNN5, VGG19, MobileNetV2, and InceptionResNetV2, respectively. Precision was identified as the most relevant performance metric for a study that focuses on the classification of vocal fold nodules. The highest performing model was MobilNetV2 with a precision of 97.7%. The average accuracy across all 4 neural networks was 90.63% showing that neural networks can be used for classifying vocal fold nodules in a clinical setting.

Federated Spectrum Learning for Reconfigurable Intelligent Surfaces-Aided Wireless Edge Networks

Increasing concerns on intelligent spectrum sensing call for efficient training and inference technologies. In this paper, we propose a novel federated learning (FL) framework, dubbed federated spectrum learning (FSL), which exploits the benefits of reconfigurable intelligent surfaces (RISs) and overcomes the unfavorable impact of deep fading channels. Distinguishingly, we endow conventional RISs with spectrum learning capabilities by leveraging a fully-trained convolutional neural network (CNN) model at each RIS controller, thereby helping the base station to cooperatively infer the users who request to participate in FL at the beginning of each training iteration. To fully exploit the potential of FL and RISs, we address three technical challenges: RISs phase shifts configuration, user-RIS association, and wireless bandwidth allocation. The resulting joint learning, wireless resource allocation, and user-RIS association design is formulated as an optimization problem whose objective is to maximize the system utility while considering the impact of FL prediction accuracy. In this context, the accuracy of FL prediction interplays with the performance of resource optimization. In particular, if the accuracy of the trained CNN model deteriorates, the performance of resource allocation worsens. The proposed FSL framework is tested by using real radio frequency (RF) traces and numerical results demonstrate its advantages in terms of spectrum prediction accuracy and system utility: a better CNN prediction accuracy and FL system utility can be achieved with a larger number of RISs and reflecting elements.

Enhancing one-way wave equation-based migration with deep learning

One advantage of one-way wave equation-based migration is its low computational cost. However, due to the limited wavefield propagation angle, it is difficult to use one-way wave equation-based migration for high-precision imaging of structures with large inclinations due to issues such as inaccurate amplitudes and migration image artifacts. In addition, when the model has large horizontal velocity differences, it is difficult for the one-way wave propagator to calculate an accurate wavefield phase. Reverse time migration (RTM) based on the two-way wave propagator has a high resolution and avoids the issues associated with one-way wave propagators; however, it has a high computational cost in practical applications. We develop a convolutional neural network (CNN) application mode that improves one migration method by learning from another one and design a CNN with a structure similar to U-net that combines the advantages of both migration methods. The CNN label is the RTM result, and the corresponding input is the result of one-way wave migration with a generalized screen propagator (GSP). The trained CNN model improves the amplitude in the one-way wave migration image and removes the errors caused by large lateral velocity perturbations. Moreover, by maintaining the high migration calculation efficiency, our CNN model allows for a high resolution, few artifacts, and accurate images of steep structures in the one-way wave migration result. With our method, the accuracy of the one-way wave migration result is close to that of the RTM result. The use of GSP-based migration in our CNN model rather than conventional RTM to generate prospecting images can considerably reduce the calculation costs.

An Ameliorated Denoising Scheme Based on Deep Learning for Φ-OTDR System With 41-km Detection Range

In recent years, denoising methods for improving the performance of the phase-sensitive optical time-domain reflectometry ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\Phi $ </tex-math></inline-formula> -OTDR) system have been restricted by the deficiencies of time-consuming and limited denoising effect. In this work, a trained convolutional neural network (CNN)-based image denoising model is proposed to greatly eliminate the unwanted noises in the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\Phi $ </tex-math></inline-formula> -OTDR-based sensing system. First, the given Rayleigh backscattering traces are acquired and preprocessed through adjacent differentiation and two-dimensionalization. Second, the 2-D preprocessed data are converted into a noisy gray-scale image and sent into the CNN model for training and testing. Third, the CNN model outputs a corresponding denoised gray-scale image, which can be further analyzed by reconverting it into a series of denoised Rayleigh backscattering traces. Finally, a series of experiments are carried out to demonstrate the effectiveness of the proposed denoising scheme. Experimental results show that, in allusion to the vibration signal with different intensities along the 41-km optical sensing fiber, the trained CNN model achieves a signal-to-noise ratio (SNR) enhancement of about 20 dB. Compared with the conventional methods based on wavelet and empirical mode decomposition (EMD), the proposed denoising scheme demonstrates characteristics of robustness, well spatial resolution reservation, and high efficiency. It is believed that the trained CNN model has great potential to be deployed on the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\Phi $ </tex-math></inline-formula> -OTDR system for real-time denoising.

Season-Invariant GNSS-Denied Visual Localization for UAVs

Localization without Global Navigation Satellite Systems (GNSS) is a critical functionality in autonomous operations of unmanned aerial vehicles (UAVs). Vision-based localization on a known map can be an effective solution, but it is burdened by two main problems: places have different appearance depending on weather and season, and the perspective discrepancy between the UAV camera image and the map make matching hard. In this work, we propose a localization solution relying on matching of UAV camera images to georeferenced orthophotos with a trained convolutional neural network model that is invariant to significant seasonal appearance difference (winter-summer) between the camera image and map. We compare the convergence speed and localization accuracy of our solution to six reference methods. The results show major improvements with respect to reference methods, especially under high seasonal variation. We finally demonstrate the ability of the method to successfully localize a real UAV, showing that the proposed method is robust to perspective changes.

Region-Based Convolutional Neural Network for Segmenting Text in Epigraphical Images

Indian history is derived from ancient writings on the inscriptions, palm leaves, copper plates, coins, and many more mediums. Epigraphers read these inscriptions and produce meaningful interpretations. Automating the process of reading is the interest of our study, and in this paper, segmentation to detect text on digitized inscriptional images is dealt in detail. Character segmentation from epigraphical images helps in optical character recognizer in training and recognition of old regional scripts. Epigraphical images are drawn from estampages containing scripts from various periods starting from Brahmi in the 3rd century BC to the medieval period of the 15th century AD. The scripts or characters present in digitized epigraphical images are illegible and have complex noisy background textures. To achieve script/text segmentation, region-based convolutional neural network (CNN) is employed to detect characters in the images. Proposed method uses selective search to identify text regions and forwards them to trained CNN models for drawing feature vectors. These feature vectors are fed to support vector machine classifiers for classification and recognize text by drawing a bounding box based on confidence score. Alexnet, VGG16, Resnet50, and InceptionV3 are used as CNN models for experimentation, and InceptionV3 performed well with good results. A total of 197 images are used for experimentation, out of which 70 samples are of printed denoised epigraphical images, 40 denoised estampage images, and 87 noisy estampage images. The segmentation result of 74.79% for printed denoised epigraphical images, 71.53 % for denoised estampage epigraphical images, and 18.11% for noisy estampage images are recorded by InceptionV3. The segmented characters are used for epigraphical applications like period/era prediction and recognition of characters. FAST and FASTER region-based design approach was also tested and illustrated in this paper.

A novel scaled-gamma-tanh (SGT) activation function in 3D CNN applied for MRI classification

Activation functions in the neural network are responsible for ‘firing’ the nodes in it. In a deep neural network they ‘activate’ the features to reduce feature redundancy and learn the complex pattern by adding non-linearity in the network to learn task-specific goals. In this paper, we propose a simple and interesting activation function based on the combination of scaled gamma correction and hyperbolic tangent function, which we call Scaled Gamma Tanh (SGT) activation. The proposed activation function is applied in two steps, first is the calculation of gamma version as y = f(x) = axα for x < 0 and y = f(x) = bxβ for x ≥ 0, second is obtaining the squashed value as z = tanh(y). The variables a and b are user-defined constant values whereas alpha and beta are channel-based learnable parameters. We analyzed the behavior of the proposed SGT activation function against other popular activation functions like ReLU, Leaky-ReLU, and tanh along with their role to confront vanishing/exploding gradient problems. For this, we implemented the SGT activation functions in a 3D Convolutional neural network (CNN) for the classification of magnetic resonance imaging (MRIs). More importantly to support our proposed idea we have presented a thorough analysis via histogram of inputs and outputs in activation layers along with weights/bias plot and t-SNE (t-Distributed Stochastic Neighbor Embedding) projection of fully connected layer for the trained CNN models. Our results in MRI classification show SGT outperforms standard ReLU and tanh activation in all cases i.e., final validation accuracy, final validation loss, test accuracy, Cohen’s kappa score, and Precision.

PERFORMANCE COMPARISON OF GRADIENT-BASED CONVOLUTIONAL NEURAL NETWORK OPTIMIZERS FOR FACIAL EXPRESSION RECOGNITION

A convolutional neural network (CNN) is one of the machine learning models that achieve excellent success in recognizing human facial expressions. Technological developments have given birth to many optimizers that can be used to train the CNN model. Therefore, this study focuses on implementing and comparing 14 gradient-based CNN optimizers to classify facial expressions in two datasets, namely the Advanced Computing Class 2022 (ACC22) and Extended Cohn-Kanade (CK+) datasets. The 14 optimizers are classical gradient descent, traditional momentum, Nesterov momentum, AdaGrad, AdaDelta, RMSProp, Adam, Radam, AdaMax, AMSGrad, Nadam, AdamW, OAdam, and AdaBelief. This study also provides a review of the mathematical formulas of each optimizer. Using the best default parameters of each optimizer, the CNN model is trained using the training data to minimize the cross-entropy value up to 100 epochs. The trained CNN model is measured for its accuracy performance using training and testing data. The results show that the Adam, Nadam, and AdamW optimizers provide the best performance in model training and testing in terms of minimizing cross-entropy and accuracy of the trained model. The three models produce a cross-entropy of around 0.1 at the 100th epoch with an accuracy of more than 90% on both training and testing data. Furthermore, the Adam optimizer provides the best accuracy on the testing data for the ACC22 and CK+ datasets, which are 100% and 98.64%, respectively. Therefore, the Adam optimizer is the most appropriate optimizer to be used to train the CNN model in the case of facial expression recognition.