Rectified Linear Unit Research Articles

Obstructive sleep apnea detection based on electrocardiogram signal using one-dimensional convolutional neural network

Obstructive sleep apnea (OSA) is a respiratory obstruction that occurs during sleep and is often known as snoring. OSA is often ignored even though it can cause cardiovascular problems. Early diagnosis is needed for prevention towards worse complications. OSA clinical diagnosis can use polysomnography (PSG) while the patient is sleeping. The PSG examination includes calculating total apnea plus hypopnea every hour during sleep. However, PSG examination tends to be high cost, takes a long time, and is impractical. Since OSA is related to breathing and heart activity, the electrocardiogram (ECG) examination is an alternative tool in OSA analysis. Therefore, this study proposes OSA detection on single lead ECG using one dimensional (1D)-convolutional neural network (CNN). The proposed CNN architecture consists of 4 convolutional layers, 4 pooling layers, 1 dropout layer, 1 flatten layers, 2 dropout layers, 1 dense layer with rectified linear unit (ReLU) activation function, and 1 dense layer with SoftMax activation function. The proposed method was then tested on the ECG sleep apnea dataset from PhysioNet. The proposed model produces an accuracy of 92.69% with the average pooling scenario. The proposed method is expected to help clinicians in diagnosing OSA based on ECG signals.

Pneumonia detection on x-ray image using improved depthwise separable convolutional neural networks

A single neural network model cannot capture intricate and diverse features due to its ability to learn only a finite set of patterns from the data. Additionally, training and utilising a single model can be computationally demanding. Experts propose incorporating multiple neural network models to address these constraints to extract complementary attributes. Previous research has highlighted challenges network models face, including difficulties in effectively capturing highly detailed spatial features, redundancy in network structure parameters, and restricted generalisation capabilities. This study introduces an innovative neural network architecture that combines the Xception module with the inverse residue structure to tackle these issues. Considering this, the paper presents a model for detecting pneumonia in X-ray images employing an improved depthwise separable convolutional network. This network architecture integrates the inverse residual structure from the MobileNetV2 model, using the rectified linear unit (ReLU) non-linear activation function throughout the entire network. The experimental results show an impressive recognition rate with a test accuracy of 97.24% on the chest x-ray dataset. This method can extract more profound and abstract image features while mitigating overfitting issues and enhancing the network's generalisation capacity.

Mobile Phone Intelligent Recognition of Human Activity Based on Deep Neural Network

Abstract. Because of the rapid development of smartphone sensor technology and the continuous progress of machine learning algorithms, it has become possible to use smartphones for human activity recognition. Sensors such as accelerometers, gyroscopes, and magnetometers built into smartphones are able to collect human motion data in different activities, which provides a rich data source for activity recognition through mobile phones. In this paper, we change the activation function of the hidden layer of the neural network to observe the effect of this variable on the mobile phone to recognize human activities. Taking Rectified Linear Unit (ReLU)function and Sigmoid function as examples, it is found that the neural network using the ReLU function shows higher accuracy (0. 9518 and 0. 9332) and lower loss value in the early stage and middle to late stage of training. In addition, This paper finds that training for 100 epochs takes significantly less time than the model with the Sigmoid activation function (47. 041 seconds vs 179. 022 seconds). The neural network system built by the relu function has faster operation speed and better performance. This study shows that different hidden layer functions have great influence on the quality of the same neural network, and selecting the best hidden layer function will be an important point for the success of the study.

Bayesian-Optimized Deep Learning for Predicting Compressive Strength of Sustainable Eggshell Foamed Concrete

<p><span lang="EN-US">The use of eggshell-based foamed concrete represents a sustainable approach to enhancing environmental friendliness in construction materials. This study investigates the predictive modelling of the compressive strength of eggshell-based foamed concrete through a deep learning model, fine-tuned via Bayesian optimization. Utilizing a dataset of 360 samples with diverse input parameters, the model was optimized with four hidden layers (28, 21, 28, and 21 neurons) and the Rectified Linear Unit (ReLU) activation function. The model demonstrated excellent predictive accuracy, achieving a mean squared error of 0.0522, a mean absolute error of 0.0382, and an R² value of 0.9548 over 200 epochs. Notably, the water/cement ratio emerged as the most influential factor in prediction accuracy. This research provides a robust, AI-driven method for predicting the compressive strength of sustainable construction materials, contributing to advancements in environmental technology and the optimization of eco-friendly construction practices.</span></p>

A practical Approach to Handwritten Digit Generation with Generative Adversarial Networks on MNIST

Implementation of a Generative Adversarial Network (GAN) utilizing the MNIST (Modified National Institute of Standards and Technology database) dataset of handwritten digits, the GAN comprises a generator creating synthetic images from random noise and a discriminator to classifying these images as real or fake. The Generator utilizes leaky ReLU (rectified linear unit) activations, Batch normalization and reshaping to produce 28x28 grayscale images, while the Discriminator uses dense layers, leaky ReLU, and dropout to enhance classification accuracy. Both networks are trained using the Adam optimizer to improve stability and performance. The GAN is trained in an adversarial setup where the Generator seeks to create convincing images and the Discriminator aims to correctly classify them. Results demonstrate the GAN's ability to generate high-quality handwritten digit images, displaying its effectiveness and providing valuable insights into best practices for GAN implementation and training.

Detection of Subarachnoid Hemorrhage Using CNN with Dynamic Factor and Wandering Strategy-Based Feature Selection.

Objectives: Subarachnoid Hemorrhage (SAH) is a serious neurological emergency case with a higher mortality rate. An automatic SAH detection is needed to expedite and improve identification, aiding timely and efficient treatment pathways. The existence of noisy and dissimilar anatomical structures in NCCT images, limited availability of labeled SAH data, and ineffective training causes the issues of irrelevant features, overfitting, and vanishing gradient issues that make SAH detection a challenging task. Methods: In this work, the water waves dynamic factor and wandering strategy-based Sand Cat Swarm Optimization, namely DWSCSO, are proposed to ensure optimum feature selection while a Parametric Rectified Linear Unit with a Stacked Convolutional Neural Network, referred to as PRSCNN, is developed for classifying grades of SAH. The DWSCSO and PRSCNN surpass current practices in SAH detection by improving feature selection and classification accuracy. DWSCSO is proposed to ensure optimum feature selection, avoiding local optima issues with higher exploration capacity and avoiding the issue of overfitting in classification. Firstly, in this work, a modified region-growing method was employed on the patient Non-Contrast Computed Tomography (NCCT) images to segment the regions affected by SAH. From the segmented regions, the wide range of patterns and irregularities, fine-grained textures and details, and complex and abstract features were extracted from pre-trained models like GoogleNet, Visual Geometry Group (VGG)-16, and ResNet50. Next, the PRSCNN was developed for classifying grades of SAH which helped to avoid the vanishing gradient issue. Results: The DWSCSO-PRSCNN obtained a maximum accuracy of 99.48%, which is significant compared with other models. The DWSCSO-PRSCNN provides an improved accuracy of 99.62% in CT dataset compared with the DL-ICH and GoogLeNet + (GLCM and LBP), ResNet-50 + (GLCM and LBP), and AlexNet + (GLCM and LBP), which confirms that DWSCSO-PRSCNN effectively reduces false positives and false negatives. Conclusions: the complexity of DWSCSO-PRSCNN was acceptable in this research, for while simpler approaches appeared preferable, they failed to address problems like overfitting and vanishing gradients. Accordingly, the DWSCSO for optimized feature selection and PRSCNN for robust classification were essential for handling these challenges and enhancing the detection in different clinical settings.

Multi-task OCTA image segmentation with innovative dimension compression

Optical Coherence Tomography Angiography (OCTA) plays a crucial role in the early detection and continuous monitoring of ocular diseases, which relies on accurate multi-tissue segmentation of retinal images. Existing OCTA segmentation methods typically focus on single-task designs that do not fully utilize the information of volume data in these images. To bridge this gap, our study introduces H2C-Net, a novel network architecture engineered for simultaneous and precise segmentation of various retinal structures, including capillaries, arteries, veins, and the fovea avascular zone (FAZ). At its core, H2C-Net consists of a plug-and-play Height-Channel Module (H2C) and an Enhanced U-shaped Network (GPC-Net). The H2C module cleverly converts the height information of the OCTA volume data into channel information through the Squeeze operation, realizes the lossless dimensionality reduction from 3D to 2D, and provides the "Soft layering" information by unidirectional pooling. Meanwhile, in order to guide the network to focus on channels for training, U-Net is enhanced with group normalization, channel attention mechanism, and Parametric Rectified Linear Unit (PReLU), which reduces the dependence on batch size and enhances the network's ability to extract salient features. Extensive experiments on two subsets of the publicly available OCTA-500 dataset have shown that H2C-Net outperforms existing state-of-the-art methods. It achieves average Intersection over Union (IoU) scores of 82.84 % and 88.48 %, marking improvements of 0.81 % and 1.59 %, respectively. Similarly, the average Dice scores are elevated to 90.40 % and 93.76 %, exceeding previous benchmarks by 0.42 % and 0.94 %. The proposed H2C-Net exhibits excellent performance in OCTA image segmentation, providing an efficient and accurate multi-task segmentation solution in ophthalmic diagnostics. The code is publicly available at: https://github.com/IAAI-SIT/H2C-Net.

Rapid and high-accuracy concentration prediction of gas mixtures based on PMH-TCN

In industrial environments, the flammable gas ethylene (C2H4) and toxic gas carbon monoxide (CO) often coexist. To ensure worker safety, it is essential to predict these gas concentrations quickly and accurately. Many studies lack prediction accuracy or speed due to insufficient time feature extraction and loss of critical features. In this work, a novel approach called the Parametric Rectified Linear Unit (PReLU)-based Multi-Head Attention Temporal Convolutional Network (PMH-TCN) model was devised to efficiently and precisely predict gas concentrations in mixtures. The model is capable of accurately predicting the gas concentration within 5 s after the introduction of the target gas. In the 5-fold cross-validation, the values of Adjusted R-Square (R2), Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) of the PMH-TCN model reached 0.9850, 0.0448, and 0.0651 respectively. It was shown that the PMH-TCN model offered a very efficient method for the quick identification of electronic noses.

ReLU surrogates in mixed-integer MPC for irrigation scheduling

Efficient water management in agriculture is important for mitigating the growing freshwater scarcity crisis. Mixed-integer Model Predictive Control (MPC) has emerged as an effective approach for addressing the complex scheduling problem in agricultural irrigation. However, the computational complexity of mixed-integer MPC still poses a significant challenge, particularly in large-scale applications. This study proposes an approach to enhance the computational efficiency of mixed-integer MPC-based irrigation schedulers by employing Rectified Linear Unit (ReLU) surrogate models to describe the soil moisture dynamics of the agricultural field. By leveraging the mixed-integer linear representation of the ReLU operator, the proposed approach transforms the mixed-integer MPC-based scheduler with a quadratic cost function into a mixed-integer quadratic program, which is the simplest class of mixed-integer nonlinear programming problems that can be efficiently solved using global optimization solvers. The effectiveness of this approach is demonstrated through comparative studies conducted on a large-scale agricultural field across two growing seasons, involving other machine learning surrogate models, specifically Long Short-Term Memory (LSTM) networks, and the triggered irrigation scheduling method. The ReLU-based approach significantly reduces solution times — by up to 99.5% — while achieving comparable performance to the LSTM approach in terms of water savings and Irrigation Water Use Efficiency (IWUE). Moreover, the ReLU-based approach achieves enhanced performance in terms of irrigation water savings and IWUE compared to the triggered approach.

Deep learning as Ricci flow.

Deep neural networks (DNNs) are powerful tools for approximating the distribution of complex data. It is known that data passing through a trained DNN classifier undergoes a series of geometric and topological simplifications. While some progress has been made toward understanding these transformations in neural networks with smooth activation functions, an understanding in the more general setting of non-smooth activation functions, such as the rectified linear unit (ReLU), which tend to perform better, is required. Here we propose that the geometric transformations performed by DNNs during classification tasks have parallels to those expected under Hamilton's Ricci flow-a tool from differential geometry that evolves a manifold by smoothing its curvature, in order to identify its topology. To illustrate this idea, we present a computational framework to quantify the geometric changes that occur as data passes through successive layers of a DNN, and use this framework to motivate a notion of 'global Ricci network flow' that can be used to assess a DNN's ability to disentangle complex data geometries to solve classification problems. By training more than 1500 DNN classifiers of different widths and depths on synthetic and real-world data, we show that the strength of global Ricci network flow-like behaviour correlates with accuracy for well-trained DNNs, independently of depth, width and data set. Our findings motivate the use of tools from differential and discrete geometry to the problem of explainability in deep learning.

Analytical Bounds on the Local Lipschitz Constants of ReLU Networks.

In this article, we determine analytical upper bounds on the local Lipschitz constants of feedforward neural networks with rectified linear unit (ReLU) activation functions. We do so by deriving Lipschitz constants and bounds for ReLU, affine-ReLU, and max pooling functions, and combining the results to determine a network-wide bound. Our method uses several insights to obtain tight bounds, such as keeping track of the zero elements of each layer, and analyzing the composition of affine and ReLU functions. Furthermore, we employ a careful computational approach which allows us to apply our method to large networks such as AlexNet and VGG-16. We present several examples using different networks, which show how our local Lipschitz bounds are tighter than the global Lipschitz bounds. We also show how our method can be applied to provide adversarial bounds for classification networks. These results show that our method produces the largest known bounds on minimum adversarial perturbations for large networks such as AlexNet and VGG-16.

Machine Learning-Based Gesture Recognition Glove: Design and Implementation.

In the evolving field of human-computer interaction (HCI), gesture recognition has emerged as a critical focus, with smart gloves equipped with sensors playing one of the most important roles. Despite the significance of dynamic gesture recognition, most research on data gloves has concentrated on static gestures, with only a small percentage addressing dynamic gestures or both. This study explores the development of a low-cost smart glove prototype designed to capture and classify dynamic hand gestures for game control and presents a prototype of data gloves equipped with five flex sensors, five force sensors, and one inertial measurement unit (IMU) sensor. To classify dynamic gestures, we developed a neural network-based classifier, utilizing a convolutional neural network (CNN) with three two-dimensional convolutional layers and rectified linear unit (ReLU) activation where its accuracy was 90%. The developed glove effectively captures dynamic gestures for game control, achieving high classification accuracy, precision, and recall, as evidenced by the confusion matrix and training metrics. Despite limitations in the number of gestures and participants, the solution offers a cost-effective and accurate approach to gesture recognition, with potential applications in VR/AR environments.

Invariant set estimation for piecewise affine dynamical systems using piecewise affine barrier function

This paper introduces an algorithm for estimating the invariant set of closed-loop controlled dynamical systems identified using single-hidden layer Rectified linear units (ReLU) neural networks or piecewise affine (PWA) functions, particularly addressing the challenge of providing safety guarantees for single-hidden layer ReLU networks commonly used in safety–critical applications. The invariant set of PWA dynamical system is estimated using single-hidden layer ReLU networks or its equivalent PWA function. This method entails formulating the barrier function as a PWA function and converting the search process into a linear optimization problem using vertices. We incorporate a domain refinement strategy to increase flexibility in case the optimization does not find a valid barrier function. Moreover, the objective of the optimization is to find a less conservative invariant set based on the current partition. Our experimental results demonstrate the effectiveness and efficiency of our approach, demonstrating its potential for ensuring the safety of PWA dynamical systems.

Gradient Descent Provably Escapes Saddle Points in the Training of Shallow ReLU Networks

AbstractDynamical systems theory has recently been applied in optimization to prove that gradient descent algorithms bypass so-called strict saddle points of the loss function. However, in many modern machine learning applications, the required regularity conditions are not satisfied. In this paper, we prove a variant of the relevant dynamical systems result, a center-stable manifold theorem, in which we relax some of the regularity requirements. We explore its relevance for various machine learning tasks, with a particular focus on shallow rectified linear unit (ReLU) and leaky ReLU networks with scalar input. Building on a detailed examination of critical points of the square integral loss function for shallow ReLU and leaky ReLU networks relative to an affine target function, we show that gradient descent circumvents most saddle points. Furthermore, we prove convergence to global minima under favourable initialization conditions, quantified by an explicit threshold on the limiting loss.

Automated detection of kidney masses lesions using a deep learning approach

<p>Deep learning has emerged as a potent tool for various tasks, such as image classification. However, in the medical domain, there exists a scarcity of data, which poses a challenge in obtaining a well-balanced and high-quality dataset. Commonly seen issues in the realm of renal health include conditions such as kidney stones, cysts, and tumors. This study is centered on the examination of deep learning models for the purpose of classifying renal computed tomography (CT)-scan pictures. State-of-the-art classification models, such as convolutional neural network (CNN) approaches, are employed to boost model performance and improve accuracy. The algorithm is comprised of six convolutional layers that progressively increase in complexity. Every layer in the network utilizes a uniform 3x3 kernel size and applies the rectified linear unit (ReLU) activation function. This is followed by a max-pooling layer that downsamples the feature maps using a 2x2 pool size. Following this, a flatten layer was implemented in order to preprocess the data for the fully linked layers. The consistent utilization of uniform kernel sizes and activation functions throughout all layers of the model facilitated the smooth extraction of complex features, thereby enhancing the model’s ability to accurately identify different kidney conditions. As a result, we achieved a high accuracy rate of 99.8%, precision is 99.8%, and F1 score of approximately 99.7%.</p>

Hybrid Feature Mammogram Analysis: Detecting and Localizing Microcalcifications Combining Gabor, Prewitt, GLCM Features, and Top Hat Filtering Enhanced with CNN Architecture.

Breast cancer is a prevalent malignancy characterized by the uncontrolled growth of glandular epithelial cells, which can metastasize through the blood and lymphatic systems. Microcalcifications, small calcium deposits within breast tissue, are critical markers for early detection of breast cancer, especially in non-palpable carcinomas. These microcalcifications, appearing as small white spots on mammograms, are challenging to identify due to potential confusion with other tissues. This study hypothesizes that a hybrid feature extraction approach combined with Convolutional Neural Networks (CNNs) can significantly enhance the detection and localization of microcalcifications in mammograms. The proposed algorithm employs Gabor, Prewitt, and Gray Level Co-occurrence Matrix (GLCM) kernels for feature extraction. These features are input to a CNN architecture designed with maxpooling layers, Rectified Linear Unit (ReLU) activation functions, and a sigmoid response for binary classification. Additionally, the Top Hat filter is used for precise localization of microcalcifications. The preprocessing stage includes enhancing contrast using the Volume of Interest Look-Up Table (VOI LUT) technique and segmenting regions of interest. The CNN architecture comprises three convolutional layers, three ReLU layers, and three maxpooling layers. The training was conducted using a balanced dataset of digital mammograms, with the Adam optimizer and binary cross-entropy loss function. Our method achieved an accuracy of 89.56%, a sensitivity of 82.14%, and a specificity of 91.47%, outperforming related works, which typically report accuracies around 85-87% and sensitivities between 76 and 81%. These results underscore the potential of combining traditional feature extraction techniques with deep learning models to improve the detection and localization of microcalcifications. This system may serve as an auxiliary tool for radiologists, enhancing early detection capabilities and potentially reducing diagnostic errors in mass screening programs.

Lumbar intervertebral disc detection and classification with novel deep learning models

Low back pain (LBP) is a prevalent spinal issue, affecting eight out of ten individuals. Notably, lumbar intervertebral disc (IVD) abnormalities frequently contribute to LBP. To diagnose LBP, Magnetic Resonance Imaging (MRI) is crucial for obtaining detailed spinal images. This paper employs deep learning (DL) to detect and locate lumbar IVD in sagittal MR images. It further classifies lumbar IVDs as healthy or herniated, utilizing both novel convolutional neural network (CNN) and conventional CNN models. The dataset utilized comprises MR images from 32 patients, with 10 exhibiting healthy discs and the remaining 22 posing a mix of healthy and herniated discs, totaling 160 lumbar discs, incorporating 112 healthy and 48 herniated discs. In this study, ResNet-50 architecture in the Novel Lumbar IVD detection (NLID) model served as the feature extractor to segment the five lumbar IVDs from MR images. The features extracted from ResNet-50 were input into YOLOv2 for the identification of the region of interest (ROI). The findings indicate that optimal performance was achieved at the 22nd Rectified Linear Unit (ReLU) activation layer, boasting a remarkable 99.59% average precision, 97.22% F1-score, 94.59% precision, and a perfect 100% recall. This commendable performance consistently held above the 85% threshold until the 22nd ReLU activation layer. Regarding imbalanced dataset classification, AlexNet emerged as the frontrunner among other pre-trained networks, boasting the highest test accuracy of 90.63%, and an impressive F1 score of 88.77%. Meanwhile, the Novel Lumbar IVD Classification (NLIC) model achieved superior results with 93.75% test accuracy, and 92.27% F1-score. In the setting of the balanced dataset, NLIC achieved 96.88% test accuracy, and 96.46% F1-score with fewer epochs compared to AlexNet, affirming the robustness of the novel trained-from-scratch network. These findings distinctly underscore the effectiveness of CNNs in both medical image segmentation and classification.

Breast Cancer Detection and Analytics Using Hybrid CNN and Extreme Learning Machine.

Early detection of breast cancer is essential for increasing survival rates, as it is one of the primary causes of death for women globally. Mammograms are extensively used by physicians for diagnosis, but selecting appropriate algorithms for image enhancement, segmentation, feature extraction, and classification remains a significant research challenge. This paper presents a computer-aided diagnosis (CAD)-based hybrid model combining convolutional neural networks (CNN) with a pruned ensembled extreme learning machine (HCPELM) to enhance breast cancer detection, segmentation, feature extraction, and classification. The model employs the rectified linear unit (ReLU) activation function to enhance data analytics after removing artifacts and pectoral muscles, and the HCPELM hybridized with the CNN model improves feature extraction. The hybrid elements are convolutional and fully connected layers. Convolutional layers extract spatial features like edges, textures, and more complex features in deeper layers. The fully connected layers take these features and combine them in a non-linear manner to perform the final classification. ELM performs classification and recognition tasks, aiming for state-of-the-art performance. This hybrid classifier is used for transfer learning by freezing certain layers and modifying the architecture to reduce parameters, easing cancer detection. The HCPELM classifier was trained using the MIAS database and evaluated against benchmark methods. It achieved a breast image recognition accuracy of 86%, outperforming benchmark deep learning models. HCPELM is demonstrating superior performance in early detection and diagnosis, thus aiding healthcare practitioners in breast cancer diagnosis.

MARNet: Multi-head attention residual network for rolling bearing fault diagnosis under noisy condition

Rolling bearings are crucial components of rotating machinery, and their health states directly affect the overall performance of the machinery. Therefore, it is exceedingly necessary to detect and diagnose bearing faults. Numerous bearing fault diagnosis methods have been successfully used for ensuring the safe operation of rotating machinery. However, in practical working environments, there is a considerable amount of noise, resulting in traditional methods incapable of achieving accurate fault diagnosis. This paper proposes a new multi-head attention residual network (MARNet) for rolling bearing fault diagnosis under noisy condition. MARNet optimizes residual units by simplifying multi-layer convolutions into a single-layer convolution and replaces the rectified linear unit (ReLU) function with the exponential linear unit (ELU) function to obtain a more appropriate activation function. Additionally, the multi-head attention mechanism is introduced into the residual block to capture correlation information between any two time sequences, enhancing the network’s feature extraction capability. The effectiveness and superiority of the MARNet in noisy environments are demonstrated through conducting the two bearing datasets from Case Western Reserve University (CWRU) and Paderborn University (PU). The experiment results show that the proposed method exhibits anti-noise characteristics and generalization capability compared with several up-to-date deep learning methods for fault diagnosis of rolling bearings.

Establishment of a novel DNNSS-MOVES prediction model for carbon emissions of trucks driving on dirt roads

This study proposes a novel prediction model to accurately quantify the carbon emissions of the trucks driving on dirt roads based on the deep learning neural network for small sample (DNNSS) and the motor vehicle emission simulator (MOVES). The application range of the MOVES model was extended to the transportation of asphalt mixtures on the temporary road. The model correction method was also established based on the rolling resistance coefficient (CR) and the correction coefficient (μ). By comparing the measured fuel consumption of the truck with the carbon emissions calculated by the MOVES model, the values of CR × μ can be back-calculated. DNNSS was constructed for estimating the CR × μ. The Adaptive Moment Estimation (Adam) algorithm was used to dynamically adjust the learning rate and accelerate the convergence of the network; the Dropout function was used to alleviate overfitting; and the Rectified Linear Unit (ReLU) function was used as the activation function to solve the gradient vanishing problem. The test results showed that the vehicle speed and load greatly influence the CR × μ. The DNNSS algorithm was better at predicting the CR × μ. The proposed DNNSS-MOVES model was more accurate than the conventional methods.