Parametric Rectified Linear Unit Research Articles

Retraction Note: Alcoholism identification via convolutional neural network based on parametric ReLU, dropout, and batch normalization

Retraction Note: Alcoholism identification via convolutional neural network based on parametric ReLU, dropout, and batch normalization

Convolutional transformer-driven robust electrocardiogram signal denoising framework with adaptive parametric ReLU.

The electrocardiogram (ECG) is a widely used diagnostic tool for cardiovascular diseases. However, ECG recording is often subject to various noises, which can limit its clinical evaluation. To address this issue, we propose a novel Transformer-based convolutional neural network framework with adaptively parametric ReLU (APtrans-CNN) for ECG signal denoising. The proposed APtrans-CNN architecture combines the strengths of transformers in global feature learning and CNNs in local feature learning to address the inadequacy of learning with long sequence time-series features. By fully exploiting the global features of ECG signals, our framework can effectively extract critical information that is necessary for signal denoising. We also introduce an adaptively parametric ReLU that can assign a value to the negative information contained in the ECG signal, thereby overcoming the limitation of ReLU to retain negative information. Additionally, we introduce a dynamic feature aggregation module that enables automatic learning and retention of valuable features while discarding useless noise information. Results obtained from two datasets demonstrate that our proposed APtrans-CNN can accurately extract pure ECG signals from noisy datasets and is adaptable to various applications. Specifically, when the input consists of ECG signals with a signal-to-noise ratio (SNR) of -4 dB, APtrans-CNN successfully increases the SNR to more than 6 dB, resulting in the diagnostic model's accuracy exceeding 96%.

Feature Extraction and Classification of Gray-Scale Images of Brain Tumor using Deep Learning

Deep Learning using CNN plays a paramount role for the classification methods applied on medical image data. With a crucial role in accurate diagnosis, treatment planning and patient management for medical and healthcare systems, CNNs won accolades in the Deep Learning research. As simple the learning model so precise are the results for decision making. The proposed Sequential model of CNN is built with Parametric ReLU with the values aligned to geometric mean, attains a specific goal of tumor classification. The additional support of ground-truth aid in deciding the shape and severity of tumor in the Grayscale MRI of brain tumor. The simple Sequential model, although a minimal version has proved achieved significant classification goals using the GMP-ReLU. Comparative results with variants of ReLU have been charted in this article standing with the proof of consistent classification model with parametric-ReLU. The proposed design is conducted on images from Kaggle and a model is trained (classifier is built), which can be considered as ideal filter for all the benchmark images. The accuracy of proposed design is considerably improved compared to normal ReLU up to 89.214%.

Mask wearing detection algorithm based on improved YOLOv7

The ongoing COVID-19 pandemic remains a significant threat, emphasizing the critical importance of mask-wearing to reduce infection risks. However, existing methods for mask detection encounter challenges such as identifying small targets and achieving high accuracy. In this paper, we present an enhanced YOLOv7 model tailored for mask-wearing detection. we employing a Generative Adversarial Network (GAN) to augment the original dataset, introducing the Convolutional Block Attention Module (CBAM) mechanism into the YOLOv7 model to enhance its small target detection capabilities, and replacing the model’s activation function with Parametric Rectified Linear Unit (FReLU) to improve overall performance. Experimental validation on a dataset showcases an average precision of 97.8% and a real-time inference speed of 64 frames per second (fps), meeting the real-time mask-wearing detection requirements effectively.

Classification and Detection of Rice Diseases Using a 3-Stage CNN Architecture with Transfer Learning Approach

Rice is a vital crop for global food security, but its production is vulnerable to various diseases. Early detection and treatment of rice diseases are crucial to minimise yield losses. Convolutional neural networks (CNNs) have shown great potential for disease detection in plant leaves, but training CNNs requires large datasets of labelled images, which can be expensive and time-consuming. Here, we have experimented a 3-Stage CNN architecture with a transfer learning approach that utilises a pre-trained CNN model fine-tuned on a small dataset of rice disease images. The proposed approach significantly reduces the required training data while achieving high accuracy. We also incorporated deep learning techniques such as progressive re-sizing and parametric rectified linear unit (PReLU) to enhance rice disease detection. Progressive re-sizing improves feature learning by gradually increasing image size during training, while PReLU reduces overfitting and enhances model performance. The proposed approach was evaluated on a dataset of 8883 and 1200 images of disease and healthy rice leaves, respectively, achieving an accuracy of 94% when subjected to the 10-fold cross-validation process, significantly higher than other methods. These simulation results for disease detection in rice prove the feasibility and efficiency and offer a cost-effective, accessible solution for the early detection of rice diseases, particularly useful in developing countries with limited resources that can significantly contribute toward sustainable food production.

Learning Bilateral Clipping Parametric Activation for Low-Bit Neural Networks

Among various network compression methods, network quantization has developed rapidly due to its superior compression performance. However, trivial activation quantization schemes limit the compression performance of network quantization. Most conventional activation quantization methods directly utilize the rectified activation functions to quantize models, yet their unbounded outputs generally yield drastic accuracy degradation. To tackle this problem, we propose a comprehensive activation quantization technique namely Bilateral Clipping Parametric Rectified Linear Unit (BCPReLU) as a generalized version of all rectified activation functions, which limits the quantization range more flexibly during training. Specifically, trainable slopes and thresholds are introduced for both positive and negative inputs to find more flexible quantization scales. We theoretically demonstrate that BCPReLU has approximately the same expressive power as the corresponding unbounded version and establish its convergence in low-bit quantization networks. Extensive experiments on a variety of datasets and network architectures demonstrate the effectiveness of our trainable clipping activation function.

DPReLU: Dynamic Parametric Rectified Linear Unit and Its Proper Weight Initialization Method

Activation functions are essential in deep learning, and the rectified linear unit (ReLU) is the most widely used activation function to solve the vanishing gradient problem. However, owing to the dying ReLU problem and bias shift effect, deep learning models using ReLU cannot exploit the potential benefits of negative values. Numerous ReLU variants have been proposed to address this issue. In this study, we propose Dynamic Parametric ReLU (DPReLU), which can dynamically control the overall functional shape of ReLU with four learnable parameters. The parameters of DPReLU are determined by training rather than by humans, thereby making the formulation more suitable and flexible for each model and dataset. Furthermore, we propose an appropriate and robust weight initialization method for DPReLU. To evaluate DPReLU and its weight initialization method, we performed two experiments on various image datasets: one using an autoencoder for image generation and the other using the ResNet50 for image classification. The results show that DPReLU and our weight initialization method provide faster convergence and better accuracy than the original ReLU and the previous ReLU variants.

Fault Diagnosis of Conventional Circuit Breaker Accessories Based on Grayscale Image of Current Signal and Improved ZFNet-DRN

Considering that the randomness of the closing phase of the coil circuit for a low-voltage conventional circuit breaker (CCB) will lead to the difference of current signal under the same operation, a fault diagnosis method of switching accessories based on the grayscale image of the current signal and deep residual network (DRN) improved by Zeiler and Fergus Net (ZFNet-DRN) is proposed in this article. First, the signal stacking method is employed to convert the original 1-D current signal into a 2-D grayscale image, so as to retain the information contained in the signal to the greatest extent and avoid the insufficient amount of data input to the subsequent network model due to the limitation of 1-D data length. Second, regarding the diagnosis model, the improved ZFNet network realizes the deep extraction of features, DRN tackles the issue that the recognition rate decreases with the increase in the network depth, and the parametric rectified linear unit (PReLU) activation function and AMSGrad optimization algorithm are introduced to enhance its performance. Finally, the experimental analysis demonstrates that the diagnosis model can overcome the influence of the randomness of the closing phase and effectively complete the fault classification.

Deep Learning-Based Beamforming for Millimeter-Wave Systems Using Parametric ReLU Activation Function

Beamforming design is a crucial stage in millimeter-wave systems with massive antenna arrays. We propose a deep learning network for the design of the precoder and combiner in hybrid architectures. The proposed network employs a parametric rectified linear unit (PReLU) activation function which improves model accuracy with almost no complexity cost compared to other functions. The proposed network accepts practical channel estimation input and can be trained to enhance spectral efficiency considering the hardware limitation of the hybrid design. Simulation shows that the proposed network achieves small performance improvement when compared to the same network with the ReLU activation function.

Classification of microscopic cervical blood cells using inception ResNet V2 with modified activation function

Cervical cancer is the most frequent and fatal malignancy among women worldwide. If this tumor is detected and treated early enough, the complications it causes can be minimized. Deep learning demonstrated significant promise when imposed on biomedical difficulties such as medical image processing and disease prognostication. Therefore, in this paper, an automatic cervical cell classification approach named IR-PapNet is developed based on Inception-ResNet which is an optimized version of Inception. The learning model’s conventional ReLu activation is replaced with the parametric-rectified linear unit (PReLu) to overcome the nullification of negative values and dying ReLu. Finally, the model loss function is minimized with the SGD optimization model by modifying the attributes of the neural network. Furthermore, we present a simple but efficient noise removal technique called 2D-Discrete Wavelet Transform (2D-DWT) algorithm for enhancing image quality. Experimental results show that this model can achieve a top-1 average identification accuracy of 99.8% on the pap smear cervical Herlev datasets, which verifies its satisfactory performance. The restructured Inception-ResNet network model can obtain significant improvements over most of the state-of-the-art models in 2-class classification, and it achieves a high learning rate without experiencing dead nodes.

Improved Variational Mode Decomposition and One-Dimensional CNN Network with Parametric Rectified Linear Unit (PReLU) Approach for Rolling Bearing Fault Diagnosis

As a critical component of rotating machinery, rolling bearings are essential for the safe and efficient operation of machinery. Sudden faults of rolling bearings can lead to unscheduled downtime and substantial economic costs. Therefore, diagnosing and identifying the equipment status is essential for ensuring the operation and decreasing the additional maintenance costs of the machines. However, extracting the features from the early bearing fault signals is challenging under background noise interference. With the purpose of solving the above problem, we propose an integrated rolling bearing fault diagnosis model based on the improved grey wolf optimized variational modal decomposition (IGVMD) and an improved 1DCNN with a parametric rectified linear unit (PReLU). Firstly, an improved grey wolf optimizer (IGWO) with the fitness function, the minimum envelope entropy, is designed for adaptively searching the optimal parameter values of the VMD model. The performance of the basic grey wolf optimizer (GWO) algorithm by introducing three improvement strategies, the non-linear convergence factor adjustment strategy, the grey wolf adaptive position update strategy, and the Levy flight strategy in the IGWO algorithm, is improved. Then, an improved 1DCNN model with the PReLU activation function is proposed, which extracts the bearing fault features, and a grid search to optimize the model parameters of the 1DCNN is introduced. Finally, the effectiveness of the proposed model is demonstrated well by employing two experimental datasets. The preliminary comparative results of the average identification accuracy in the proposed method in two datasets are 99.98% and 99.50%, respectively, suggesting that this proposed method has a relatively higher recognition accuracy and application values.

Deep-learning-based surrogate model for fast and accurate simulation in pipeline transport

A new deep-learning-based surrogate model is developed and applied for predicting dynamic temperature, pressure, gas rate, oil rate, and water rate with different boundary conditions in pipeline flow. The surrogate model is based on the multilayer perceptron (MLP), batch normalization and Parametric Rectified Linear Unit techniques. In training, the loss function for data mismatch is considered to optimize the model parameters with means absolute error (MAE). In addition, we also use the dynamic weights, calculated by the input data value, to increase the contribution of smaller inputs and avoid errors caused by large values eating small values in total loss. Finally, the surrogate model is applied to simulate a complex pipeline flow in the eastern part of the South China Sea. We use flow and pressure boundary as the input data in the numerical experiment. A total of 215690 high-fidelity training simulations are performed in the offline stage with commercial software LeadFlow, in which 172552 simulation runs are used for training the surrogate model, which takes about 240 min on an RTX2060 graphics processing unit. Then the trained model is used to provide pipeline flow forecasts under various boundary conduction. As a result, it is consistent with those obtained from the high-fidelity simulations (e.g., the media of relative error for temperature is 0.56%, pressure is 0.79%, the gas rate is 1.02%, and oil rate is 1.85%, and water is 0.80%, respectively). The online computations from our surrogate model, about 0.008 s per run, achieve speedups of over 1,250 relative to the high-fidelity simulations, about 10 s per run. Overall, this model provides reliable and fast predictions of the dynamic flow along the pipeline.

A Semi-Automatic Wheelchair with Navigation Based on Virtual-Real 2D Grid Maps and EEG Signals

A semi-automatic wheelchair allows disabled people to possibly control in an indoor environment with obstacles and targets. The paper proposes an EEG-based control system for the wheelchair based on a grid map designed to allow disabled people to reach any preset destination. In particular, the grid map is constructed by dividing it into grid cells that may contain free spaces or obstacles. The map with the grid cells is simulated to find the optimal paths to the target positions using a Deep Q-Networks (DQNs) model with the Parametric Rectified Linear Unit (PReLU) activation function, in which a novel algorithm for finding the optimal path planning by converting wheelchair actions is applied using the output parameters of the DQNs. For the wheelchair movement in one real indoor environment corresponding to the virtual 2D grid map, the initial position of the wheelchair will be determined based on natural landmarks and a graphical user interface designed for on-screen display can support disabled people in selecting the desired destination from a list of predefined locations using Electroencephalogram (EEG) signals by blinking eyes. Therefore, one user can easily and safely control the wheelchair using an EEG system to reach the desired target when the wheelchair position and destination are determined in the indoor environment. As a result, a grid map was developed and experiments for the semi-automatic wheelchair control were performed in real indoor environments to illustrate the effectiveness of the proposed method. In addition, the system is a platform to develop different types of controls depending on the types of user disabilities and different environmental maps built.

Binarized Aggregated Network With Quantization: Flexible Deep Learning Deployment for CSI Feedback in Massive MIMO Systems

Massive multiple-input multiple-output (MIMO) is one of the key techniques to achieve better spectrum and energy efficiency in 5G system. The channel state information (CSI) needs to be fed back from the user equipment to the base station in frequency division duplexing (FDD) mode. However, the overhead of the direct feedback is unacceptable due to the large antenna array in massive MIMO system. Recently, deep learning is widely adopted to the compressed CSI feedback task and proved to be effective. In this paper, a novel network named aggregated channel reconstruction network (ACRNet) is designed to boost the feedback performance with network aggregation and parametric rectified linear unit (PReLU) activation. The practical deployment of the feedback network in the communication system is also considered. Specifically, the elastic feedback scheme is proposed to flexibly adapt the network to meet different resource limitations. Besides, the network binarization technique is combined with the feature quantization for lightweight and practical deployment. Experiments show that the proposed ACRNet outperforms loads of previous state-of-the-art networks, providing a neat feedback solution with high performance, low cost and impressive flexibility.

Deep Neural Network with Adaptive Parametric Rectified Linear Units and its Fast Learning

The adaptive parametric rectified linear unit (AdPReLU) as an activation function of the deep neural network is proposed in the article. The main benefit of the proposed system is adjusted activation function whose parameters are tuning parallel with synaptic weights in online mode. The algorithm of the simultaneous learning of all neurons parameters with AdPReLU and the modified backpropagation procedure based on this algorithm is introduced. The approach under consideration permits to reduce volume of the training data set and increase tuning speed of the DNN with AdPReLU. The proposed approach could be applied in the deep convolutional neural networks (CNN) in conditions of the small value of training data sets and additional requirements for system performance. The main feature of DNN under consideration is possibility to tune not only synaptic weights but the parameters of activation function too. The effectiveness of this approach is proved by experimental modeling.

Progressive 3-Layered Block Architecture for Image Classification

Convolutional Neural Networks (CNNs) have been used to handle a wide range of computer vision problems, including image classification and object detection. Image classification refers to automatically classifying a huge number of images and various techniques have been developed for accomplishing this goal. The focus of this article is to enhance image classification accuracy implemented on CNN models by using the concept of transfer learning and progressive resizing with split and train strategy. Furthermore, the Parametric Rectified Linear Unit (PReLU) activation function, which generalizes the standard traditional rectified unit, has also been applied on dense layers of the model. PReLU enhances model fitting with almost little significant computational cost and low over-fitting hazard. A “Progressive 3-Layered Block Architecture" model is proposed in this paper which considers the fine-tuning of hyperparameters and optimizers of the Deep network to achieve state-of-the-art accuracy on benchmark datasets with fewer parameters.

Dual Attention Convolutional Neural Network Based on Adaptive Parametric ReLU for Denoising ECG Signals with Strong Noise.

Electrocardiogram (ECG) signal is one of the most important methods for diagnosing cardiovascular diseases but is usually affected by noises. Denoising is therefore necessary before further analysis. Deep learning-related methods have been applied to image processing and other domains with great success but are rarely used for denoising ECG signals. This paper proposes an effective and simple model of encoder-decoder structure for denoising ECG signals (APR-CNN). Specifically, Adaptive Parametric ReLU (APReLU) and Dual Attention Module (DAM) are introduced in the model. Rectified Linear Unit (ReLU) is replaced with the APReLU for better negative information retainment. The DAM is an attention-based module consisting of a channel attention module and spatial attention module, through which the inter-spatial and inter-channel relationship of the input data are exploited. We tested our model on the MIT-BIH dataset, and the results show that the APR-CNN can handle ECG signals with a different signal-to-noise ratio (SNR). The comparative experiment proves our model is better than other deep learning and traditional methods.Clinical Relevance- This paper proposed a method capable of denoising ECG signals with strong noise to alleviate difficulties for further medical analysis.

Adaptive Levenberg–Marquardt Algorithm: A New Optimization Strategy for Levenberg–Marquardt Neural Networks

Engineering data are often highly nonlinear and contain high-frequency noise, so the Levenberg–Marquardt (LM) algorithm may not converge when a neural network optimized by the algorithm is trained with engineering data. In this work, we analyzed the reasons for the LM neural network’s poor convergence commonly associated with the LM algorithm. Specifically, the effects of different activation functions such as Sigmoid, Tanh, Rectified Linear Unit (RELU) and Parametric Rectified Linear Unit (PRLU) were evaluated on the general performance of LM neural networks, and special values of LM neural network parameters were found that could make the LM algorithm converge poorly. We proposed an adaptive LM (AdaLM) algorithm to solve the problem of the LM algorithm. The algorithm coordinates the descent direction and the descent step by the iteration number, which can prevent falling into the local minimum value and avoid the influence of the parameter state of LM neural networks. We compared the AdaLM algorithm with the traditional LM algorithm and its variants in terms of accuracy and speed in the context of testing common datasets and aero-engine data, and the results verified the effectiveness of the AdaLM algorithm.

Denoising of magnetic resonance images using discriminative learning-based deep convolutional neural network.

The noise in magnetic resonance (MR) images causes severe issues for medical diagnosis purposes. In this paper, we propose a discriminative learning based convolutional neural network denoiser to denoise the MR image data contaminated with noise. The proposed method incorporates the use of depthwise separable convolution along with local response normalization with modified hyperparameters and internal skip connections to denoise the contaminated MR images. Moreover, the addition of parametric RELU instead of normal conventional RELU in our proposed architecture gives more stable and fine results. The denoised images were further segmented to test the appropriateness of the results. The network is trained on one dataset and tested on other dataset produces remarkably good results. Our proposed network was used to denoise the images of different noise levels, and it yields better performance as compared with various networks. The SSIM and PSNR showed an average improvement of (7.2 ± 0.002) % and (8.5 ± 0.25) % respectively when tested on different datasets without retaining the network. An improvement of 5% and 6% was achieved in the values of mean intersection over union (mIoU) and BF score when the denoised images were segmented for testing the relevancy in biomedical imaging applications. The statistical test suggests that the obtained results are statistically significant as p< 0.05. The denoised images obtained are more clinically suitable for medical image diagnosis purposes, as depicted by the evaluation parameters. Further, external clinical validation was performed by an experienced radiologist for testing the validation of the resulting images.

An optimized adaptive PReLU-DBN for rolling element bearing fault diagnosis

Rolling element bearings are critical components in industrial rotating machines. Faults and failures of bearings can cause degradation of machine performance or even a catastrophe. Therefore, it is significant to perform bearing fault diagnosis accurately and effectively. Deep Learning based approaches are promising for bearing diagnosis. They can extract fault information efficiently and conduct accurate diagnosis. However, the structure of deep learning networks is often determined by trial and error, which is time consuming and lacks theoretical guidance. Besides, the traditional deep learning approaches have low diagnosis accuracy and learning efficiency. To address these problems, this paper proposes a rolling element bearing fault diagnosis approach based on principal component analysis and adaptive deep belief network with Parametric Rectified Linear Unit activation layers. In the proposed approach, particle swarm optimization is integrated to obtain an optimal DBN structure with high accuracy and convergence rate. Experiments on tapered roller bearings and comparison studies with state-of-the-art methods are conducted to demonstrate the effectiveness and accuracy of the proposed approach.