Deep Learning Neural Network Research Articles

A reconfigurable multi-precision quantization-aware nonlinear activation function hardware module for DNNs

In recent years, the increasing variety of nonlinear activation functions (NAFs) in deep neural networks (DNNs) has led to higher computational demands. However, hardware implementation faces challenges such as lack of flexibility, high hardware cost, and limited accuracy. This paper proposes a highly flexible and low-cost hardware solution for implementing activation functions to overcome these issues. Based on the piecewise linear (PWL) approximation method, our method supports NAFs with different accuracy configurations through a customized implementation strategy to meet the requirements in different scenario applications. In this paper, the symmetry of the activation function is investigated, and incorporate curve translation preprocessing and data quantization to significantly reduce hardware storage costs. The modular hardware architecture proposed in this study supports NAFs of multiple accuracies, which is suitable for designing deep learning neural network accelerators in various scenarios, avoiding the need to design dedicated hardware circuits for the activation function layer and enhances circuit design efficiency. The proposed hardware architecture is validated on the Xilinx XC7Z010 development board. The experimental results show that the average absolute error (AAE) is reduced by about 35.6 % at a clock frequency of 312.5 MHz. Additionally, the accuracy loss of the model is maximized to −0.684 % after replacing the activation layer function of DNNs under the PyTorch framework.

Enhancing effort estimation in global software development using a unique combination of Neuro Fuzzy Logic and Deep Learning Neural Networks (NFDLNN)

ABSTRACT Effective project planning and management in the global software development landscape relies on addressing major issues like cost estimation and effort allocation. Timely estimation of software development is a critical focus in software engineering research. With the industry increasingly relying on diverse teams worldwide, accurate estimation becomes vital. Software size serves as a common measure for costs and schedules, but advanced estimation methods consider various variables, such as project purpose, personnel expertise, time and efficiency constraints, and technology requirements. Estimating software costs involve significant financial and strategic commitments, making it crucial to address complexity and versatility related to cost drivers. To achieve enhanced accuracy and convergence, we employ the cuckoo algorithm in our proposed NFDLNN (Neuro Fuzzy Logic and Deep Learning Neural Networks) model. Through extensive validation with industrial project data, using Function Point Analysis as the algorithmic models, our NFA model demonstrates high accuracy in software cost approximation, outperforming existing methods insights of MRE of 3.33, BRE of 0.13, and PI of 74.48. Our research contributes to improved project planning and decision-making processes in global software development endeavours.

Predicting air quality index and fine particulate matter levels in Bagdad city using advanced machine learning and deep learning techniques

Particulate matter pollution is recognized globally as one of the most hazardous forms of air pollution, profoundly impacting environmental integrity and public health. Key metrics for assessing this pollution include the Air Quality Index (AQI) and fine particulate matter with diameters ≤2.5 μm (PM2.5). These indicators are closely associated with severe health consequences, such as premature death from chronic exposure. While traditional statistical methods have been employed in some studies to evaluate AQI and PM2.5, the application of advanced machine learning techniques has been limited. This research employs deep learning and artificial neural networks (ANN) to forecast AQI and PM2.5 levels in Baghdad, Iraq. The study utilizes an extensive dataset from July 1, 2016, to December 12, 2021, comprising over 48,000 data points for AQI and PM2.5. Time serves as an independent input variable influencing these dependent variables. The analysis employs a diverse set of machine learning algorithms, including random forest, decision tree, K-Nearest Neighbors (KNN), Multi-Layer Perceptron (MLP), and Long Short-Term Memory networks (LSTM). The findings demonstrate that MLP and LSTM models outperform other methods, providing the most accurate predictions. The correlation coefficients were 0.977 and 0.983 for the prediction of AQI and 0.973 and 0.985 for the prediction of PM2.5 using MLP and LSTM, respectively. In addition, the outcomes showed that both AQI and PM2.5 were within the moderate to unhealthy ranges, and their distribution levels pointed to the need for addressing air quality in Baghdad city. Furthermore, this study contributes to the burgeoning field of machine learning applications in environmental science by establishing a robust and nuanced predictive framework for evaluating air quality. It highlights the potential of deep learning in public health applications and offers actionable insights for policymaking to mitigate air pollution and its adverse effects.

Predicting Structural Temperature of Ancient City Walls: A Case Study Using Ambient Temperature and Deep Learning

ABSTRACT Accurate prediction of the structural temperature field is required for the performance assessment of engineering structures and for maintaining structural safety. However, there are complex time-lag effects between ambient temperature and structural temperature. It is difficult to accurately describe and predict the structural temperature using traditional linear and nonlinear fitting methods. Hence, this study uses the advantages of gated recurrent unit (GRU) neural networks in dealing with complex nonlinear mapping issues, further considering solar radiation and developing a structural temperature prediction framework using ambient temperature and the deep learning model. The framework is applied in structural temperature prediction for an ancient city wall in China. The solar altitude angle at the location of the city wall is calculated, and the input and output structures of the deep learning neural network model are optimized. In the long-term prediction of the structural temperature of the city wall, the root mean square error (RMSE) of the GRU model is less than 0.6°C, which solves the problem of low accuracy caused by time-lag effects.

Quantitative modelling of Plato and total flavonoids in Qingke wort at mashing and boiling stages based on FT-IR combined with deep learning and chemometrics

Craft beer brewers need to learn process control strategies from traditional industrial production to ensure the consistent quality of the finished product. In this study, FT-IR combined with deep learning was used for the first time to model and analyze the Plato degree and total flavonoid content of Qingke beer during the mashing and boiling stages and to compare the effectiveness with traditional chemometrics methods. Two deep learning neural networks were designed, the effect of variable input methods on the effectiveness of the models was discussed. The experimental results showed that the CARS-LSTM model had the best predictive performance, not only as the best quantitative model for Plato in the mashing (R2p = 0.9368) and boiling (R2p = 0.9398) phases but also as the best model for TFC in the boiling phase (R2p = 0.9154). This study demonstrates the great potential of deep learning and provides a new approach to quality control analysis in beer brewing.

Prediction of Accident Risk Levels in Traffic Accidents Using Deep Learning and Radial Basis Function Neural Networks Applied to a Dataset with Information on Driving Events

A complex AI system must be worked offline because the training and execution phases are processed separately. This process often requires different computer resources due to the high model requirements. A limitation of this approach is the convoluted training process that needs to be repeated to obtain models with new data continuously incorporated into the knowledge base. Although the environment may be not static, it is crucial to dynamically train models by integrating new information during execution. In this article, artificial neural networks (ANNs) are developed to predict risk levels in traffic accidents with relatively simpler configurations than a deep learning (DL) model, which is more computationally intensive. The objective is to demonstrate that efficient, fast, and comparable results can be obtained using simple architectures such as that offered by the Radial Basis Function neural network (RBFNN). This work led to the generation of a driving dataset, which was subsequently validated for testing ANN models. The driving dataset simulated the dynamic approach by adding new data to the training on-the-fly, given the constant changes in the drivers’ data, vehicle information, environmental conditions, and traffic accidents. This study compares the processing time and performance of a Convolutional Neural Network (CNN), Random Forest (RF), Radial Basis Function (RBF), and Multilayer Perceptron (MLP), using evaluation metrics of accuracy, Specificity, and Sensitivity-recall to recommend an appropriate, simple, and fast ANN architecture that can be implemented in a secure alert traffic system that uses encrypted data.

Threshold Active Learning Approach for Physical Violence Detection on Images Obtained from Video (Frame-Level) Using Pre-Trained Deep Learning Neural Network Models

Threshold Active Learning Approach for Physical Violence Detection on Images Obtained from Video (Frame-Level) Using Pre-Trained Deep Learning Neural Network Models

Evaluation of Denoising Performance of ResNet Deep Learning Model for Ultrasound Images Corresponding to Two Frequency Parameters.

Ultrasound imaging is widely used for accurate diagnosis due to its noninvasive nature and the absence of radiation exposure, which is achieved by controlling the scan frequency. In addition, Gaussian and speckle noises degrade image quality. To address this issue, filtering techniques are typically used in the spatial domain. Recently, deep learning models have been increasingly applied in the field of medical imaging. In this study, we evaluated the effectiveness of a convolutional neural network-based residual network (ResNet) deep learning model for noise reduction when Gaussian and speckle noises were present. We compared the results with those obtained from conventional filtering techniques. A dataset of 500 images was prepared, and Gaussian and speckle noises were added to create noisy input images. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. The ResNet deep learning model, comprising 16 residual blocks, was trained using optimized hyperparameters, including the learning rate, optimization function, and loss function. For quantitative analysis, we calculated the normalized noise power spectrum, peak signal-to-noise ratio, and root mean square error. Our findings showed that the ResNet deep learning model exhibited superior noise reduction performance to median, Wiener, and median-modified Wiener filter algorithms.

The development history and future prospects of artificial intelligence in facial recognition technology

In recent decades, with the fast development of various algorithms and the rapid improvement of computer hardware performance, the development of facial recognition technology has been very rapid. This technology used to only appear in science fiction movies, but now it has been widely used in mobile payments, identity recognition, and other fields, and it also has very reliable performance. This article introduces the different facial recognition methods used in different periods of facial recognition technology from the 1960s and 1970s to the present day, to help people understand the development process of facial recognition technology. Among them, the focus is on early methods based on geometric features, grayscale information, and feature extraction, as well as recent applications based on deep learning and convolutional neural networks. In response to some challenges faced by current technology, some speculations on possible future development directions are proposed.

Wavelet Transform, Reconstructed Phase Space, and Deep Learning Neural Networks for EEG-Based Schizophrenia Detection.

This study proposes an innovative expert system that uses exclusively EEG signals to diagnose schizophrenia in its early stages. For diagnosing psychiatric/neurological disorders, electroencephalogram (EEG) testing is considered a financially viable, safe, and reliable alternative. Using the reconstructed phase space (RPS) and the continuous wavelet transform, the researchers maximized the differences between the EEG nonstationary signals of normal and schizophrenia individuals, which cannot be observed in the time, frequency, or time-frequency domains. This reveals significant information, highlighting more distinguishable features. Then, a deep learning network was trained to enhance the accuracy of the resulting image classification. The algorithm's efficacy was confirmed through three distinct methods: employing 70% of the dataset for training, 15% for validation, and the remaining 15% for testing. This was followed by a 5-fold cross-validation technique and a leave-one-out classification approach. Each method was iterated 100 times to ascertain the algorithm's robustness. The performance metrics derived from these tests - accuracy, precision, sensitivity, F1 score, Matthews correlation coefficient, and Kappa - indicated remarkable outcomes. The algorithm demonstrated steady performance across all evaluation strategies, underscoring its relevance and reliability. The outcomes validate the system's accuracy, precision, sensitivity, and robustness by showcasing its capability to autonomously differentiate individuals diagnosed with schizophrenia from those in a state of normal health.

Transfer learning enables the rapid design of single crystal superalloys with superior creep resistances at ultrahigh temperature

Accelerating the design of Ni-based single crystal (SX) superalloys with superior creep resistance at ultrahigh temperatures is a desirable goal but extremely challenging task. In the present work, a deep transfer learning neural network with physical constraints for creep rupture life prediction at ultrahigh temperatures is constructed. Transfer learning enables deep learning model breaks through the generalization performance barrier in the extrapolation space of ultrahigh temperature creep properties in the case of a very small dataset, which is the key to achieving the above design goal. Transfer learning is demonstrated to be effective in utilizing the prior compositional sensitivities information contained in the pre-trained model, and motivates the fine-tuned model to capture the particular relationship between composition and creep rupture life at ultrahigh temperature. Aiming to find advanced SX superalloys applied at 1200 °C, the proposed transfer learning-based model guides us to design a superalloy with a verified creep rupture life of ~170 h at 80 MPa, which exceeds the state-of-art value by 30%. The improved γ/γ′ interface strengthening, which is effectively regulated by the Mo/Ta ratio to form γ′ rafting with longer, flatter interfaces and achieve stronger interfacial bonding, is revealed as the dominant mechanism behind combining experiments and first-principles calculations. Moreover, the excellent extrapolation ability of the proposed model is further confirmed to enhance the efficiency of active learning by reducing its dependence on the initial dataset size. This study provides a pioneering AI-driven approach for the rapid development of Ni-based SX superalloys applied in advanced aero-engine blades.

Enhancing water quality monitoring through the integration of deep learning neural networks and fuzzy method

The escalating growth of the global population has led to degraded water quality, particularly in seawater environments. Water quality monitoring is crucial to understanding the dynamic changes and implementing effective management strategies. In this study, water samples from the southwestern regions of Iran were spatially analyzed in a GIS environment using geostatistical methods. Subsequently, a water quality map was generated employing large and small fuzzy membership functions. Additionally, advanced prediction models using neural networks were employed to forecast future water pollution trends. Fuzzy method results indicated higher pollution levels in the northern regions of the study area compared to the southern parts. Furthermore, the water quality prediction models demonstrated that the LSTM model exhibited superior predictive performance (R2 = 0.93, RMSE = 0.007). The findings also underscore the impact of urbanization, power plant construction (2010 to 2020), and inadequate urban wastewater management on water pollution in the studied region.

B cells in perivascular and peribronchiolar granuloma-associated lymphoid tissue and B-cell signatures identify asymptomatic Mycobacterium tuberculosis lung infection in Diversity Outbred mice.

Because most humans resist Mycobacterium tuberculosis infection, there is a paucity of lung samples to study. To address this gap, we infected Diversity Outbred mice with M. tuberculosis and studied the lungs of mice in different disease states. After a low-dose aerosol infection, progressors succumbed to acute, inflammatory lung disease within 60 days, while controllers maintained asymptomatic infection for at least 60 days, and then developed chronic pulmonary tuberculosis (TB) lasting months to more than 1 year. Here, we identified features of asymptomatic M. tuberculosis infection by applying computational and statistical approaches to multimodal data sets. Cytokines and anti-M. tuberculosis cell wall antibodies discriminated progressors vs controllers with chronic pulmonary TB but could not classify mice with asymptomatic infection. However, a novel deep-learning neural network trained on lung granuloma images was able to accurately classify asymptomatically infected lungs vs acute pulmonary TB in progressors vs chronic pulmonary TB in controllers, and discrimination was based on perivascular and peribronchiolar lymphocytes. Because the discriminatory lesion was rich in lymphocytes and CD4 T cell-mediated immunity is required for resistance, we expected CD4 T-cell genes would be elevated in asymptomatic infection. However, the significantly different, highly expressed genes were from B-cell pathways (e.g., Bank1, Cd19, Cd79, Fcmr, Ms4a1, Pax5, and H2-Ob), and CD20+ B cells were enriched in the perivascular and peribronchiolar regions of mice with asymptomatic M. tuberculosis infection. Together, these results indicate that genetically controlled B-cell responses are important for establishing asymptomatic M. tuberculosis lung infection.

A Comprehensive Review on Diabetic Retinopathy Detection Techniques using Neural Network Architectures

Diabetic retinopathy (DR) is a significant complication arising from diabetes, affecting the eyes and potentially causing vision loss if not identified and addressed promptly. Over the years, there has been a significant advancement in the field of DR detection, primarily driven by advancements in imaging techniques and machine learning algorithms. This review paper presents a comprehensive overview of different techniques and advancements in the detection of diabetic retinopathy using deep learning and several neural network architectures. The comparative study of the existing datasets for the DR detection with the benefits, challenges and possible solutions for each dataset is also provided. The paper discusses the methods, preprocessing, implementation platforms and results of various implementation of CNN architectures like Deep CNN, CNN with Transfer Learning, Capsule Networks and DNN. The objective of this paper is to furnish researchers and clinicians with a thorough understanding of the present status of diabetic retinopathy detection, highlight the strengths and limitations of existing approaches, and identify future research directions in this vital area of healthcare.

Mapping and modeling groundwater potential using machine learning, deep learning and ensemble learning models in the Saiss basin (Fez-Meknes region, Morocco)

The Saïss basin in the Fez-Meknes region of Morocco, covering approximately 2100 km2, faces increased water demand due to population growth, economic development, and climate change, making groundwater a crucial resource. This study aims to delineate areas with groundwater potential (GWP) and evaluate the performance of various machine learning, deep learning, and hybrid ensemble models in predicting GWP. Using a dataset of 440 springs and wells, and 20 groundwater conditioning factors (GWCF) including topographical, hydrological, geological, and hydrogeological features, the study employed multi-collinearity analysis, variance inflation factor (VIF), tolerance (Tol) assessments, and an Information Gain (IG) test to analyze these factors. The study compared the performance of three machine learning algorithms (Gaussian Naive Bayes (GNB), k-Nearest Neighbors (KNN), Gradient Boosting Classifier (GBC)), three deep learning algorithms (Deep Learning Neural Networks (DLNN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN)), and a hybrid ensemble model (Random Forests (RF), Support Vector Machine (SVM), Logistic Regression (LR)) using the area under the receiver operating characteristic curve (ROC-AUC) as the evaluation metric. The results showed that the hybrid ensemble model had the highest AUC of 0.86, followed by GBC (AUC = 0.85), DLNN (AUC = 0.84), CNN (AUC = 0.83), KNN (AUC = 0.79), RNN (AUC = 0.78), and GNB (AUC = 0.75). The study revealed that 45% of the Saïss Basin exhibits high to very high GWP, particularly in Ain Taoujdat, Haj Kaddour, and Boufekrane districts, with lithology, slope, and transmissivity being the most influential factors. The resulting GWP map can guide decision-makers in planning well and borehole drilling for drinking water and agriculture, as well as artificial recharge projects, thus promoting sustainable groundwater management in the Saïss basin.

Sea clutter suppression using neural network

Sea clutter suppression is a critical task in radar systems to enhance target detection performance in complex naval environments and at coastlines. This paper discusses the use of neural networks for marine clutter suppression and coastal surveillance radar clutter suppression. Effective maritime clutter suppression is made possible by the Feed Forward Neural Network (FFNN) and Principal Component Analysis (PCA) based clutter reduction method, which uses neural network deep learning capabilities to automatically identify and extract features and patterns from raw radar data. Support Vector Machine (SVM) is utilized for clutter suppression along the shoreline. To train and test the network model, a sizable collection of radar measurements, including clutter and target echoes, is gathered. After pre-processing, the gathered data is used in a specially created model, which uses its underlying patterns to distinguish between target echoes and clutter. Then, clutter in real-time radar signals is suppressed using the learned neural network models, improving the detection of targets on the sea and at the coastline. Performance measures Structural Similarity (SSIM) and Signal to Noise Ratio (SNR) shows that the proposed method provides improved clutter reduction. Highlights Maritime clutter suppression is required for better visibility of targets Neural Networks provides better clutter suppression Clutter reduction using PPI images helps in Surveillance applications Objective performance measures SNR and SSI are used for performance analysis Better training and good dataset helps in improved clutter reduction for neural network approaches

<i>De novo</i> design and virtual screening of potential Bcr-Abl tyrosine kinase inhibitors using deep learning and molecular modeling technologies

De novo design and virtual screening of small-molecule compounds with a high potential inhibitory activity against the Bcr-Abl tyrosine kinase playing a key role in the pathogenesis of chronic myeloid leukemia (CML) were carried out by an integrated computational approach including technologies of deep learning and molecular modeling. As a result, according to the calculation data we identified 5 compounds exhibiting low values of binding free energy to the enzyme comparable with those predicted for imatinib, nilotinib and ponatinib, anticancer drugs widely used in the clinic to treat patients with CML. It was shown that these compounds are able to form stable complexes with the ATP-binding sites of the Bcr-Abl tyrosine kinase and its mutant form T315I, which is confirmed by the analysis of the profiles of binding affinity and intermolecular interactions responsible for their energy stabilization. Based on the obtained data, these compounds, which have been generated by the deep learning neural network, are assumed to form promising basic structures for development of new effective drugs for treatment of patients with CML.

Assessment of Pelvic Tilt in Anteroposterior Radiographs by Area Ratio Based on Deep Learning.

Diagnostics. Based on deep learning semantic segmentation model, we sought to assess pelvic tilt by area ratio of the lesser pelvic and the obturator foramen in anteroposterior (AP) radiographs. Pelvic tilt is a critical factor in hip and spinal surgery, commonly evaluated by medical professionals through sagittal pelvic radiographs. The inherent pelvic asymmetry, as well as potential obstructions from clothing and musculature in roentgenography, may result in ghosting and blurring artifacts, thereby complicating precise measurement. PT directly affects the area ratio of the lesser pelvis to the obturator foramen in AP radiographs. An exponential regression analysis of simulated radiographs from ten male and ten female pelvises in specific tilt positions derived a formula correlating this area ratio with PT. Two blinded investigators evaluated this formula using 161 simulated AP pelvic radiographs. A deep learning semantic segmentation model was then fine-tuned to automatically calculate the area ratio, enabling intelligent PT evaluation. This model and the regression function were integrated for automated PT measurement and tested on a dataset of 231 clinical cases. We observed no disparity between men and women in the aforementioned area ratio. The test results from two blinded investigators analyzing 161 simulated radiographs revealed a mean absolute error of 0.19° (SD ±4.71°), with a correlation coefficient between them reaching 0.96. In addition, the mean absolute error obtained from testing 231 clinical AP radiographs using the fine-tuned semantic segmentation model mentioned earlier is -0.58° (SD ±5.97°). We found that using deep learning neural networks enabled a more accurate and robust automatic measurement of PT through the area ratio of the lesser pelvis and obturator foramen.

Establishment of a novel DNNSS-MOVES prediction model for carbon emissions of trucks driving on dirt roads

This study proposes a novel prediction model to accurately quantify the carbon emissions of the trucks driving on dirt roads based on the deep learning neural network for small sample (DNNSS) and the motor vehicle emission simulator (MOVES). The application range of the MOVES model was extended to the transportation of asphalt mixtures on the temporary road. The model correction method was also established based on the rolling resistance coefficient (CR) and the correction coefficient (μ). By comparing the measured fuel consumption of the truck with the carbon emissions calculated by the MOVES model, the values of CR × μ can be back-calculated. DNNSS was constructed for estimating the CR × μ. The Adaptive Moment Estimation (Adam) algorithm was used to dynamically adjust the learning rate and accelerate the convergence of the network; the Dropout function was used to alleviate overfitting; and the Rectified Linear Unit (ReLU) function was used as the activation function to solve the gradient vanishing problem. The test results showed that the vehicle speed and load greatly influence the CR × μ. The DNNSS algorithm was better at predicting the CR × μ. The proposed DNNSS-MOVES model was more accurate than the conventional methods.

Machine learning opportunities to predict obstetric haemorrhages

Obstetric hemorrhages (OH) are the main preventable cause of morbidity, mortality and cases of "near miss" among obstetric complications worldwide. Early preventive measures based on the OH prediction allow to profoundly reduce the rate of female mortality and morbidity as well as prevent the economic costs of patient intensive care, blood transfusion, surgical treatment and long-term hospitalization. Postpartum haemorrhage (PPH) is the most frequent obstetric haemorrhage determined by one of the four causes: a uterine tonus disorder, maternal birth trauma, retention of placenta parts and blood-clotting disorder. There is still a need for the continued search for an accurate and reliable prediction method despite multiple attempts to develop an effective system for predicting OH. The solution to this may be reasonably considered an innovative method such as artificial intelligence (AI) including computer technologies capable of obtaining conclusions similar to human thinking. One of the particular AI variants is presented by machine learning (ML), which develops accurate predictive models using computer analysis. Machine learning is based on computer algorithms, the most common among them in medicine are the decision tree (DT), naive Bayes classifier (NBC), random forest (RF), support vector machine (SVM), artificial neural network (ANNs), deep neural network (DNN) or deep learning (DL) and convolutional neural network (CNN). Here, we review the main stages of ML, the principles of algorithms action, and the prospects for using AI to predict OH in real-life clinical practice.